Pkg "strict" mode for well-behaved `[compat]` + precompilation

[topolarity]

Julia's stacked environments are very cool because they let you quickly jump between multiple active projects in the same session, but they have the downside that they interact quite poorly with Pkg.precompile and pkgimages in general since Pkg often cannot predict loading behavior with them.

I love the new [workspaces] feature, because it solves a ton of problems associated with this workflow:

• developers can still quickly hop between projects without a restart
  (this is even faster than with stacked environments, since precompilation is shared!)
• [compat] is not violated since the loaded environment actually went through a ] resolve
• ] precompile works equally well for tests and the main project (and overall compiles much less!)
• Pkg.test() failures are easily reproduced and have a clear environment (no need for TestEnv.jl!)

but there are still a few gaps:

1. Dynamic environment changes are still an issue. The warning improved in precompilation: improve warn_loaded messaging for transitive deps julia#61293 is nice, but the end-to-end Pkg workflow encourages you to do the "easy" thing and attempt a using only to immediately hit a ton of unexpected pre-compilation. The problem is that the environment was "torn" since it now has packages of a different version loaded which do not match your environment (or what will load next session). Since your environment is now "torn", this is completely one-off, never-to-be-reused-again work. Furthermore, since you are running in a potentially non-[compat] (and non-reproducible environment), so you may hit bugs and choose to back out anyway.
2. Loading is still affected by ] activate if you hop between stacked environments instead of between [workspace] projects. That leads to the same "torn environment" issues alluded to above when you intersperse using with activate changes. (i.e. moving between stacked environments is much worse than moving between workspace projects)
3. Implicit environments such as package directories still exist.

The problem is that these dynamic / implicit environment changes (1) lead to many "unique" environments that require one-off pre-compilation work and are often broken anyway due to [compat] issues, and (2) prevent Pkg.precompile from being able to predict which packages your project(s) will actually need / want to load at run time.

As a developer, it'd be useful to me to have a "strict" Pkg workflow which:

• doesn't allow me to ] activate anything that would "tear" my environment
• after any package changes / upgrades that "tear" the environment, "disconnects" the updated Manifest and loads based on only the "old" environment
  (i.e. require a restart if you want to use the new packages, or keep running with the old one)
• disables problematic features such as package directories / implicit environments

That the environment is explicit and static (rather than torn dynamically) means that Pkg.precompile() can actually work and guarantee that it precompiles the packages relevant for your project(s). It also means that we could make it an error to end up in precompilation in tests, etc. and that "serial precompilation" never happens in this mode (i.e. #4632 could be disabled if you are in "strict" Pkg mode)

[IanButterworth]

#4507 helps with this

[topolarity]

#4507 helps with this

helps both w/ and w/o "strict" mode IMO

In non-"strict" mode that change means much less precompilation. In "strict" mode, it means you need fewer restarts and less precompilation too.

[KristofferC]

Continuing from #4507 (comment), if loading were made so that e.g. locate_package returned the path to the loaded package, instead of the one given by the current manifest, then precompilation work would never be wasted in a given julia session, right? Sure, if you restart julia and e.g. disable your startup file, you may need to precompile again, but you would never have auto precompile have to run twice on the same package in the same session?

[topolarity]

Continuing from #4507 (comment), if loading were made so that e.g. locate_package returned the path to the loaded package, instead of the one given by the current manifest, then precompilation work would never be wasted in a given julia session, right?

We could try to teach precompile what the current loading "frankenstein" environment is (and hope that the same environment happens at the next run), but since it's encoded in your startup.jl and only discovered via actually loading we really can't "predict" anything and we can't fix the [compat] tearing - that's much further on the "be lazy, and just hope things don't change" side of the spectrum

It's pretty hard to distinguish "loaded by startup.jl and will be loaded next time" vs. "loaded by the user / from an old environment, so won't be reloaded", so I'm skeptical that it would behave well in practice. I'd expect lots of scenarios where you hit surprising precompilation at runtime.

[KristofferC]

We could try to teach precompile what the loading "frankenstein" environment will be

I think hooking into https://github.com/JuliaLang/julia/blob/3360c8ca7cc3a0dac2fa3eafe980591a94a774f4/base/loading.jl#L1270 would more or less do this (having it return the path of the loaded package) would suffice for everything.

that's much further on the "be lazy, and just hope things don't change" side of the spectrum

Well, it can be argued that spending a long time creating precompile files that are unusable in the current session is unlikely to be very useful in practice because you really hope that:

• the user will immediately exit the session after the precompilation and not try to load the packages that were precompiled
• the user will restart julia with a pristine session with none of the packages loaded that were loaded when the last precompile was run

And I don't think that happens very often in practice.

we really can't "predict" anything and we can't fix the [compat] tearing

Yes, but as long as you can just run around with activate in completely decoupled environments, that will always be the case.

---

I think there are three types of features that can be added to help with the problem:

• Features that reduce the need to do environment switching, e.g., workspaces.
• Features that warn / prevent you from ending up in the bad case (like this suggests)
• Features that try to reduce the harm of environment switching (Prefer adding already loaded dep versions #4507)

[topolarity]

Isn't that inconsistent with how loading behaves next session anyway? I thought that your startup.jl would next session load the "new" Preferences.jl v1.5.2 because that's what's in the top of your (stacked) manifest.

Yes, but as long as you can just run around with activate in completely decoupled environments, that will always be the case.

100%. That's why this issue proposes banning that:

a "strict" Pkg workflow which:

• doesn't allow me to ] activate anything that would "tear" my environment

One of the things I didn't detail in the proposal, but which I think is entailed by that restriction is that for stacked environments to be editable in "strict" mode they have to behave a lot more like [workspaces] (i.e. the activate switches the Project.toml you are operating on, but not the Manifest.toml which remains your root workspace). They'd also need to be included into the resolve in the first place if we want to have them be [compat]-consistent.

But the "basic" version ignores that and says it's enough to just not allow them to be edited / activated in strict mode

[KristofferC]

Isn't that inconsistent with how loading behaves next session anyway? I

It depends how you activate the new env, via --project or a manual activate.

to be editable in "strict" mode they have to behave a lot more like [workspaces] (i.e. the activate switches the Project.toml you are operating on, but not the Manifest.toml which remains your root workspace).

Yes, I've thought a bit about the concept of an "overlay project" which is like a project you set (as a global perhaps) that would more or less function so that this project is always considered part of your workspace. It would never have a manifest on its own but would be resolved together with every environment you have active. You would put debug tools in there. It would kind of replace what most people use the "global" environment for. I wouldn't help the case where you go around and manually activate isolated environments but might be an improvement.

[StefanKarpinski]

What about detecting when the dependencies of a package are "torn" in the sense that they cross load path entries and an earlier entry provides a different version than the later one would, and skip precompilation? After all, we don't need to precompile all the time, we can just load packages without doing that. The strict option could also be nice to prevent that from even happening, but it seems like a gentle improvement would be just not trying to precompile that work.

Kristoffer, your overlay project idea reminds me of a spat I had with someone new to Pkg who was incensed(!) that Pkg environments didn't work like Python ones (ok). The specific thing I seem to recall they wanted was that Pkg environments be local in the way that venv does it—literally a fully separate package tree in a local directory. Now, I don't think that's better, but I do think the idea of having a local dev workspace is nice so that you can add project-specific tools to it and have them be resolved together with the main project. But it's annoying to have to add every dev tool in every project, so maybe the overlay project would be a kind of "here are things you can load and they'll just be automatically added to the current dev workspace." And it could try to be smart about picking a version that's already installed and/or trying to pick versions of its dependencies that already have usable precompile files. Now, that's a lot of smarts, but I'm just trying to imagine and ideal workflow that we could build towards.