-
Notifications
You must be signed in to change notification settings - Fork 415
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
nix: nixBuildPackage’s depRoots are costly on the store #2508
Comments
Yes, this is definitely something I've considered changing before. I don't remember all that many details but one weak reason was that changing it would make
Not right now, but with a module system that would change.
Heh, I would not be surprised. We're taking Nix out of its comfort zone in various ways here. |
The Nix build has been officially deprecated #4895 |
I recently ran out of inodes on my
/nix
store partition, and I thinknixBuildPackage
’sdepRoot
derivations are the cause.Status quo
Consider a module dependency graph
A ← B ← C
. Right now,nixBuildPackage
will create 6 store paths, roughly like this:So for each module, there is a
-depRoot
derivation that contains a symlinks to all theolean
’s of its (transitive) dependencies. So if we have a dependency chain of length n, we’ll add O(n²) symlinks to the store.The problem
These symlinks are small, but still take up considerable filesystem resources.
Moreover, these
depRoot
store paths don’t seem to be that useful:Mod-depRoot
is only re-used whenMod
changes. For all unchanged modules, the-depRoot
path is not used again. And for all modules whose dependencies change, the-depRoot
is not used either.Moreover, having two store paths for each module doubles the time spent on querying nix caches when building.
A possible fix
Given that analysis (if it turns out to be accurate), it seems reasonable to just not persist the
depRoot
store path used when building a module, and instead just create it temporarily when building the module.It’s still useful to have such a store path for the overall library (the
foo.modRoot
output).Extension: lean-deps file
Right now, the
/nix/store/…-C
store path does not depend on anything. Is it ever useful to haveC.olean
withoutA.olean
andB.olean
? It might be prudent to add a text filelean-deps
to/nix/store/…-C
that lists the (direct) dependencies, so that nix knows about that relation.Also, the derivation build to build, say,
Mathlib-depRoot
(or, with this proposal,Mathlib
) is rather large (1.3MB), because it lists all it’s dependencies, twice, indeps
anddepRoots
, and then again twice ininputDrvs
. I don't have evidence, but I wonder if this might be rather costly in terms of nix evaluation etc. Probably the derivation doesn’t have to mention the indirect dependencies, and could use thelean-deps
text file to list all the dependencies.As before, this is not highly critical, but I wanted to write it down. If @Kha roughly agrees with the analysis and conclusion, I might give it a try at some point.
The text was updated successfully, but these errors were encountered: