Funny to see this now. I’ve got a private branch I am in the process of shaping up this weekend to show the compiler team. For deep nested projects like rust analyzer, if you emit the meta data about function types earlier for downstream slots to use, before successful type checking, you can start other crates earlier and use all the slots you have instead of sitting around waiting for the full type checking of the bodies (which other crates largely don’t care about). Something like 40% wall time speed up, maybe 10% to 15% if you have the parallel frontend on.
Sounds lovely, give it a try on the codex-rs which seems to be getting close to using 2K crates in their repository any time now. Takes like 30 minutes to compile on a my beefy machine and it's a goddamn TUI, not sure what's going on, and not too interested in diving into that beast either.
> Incremental compilation is not cleaned up. Older deps pile up and don’t get cleaned.
This is one of those problems that’s both huge/important, and probably impossible to solve. It bites me all the time, I have a measly 1TB nvme disk and I basically have to wipe my target directory and docker build cache (I have a shared cargo cache volume across builds) every other day. I’ve had this workstation for 2.5 years now and I’m honestly worried about nand endurance coming to bite me soon.
How would one even solve this? When do you know a cache entry won’t be used any more? Trace back provenance of artifacts to their sources, and if the mtimes have been bumped just delete the artifacts? It’s not like resetting source files to older mtimes is a good idea anyway…
I disagree it’s impossible to solve. Huge C++ projects don’t have continuously growing build directories and can still compile incrementally correctly.
That’s because they overwrite the old artifacts with new ones every time, which has its own problems: you could end up reusing stale artifacts if your compiler version changes, or if different compiler args are used for the same artifact, and a bunch of other stuff. Rust computes a digest of all the relevant build environment settings that may need a different artifact to be built, and uses that digest in the file name. That means you basically never have issues like “oh crap the incremental build broke, do a clean build instead”. (Amongst a whole lot of other benefits I’m sure.)
But this technique errs on the side of not using invalid artifacts, at the cost of not having a good system for expiring them. To maintain the goal of never having to worry about when to require a clean rebuild, solving cleaning of stale artifacts becomes a much harder problem.
knuckleheads · · focus · HN ↗
embedding-shape · · focus · HN ↗
yearolinuxdsktp · · focus · HN ↗
Incremental compilation is not cleaned up. Older deps pile up and don’t get cleaned.
Run out of disk space? Oh it’s just the 200+ GB codex-rs target folder.
Forget about doing worktrees.
Rust has a lot of work to do.
ninkendo · · focus · HN ↗
This is one of those problems that’s both huge/important, and probably impossible to solve. It bites me all the time, I have a measly 1TB nvme disk and I basically have to wipe my target directory and docker build cache (I have a shared cargo cache volume across builds) every other day. I’ve had this workstation for 2.5 years now and I’m honestly worried about nand endurance coming to bite me soon.
How would one even solve this? When do you know a cache entry won’t be used any more? Trace back provenance of artifacts to their sources, and if the mtimes have been bumped just delete the artifacts? It’s not like resetting source files to older mtimes is a good idea anyway…
yearolinuxdsktp · · focus · HN ↗
ninkendo · · focus · HN ↗
But this technique errs on the side of not using invalid artifacts, at the cost of not having a good system for expiring them. To maintain the goal of never having to worry about when to require a clean rebuild, solving cleaning of stale artifacts becomes a much harder problem.