In-repo build artifacts survive the CI cache - #75
Merged
Conversation
The gate rebuilds every crate that lives in the repository on every run: our two host binaries with their LTO links (~2m20), the pinned iroh checkout's crates in the host profile (~50s), and the upstream relay (~2m20) — about 5 of the gate's 7.5 minutes. Two mechanisms held them there, and neither is fixed by the other. Swatinem/rust-cache drops, from each cached target directory, every package whose manifest lives inside that workspace root, which is all of them; cache-workspace-crates keeps the members. And cargo decides a path dependency is stale by mtime, while a checkout (ours) or a clone (setup.sh's) writes every source with the current time — so a restored artifact is always older than the sources it was built from, and keeping it changes nothing. scripts/restore-mtimes.py dates each tracked file by the commit that last touched it, which is stable across runs and machines, so unchanged sources stay older than the artifacts built from them. Files that differ from HEAD keep their mtimes: backdating a modified file would hide the modification from cargo. The full history the dating needs also arrives now (fetch-depth: 0); a depth-1 checkout knows only the tip commit and would date every file alike. The workflow joins the cache key, because it decides what the cache holds and a save skipped on an exact-key hit would otherwise pin the contents chosen by a superseded configuration — this change included.
lannbot
pushed a commit
that referenced
this pull request
Aug 13, 2026
rust-cache keeps the cargo home; `actions/cache` keeps `target` and `.deps/iroh/target` whole. The action's own target caching cannot serve this gate: it saves a build directory only after deleting every package whose manifest lives inside the workspace root, which is every crate the gate spends its time on — ours and the pinned checkouts'. Keeping the workspace members (#75) did not help, because they sit on top of the path dependencies it still deleted, and a missing dependency output rebuilds everything above it. Even among the packages it keeps, artifacts whose file name carries no build hash are dropped: cargo's own `deps/libiroh_relay.rlib` was the reason the relay rebuilt every run. A green gate is what justifies saving, so the save runs after it, on main only, and only when the restore did not already hold the key. A key that misses still restores the newest entry under the `build-dirs-` prefix, so cargo rebuilds what changed rather than everything. Measured on this branch, seeded and then re-run: the gate compiles nothing and takes 3m12, against 7m28 before; the two cache entries together are 1.85 GB where the single one was 1.78 GB, and restoring them costs 26s.
lannbot
pushed a commit
that referenced
this pull request
Aug 13, 2026
rust-cache keeps the cargo home; `actions/cache` keeps `target` and `.deps/iroh/target` whole. The action's own target caching cannot serve this gate: it saves a build directory only after deleting every package whose manifest lives inside the workspace root, which is every crate the gate spends its time on — ours and the pinned checkouts'. Keeping the workspace members (#75) did not help, because they sit on top of the path dependencies it still deleted, and a missing dependency output rebuilds everything above it. Even among the packages it keeps, artifacts whose file name carries no build hash are dropped: cargo's own `deps/libiroh_relay.rlib` was the reason the relay rebuilt every run. A green gate is what justifies saving, so the save runs after it, on main only, and only when the restore did not already hold the key. A key that misses still restores the newest entry under the `build-dirs-` prefix, so cargo rebuilds what changed rather than everything. Measured on this branch, seeded and then re-run: the gate compiles nothing and takes 3m12, against 7m28 before; the two cache entries together are 1.85 GB where the single one was 1.78 GB, and restoring them costs 26s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
cache-workspace-crates: true— rust-cache keeps the artifacts of crates that live in the repository (our members and the pinned.deps/irohmembers) instead of deleting them on save.scripts/restore-mtimes.py(new, run in CI aftersetup.sh) dates every tracked file by the commit that last touched it, in this repo and each.depscheckout.fetch-depth: 0on checkout, and.github/workflows/ci.ymljoins the cache key.Why
After #74 the gate is 448 s, of which ~5 min is rebuilding crates that live in the repository — measured from run 31711380488:
iroh-peerdominates)iroh-peerrarely changesiroh-relayrelease build (.deps/iroh/target).depscrates in the host profileTwo independent mechanisms keep them there:
cleanTargetDirkeeps only packages whose manifest is outside the workspace root, so every crate under the repo is deleted from the cache on save.cache-workspace-crates: truekeeps the members of each registered workspace — ours, and iroh's in.deps/iroh/target. (The.deps/*sibling crates are path dependencies but not members of either workspace, so the ~52 s row is not addressed here.)-Z checksum-freshness), and bothactions/checkoutandsetup.sh's clones write sources with the current time. Verified locally:touching.deps/iroh's sources with no content change recompilesiroh,iroh-relay,iroh-peer. So restored in-repo artifacts are rejected on sight, and pruning is not even the binding constraint.Neither half is worth landing alone — caching artifacts cargo will reject buys nothing, and dating sources with nothing cached buys nothing — so they are one change.
Verified locally, on the real
.deps/irohcheckout:touchevery file, thenrestore-mtimes.py→just build-hostsrecompiles nothing (without the script: 3 crates).iroh-relay/src/lib.rs, thenrestore-mtimes.py→ the file is skipped (230 of 231 stamped) andiroh-relay,iroh,iroh-peerrebuild. Backdating a modified file would produce a binary that does not match its source; this is the property that must not break..gitcontents are irrelevant to cargo's decisions (touching only.gitrebuilds nothing).just checkpasses; the gate itself is this PR's CI run.What to expect
The key rotates (the workflow is in it), so this PR's run and the first run on main are cold — the first main run re-seeds the cache with in-repo artifacts kept. The measurement is the dispatch after that: expect the three rows above to mostly vanish on runs that do not touch their sources, and the cache entry to grow from ~1.8 GB (quota is 10 GB per repo).
Risk
We are trusting commit dates for freshness in CI. The exposure is a file whose content differs from HEAD getting backdated, which the script refuses to do (
git diff --name-only HEADis the skip list), and CI trees are pristine checkouts at a known commit. The script is not wired into anyjustrecipe, so local builds keep using real mtimes.