Assorted bootstrap LLVM refactors (part 1/N)

[Kobzol]

This PR continues my LLVM and download-ci-llvm bootstrap refactors (it took me almost a year to get back to them, lol), with the goals of:

• Remove dependency on implicit paths with artifacts. Ideally, everything should only depend on output paths from executed steps.
• Reduce eager and implicit downloads and other shenanigans happening in config parsing, particularly around download-ci-llvm.
• Allow downloading LLVM from CI for non-host targets. There is not really any reason why bootstrap shouldn't be able to download LLVM from CI if it's there for any given target, but right now it can only do so for the host target. This would be useful for debugging bootstrap itself, and perhaps for some people also for development.
• Centralize the handling of LLVM into the Llvm step, and reduce overall knowledge between locally built and downloaded LLVM. Ideally, most of bootstrap shouldn't know about whether it uses a LLVM that was built locally or downloaded. But that is of course not the case today, because there are many places in bootstrap that ad-hoc work with some LLVM paths, and do things that depend on some part of bootstrap randomly modifying a config here or there, or doing some side effect, which then makes the other activity "work". But of course, this is very fragile, as we know.
• If I manage to reach the previous goal, this should massively help unblock further unborking of download-ci-rustc. Those two features are quite interrelated, and sadly both are scattered across the codebase. I actually first wanted to start with refactoring download-ci-rustc, but I couldn't find a way to do that without first improving download-ci-llvm.

This PR contains a bunch of commits that slowly move us towards these goals. The refactors were quite explanatory, they are mostly a preparation for larger refactors that I had in mind, so nothing major. But as always, step by step.

As usually, best reviewed commit-by-commit.

Some notes for review:

• The last commit might case a behavior change (well, or an error) on MSVC without Ninja, when filecheck is used without download-ci-llvm. As per the experiment on Zulip, this should be fine. And I'm not sure if anyone actually uses MSVC without Ninja in bootstrap, it seems to be quite broken.
• 07b88f2 removes a sanity check for FileCheck being present. It was a bit hacky, because it was only checking it if it wasn't locally built (or downloaded from CI...). To avoid introducing more hacks, I just removed this sanity check. If the code will need filecheck and it won't be available, it will error out at the usage site (or I could add an assert that the FileCheck binary is present in the FileCheck step if you want).

r? jieyouxu

[rustbot]

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

This PR changes how LLVM is built. Consider updating src/bootstrap/download-ci-llvm-stamp.

[rustbot]

jieyouxu is currently at their maximum review capacity.
They may take a while to respond.

[jieyouxu]

(Will take a look later today)

[jieyouxu]

Thanks, r=me after a rebase.
@bors rollup=never note="bootstrap llvm refactors"

View changes since this review

[jieyouxu]

@rustbot author

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[Kobzol]

@bors r=jieyouxu

[rust-bors]

📌 Commit 7a1d067 has been approved by jieyouxu

It is now in the queue for this repository.

[rust-bors]

☀️ Test successful - CI
Approved by: jieyouxu
Duration: 3h 9m 24s
Pushing ef20314 to main...

[github-actions]

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing ea06042 (parent) -> ef20314 (this PR)

Test differences

No test diffs found

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard ef20314466010b8b9259ec5f86230c530ca08661 --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

1. x86_64-gnu-gcc-core-tests: 14m 53s -> 6m 27s (-56.6%)
2. aarch64-apple-macos-26-2: 1h 49m -> 2h 38m (+44.7%)
3. x86_64-gnu-stdlib-semver-check: 9m 53s -> 5m 36s (-43.4%)
4. dist-powerpc64le-linux-musl: 1h 7m -> 1h 36m (+43.3%)
5. dist-ohos-aarch64: 53m 37s -> 1h 15m (+40.3%)
6. x86_64-mingw-2: 2h 41m -> 1h 36m (-39.9%)
7. x86_64-mingw-1: 2h 49m -> 1h 46m (-37.1%)
8. aarch64-apple-1: 1h 31m -> 2h 3m (+35.3%)
9. x86_64-gnu-parallel-frontend: 2h 5m -> 1h 27m (-30.5%)
10. pr-check-2: 40m 31s -> 28m 26s (-29.8%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

[rust-timer]

Finished benchmarking commit (ef20314): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This perf run didn't have relevant results for this metric.

Max RSS (memory usage)

Results (primary -2.9%, secondary 0.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.0%	[0.4%, 6.1%]	4
Improvements ✅ (primary)	-2.9%	[-2.9%, -2.9%]	1
Improvements ✅ (secondary)	-0.6%	[-0.9%, -0.5%]	6
All ❌✅ (primary)	-2.9%	[-2.9%, -2.9%]	1

Cycles

Results (secondary 0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	1.0%	[0.4%, 2.7%]	11
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.7%	[-1.3%, -0.4%]	12
All ❌✅ (primary)	-	-	0

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 453.486s -> 455.478s (0.44%)
Artifact size: 398.53 MiB -> 398.50 MiB (-0.01%)