Skip to content

Commit 9f2d940

Browse files
committed
feat(build): resolve the schedule once, record it in the graph, report why
—— 决策、并发与可观测性收敛到一处;图的形态尚未改变 `schedule::decide()` 在 prepare 里求值一次,结果落在 `BuildPlan` 上: 后端据此写图、图头记下 tag、`--verbose` 打印理由 —— 三个读者,一次推导。 # mcpp:graph=normal;schedule=detach-codegen build: schedule: detach-codegen — gcc: publishes the BMI with rename() at ~22% of the compile, so importers can start before code generation **并发解析也搬进 policy**。「跑几个」和「什么形状」是同一类决策,分开放就会再次 出现两处推导 —— 这正是 BMI 等价判断(藏在 ninja 命令里的 POSIX shell,Windows 整段没有) 和作业数(ninja 默认值)当初漂开的原因。`execute` 现在读 `plan.scheduleNinjaJobs`, 不再自己解析;`resolve_jobs` 用回调报告非法值,因而不依赖 UI、可单测。 **失效靠指纹,不靠守卫**。两条快路径跑在 plan 之前,拿不到工具链, 所以不可能在那里推导出「本次应有的调度」——给它传参数就等于第二次推导。 改为把开关折进指纹:换了调度就换构建目录,旧形状的图**结构上不可达**。 图头那行 tag 因此是给人看的(和 `mcpp explain`),不承担失效职责。 只在开关非默认时才折入,已有构建目录的身份不受影响。 顺带:`[build] schedule`(auto|on|off)+ `MCPP_BMI_SCHEDULE`,与 `jobs` 同样存为文本 —— "auto" 的含义取决于做构建的那台机器,parse 期解析等于把一台机器的答案冻进 manifest。
1 parent 065d995 commit 9f2d940

8 files changed

Lines changed: 168 additions & 49 deletions

File tree

src/build/execute.cppm

Lines changed: 6 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -377,46 +377,11 @@ compute_subos_env(const mcpp::build::BuildPlan& plan) {
377377
// so the latter doesn't call prepare_build twice (and re-print the toolchain
378378
// resolution banner).
379379
// How many compiles to run at once.
380-
//
381-
// Precedence: `--jobs` (arriving as MCPP_JOBS, same channel --offline uses and
382-
// for the same reason — the consumers span subsystems) > `[build] jobs` >
383-
// 0, which means "say nothing" and leaves ninja's own default (nproc + 2).
384-
// The default is deliberately unchanged: altering everyone's concurrency is a
385-
// behaviour change, and this lands as an opt-in first.
386-
//
387-
// `auto` is resolved HERE, against the machine doing the build, never frozen
388-
// into a manifest. Measured on this repository: the cold self-build takes
389-
// 81.0s at -j8 and 79.9s at -j32 — 4x the workers for 1.4%, because the build
390-
// is latency-bound. Meanwhile a single module compile peaks at 0.5-1.0 GB, so
391-
// the extra jobs are pure memory pressure; on a high-core, modest-RAM machine
392-
// ninja's default swaps.
393-
std::size_t resolve_parallel_jobs(const mcpp::build::BuildPlan& plan) {
394-
auto from_text = [&](std::string_view v) -> std::optional<std::size_t> {
395-
if (v.empty()) return std::nullopt;
396-
if (v == "auto") {
397-
const auto cap = mcpp::platform::capacity::host_capacity();
398-
return static_cast<std::size_t>(
399-
mcpp::platform::capacity::recommended_jobs(cap));
400-
}
401-
std::size_t n = 0;
402-
const auto* first = v.data();
403-
const auto* last = v.data() + v.size();
404-
if (auto [p, ec] = std::from_chars(first, last, n);
405-
ec == std::errc{} && p == last && n > 0)
406-
return n;
407-
// A malformed value must not silently become "use the default" — that
408-
// is how a typo turns into a build that is mysteriously slower.
409-
mcpp::ui::warning(std::format(
410-
"ignoring invalid job count '{}' (expected a positive number or 'auto')", v));
411-
return std::nullopt;
412-
};
413-
414-
if (const char* e = std::getenv("MCPP_JOBS"))
415-
if (auto n = from_text(e)) return *n;
416-
if (auto n = from_text(plan.manifest.buildConfig.jobs)) return *n;
417-
return 0;
418-
}
419-
380+
// Concurrency and the module-edge schedule are resolved together in
381+
// mcpp.build.schedule.policy and stamped onto the plan, so this reads one value
382+
// instead of re-deriving it. `scheduleNinjaJobs` is NOT the compiler cap under
383+
// detach-codegen: a detached compiler stops holding a ninja slot, so ninja is
384+
// handed a larger number on purpose.
420385
export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache,
421386
std::string_view targetOverride = "") {
422387
// `--cache=off` means a cold build: no global cache, and target/ cleared —
@@ -488,7 +453,7 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache,
488453

489454
mcpp::build::BuildOptions opts;
490455
opts.verbose = verbose;
491-
opts.parallelJobs = resolve_parallel_jobs(ctx.plan);
456+
opts.parallelJobs = static_cast<std::size_t>(ctx.plan.scheduleNinjaJobs);
492457
auto r = be->build(ctx.plan, opts);
493458
if (!r) {
494459
std::fflush(stdout);

src/build/graph_shape.cppm

Lines changed: 46 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,15 @@ std::string_view to_string(GraphShape shape) {
4747

4848
// The marker line, without its newline. A ninja comment, so it costs nothing
4949
// and older ninja versions do not care.
50-
std::string header_line(GraphShape shape) {
51-
return std::format("# mcpp:graph={}", to_string(shape));
50+
// `scheduleTag` names the SHAPE OF THE MODULE EDGES (see
51+
// mcpp.build.schedule.policy): "none", "two-phase", "detach-codegen". It rides
52+
// the same line for the same reason the shape does — build.ninja is shared
53+
// mutable state and the fast path replays it, so a graph built under one
54+
// schedule must not be replayed under another. Flipping the switch has to
55+
// invalidate the graph, and the only way that cannot be forgotten is if the
56+
// graph says which schedule produced it.
57+
std::string header_line(GraphShape shape, std::string_view scheduleTag) {
58+
return std::format("# mcpp:graph={};schedule={}", to_string(shape), scheduleTag);
5259
}
5360

5461
// Read the shape back. `nullopt` means "this file does not say" — a build.ninja
@@ -68,6 +75,10 @@ std::optional<GraphShape> read_shape(const std::filesystem::path& ninjaPath) {
6875
auto value = std::string_view(line).substr(prefix.size());
6976
while (!value.empty() && (value.back() == '\r' || value.back() == ' '))
7077
value.remove_suffix(1);
78+
// `graph=<shape>[;schedule=<tag>]`. Split before comparing, so adding
79+
// the schedule field does not turn every existing graph into "unknown".
80+
if (const auto semi = value.find(';'); semi != std::string_view::npos)
81+
value = value.substr(0, semi);
7182
if (value == "normal") return GraphShape::Normal;
7283
if (value == "test") return GraphShape::WithTests;
7384
// A shape this binary does not know is not `Normal`. An older mcpp
@@ -77,7 +88,40 @@ std::optional<GraphShape> read_shape(const std::filesystem::path& ninjaPath) {
7788
return std::nullopt;
7889
}
7990

91+
// The schedule tag this graph was written with. Empty means the file predates
92+
// the field — which is NOT the same as "none": an unlabelled graph is exactly
93+
// the case that must not be replayed blind, so callers compare and miss.
94+
std::string read_schedule(const std::filesystem::path& ninjaPath) {
95+
std::ifstream input(ninjaPath);
96+
if (!input) return {};
97+
std::string line;
98+
for (int i = 0; i < 8 && std::getline(input, line); ++i) {
99+
constexpr std::string_view prefix = "# mcpp:graph=";
100+
if (!line.starts_with(prefix)) continue;
101+
auto value = std::string_view(line).substr(prefix.size());
102+
while (!value.empty() && (value.back() == '\r' || value.back() == ' '))
103+
value.remove_suffix(1);
104+
const auto semi = value.find(';');
105+
if (semi == std::string_view::npos) return {};
106+
auto rest = value.substr(semi + 1);
107+
constexpr std::string_view schedPrefix = "schedule=";
108+
if (!rest.starts_with(schedPrefix)) return {};
109+
return std::string(rest.substr(schedPrefix.size()));
110+
}
111+
return {};
112+
}
113+
80114
// The one question every fast path asks.
115+
//
116+
// It deliberately does NOT compare the schedule tag. The fast paths run BEFORE
117+
// a plan exists, so they have no toolchain to derive the expected schedule
118+
// from — and passing one in would mean deriving the same decision a second
119+
// time, in a place that cannot see the compiler.
120+
//
121+
// Instead the schedule SWITCH is part of the toolchain fingerprint, so flipping
122+
// it lands in a different build directory: a graph written under one schedule
123+
// is structurally unreachable from a build configured with another. The tag on
124+
// the line is then for humans and for `mcpp explain`, not for invalidation.
81125
bool is_plain_build_graph(const std::filesystem::path& ninjaPath) {
82126
return read_shape(ninjaPath) == GraphShape::Normal;
83127
}

src/build/ninja_backend.cppm

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -407,7 +407,7 @@ std::string emit_ninja_string(const BuildPlan& plan) {
407407
// #407: the graph declares which mode produced it, because three modes
408408
// write this one file and the fast path has to know what it is about to
409409
// replay. Must stay within the first few lines — see read_shape.
410-
append(mcpp::build::header_line(plan.graphShape) + "\n");
410+
append(mcpp::build::header_line(plan.graphShape, plan.scheduleTag) + "\n");
411411
append("ninja_required_version = 1.11\n\n");
412412

413413
// All compile/link flags are computed once via flags.cppm.

src/build/plan.cppm

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,18 @@ struct BuildPlan {
145145
// share an output directory and overwrite each other's graph; this is what
146146
// lets a fast path tell them apart (mcpp#407, mcpp.build.graph_shape).
147147
GraphShape graphShape = GraphShape::Normal;
148+
// The module-edge schedule this plan will emit, resolved ONCE (see
149+
// mcpp.build.schedule.policy). The backend writes the graph in this shape,
150+
// the graph records the tag, and the fast path compares against it — three
151+
// readers, one derivation. Deriving it separately in the backend and in the
152+
// executor is how the BMI-equivalence check and the job count drifted into
153+
// disagreeing about what a module edge is.
154+
std::string scheduleTag = "none";
155+
// What to hand ninja. Under detach-codegen a compiler stops holding a slot
156+
// when it publishes, so this must exceed the real compiler cap or the ready
157+
// frontier starves — see the hazard note in schedule/detach_codegen.
158+
int scheduleNinjaJobs = 0;
159+
int scheduleCompilerCap = 0;
148160
// One immutable snapshot selected before workspace member substitution.
149161
// Build/run/test and cache fast paths consume this value; none may re-read
150162
// xlings active/current state.

src/build/prepare.cppm

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ import mcpp.toolchain.post_install;
3333
import mcpp.toolchain.abi;
3434
import mcpp.toolchain.triple;
3535
import mcpp.build.plan;
36+
import mcpp.build.schedule.policy;
3637
import mcpp.build.graph_shape; // #407: the graph says which mode wrote it
3738
import mcpp.build.runtime_validation; // declared artifact -> identity verdict
3839
import mcpp.build.cache_key;
@@ -5226,6 +5227,17 @@ prepare_build(bool print_fingerprint,
52265227
fpi.cppStandard = m->package.standard;
52275228
fpi.compileFlags = canonical_compile_flags(*m)
52285229
+ canonical_package_build_metadata(packages);
5230+
// The module-edge schedule changes the SHAPE of build.ninja, and the fast
5231+
// path replays that file without a plan to compare against. Folding the
5232+
// switch into the fingerprint puts a differently-scheduled build in a
5233+
// different directory, which makes replaying the wrong shape structurally
5234+
// impossible instead of merely guarded. Only appended when non-default, so
5235+
// existing build directories keep their identity.
5236+
if (const auto sched = mcpp::build::schedule::requested_switch(*m);
5237+
sched != "auto") {
5238+
fpi.compileFlags += " #schedule=";
5239+
fpi.compileFlags += sched;
5240+
}
52295241
if (m->cppStandard.experimental) {
52305242
// c++fly gate flags are derived (not manifest-declared): fold them in
52315243
// so a cppfly table change across mcpp versions re-fingerprints.
@@ -5311,6 +5323,24 @@ prepare_build(bool print_fingerprint,
53115323
ctx.plan.graphShape = (includeDevDeps || !extraTargets.empty())
53125324
? mcpp::build::GraphShape::WithTests
53135325
: mcpp::build::GraphShape::Normal;
5326+
// Resolve the module-edge schedule ONCE, here, where both the toolchain and
5327+
// the manifest are in hand. The backend writes the graph in this shape, the
5328+
// graph records the tag, and `mcpp build --verbose` prints the reason — all
5329+
// three read this, none of them re-derives it.
5330+
{
5331+
const auto decision = mcpp::build::schedule::decide(
5332+
ctx.plan.toolchain,
5333+
mcpp::build::schedule::requested_switch(*m),
5334+
mcpp::build::schedule::resolve_jobs(*m, [](std::string_view bad) {
5335+
mcpp::ui::warning(std::format(
5336+
"ignoring invalid job count '{}' (expected a positive number or 'auto')", bad));
5337+
}));
5338+
ctx.plan.scheduleTag = std::string(mcpp::build::schedule::to_string(decision.strategy));
5339+
ctx.plan.scheduleNinjaJobs = decision.ninjaJobs;
5340+
ctx.plan.scheduleCompilerCap = decision.compilerCap;
5341+
mcpp::log::verbose("build", std::format("schedule: {} — {}",
5342+
ctx.plan.scheduleTag, decision.reason));
5343+
}
53145344
ctx.plan.runtimeBinding = runtimeBindingSnapshot;
53155345
mcpp::build::merge_runtime_binding_contract(
53165346
ctx.plan, runtimeBindingSnapshot);

src/build/schedule/policy.cppm

Lines changed: 66 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,13 @@
77
// whatever ninja defaulted to. Deriving the same decision in two places is how
88
// the two halves drifted apart. This module is the only place it is derived.
99
//
10-
// PURE. No filesystem, no processes, no environment reads — a caller passes the
11-
// facts in and gets a Decision plus the sentence explaining it. That is what
12-
// makes the policy unit-testable without a toolchain, and what lets the reason
13-
// be printed, logged and written into build.ninja unchanged.
10+
// `decide()` IS PURE. No filesystem, no processes, no environment — a caller
11+
// hands it facts and gets a Decision plus the sentence explaining it, so the
12+
// table is unit-testable without a toolchain and the reason can be printed,
13+
// logged and written into build.ninja unchanged. `requested_switch()` is the
14+
// one impure function here, and it is impure on purpose: the switch has to be
15+
// read somewhere, and two callers each doing env-then-manifest in their own
16+
// order is exactly the duplicate derivation this module exists to prevent.
1417
//
1518
// THE MEASUREMENTS BEHIND THE TABLE (2026-08-13, mcpp building itself: 138
1619
// module interface units, 57k lines, i9-13900K, gcc@16.1.0 / llvm@22.1.8):
@@ -49,6 +52,8 @@ export module mcpp.build.schedule.policy;
4952

5053
import std;
5154
import mcpp.toolchain.model;
55+
import mcpp.manifest;
56+
import mcpp.platform.capacity;
5257

5358
export namespace mcpp::build::schedule {
5459

@@ -84,6 +89,32 @@ struct Decision {
8489
int ninjaJobs = 0;
8590
};
8691

92+
// The one place the switch is READ. `decide` above stays pure — a caller hands
93+
// it facts — but the switch itself has to come from somewhere, and having two
94+
// callers each read env-then-manifest in their own order is precisely the
95+
// duplicate-derivation this module exists to prevent.
96+
//
97+
// Precedence matches every other mcpp switch: environment beats manifest.
98+
std::string requested_switch(const manifest::Manifest& m);
99+
100+
// How many compilers this machine should run at once.
101+
//
102+
// Precedence: MCPP_JOBS (where `--jobs` lands) > `[build] jobs` > 0, meaning
103+
// "say nothing" and leave the backend's own default. The default is unchanged
104+
// on purpose: altering everyone's concurrency is a behaviour change.
105+
//
106+
// `auto` is resolved HERE, against the machine doing the build, never frozen
107+
// into a manifest. Measured on this repository: the cold self-build takes 81.0s
108+
// at -j8 and 79.9s at -j32 — 4x the workers for 1.4%, because the build is
109+
// latency-bound — while a single module compile peaks at 0.5–1.0 GB, so the
110+
// extra jobs are pure memory pressure. On a high-core, modest-RAM machine the
111+
// backend default swaps.
112+
//
113+
// `onInvalid` is called with the offending text instead of warning directly, so
114+
// this stays free of any UI dependency and remains testable.
115+
int resolve_jobs(const manifest::Manifest& m,
116+
const std::function<void(std::string_view)>& onInvalid = {});
117+
87118
// `requested` is the user's switch: "auto" (default), "on", "off". `hostJobs` is
88119
// the already-resolved parallelism (`--jobs`, `[build] jobs`, or the backend
89120
// default), i.e. how many compilers this machine should run at once.
@@ -137,4 +168,35 @@ Decision decide(const toolchain::Toolchain& tc, std::string_view requested, int
137168
return d;
138169
}
139170

171+
int resolve_jobs(const manifest::Manifest& m,
172+
const std::function<void(std::string_view)>& onInvalid) {
173+
auto from_text = [&](std::string_view v) -> std::optional<int> {
174+
if (v.empty()) return std::nullopt;
175+
if (v == "auto") {
176+
const auto cap = platform::capacity::host_capacity();
177+
return platform::capacity::recommended_jobs(cap);
178+
}
179+
int n = 0;
180+
const auto* first = v.data();
181+
const auto* last = v.data() + v.size();
182+
if (auto [p, ec] = std::from_chars(first, last, n);
183+
ec == std::errc{} && p == last && n > 0)
184+
return n;
185+
// A malformed value must not silently become "use the default" — that
186+
// is how a typo turns into a build that is mysteriously slower.
187+
if (onInvalid) onInvalid(v);
188+
return std::nullopt;
189+
};
190+
if (const char* e = std::getenv("MCPP_JOBS"))
191+
if (auto n = from_text(e)) return *n;
192+
if (auto n = from_text(m.buildConfig.jobs)) return *n;
193+
return 0;
194+
}
195+
196+
std::string requested_switch(const manifest::Manifest& m) {
197+
if (const char* e = std::getenv("MCPP_BMI_SCHEDULE"); e && *e) return std::string(e);
198+
if (!m.buildConfig.schedule.empty()) return m.buildConfig.schedule;
199+
return "auto";
200+
}
201+
140202
} // namespace mcpp::build::schedule

src/manifest/toml.cppm

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1046,6 +1046,7 @@ std::expected<Manifest, ManifestError> parse_string(std::string_view content,
10461046
// where they are used, so a bad value warns at build time instead of making
10471047
// the whole manifest unloadable. (A published package carrying an unknown
10481048
// key must never break an older mcpp — same rule the dependency keys follow.)
1049+
if (auto v = doc->get_string("build.schedule")) m.buildConfig.schedule = *v;
10491050
if (auto v = doc->get_string("build.jobs")) m.buildConfig.jobs = *v;
10501051
else if (auto n = doc->get_int("build.jobs")) m.buildConfig.jobs = std::to_string(*n);
10511052
if (auto v = doc->get_string("build.default-profile")) m.buildConfig.defaultProfile = *v;
@@ -1080,7 +1081,7 @@ std::expected<Manifest, ManifestError> parse_string(std::string_view content,
10801081
"allow_host_libs", "build_program_timeout", "c_standard", "cache",
10811082
"cflags", "cxxflags", "cxx_runtime", "default-profile", "defines",
10821083
"dialect_cxxflags", "flags", "include_dirs", "include_dirs_after",
1083-
"jobs", "ldflags", "macos_deployment_target", "module_extensions", "profile",
1084+
"jobs", "ldflags", "schedule", "macos_deployment_target", "module_extensions", "profile",
10841085
"sources", "static_stdlib", "target",
10851086
};
10861087
if (auto* bt = doc->get_table("build")) {

src/manifest/types.cppm

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -360,6 +360,11 @@ struct BuildConfig : BuildInputs {
360360
// that actually runs: resolving it at parse time would freeze one machine's
361361
// core count into a value that then travels with the manifest.
362362
std::string jobs;
363+
// `[build] schedule` — the module-edge shape: "auto" (default), "on",
364+
// "off". Text for the same reason `jobs` is: the meaning of "auto" depends
365+
// on the compiler doing the build, and resolving it at parse time would
366+
// freeze one machine's answer into a manifest that travels.
367+
std::string schedule;
363368
// feature name → extra source globs gated by that feature. A glob listed
364369
// here is EXCLUDED from the default build and only compiled/linked when the
365370
// feature is active for this package (resolved in prepare_build). Lets a

0 commit comments

Comments
 (0)