Release 2.0.0 (tag v2.0.0 at 629429f, PR #155) reached the release pipeline with everything green except the CleanBench T5 runtime gate, which failed at +27.68% against the 20% budget (run 29736705242). Nothing was published: PyPI and GitHub Releases still have 1.1.1 as latest. This issue documents why the failure is real, what causes it, and the decision needed to unblock.
The gate failure is deterministic, not runner variance
The July 18 v1.2.0 post-mortem attributed the second failed attempt to a "±30% runner lottery vs absolute baseline". The full nightly cleanbench-full history disproves that:
| Date (nightly, main) |
Runtime vs baseline_v1.json |
| Jul 5 |
+21.4% (marginal fail) |
| Jul 6, 8, 10 |
pass |
| Jul 12 |
+20.4% (marginal fail) |
| Jul 14–16 |
+50–52% |
| Jul 17 |
+31.3% |
| Jul 18 |
+28.3% |
| Jul 19 |
+26.8% |
| Jul 20 |
+34.0% |
| Jul 18 release (v1.2.0) |
fail |
| Jul 20 release (v2.0.0) |
+27.7% |
The gate passed on CI runners as recently as Jul 10, so the committed baseline (0.3878s / 50k rows) is CI-achievable and the instrument works. A regression landed Jul 13, was partially recovered (incl. #151's vectorized sci-notation guard), and the code has sat at a stable **+27–34% on CI** for seven consecutive runs. Re-running the workflow is dice-rolling, not a fix.
Where the time goes
Controlled same-machine A/B (T5 fixture, 50k rows, min-of-5, interleaved rounds, arm64):
| Revision |
seconds |
vs Jul 5 code |
| 4506c08 (baseline-era, v1.0.0) |
0.380 |
— |
| 9368ea4 (v1.2.0) |
0.411 |
+8.2% |
| 629429f (2.0.0) |
0.435 |
+14.5% |
cProfile attribution of the delta on the default path:
engine/cache.py:build_engine_cache (~0.08s new): column contexts + numeric correlation for the engine-backed auto_missing/auto_outliers default steps — intended safety-by-default work, and the correlation matrix is genuinely consumed by engine/missing.py.steps/dtypes.py:_try_numeric roughly doubled (10 pd.to_numeric passes vs 5): the guard-then-coerce pattern from the coerced-cell quarantine work.- Assorted smaller costs from typography protection, duplicate detection without dropping, and quarantine bookkeeping.
CI (x86) shows ~2× the slowdown of arm64 (+27–34% vs +14.5%), consistent with string/regex-heavy guard code costing relatively more there.
The actual conflict
The gate budgets +20% vs a "v1.0-equivalent default path", but 2.0.0's entire direction — decided in the production audit and shipped in #154 — is that the default path deliberately does more work (detection without mutation, quarantine, provenance). The budget was never revisited when that direction was set. The gate is now enforcing a v1.0-era performance contract against a v2.0-era safety contract; one of them has to give, and that is a product decision, not a release-engineering one. (Related history: #152/#153 neutering the gate with || true was itself flagged and reverted in #154 — silently accommodating the gate is not an option.)
Options
A. Accept the 2.0.0 default-path cost. Re-record benchmarks/cleanbench/results/baseline_v1.json on a CI-class runner at the release commit (or explicitly raise FULL_GATE_RUNTIME_SLOWDOWN), with the decision recorded in the changelog. Requires a commit, so the release becomes either 2.0.1 or a re-tag of v2.0.0 (see tag note below).
B. Claw the cost back first. Profile-guided candidates: share one to_numeric pass between guard and coercion in _try_numeric; dedupe context profiling between build_engine_cache and the step-level profilers. Estimated recovery ~10–15% on CI, which would bring the code back under the existing budget. Then release as above.
C. Redesign the gate (the durable fix). Measure the v1.0-reference implementation in the same run and gate on the relative ratio, eliminating cross-runner drift entirely. This was already identified as the open question in the v1.2.0 post-mortem; the Jul 5–12 marginal flip-flops (+20.4%/+21.4% fails around genuine passes) show the absolute design has near-zero noise margin even when the code is fine.
Tag state
Two tags now point at commits that were never published: v1.2.0 (9368ea4) and v2.0.0 (629429f). Since no artifacts were ever produced from either, deleting and re-tagging v2.0.0 on the eventual release commit is defensible; going to 2.0.1/2.1.0 and leaving them is the conservative alternative. Either way, main's version metadata (2.0.0 in pyproject.toml/__init__.py, changelog dated 2026-07-20) is release-ready and validated: full test matrix green on PR #155, artifacts built and twine check-clean, wheel smoke-tested in a clean environment.
Exact unblock path (if Option A is chosen)
- On a branch from
main: regenerate baseline_v1.json via the documented baseline harness on a ubuntu-latest runner (python -m benchmarks.cleanbench --update-baseline, in CI, not locally), commit with the rationale, PR, merge.
- Delete and re-create
v2.0.0 on the merge commit (or bump to 2.0.1 and tag that).
- The tag push runs
release.yml end-to-end: trusted publishing to PyPI, GitHub Release with artifacts, checksums, and attestations — no further manual steps.
Release 2.0.0 (tag
v2.0.0at 629429f, PR #155) reached the release pipeline with everything green except the CleanBench T5 runtime gate, which failed at +27.68% against the 20% budget (run 29736705242). Nothing was published: PyPI and GitHub Releases still have 1.1.1 as latest. This issue documents why the failure is real, what causes it, and the decision needed to unblock.The gate failure is deterministic, not runner variance
The July 18 v1.2.0 post-mortem attributed the second failed attempt to a "±30% runner lottery vs absolute baseline". The full nightly
cleanbench-fullhistory disproves that:baseline_v1.jsonThe gate passed on CI runners as recently as Jul 10, so the committed baseline (0.3878s / 50k rows) is CI-achievable and the instrument works. A regression landed
Jul 13, was partially recovered (incl. #151's vectorized sci-notation guard), and the code has sat at a stable **+27–34% on CI** for seven consecutive runs. Re-running the workflow is dice-rolling, not a fix.Where the time goes
Controlled same-machine A/B (T5 fixture, 50k rows, min-of-5, interleaved rounds, arm64):
cProfile attribution of the delta on the default path:
engine/cache.py:build_engine_cache(~0.08s new): column contexts + numeric correlation for the engine-backedauto_missing/auto_outliersdefault steps — intended safety-by-default work, and the correlation matrix is genuinely consumed byengine/missing.py.steps/dtypes.py:_try_numericroughly doubled (10pd.to_numericpasses vs 5): the guard-then-coerce pattern from the coerced-cell quarantine work.CI (x86) shows ~2× the slowdown of arm64 (+27–34% vs +14.5%), consistent with string/regex-heavy guard code costing relatively more there.
The actual conflict
The gate budgets +20% vs a "v1.0-equivalent default path", but 2.0.0's entire direction — decided in the production audit and shipped in #154 — is that the default path deliberately does more work (detection without mutation, quarantine, provenance). The budget was never revisited when that direction was set. The gate is now enforcing a v1.0-era performance contract against a v2.0-era safety contract; one of them has to give, and that is a product decision, not a release-engineering one. (Related history: #152/#153 neutering the gate with
|| truewas itself flagged and reverted in #154 — silently accommodating the gate is not an option.)Options
A. Accept the 2.0.0 default-path cost. Re-record
benchmarks/cleanbench/results/baseline_v1.jsonon a CI-class runner at the release commit (or explicitly raiseFULL_GATE_RUNTIME_SLOWDOWN), with the decision recorded in the changelog. Requires a commit, so the release becomes either 2.0.1 or a re-tag ofv2.0.0(see tag note below).B. Claw the cost back first. Profile-guided candidates: share one
to_numericpass between guard and coercion in_try_numeric; dedupe context profiling betweenbuild_engine_cacheand the step-level profilers. Estimated recovery ~10–15% on CI, which would bring the code back under the existing budget. Then release as above.C. Redesign the gate (the durable fix). Measure the v1.0-reference implementation in the same run and gate on the relative ratio, eliminating cross-runner drift entirely. This was already identified as the open question in the v1.2.0 post-mortem; the Jul 5–12 marginal flip-flops (+20.4%/+21.4% fails around genuine passes) show the absolute design has near-zero noise margin even when the code is fine.
Tag state
Two tags now point at commits that were never published:
v1.2.0(9368ea4) andv2.0.0(629429f). Since no artifacts were ever produced from either, deleting and re-taggingv2.0.0on the eventual release commit is defensible; going to 2.0.1/2.1.0 and leaving them is the conservative alternative. Either way,main's version metadata (2.0.0 inpyproject.toml/__init__.py, changelog dated 2026-07-20) is release-ready and validated: full test matrix green on PR #155, artifacts built andtwine check-clean, wheel smoke-tested in a clean environment.Exact unblock path (if Option A is chosen)
main: regeneratebaseline_v1.jsonvia the documented baseline harness on aubuntu-latestrunner (python -m benchmarks.cleanbench --update-baseline, in CI, not locally), commit with the rationale, PR, merge.v2.0.0on the merge commit (or bump to 2.0.1 and tag that).release.ymlend-to-end: trusted publishing to PyPI, GitHub Release with artifacts, checksums, and attestations — no further manual steps.