Skip to content

ENG-1796: the pod image reports the version it was built from - #404

Merged
alecantu7 merged 3 commits into
stagingfrom
alejandrocantu/eng-1796-pod-image-reports-its-real-version
Aug 27, 2026
Merged

ENG-1796: the pod image reports the version it was built from#404
alecantu7 merged 3 commits into
stagingfrom
alejandrocantu/eng-1796-pod-image-reports-its-real-version

Conversation

@alecantu7

@alecantu7 alecantu7 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

What

anton/Dockerfile hardcoded SETUPTOOLS_SCM_PRETEND_VERSION=2.0.0, so every turn a scratchpad pod ever served reported anton_version=2.0.0. The version is now resolved from tags on the runner and passed in as a build arg.

Why it was invisible

Nothing failed. The build succeeded, the pod ran, and 2.0.0 is a well-formed release number — so a version breakdown read it as a legitimate cohort rather than as a null. And bumping the pod image pin produced no observable change, which makes "did the deploy actually land?" unanswerable for the one image that is pinned by digest and bumped by hand.

Premise re-measured before building (prod PostHog 424726, 3d, real turns only per ENG-1692's filter). It had grown, not decayed:

anton_version turns installs share
2.0.0 166 109 46% (was 35% on 2026-08-20)
2.26.8.23.1 985 90 38%
2.26.8.20.3 168 17 7%

One caution for anyone quoting that: 166 turns across 109 "installs" is 1.5 each, consistent with near-ephemeral pod install-ids. That cohort is closer to pods than people.

How

The constant existed for a real reason — .dockerignore excludes .git to keep the context lean, so hatch-vcs has nothing to describe. Rather than un-ignore it, the version is resolved on the runner and handed to the build.

Resolution lives in its own version job on ubuntu-latest. The build runs on the self-hosted mdb-dev runner, and nothing in this repo has ever run setup-uv there — every existing usage is on ubuntu-latest. Putting the resolver in the build job would have placed an unproven dependency in front of the image. Splitting it also leaves the build's checkout shallow, since only the resolver needs history.

An empty arg now fails the build rather than falling back. A build that quietly substitutes something plausible is precisely what 2.0.0 was.

Blast radius — three things that could have broken, and don't

Risk Verdict
uv sync --frozen rejects a version that no longer matches the lock uv.lock records anton-agent with source = { editable = "." } and no version = line (dynamic = ["version"]). 2.0.0 appears nowhere in it
A consumer parses or compares anton_version ✅ Every consumer displays it — cli.py:1620, channel/branding.py, cowork's UpdatesSection.jsx, server-process.ts. No comparisons, no gates
The shared build-push-ecr action can't pass build args extra-build-args is already an input; no change to the org-wide action

Scope is the pod image only. This Dockerfile is referenced by one workflow; the PyPI wheel is built separately by release.yml / publish-staging.yml, which already resolve the version correctly.

SETUPTOOLS_SCM_PRETEND_VERSION is kept as ENV rather than narrowed to the build stage, deliberately — that is the conservative choice, identical in shape to today. See "follow-up" below.

Verification

Executed:

  • uvx hatch version resolves correctly here and prints only the version to stdout, so $( ) captures exactly it — checked explicitly, since a captured progress line would be baked into the image: captured=[2.26.8.26.2rc3] lines=0
  • The empty-arg guard, both branches. A || B && C precedence is a classic silent inversion, so it was run rather than reasoned about: version set → reaches the install step, exit 0; version empty → prints the error, exit 1, install never reached
  • 6 build assertions, all mutation-verified, each mutation asserted to have actually modified the file first

Measured during adversarial self-review (these drove 4c685f4):

  • With no tags, hatch-vcs resolves 2.0.0.dev1+g19c6e7515; with no .git, 2.0.0-dev. Both well-formed, both accepted by the original non-empty check — so losing fetch-depth: 0 would have rebuilt the 2.0.0 family silently. Now rejected in the Dockerfile and the workflow, so the guard holds however the image is built (mirroring cowork-server's 0.0.0* guard)
  • pipefail confirmed load-bearing: without it, a failing resolver is masked by tail and yields an empty version

NOT executed: the image itself is not built here — the local Docker daemon is not running. The build path is exercised by CI on this PR.

Automated coverage

tests/test_pod_image_version.py — build assertions rather than behaviour tests, because there is no runtime symptom to assert on. That is the same property that let this survive so long.

Mutation Caught by
version back to the literal 2.0.0 2 tests
drop the ARG declaration 1
drop the empty-arg guard 1
drop fetch-depth: 0 1
build no longer needs the version job 1
stop passing the version to the build 1
Dockerfile stops rejecting the 2.0.0* fallback 1
back to a prefix-only shape check 1
workflow stops rejecting the fallback 1
drop the single-line capture 1

One of these tests was vacuous and mutation caught it, not review. The single-line-capture assertion matched tail -n 1 anywhere in the run block — and the explanatory comment above the pipeline contains that string, so deleting the pipe left the test green. It now asserts on the pipeline itself. Recording it because a guard that cannot fail is worse than no guard: it reads as coverage.

Security check

Read-only build-metadata change. No new endpoint, no auth surface, no user input. No secrets are read, logged, or baked. .git still does not enter the build context or the image.

The one surface worth stating: the resolved version is expanded unquoted by the shared action into docker buildx build, so it is a value that reaches a shell. An earlier revision of this PR validated it with a prefix match (^[0-9]+\.[0-9]) and this section claimed that "the check rejects" whitespace. That claim was false2.26.8 --build-arg EVIL=1 passes a prefix match and becomes extra arguments to docker buildx build. Found by adversarially reviewing this PR and fixed in 4c685f4: validation is now grep -Eqx '[0-9]+\.[0-9][0-9A-Za-z.!+]*', whole-line over PEP 440 characters only, so no whitespace or shell metacharacter can survive it. The value's origin is a git tag, so exploiting it required a hostile or garbled tag — but the claim was load-bearing and wrong, which is why it is corrected here rather than quietly edited.

Merge order

None — this and the cowork-server half of ENG-1796 are independent repos with no shared contract. Either can land first.

Follow-up, not in this PR

SETUPTOOLS_SCM_PRETEND_VERSION is set as ENV, so it persists at runtime, where scratchpad_boot shells out to uv pip install for missing packages. Any package built from an sdist in a pod inherits anton's version. Narrowing it to the build stage would fix that, but it changes runtime environment as a side effect of a reporting fix, so it is left out deliberately and noted on the ticket.

…1796)

`Dockerfile` hardcoded `SETUPTOOLS_SCM_PRETEND_VERSION=2.0.0`, so every turn
a scratchpad pod ever served reported `anton_version=2.0.0`. Not a fallback
firing occasionally — the only value cloud ever reported, and re-measured at
46% of the install population on 2026-08-26 (up from 35% six days earlier).

Two consequences, both silent. A version breakdown read `2.0.0` as a
legitimate cohort rather than as a null, because it is a well-formed release
number. And bumping the pod image pin produced no observable change at all,
so "did the deploy land?" was unanswerable for the one image that is pinned
by digest and deployed by hand.

The constant was there for a real reason: `.dockerignore` excludes `.git` to
keep the context lean, so hatch-vcs has nothing to describe. Rather than
un-ignore it, the version is resolved on the runner and passed in.

Resolution lives in its own `version` job on ubuntu-latest. The build runs on
the self-hosted mdb-dev runner and nothing in this repo has ever run setup-uv
there, so resolving it in the build job would put an unproven dependency in
front of the image; splitting it also leaves the build's checkout shallow,
since only the resolver needs history.

An empty arg now fails the build rather than falling back. A build that
quietly substitutes something plausible is precisely what 2.0.0 was.

`uv.lock` records anton-agent with no `version =` line (`dynamic = ["version"]`)
and `2.0.0` appears nowhere in it, so `uv sync --frozen` is unaffected. Every
consumer of `anton_version` displays it; none compare or gate on it.

Guarded by six build assertions rather than convention — there is no runtime
symptom to assert on, which is why this survived so long. All six
mutation-verified, each mutation checked to have actually modified the file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alecantu7
alecantu7 requested a review from a team as a code owner August 26, 2026 22:00
@alecantu7

Copy link
Copy Markdown
Contributor Author

CI closed the one gap in the PR body

The body said the image is not built locally because the Docker daemon isn't running. This PR's own CI builds it, so that gap is now closed with executable evidence rather than left standing.

version job:

anton version: 2.26.8.26.2rc4.dev2+ga235462d5

Real, tag-derived, and carrying the exact commit — against 2.0.0 for every pod turn ever served. build then succeeded with that value, so the whole chain is exercised end to end: tags fetched → uvx hatch version on ubuntu-latest → job output → extra-build-argsARGSETUPTOOLS_SCM_PRETEND_VERSION → image.

That also settles the one assumption I flagged as unproven. Splitting resolution into its own ubuntu-latest job was a guess that setup-uv on the self-hosted mdb-dev runner was risky — untested either way. It is now moot: the split works, and the build job stayed unchanged apart from the extra arg.

Full status: run-tests, version, build, scope, CodeQL — all green.

Still not verified, and not verifiable here: that a pod running this image reports the new value on a real trace. That needs SCRATCHPAD_CONTROLLER__SCRATCHPAD_IMAGE re-pinned to a build containing this — the same manual gate that held ENG-1459 for four days. It is recorded on the ticket as the after-measurement, not claimed here.

alecantu7 and others added 2 commits August 26, 2026 15:42
…ole-line

Adversarial self-review of this PR found three defects in it. All measured,
not reasoned about.

1. The fallback rebuilds the bug. With no tags reachable hatch-vcs resolves
   `2.0.0.dev1+g<sha>`; with no .git at all, `2.0.0-dev`. Both are well-formed
   and both sailed through a non-empty check -- so losing `fetch-depth: 0`
   would have baked the 2.0.0 family straight back in, silently. That is the
   entire bug, reintroduced through the front door. anton is CalVer, so a
   2.0.0 version always means the tags did not arrive. Rejected in the
   Dockerfile as well as the workflow, so it holds however the image is built
   -- matching the 0.0.0 guard on the cowork-server side.

2. The shape check was a prefix match, and the value reaches a shell
   unquoted. The shared action expands `extra-build-args` unquoted into
   `docker buildx build`, and `2.26.8 --build-arg EVIL=1` passes
   `^[0-9]+\.[0-9]` -- becoming extra arguments to that command. This PR's
   body claimed the check "would have rejected it"; that claim was false.
   `grep -Eqx` over PEP 440 characters anchors both ends and makes it true.

3. A multi-line capture would corrupt $GITHUB_OUTPUT rather than fail, since
   the second line parses as another key=value pair. `tail -n 1` plus the
   whole-line check closes it. `pipefail` verified to be load-bearing here:
   without it a failing resolver is masked by the pipe.

Also fixes a VACUOUS test caught by mutation, not by reading: the assertion
for (3) matched `tail -n 1` anywhere in the `run` block, and the explanatory
comment above the pipeline contains that string -- so deleting the pipe left
the test green. It now asserts on the pipeline itself.

Four new mutations, all caught. Ten total on this file now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review finding, and it is the wrong-quietly one this ticket is about.

anton's image is SINGLE-STAGE. `COPY . /app` lands directly in the runtime
image, so `.dockerignore` is the only thing keeping the repo's history out of
every scratchpad pod -- and that exclusion had no guard and no test.

What makes it the ticket's own failure class rather than a tidiness nit:
un-ignoring `.git` produces no signal at all. Measured -
`SETUPTOOLS_SCM_PRETEND_VERSION` wins over VCS discovery (`9.9.9.9` overrode a
real `2.26.8.12.1rc6.dev8+g19c6e7515`), so the version stays correct, every
guard added in this PR still passes, the build goes green, and the pod runs
fine. The only symptom is full git history inside the image, which nothing
reports.

The asymmetry was backwards, too. cowork-server is multi-stage and deletes
.git in the builder, and its test asserts that -- so the repo with a
structural safety net was the covered one, while the repo where the exclusion
is load-bearing was not.

Two guards, matching the split used on the cowork-server side:

- a test asserting `.dockerignore` still carries a bare `.git` line, and
- a build-time `test ! -e /app/.git`, which catches .git arriving by any
  route (a negation pattern, a different context, a build that bypasses
  .dockerignore) rather than only the one spelling written in that file.

Mutations 11-13, all caught: removing the line, negating it to `!.git`, and
dropping the build guard.

Separately: the reported `fetch-depth: 0` gap does not reproduce at 4c685f4.
`test_the_version_job_fetches_tags` fails on all four ways of expressing it --
deleting the line, setting it to 1, deleting the whole `with:` block, and
commenting it out. Left as is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@pnewsam pnewsam left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

Verdict: APPROVE
Model: claude-opus-4-8[1m]

Reviewed the full diff (.dockerignore, scratchpad-dev-build.yml, Dockerfile, new test_pod_image_version.py). The one reason it holds: the version is resolved from tags in an isolated job, whole-line validated against PEP 440 before an unquoted shell expansion, and both the empty-arg and 2.0.0* fallbacks fail the build in the Dockerfile and the workflow — with the .git exclusion now guarded too. CI runs the resolver and the real image build green. No actionable findings.

Validation

  • Confirmed anton declares pyyaml>=6.0 as a direct runtime dependency (pyproject.toml:30), so the test file's import yaml / yaml.safe_load is safe here — unlike cowork-server, which correctly avoids it. run-tests passing confirms this at runtime.
  • Ran uvx hatch version in a local anton clone: resolves a single-line version with only the version on stdout (progress goes to stderr). The value passes the grep -Eqx '[0-9]+\.[0-9][0-9A-Za-z.!+]*' shape check and is not 2.0.0*.
  • Traced the Dockerfile guard precedence A || {…;exit 1} && case… && {test ! -e /app/.git || …} && uv venv…: the success path skips the || branch, and every failure branch exit 1s the shell, so the classic A||B&&C inversion does not occur. Verified empty-arg, 2.0.0*, and .git-present all fail correctly.
  • Confirmed the security fix: extra-build-args reaches docker buildx build unquoted, and whole-line PEP 440 validation blocks whitespace/metacharacter injection (2.26.8 --build-arg EVIL=1 is rejected). Value origin is a git tag.
  • CI: version pass, build pass, run-tests pass, CodeQL/Snyk pass. The resolver job and the real image build both run green.

No actionable findings. One noted observation (not a defect, and already deferred on the ticket): SETUPTOOLS_SCM_PRETEND_VERSION remains an ENV, so it persists at runtime where pods uv pip install from sdists. This is pre-existing behavior — the value merely changes from 2.0.0 to the real CalVer — so it is out of scope for this reporting fix.

@alecantu7
alecantu7 merged commit 47fc554 into staging Aug 27, 2026
11 checks passed
@alecantu7
alecantu7 deleted the alejandrocantu/eng-1796-pod-image-reports-its-real-version branch August 27, 2026 01:16
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 27, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants