Skip to content

ci: stop the rocm container build from timing out - #316

Open
lstein wants to merge 1 commit into
mainfrom
ci/rocm-container-build-timeout
Open

lstein wants to merge 1 commit into
mainfrom
ci/rocm-container-build-timeout

Conversation

@lstein

@lstein lstein commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Problem

The rocm leg of build container image has timed out four times — 2026-07-28, 2026-09-17 (×2) and 2026-09-22 (run 35724289637). cuda and cpu are unaffected.

This is not a regression. Run 35724289637 (42m37s, timed out) and run 35724304187 (16m25s, green) were pushed ten seconds apart on near-identical trees. Every phase of the slow run was ~2.4× slower:

phase healthy run timed-out run
apt install 59s 125s
uv sync --extra rocm 296s 529s
snapshot before SBOM 54s 353s
SBOM (syft) 44s 178s
exporting layers 386s 927s
attestation export ~60s 178s
pushing layers 120s killed 52s in
total ~17.5 min >40 min

It was runner quality, and the job had no headroom to absorb it.

Why there's no headroom

The rocm image is big, and its bulk is in one layer — compressed sizes from ghcr:

variant total biggest layer (the venv)
rocm 7.87 GB 7.22 GB
cuda 5.16 GB 4.52 GB
cpu 1.46 GB 0.82 GB

BuildKit gzips a single layer on one thread, and the build cache is disabled, so ~6.5 min of the budget is one gzip stream even on a healthy runner. A ~17 min median against a 40 min cap dies at a 2.3× slowdown. cuda does the same work in 244s and has 4× headroom.

Changes

  1. timeout-minutes: 4090. Removes the flakiness outright, costs nothing on healthy runs, still fails fast on a genuine hang.

  2. Export at gzip -1 instead of BuildKit's default gzip -6 (~19 MB/s over that one layer). 2-3× faster compression for ~15% larger layers — a good trade when the bottleneck is CI wall-clock, not registry bandwidth.

  3. Fix the paths filter, which listed workflows/build-container.yml without the .github/ prefix — a path that does not exist in the repo, so edits to this workflow never triggered it. Without this, the change would not run until some unrelated push happened to touch invokeai/** or the Dockerfile.

force-compression stays off, so base-image layers keep their existing compression and only newly built layers are affected.

One trap worth calling out

Setting a compression level means using outputs instead of push, and the condition push carried cannot move verbatim. github.event.inputs.* is always a string, and every non-empty string is truthy in an expression, so the bare github.event.inputs.push-to-registry the old condition ended in evaluates as true when the box is unchecked. push: absorbed that because build-push-action reads it with core.getBooleanInput, which maps "false"false. outputs: is read with getInputList and passes any non-empty string straight to --output, so without an explicit == 'true' a manual run from a non-main branch with the box unchecked would have published that branch to ghcr.

An adversarial review of the first draft of this PR caught exactly that. All seven trigger combinations were then checked against the old push decision:

trigger old new
push to main push push
push tag v*.*.* push push
dispatch from main, checked push push
dispatch from main, unchecked push push (pre-existing: main short-circuits the box)
dispatch from branch X, checked push push
dispatch from branch X, unchecked no push no push
dispatch from a tag ref, unchecked push push

Follow-up (not in this PR)

A registry layer cache — cache-from: type=registry,ref=ghcr.io/invoke-ai/invokeai-7:main-<variant> plus cache-to: ...,mode=min — would let the 7.2 GB layer be reused by digest on the majority of pushes that don't touch pyproject.toml/uv.lock, cutting the common case to a couple of minutes. Worth doing separately. Note the commented-out type=gha lines in this file can't serve that purpose: 3 variants × ~8 GB is far past the 10 GB per-repo GHA cache limit.

Verification

First main build after merge should be compared against the 7.87 GB baseline above; if gzip -1 costs materially more than ~15%, level 3 is the fallback.

Note that changing the compression level changes the freshly built layers' digests, so that first push re-uploads them as new blobs and every existing puller re-downloads them once. Base-image layers are unaffected.

🤖 Generated with Claude Code

The rocm leg of `build container image` has timed out four times
(2026-07-28, 2026-09-17 x2, 2026-09-22). It is not a regression: run
35724289637 (42m37s, timed out) and run 35724304187 (16m25s, green) were
pushed ten seconds apart on near-identical trees. Every phase of the slow
run was ~2.4x slower than the fast one - apt 59s->125s, `uv sync`
296s->529s, sbom 44s->178s, layer export 386s->927s. It is runner
quality, and the job had no headroom to absorb it.

It has no headroom because the rocm image is big: its venv layer is
7.2GB compressed (cuda 4.5GB, cpu 0.8GB), BuildKit gzips a single layer
on one thread, and there is no build cache, so ~6.5 min of the budget is
one gzip stream even on a healthy runner. A ~17 min median against a 40
min cap dies at a 2.3x slowdown.

So: raise the cap to 90 min, and drop the export from BuildKit's default
gzip -6 (~19MB/s here) to gzip -1, which is 2-3x faster for ~15% larger
layers. The cap is what actually fixes this - gzip -1 attacks the single
largest slice, but the build phases ahead of the export are untouched by
it.

Setting a compression level means using `outputs` instead of `push`, so
the condition `push` carried moves into the expression. It does not move
verbatim: `github.event.inputs.*` is always a string and every non-empty
string is truthy, so the bare reference the old expression ended in
evaluated as TRUE for an unchecked box. `push:` absorbed that because the
action ran it through core.getBooleanInput, which maps "false" to false.
`outputs:` is read with getInputList and passes any non-empty string
straight to --output, so the reference needs an explicit `== 'true'` or a
manual run from a non-main branch with the box unchecked would publish
that branch to ghcr. All seven trigger combinations were checked against
the old push decision and match.

Also fix the paths filter, which listed `workflows/build-container.yml`
without the `.github/` prefix - a path that does not exist, so edits to
this workflow never triggered it. Without this the change would not run
until some unrelated push happened to touch `invokeai/**` or the
Dockerfile.

`force-compression` stays off, so base-image layers keep their existing
compression and only newly built layers are affected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant