Conversation
The rocm leg of `build container image` has timed out four times (2026-07-28, 2026-09-17 x2, 2026-09-22). It is not a regression: run 35724289637 (42m37s, timed out) and run 35724304187 (16m25s, green) were pushed ten seconds apart on near-identical trees. Every phase of the slow run was ~2.4x slower than the fast one - apt 59s->125s, `uv sync` 296s->529s, sbom 44s->178s, layer export 386s->927s. It is runner quality, and the job had no headroom to absorb it. It has no headroom because the rocm image is big: its venv layer is 7.2GB compressed (cuda 4.5GB, cpu 0.8GB), BuildKit gzips a single layer on one thread, and there is no build cache, so ~6.5 min of the budget is one gzip stream even on a healthy runner. A ~17 min median against a 40 min cap dies at a 2.3x slowdown. So: raise the cap to 90 min, and drop the export from BuildKit's default gzip -6 (~19MB/s here) to gzip -1, which is 2-3x faster for ~15% larger layers. The cap is what actually fixes this - gzip -1 attacks the single largest slice, but the build phases ahead of the export are untouched by it. Setting a compression level means using `outputs` instead of `push`, so the condition `push` carried moves into the expression. It does not move verbatim: `github.event.inputs.*` is always a string and every non-empty string is truthy, so the bare reference the old expression ended in evaluated as TRUE for an unchecked box. `push:` absorbed that because the action ran it through core.getBooleanInput, which maps "false" to false. `outputs:` is read with getInputList and passes any non-empty string straight to --output, so the reference needs an explicit `== 'true'` or a manual run from a non-main branch with the box unchecked would publish that branch to ghcr. All seven trigger combinations were checked against the old push decision and match. Also fix the paths filter, which listed `workflows/build-container.yml` without the `.github/` prefix - a path that does not exist, so edits to this workflow never triggered it. Without this the change would not run until some unrelated push happened to touch `invokeai/**` or the Dockerfile. `force-compression` stays off, so base-image layers keep their existing compression and only newly built layers are affected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
rocmleg of build container image has timed out four times — 2026-07-28, 2026-09-17 (×2) and 2026-09-22 (run 35724289637).cudaandcpuare unaffected.This is not a regression. Run
35724289637(42m37s, timed out) and run35724304187(16m25s, green) were pushed ten seconds apart on near-identical trees. Every phase of the slow run was ~2.4× slower:apt installuv sync --extra rocmIt was runner quality, and the job had no headroom to absorb it.
Why there's no headroom
The rocm image is big, and its bulk is in one layer — compressed sizes from ghcr:
BuildKit gzips a single layer on one thread, and the build cache is disabled, so ~6.5 min of the budget is one gzip stream even on a healthy runner. A ~17 min median against a 40 min cap dies at a 2.3× slowdown.
cudadoes the same work in 244s and has 4× headroom.Changes
timeout-minutes: 40→90. Removes the flakiness outright, costs nothing on healthy runs, still fails fast on a genuine hang.Export at gzip -1 instead of BuildKit's default gzip -6 (~19 MB/s over that one layer). 2-3× faster compression for ~15% larger layers — a good trade when the bottleneck is CI wall-clock, not registry bandwidth.
Fix the
pathsfilter, which listedworkflows/build-container.ymlwithout the.github/prefix — a path that does not exist in the repo, so edits to this workflow never triggered it. Without this, the change would not run until some unrelated push happened to touchinvokeai/**or the Dockerfile.force-compressionstays off, so base-image layers keep their existing compression and only newly built layers are affected.One trap worth calling out
Setting a compression level means using
outputsinstead ofpush, and the conditionpushcarried cannot move verbatim.github.event.inputs.*is always a string, and every non-empty string is truthy in an expression, so the baregithub.event.inputs.push-to-registrythe old condition ended in evaluates as true when the box is unchecked.push:absorbed that because build-push-action reads it withcore.getBooleanInput, which maps"false"→false.outputs:is read withgetInputListand passes any non-empty string straight to--output, so without an explicit== 'true'a manual run from a non-main branch with the box unchecked would have published that branch to ghcr.An adversarial review of the first draft of this PR caught exactly that. All seven trigger combinations were then checked against the old push decision:
mainv*.*.*main, checkedmain, uncheckedmainshort-circuits the box)Follow-up (not in this PR)
A registry layer cache —
cache-from: type=registry,ref=ghcr.io/invoke-ai/invokeai-7:main-<variant>pluscache-to: ...,mode=min— would let the 7.2 GB layer be reused by digest on the majority of pushes that don't touchpyproject.toml/uv.lock, cutting the common case to a couple of minutes. Worth doing separately. Note the commented-outtype=ghalines in this file can't serve that purpose: 3 variants × ~8 GB is far past the 10 GB per-repo GHA cache limit.Verification
First
mainbuild after merge should be compared against the 7.87 GB baseline above; if gzip -1 costs materially more than ~15%, level 3 is the fallback.Note that changing the compression level changes the freshly built layers' digests, so that first push re-uploads them as new blobs and every existing puller re-downloads them once. Base-image layers are unaffected.
🤖 Generated with Claude Code