Skip to content

fix: stop binding SIGPIPE to the root context so registry resets don't cancel builds#10107

Open
DionJones615 wants to merge 1 commit into
GoogleContainerTools:mainfrom
DionJones615:fix/sigpipe-build-cancellation
Open

fix: stop binding SIGPIPE to the root context so registry resets don't cancel builds#10107
DionJones615 wants to merge 1 commit into
GoogleContainerTools:mainfrom
DionJones615:fix/sigpipe-build-cancellation

Conversation

@DionJones615

Copy link
Copy Markdown

Fixes: #10106

Description

Run() in cmd/skaffold/app/skaffold.go bound syscall.SIGPIPE to the root cancellation context:

ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM, syscall.SIGINT, syscall.SIGPIPE)

Per the Go os/signal docs, once a program registers SIGPIPE via Notify/NotifyContext the runtime delivers it for every file descriptor (not just stdout/stderr). So a SIGPIPE raised when a registry such as mcr.microsoft.com resets an idle connection mid-build gets delivered to the root context and is treated as a request to cancel the whole run. Multi-stage builds whose first stage runs longer than the registry's ~60s idle timeout are cancelled with reason broken pipe signal received (the strace in #10106 shows the ECONNRESETEPIPESIGPIPE sequence). docker build of the same Dockerfile succeeds because only skaffold turns that SIGPIPE into a cancellation.

This change stops treating SIGPIPE as a cancellation signal: it removes SIGPIPE from NotifyContext and signal.Ignores it instead (extracted into a small rootContext() helper so it can be unit-tested).

Ignore is deliberate rather than simply dropping SIGPIPE from the list. SIGPIPE was originally caught on purpose (#510 / #515) so that a closed output pipe (e.g. skaffold ... | head) would not abruptly kill skaffold before graceful cleanup finished. Ignoring it preserves that intent — broken-pipe writes return EPIPE to the caller instead of either cancelling the run (this bug) or terminating the process via the default disposition (what #515 set out to avoid).

Testing

  • Added a regression test (TestRootContextIgnoresSIGPIPE, cmd/skaffold/app/skaffold_signal_test.go, gated !windows) asserting a SIGPIPE delivered to the process does not cancel rootContext(). It fails against the previous code (SIGPIPE in NotifyContext) and passes with this change. make quicktest / package tests pass; gofmt and go vet clean.
  • End-to-end with the exact Dockerfile from this issue (mcr.microsoft.com/dotnet/sdk:10.0 + RUN … sleep 90, then aspnet:10.0) via the cluster (Kaniko) builder on Linux, plain skaffold build:
    • Before (v2.18.0): canceled ≈68s in — during sleep 90, just after the registry's ~60s idle reset — Build [test] was canceled, exactly as reported.
    • After: rides through the sleep and both stages to Build [test] succeeded (~2 min). No cancellation.

User facing changes

  • Before: long-running multi-stage builds against registries that reset idle connections (e.g. mcr.microsoft.com) are cancelled mid-build with broken pipe signal received.
  • After: such a SIGPIPE is ignored; the build runs to completion. Closing skaffold's output pipe (e.g. skaffold ... | head) still shuts down gracefully — writes return EPIPE rather than cancelling the run or killing the process.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a rootContext helper function in cmd/skaffold/app/skaffold.go to explicitly ignore syscall.SIGPIPE signals, preventing them from triggering context cancellation and aborting long-running builds when registry connections are reset. A corresponding regression test TestRootContextIgnoresSIGPIPE has been added in cmd/skaffold/app/skaffold_signal_test.go to verify this behavior. There are no review comments, so I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@DionJones615 DionJones615 force-pushed the fix/sigpipe-build-cancellation branch from 71a015e to 3e42b5f Compare June 10, 2026 17:56
The root command binds syscall.SIGPIPE to the cancellation context in
cmd/skaffold/app/skaffold.go. Once a process registers SIGPIPE via
signal.Notify/NotifyContext, the Go runtime delivers it for every file
descriptor, so a SIGPIPE raised when a registry (e.g. mcr.microsoft.com)
resets an idle connection mid-build is interpreted as a request to
cancel the whole run. Multi-stage builds whose first stage runs longer
than the registry's ~60s idle timeout are cancelled with the reason
"broken pipe signal received".

Stop treating SIGPIPE as cancellation: remove it from NotifyContext and
signal.Ignore it instead. Ignoring it (rather than just dropping it)
preserves the original reason SIGPIPE was caught in GoogleContainerTools#510 -- broken-pipe
writes during shutdown return EPIPE instead of killing the process, so
cleanup still completes when skaffold's output pipe is closed.

Fixes GoogleContainerTools#10106
@DionJones615 DionJones615 force-pushed the fix/sigpipe-build-cancellation branch from 3e42b5f to e5576a5 Compare June 10, 2026 18:02
@DionJones615 DionJones615 marked this pull request as ready for review June 10, 2026 18:05
@DionJones615 DionJones615 requested a review from a team as a code owner June 10, 2026 18:05
@foriequal0

Copy link
Copy Markdown
Contributor

I understand it could fix the symptom. But I wonder why we get ECONNRESET and EPIPE errors in the first place.

@lucytesco

Copy link
Copy Markdown

Any update on when this might get merged?

@pikuman83

Copy link
Copy Markdown

Confirming this with a broader reproduction that I think strengthens the case for #10107.

Environment: Google Cloud Build (not local) running skaffold run, deploying to GKE.
Reproduced on skaffold v2.4.0, v2.16.0, v2.22.0, with both the googleCloudBuild
builder and the default local docker builder, and with both LEGACY and
CLOUD_LOGGING_ONLY Cloud Build logging.

Symptom: ~60–130s into the build, every in-flight artifact prints
Build [X] was canceled, skaffold prints its normal version/exit banner and
exits 0, so Cloud Build records the step as SUCCESS but the deploy phase never
runs — a silently-green build that deployed nothing. gcloud builds describe shows
status: SUCCESS, no failureInfo, BUILD phase ~98s, timeout (5400s) nowhere near.

Key data points supporting the SIGPIPE/idle-reset root cause in this PR:

  • It is not limited to mcr.microsoft.com. One of our images builds FROM mcr.microsoft.com/dotnet/core/sdk:2.2 and is the classic case, but a run with the
    local builder was also cancelled mid-build on an image based on
    google/cloud-sdk — i.e. any registry that resets an idle connection during a

    60s stage triggers it.

  • It is builder- and version-independent, which matches "SIGPIPE bound to the
    root context" rather than anything in a specific builder.
  • Non-skaffold processes in the same Cloud Build step (bash, docker, gcloud,
    kubectl) run fine well past the cancellation — consistent with it being skaffold's
    signal handler, not the environment.
  • Onset was sudden (~2026-06-08) with no skaffold/config change on our side,
    consistent with a registry changing its idle-connection reset behaviour.

So signal.Ignore(SIGPIPE) in #10107 should fix this class of failure broadly, not
just the MCR case. Happy to test a build of the PR branch against our Cloud Build
setup if that helps validate. Could this also get backported to a 2.x patch
release? It currently breaks deploys with a false SUCCESS, which is especially
dangerous in CI.

@pikuman83

Copy link
Copy Markdown

Hi! Just following up on this PR. Is there any update on whether it might be reviewed or merged?

We're currently blocked by this issue, and this PR appears to fix the exact problem we're seeing. We'd be happy to test it further or provide any additional information if that would help move it forward.

Thanks for your work on Skaffold!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

long-running mutistage build gets canceled around 1min if stages are from mcr.microsoft.com

4 participants