fix: stop binding SIGPIPE to the root context so registry resets don't cancel builds#10107
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a rootContext helper function in cmd/skaffold/app/skaffold.go to explicitly ignore syscall.SIGPIPE signals, preventing them from triggering context cancellation and aborting long-running builds when registry connections are reset. A corresponding regression test TestRootContextIgnoresSIGPIPE has been added in cmd/skaffold/app/skaffold_signal_test.go to verify this behavior. There are no review comments, so I have no feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
71a015e to
3e42b5f
Compare
The root command binds syscall.SIGPIPE to the cancellation context in cmd/skaffold/app/skaffold.go. Once a process registers SIGPIPE via signal.Notify/NotifyContext, the Go runtime delivers it for every file descriptor, so a SIGPIPE raised when a registry (e.g. mcr.microsoft.com) resets an idle connection mid-build is interpreted as a request to cancel the whole run. Multi-stage builds whose first stage runs longer than the registry's ~60s idle timeout are cancelled with the reason "broken pipe signal received". Stop treating SIGPIPE as cancellation: remove it from NotifyContext and signal.Ignore it instead. Ignoring it (rather than just dropping it) preserves the original reason SIGPIPE was caught in GoogleContainerTools#510 -- broken-pipe writes during shutdown return EPIPE instead of killing the process, so cleanup still completes when skaffold's output pipe is closed. Fixes GoogleContainerTools#10106
3e42b5f to
e5576a5
Compare
|
I understand it could fix the symptom. But I wonder why we get ECONNRESET and EPIPE errors in the first place. |
|
Any update on when this might get merged? |
|
Confirming this with a broader reproduction that I think strengthens the case for #10107. Environment: Google Cloud Build (not local) running Symptom: ~60–130s into the build, every in-flight artifact prints Key data points supporting the SIGPIPE/idle-reset root cause in this PR:
So |
|
Hi! Just following up on this PR. Is there any update on whether it might be reviewed or merged? We're currently blocked by this issue, and this PR appears to fix the exact problem we're seeing. We'd be happy to test it further or provide any additional information if that would help move it forward. Thanks for your work on Skaffold! |
Fixes: #10106
Description
Run()incmd/skaffold/app/skaffold.goboundsyscall.SIGPIPEto the root cancellation context:Per the Go
os/signaldocs, once a program registers SIGPIPE viaNotify/NotifyContextthe runtime delivers it for every file descriptor (not just stdout/stderr). So a SIGPIPE raised when a registry such asmcr.microsoft.comresets an idle connection mid-build gets delivered to the root context and is treated as a request to cancel the whole run. Multi-stage builds whose first stage runs longer than the registry's ~60s idle timeout are cancelled with reasonbroken pipe signal received(the strace in #10106 shows theECONNRESET→EPIPE→SIGPIPEsequence).docker buildof the same Dockerfile succeeds because only skaffold turns that SIGPIPE into a cancellation.This change stops treating SIGPIPE as a cancellation signal: it removes SIGPIPE from
NotifyContextandsignal.Ignores it instead (extracted into a smallrootContext()helper so it can be unit-tested).Ignoreis deliberate rather than simply dropping SIGPIPE from the list. SIGPIPE was originally caught on purpose (#510 / #515) so that a closed output pipe (e.g.skaffold ... | head) would not abruptly kill skaffold before graceful cleanup finished. Ignoring it preserves that intent — broken-pipe writes returnEPIPEto the caller instead of either cancelling the run (this bug) or terminating the process via the default disposition (what #515 set out to avoid).Testing
TestRootContextIgnoresSIGPIPE,cmd/skaffold/app/skaffold_signal_test.go, gated!windows) asserting a SIGPIPE delivered to the process does not cancelrootContext(). It fails against the previous code (SIGPIPE inNotifyContext) and passes with this change.make quicktest/ package tests pass;gofmtandgo vetclean.mcr.microsoft.com/dotnet/sdk:10.0+RUN … sleep 90, thenaspnet:10.0) via the cluster (Kaniko) builder on Linux, plainskaffold build:sleep 90, just after the registry's ~60s idle reset —Build [test] was canceled, exactly as reported.Build [test] succeeded(~2 min). No cancellation.User facing changes
mcr.microsoft.com) are cancelled mid-build withbroken pipe signal received.skaffold ... | head) still shuts down gracefully — writes returnEPIPErather than cancelling the run or killing the process.