Skip to content

fix(watch): coalesce rebuilds triggered during a burst of file changes - #14202

Open
glours wants to merge 3 commits into
docker:mainfrom
glours:watch-rebuild-coalescing
Open

fix(watch): coalesce rebuilds triggered during a burst of file changes#14202
glours wants to merge 3 commits into
docker:mainfrom
glours:watch-rebuild-coalescing

Conversation

@glours

@glours glours commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

What I did
compose watch triggered one rebuild per debounced batch, so a burst of edits arriving faster than the debounce window (e.g. from a coding agent editing several files) queued up several rebuilds in a row instead of a single consolidated one.

Rebuilds now run asynchronously through a small scheduler that coalesces requests received while a rebuild is in flight into one trailing rebuild, without adding latency to an isolated edit. This also closes a race where a plain restart could run concurrently with an asynchronous rebuild of the same service.

Related issue
N/A

(not mandatory) A picture of a cute animal, if possible in relation to what you did
image

@glours
glours requested review from a team as code owners September 9, 2026 07:35
@glours
glours requested a review from ndeloof September 9, 2026 07:35
compose watch triggered one rebuild per debounced batch, so a burst of
edits arriving faster than the debounce window (e.g. from a coding
agent editing several files) queued up several rebuilds in a row
instead of a single consolidated one.

Rebuilds now run asynchronously through a small scheduler that
coalesces requests received while a rebuild is in flight into one
trailing rebuild, without adding latency to an isolated edit. This
also closes a race where a plain restart could run concurrently with
an asynchronous rebuild of the same service.

Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
@glours
glours force-pushed the watch-rebuild-coalescing branch from a840592 to 3e3f0eb Compare September 9, 2026 10:51
@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.52239% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/compose/watch.go 83.33% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@ndeloof ndeloof left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid work overall: the scheduler is minimal and correct (decision taken synchronously under the lock, so requests racing a run's completion are neither lost nor doubled), the defer ordering in watchEvents guarantees no rebuild goroutine outlives the function, and the test suite is exemplary — deterministic channel-driven fake with the no-reentrancy invariant asserted inside the fake itself, and the assertNoRebuildStarts justification is exactly right. -race passes locally on the new tests.

One behavioral edge worth an explicit decision before merge, then small things:

[medium] A sync+restart change arriving while a rebuild of the same service is ACTIVE is silently lost.
Timeline: batch 1 triggers rebuild(A); while the image builds, batch 2 carries a sync+restart change for A. The sync lands in the doomed old container (the documented tradeoff), the restart is skipped by InFlightOrPending, and the new container comes from a build context snapshotted before the change. Net effect: the edit is silently dropped until the user touches the file again. Before this PR the synchronous rebuild made batch 2 converge; and the burst-editing regime this PR targets makes the overlap window common, not exotic.
The skip is fully correct for a pending rebuild (it will re-read the context, so the change is included and the restart is redundant). It is only the active case that loses data. Suggested fix: distinguish the two — skip on pending as today, and on active either requeue the restart to run after the current rebuild completes, or requeue a trailing scheduler.Request([service]) (converges via context re-read, at the cost of an extra — usually cache-warm — rebuild in the rare overlap).

[low] After Ctrl-C, run() still drains the pending set with a cancelled ctx, so the user log gets an extra "Build failed. Error: context canceled". A ctx.Err() check before each drain iteration would make shutdown silent.

[low] _ = s.rebuild(services) is actually fine — s.rebuild reports everything to options.LogTo itself — but that contract deserves a one-line comment on the discard, or the next reader will read it as a swallowed error.

[question] s.rebuild (build/create/start) now runs concurrently with the event loop's own uses of the same *types.Project (restart, syncer, exec hooks). Worth double-checking none of those paths mutates the project in place — the scheduler serializes rebuilds against each other, but not against the loop.

[nit] The comment on TestHandleWatchBatch_SkipsRestartForServiceWithRebuildInFlight references "#14051 (follow-up)" — that's the TTY renderer PR; stale reference.
[nit] In TestRebuildScheduler_NoConcurrentRebuilds, the local max shadows the builtin.

A rebuild whose build context was snapshotted before a newer change to
one of its own services is doomed: its output will be replaced by the
trailing rebuild anyway, and meanwhile a sync+restart change landing in
that window was silently lost — the sync went to the doomed container,
the restart was skipped, and the replacement image predated the edit.

The scheduler now gives each run its own cancellable context and
upholds one invariant: every request ends up covered by a rebuild whose
snapshot postdates it. A request naming an actively-rebuilt service
interrupts the run and folds its whole active set back into pending
(interruption kills the other services' rebuild too), so the trailing
rebuild redoes everything on a fresh snapshot. Requests for other
services keep coalescing without interrupting. handleWatchBatch drops a
restart only when the rebuild is pending (that one converges on its
own) and folds it into a Request when the rebuild is in flight —
recreate + start is the restart intent, converged. The run loop also
stops draining once the watch context is cancelled, which silences the
spurious "Build failed: context canceled" on Ctrl-C.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
@ndeloof

ndeloof commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Concrete, tested proposal for the [medium] finding above: glours#124 — targets this PR's branch, so merging it updates this PR. Design in one line: a rebuild made stale by a newer request for one of its own services is interrupted (per-run cancellable context) and its whole active set folds back into pending, upholding the invariant that every request ends up covered by a rebuild whose context snapshot postdates it. Also turns restart-during-in-flight into a Request (converged restart intent) and stops the drain on watch shutdown ([low] Ctrl-C noise). go test -race green.

@ndeloof ndeloof left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All findings from my earlier review are addressed at this head — the [medium] sync+restart convergence loss (a rebuild made stale by a newer request is now interrupted, its active set folded back into pending, and a restart landing on an in-flight rebuild folds into a Request), the shutdown-drain noise, and the stale test reference. I validated the merged change locally before it landed here (full pkg/compose suite with -race, lint clean), and CI is green on this head. The scheduler invariant — every request ends up covered by a rebuild whose context snapshot postdates it — is exactly the right contract for watch. LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants