fix(cli): enforce the worker time limit at the process level - #3432
Merged
Conversation
chubes4
force-pushed
the
fix/issue-3431-data-machine
branch
from
August 27, 2026 19:58
e830cf6 to
0b233b5
Compare
`worker run --once --time-limit=60` printed its terminal summary at 48s and
then never exited, staying resident and burning CPU while making no progress.
The loop-level bounds stop the work, but nothing bounds the process, so a hung
drain pass, output step, or shutdown callback keeps it alive indefinitely.
This matters because the starved-scheduler remedy Data Machine itself
recommends is running this command on a cadence:
"condition": "scheduler_dispatcher_starved",
"recommendation": { "command": "wp datamachine worker run --once" }
A command that never returns cannot be scheduled. Under any "skip if the
previous run is still going" supervisor it runs exactly once and never again,
while the orphaned process accumulates CPU.
Arm a SIGALRM watchdog for the configured budget. When it fires the process
warns and exits with the conventional timeout status 124; if that exit is
trapped by a hung shutdown path, a short follow-up alarm escalates to SIGKILL
so the bound still holds. The guard restores the caller's previous signal
state, refuses to clobber a pending alarm, and no-ops on hosts without pcntl,
where the existing loop-level bounds are unchanged.
Refs #3431
chubes4
force-pushed
the
fix/issue-3431-data-machine
branch
from
August 27, 2026 20:27
0b233b5 to
e7e9c9f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3431.
Problem
worker run --once --time-limit=60prints its terminal summary at 48s and then never exits:Ten minutes later it is still resident and still burning CPU, with no progress — pending jobs unchanged at 420 across the whole window. It only stops when killed.
The loop-level bounds correctly stop the work. Nothing bounds the process, so a hung drain pass, output step, or shutdown callback keeps it alive indefinitely.
This is load-bearing, because the remedy Data Machine itself recommends for a starved scheduler is running this command on a cadence:
A command that never returns cannot be scheduled. Under any "skip if the previous run is still going" supervisor it runs exactly once and never again, while the orphan accumulates CPU. On the affected host that left 427 pending jobs undrained, the oldest from 2026-05-10.
Change
WorkerProcessDeadlinearms a SIGALRM watchdog for the configured budget:exit(124), the conventional timeout status.SIGKILL, so the bound holds even when the exit itself is blocked. This is the observed failure mode — the summary had already printed, so the hang was after the loop finished.pcntlis unavailable; existing loop-level bounds are unchanged.Armed only when
--time-limit > 0, preserving the documented0 means no time limit.Test
tests/worker-process-exit-smoke.phpre-executes itself as child processes modelling both real failure shapes, and asserts each terminates on its own within 8s with status 124:once— prints the terminal summary, then aregister_shutdown_functionthat spins for 60s. This is the observed worker run --once does not exit and ignores --time-limit at the process level #3431 hang: work finished, shutdown never returns.time-limit— a non-terminating work loop that ignores every cooperative bound.I verified the behavioural assertions are meaningful rather than shape checks by running the identical hang with no guard armed:
With the guard the equivalent child exits by itself in under 8 seconds. So the guard is demonstrably what produces the bound.
The suite skips cleanly with a reported reason on hosts without
pcntl.Verification
Reviewer notes
Two things worth your judgement:
assert_containschecks (and adds one toworker-cli-smoke.php). The child-process assertions are the ones carrying the weight; the string checks are weaker and I am happy to drop them if you would rather not have shape assertions in the suite.124, which should make it far easier to catch.Provenance
Drafted by
homeboy agent-task fanout cook-batch(OpenCode, glm-5.3). The cook exited without promoting, so I reviewed the diff, proved the behavioural boundary with the no-guard control above, ran the surrounding suites, and shipped it deliberately.AI assistance
homeboy agent-task cook-batch