Skip to content

fix(cli): enforce the worker time limit at the process level - #3432

Merged
chubes4 merged 1 commit into
mainfrom
fix/issue-3431-data-machine
Aug 27, 2026
Merged

fix(cli): enforce the worker time limit at the process level#3432
chubes4 merged 1 commit into
mainfrom
fix/issue-3431-data-machine

Conversation

@chubes4

@chubes4 chubes4 commented Aug 27, 2026

Copy link
Copy Markdown
Member

Closes #3431.

Problem

worker run --once --time-limit=60 prints its terminal summary at 48s and then never exits:

passes  duration_seconds  stop_reason  action_completions  pending_jobs
1       48                once         1                   420

Ten minutes later it is still resident and still burning CPU, with no progress — pending jobs unchanged at 420 across the whole window. It only stops when killed.

The loop-level bounds correctly stop the work. Nothing bounds the process, so a hung drain pass, output step, or shutdown callback keeps it alive indefinitely.

This is load-bearing, because the remedy Data Machine itself recommends for a starved scheduler is running this command on a cadence:

"condition": "scheduler_dispatcher_starved",
"recommendation": { "code": "run_supported_worker",
                    "command": "wp datamachine worker run --once" }

A command that never returns cannot be scheduled. Under any "skip if the previous run is still going" supervisor it runs exactly once and never again, while the orphan accumulates CPU. On the affected host that left 427 pending jobs undrained, the oldest from 2026-05-10.

Change

WorkerProcessDeadline arms a SIGALRM watchdog for the configured budget:

  • On fire: warn and exit(124), the conventional timeout status.
  • If that exit is trapped by a hung shutdown path, a short follow-up alarm escalates to SIGKILL, so the bound holds even when the exit itself is blocked. This is the observed failure mode — the summary had already printed, so the hang was after the loop finished.
  • Restores the caller's previous SIGALRM handler and async-signal mode on disarm.
  • Refuses to arm over a pending alarm rather than clobbering another owner's.
  • No-ops when pcntl is unavailable; existing loop-level bounds are unchanged.

Armed only when --time-limit > 0, preserving the documented 0 means no time limit.

Test

tests/worker-process-exit-smoke.php re-executes itself as child processes modelling both real failure shapes, and asserts each terminates on its own within 8s with status 124:

I verified the behavioural assertions are meaningful rather than shape checks by running the identical hang with no guard armed:

$ timeout 12 php control_no_guard.php     # same shutdown-spin, no deadline
elapsed=12s rc=124                        # never self-terminated; killed externally

With the guard the equivalent child exits by itself in under 8 seconds. So the guard is demonstrably what produces the bound.

The suite skips cleanly with a reported reason on hosts without pcntl.

Verification

worker-process-exit-smoke   OK (42 assertions)
worker-cli-smoke            OK (55 assertions)
worker-lock-smoke           OK (33 assertions)
job-liveness-cli-smoke      26 assertions, 0 failures

Reviewer notes

Two things worth your judgement:

  1. The new test mixes real behavioural coverage with a few source-text assert_contains checks (and adds one to worker-cli-smoke.php). The child-process assertions are the ones carrying the weight; the string checks are weaker and I am happy to drop them if you would rather not have shape assertions in the suite.
  2. This bounds the process, which is what the issue asked for. It does not explain why the pass was spinning without progress in the first place. That underlying behaviour may deserve its own investigation — the guard turns an unbounded hang into a clean, observable 124, which should make it far easier to catch.

Provenance

Drafted by homeboy agent-task fanout cook-batch (OpenCode, glm-5.3). The cook exited without promoting, so I reviewed the diff, proved the behavioural boundary with the no-guard control above, ran the surrounding suites, and shipped it deliberately.

AI assistance

  • AI assistance: Yes
  • Tool(s): Claude Sonnet 4.6 via Claude Code; patch drafted by OpenCode (glm-5.3) under homeboy agent-task cook-batch
  • Used for: I diagnosed and filed the underlying issue from a live host, cooked the fix, then reviewed the patch, verified the guard is what bounds the process, ran the gates, and wrote this description. Reviewed by me before opening.

`worker run --once --time-limit=60` printed its terminal summary at 48s and
then never exited, staying resident and burning CPU while making no progress.
The loop-level bounds stop the work, but nothing bounds the process, so a hung
drain pass, output step, or shutdown callback keeps it alive indefinitely.

This matters because the starved-scheduler remedy Data Machine itself
recommends is running this command on a cadence:

    "condition": "scheduler_dispatcher_starved",
    "recommendation": { "command": "wp datamachine worker run --once" }

A command that never returns cannot be scheduled. Under any "skip if the
previous run is still going" supervisor it runs exactly once and never again,
while the orphaned process accumulates CPU.

Arm a SIGALRM watchdog for the configured budget. When it fires the process
warns and exits with the conventional timeout status 124; if that exit is
trapped by a hung shutdown path, a short follow-up alarm escalates to SIGKILL
so the bound still holds. The guard restores the caller's previous signal
state, refuses to clobber a pending alarm, and no-ops on hosts without pcntl,
where the existing loop-level bounds are unchanged.

Refs #3431
@chubes4
chubes4 force-pushed the fix/issue-3431-data-machine branch from 0b233b5 to e7e9c9f Compare August 27, 2026 20:27
@chubes4
chubes4 merged commit db68f93 into main Aug 27, 2026
30 checks passed
@chubes4
chubes4 deleted the fix/issue-3431-data-machine branch August 27, 2026 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

worker run --once does not exit and ignores --time-limit at the process level

1 participant