fix: reap ghost jobs whose worker died without recording a result

[Greenbauer]

Problem

A detached task worker that dies without throwing — an unhandled rejection, an OOM SIGKILL, a native crash — never reaches runTrackedJob's catch, so the job file is never rewritten and the job stays status: "running" forever. /codex:status then reports a ghost job indefinitely.

Observed in practice (2026-07-03): a real background task died about 2 minutes in, and /codex:status kept reporting it as running for 28+ minutes until a human noticed the log file's mtime had stopped advancing and found no live worker process.

Fix

Two complementary guards, so the job record converges to the truth from either side:

1. Worker side — registerWorkerCrashGuard(workspaceRoot, jobId, logFile) (lib/tracked-jobs.mjs, installed in handleTaskWorker before runTrackedJob): on uncaughtException, unhandledRejection, SIGTERM, SIGINT, or SIGHUP, appends the reason to the job log, marks the job file failed, and exits.

2. Reader side — reapDeadJobs(workspaceRoot, jobs) (lib/tracked-jobs.mjs, wrapping every listJobs call site in lib/job-control.mjs): any queued/running job whose recorded pid fails a process.kill(pid, 0) liveness probe is rewritten as failed with a resume hint (task --resume-last). This catches the cases no in-process handler can (SIGKILL, hard native crashes). Details:

   • ESRCH ⇒ dead; EPERM ⇒ process exists but isn't ours, treated as alive.
   • The job file is re-read before rewriting, so a job that finished between the caller's read and the probe keeps its real result.
   • Jobs with no recorded pid are left untouched.

Testing

• New tests/tracked-jobs.test.mjs: dead-pid job is reaped to failed (job file + state index), live-pid job untouched, pid-less job untouched, finished-during-probe race keeps the stored result, and a spawned worker with the crash guard that hits an unhandled rejection exits 1 with the job file marked failed.
• npm test: the 5 new tests pass; 91/95 overall — the 4 failures (status shows phases…, status preserves adversarial…, result returns the stored output…, resolveStateDir uses a temp-backed…) fail identically on unmodified main on macOS and are unrelated to this change.
• npm run build: clean.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8a7d9bf376

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[rajpratham1]

This fixes a real problem and has solid test coverage, but I agree with the reviewer's concern and found one issue that should be addressed before merge.

Blocking issue

updatedAt is stale after reaping

markJobDead() creates:

const completedAt = nowIso();
const record = {
...
completedAt
};

Then upsertJob() updates the persisted state with a fresh updatedAt, but the returned record still contains the old updatedAt (or none at all).

Immediately afterwards, callers do:

sortJobsNewestFirst(reapDeadJobs(...))

So the in-memory list is sorted using stale timestamps, meaning a freshly reaped failed job may still appear far down the list until the next /codex:status.

The returned object should include:

updatedAt: completedAt

or return the fully updated state after upsertJob().

What's good
Fixes "ghost" jobs that stay in running forever after crashes.
Uses process.kill(pid, 0) correctly to probe process existence.
Treats EPERM as "alive", avoiding false failures.
Handles race conditions where another process finishes the job before it is reaped.
Crash guard covers:
uncaughtException
unhandledRejection
SIGTERM
SIGINT
SIGHUP
Comprehensive regression tests for:
dead PID
live PID
missing PID
race with completed job
unhandled rejection
Minor suggestion

reapDeadJobs(workspaceRoot, listJobs(workspaceRoot)) is invoked in several places. If job counts grow, consider reaping once per command and reusing the result instead of repeatedly reading and rewriting job state.

[Greenbauer]

Thanks @rajpratham1 — good catch, both fixed/addressed in dc005e8.

Blocking issue — stale updatedAt after reaping: fixed. markJobDead now sets updatedAt: completedAt on the returned record, so the reaped job carries a current timestamp in both the in-memory list sortJobsNewestFirst sorts on and the per-job file written by writeJobFile. A freshly reaped failed job now sorts newest-first in the same read that recorded it. Added a regression test asserting the reaped job's updatedAt is refreshed and matches completedAt.

Minor suggestion — reap once per command: each of the four reapDeadJobs(workspaceRoot, listJobs(...)) sites lives in a separate command entry point (buildStatusSnapshot, buildSingleJobSnapshot, resolveResultJob, resolveCancelableJob) — they don't call one another, so a single /codex:status, /codex:result, or /codex:cancel invocation reaps exactly once. The liveness probe is a bare process.kill(pid, 0) per active job (no I/O), and a rewrite only happens for jobs actually found dead, so the steady-state cost is one syscall per active job. Happy to hoist it to a single shared reap if you'd prefer, but as-is there's no repeated reaping within one command.

[Greenbauer]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dc005e8815

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve cancelled jobs when SIGTERM is delivered

When cancelling a background task, handleCancel sends SIGTERM via terminateProcessTree and then writes the job as cancelled; with this new handler installed, the worker can process that same SIGTERM after the cancel command writes its update and markJobDead will rewrite the job back to failed. This makes normal /codex:cancel operations race into a failed status/result instead of cancelled, so intentional teardown signals should not use the crash-failure path or should detect an existing cancellation before writing.

Useful? React with 👍 / 👎.