fix(workers): serialize reaping so a killed worker never looks alive

[amrit110]

Problem

PR #53's CI failed test_workers_close_together_within_one_grace_period (assert not any(s._proc.is_alive() ...)) on a docs-only change. The cause is a real race, not the test.

Two threads reap a dying worker: its reader thread (on pipe EOF) and close(). Both join() / is_alive() call waitpid. When two threads call it at once, the loser gets ECHILD, and CPython's Popen.poll returns None ("still running") until the winner stores the exit code. So:

• close() could see a worker it just killed as alive and skip _release_host_memory(), leaving that worker's host reservations held until the parent exits.
• alive, death_reason and the exit code in log messages can be read wrong in the same window.

Fix

Every wait and liveness check in ProcessShard now goes through _exited(wait_s):

• The process sentinel decides whether the worker exited. It is waited on outside any lock, so a health check never blocks behind someone else's wait.
• The reap (join()) is serialized under _reap_lock, so the exit code is recorded by the time _exited returns True. _exit_code reads under the same lock.

Checking is_alive() under a lock is not enough. The sentinel can close a moment before the kernel makes the exit status available, so a non-blocking waitpid still says "running". That is why CPython's own join does a blocking waitpid once the sentinel is ready, and _exited relies on that.

Tests

• New test_concurrent_waits_all_see_a_dead_worker_exit: 7 threads wait on a worker as it is killed, and all must see it exit, with an exit code recorded.
  • No lock: fails 10 of 15 runs.
  • Lock around is_alive(): fails 12 of 15.
  • This fix: passes 20 of 20.
• tests/unit/test_workers.py: 15 of 15 clean runs. Full unit suite: 498 pass. ruff and pre-commit are clean.

No architecture change (internal to serve/workers.py), so the README and docs are unchanged. The CHANGELOG has an Unreleased/Fixed entry.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 86.66667% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.58%. Comparing base (f133977) to head (c96a62a).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/tabctx/serve/workers.py	86.66%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #55      +/-   ##
==========================================
+ Coverage   79.36%   79.58%   +0.22%     
==========================================
  Files          41       41              
  Lines        3193     3209      +16     
==========================================
+ Hits         2534     2554      +20     
+ Misses        659      655       -4     

Flag	Coverage Δ
integration	30.13% <16.66%> (+<0.01%)	⬆️
unit	76.09% <86.66%> (+0.24%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/tabctx/serve/workers.py	71.81% <86.66%> (+2.33%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[amrit110]

Review fixes pushed (independent review agent):

• Held sentinel (real regression): a forked grandchild that inherits the worker's sentinel kept a dead worker "alive" forever. _exited now falls back to a non-blocking reap under the lock. New test test_a_dead_worker_exits_while_a_grandchild_holds_its_sentinel: fails on the previous version, passes now.
• Bounded reap: after the sentinel is ready, join(5) instead of an unbounded join().
• Locked kills: kills go through _kill, under the lock, so a reaped pid is never signalled.
• Startup exit code: a worker that dies at startup is reaped before its exit code is reported.
• Race test: the exit code is now checked inside each racing thread. Without the lock it fails 11 of 15 runs; previously it couldn't detect removing the lock.

Unit tests: 499 pass, and test_workers.py ran clean 5 of 5 times.