Skip to content

fix(parallel): fail fast on permanent shard validation errors - #1

Open
rollroyces wants to merge 1 commit into
deeplethe:mainfrom
rollroyces:fix/parallel-runner-shard-validation-grace-period
Open

rollroyces wants to merge 1 commit into
deeplethe:mainfrom
rollroyces:fix/parallel-runner-shard-validation-grace-period

Conversation

@rollroyces

Copy link
Copy Markdown

Bug

wait_for_shard_result previously caught every ValueError raised by
validate_shard_result and retried it for the full 5-second grace period.
Permanent semantic validation failures (wrong number of result IDs, missing
completion timestamp, malformed payload) can never resolve themselves during
the grace window — the worker has already exited — but the old code still
spun on them, wasting a worker slot and up to grace_seconds per failed shard.

In the worst case (e.g. a worker that wrote a malformed result.json with
the wrong question IDs), the parallel runner silently held the slot for 5 s
before the scheduler was allowed to relaunch or fail the shard.

Fix

Introduce TransientShardResultError (an OSError subclass) as the only
"spin me" signal inside validate_shard_result. Semantic validation
failures keep raising plain ValueError, which wait_for_shard_result
now propagates immediately. Call sites that previously caught
ValueError (run_shards on the initial scheduling pass, the failure
branch after a worker exits, and merge_results) now catch both
TransientShardResultError and ValueError so transient races and
permanent failures both continue to relaunch cleanly.

Tests

Adds backend/tests/test_wait_for_shard_permanent_failure.py with five
regression tests:

  • test_wait_for_shard_result_raises_immediately_on_wrong_ids — wrong
    number of question IDs propagates immediately, well under the grace
    window.
  • test_wait_for_shard_result_raises_immediately_on_missing_timestamp
    missing config.finished_at propagates immediately.
  • test_wait_for_shard_result_retries_transient_missing_file — a missing
    result.json is retried until it appears (grace period still works).
  • test_validate_shard_result_raises_transient_when_result_missing — the
    missing-file signal is the new TransientShardResultError, which is
    still catchable as OSError for back-compat.
  • test_run_shards_treats_transient_and_permanent_during_relaunch
    end-to-end check that the scheduler relaunches a shard whose cached
    result.json is invalid rather than crashing the run.

Updates test_result_publication_grace_retries_transient_missing_artifact
to use TransientShardResultError instead of relying on the old generic
ValueError retry behaviour.

Verification

$ python -m pytest tests/
197 passed, 8 xfailed

The 8 xfails are pre-existing experimental tests that are pinned to the
r48 leaderboard runner; they are unrelated to this change. All frozen
artifacts and RESULTS.md are untouched.

Risk

The behavioural change is purely in error handling: permanent validation
failures now raise faster. No production data flow touches
validate_shard_result outside the parallel runner itself, and the
"happy path" (worker exits cleanly, result.json is valid) is
unchanged. The change is deterministic — same inputs, same outputs.

wait_for_shard_result previously caught every ValueError raised by
validate_shard_result and retried it for the full grace period. That
spun on permanent validation failures (wrong number of result IDs,
missing completion timestamp) even though the worker process had
already exited and the result could never become valid during the
grace window.

Introduce TransientShardResultError (an OSError subclass) as the only
'spin me' signal and let semantic validation failures propagate
immediately. Call sites that previously caught ValueError now catch
both TransientShardResultError and ValueError so transient races and
permanent failures both relaunch cleanly.

Adds backend/tests/test_wait_for_shard_permanent_failure.py covering:
- immediate raise on wrong IDs
- immediate raise on missing completion timestamp
- retry of missing-file (TransientShardResultError)
- run_shards scheduler handling transient cached results

Update test_result_publication_grace_retries_transient_missing_artifact
to use the new TransientShardResultError instead of relying on the
old generic ValueError behaviour.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant