fix(parallel): fail fast on permanent shard validation errors - #1
Open
rollroyces wants to merge 1 commit into
Open
rollroyces wants to merge 1 commit into
rollroyces wants to merge 1 commit into
Conversation
wait_for_shard_result previously caught every ValueError raised by validate_shard_result and retried it for the full grace period. That spun on permanent validation failures (wrong number of result IDs, missing completion timestamp) even though the worker process had already exited and the result could never become valid during the grace window. Introduce TransientShardResultError (an OSError subclass) as the only 'spin me' signal and let semantic validation failures propagate immediately. Call sites that previously caught ValueError now catch both TransientShardResultError and ValueError so transient races and permanent failures both relaunch cleanly. Adds backend/tests/test_wait_for_shard_permanent_failure.py covering: - immediate raise on wrong IDs - immediate raise on missing completion timestamp - retry of missing-file (TransientShardResultError) - run_shards scheduler handling transient cached results Update test_result_publication_grace_retries_transient_missing_artifact to use the new TransientShardResultError instead of relying on the old generic ValueError behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
wait_for_shard_resultpreviously caught everyValueErrorraised byvalidate_shard_resultand retried it for the full 5-second grace period.Permanent semantic validation failures (wrong number of result IDs, missing
completion timestamp, malformed payload) can never resolve themselves during
the grace window — the worker has already exited — but the old code still
spun on them, wasting a worker slot and up to
grace_secondsper failed shard.In the worst case (e.g. a worker that wrote a malformed
result.jsonwiththe wrong question IDs), the parallel runner silently held the slot for 5 s
before the scheduler was allowed to relaunch or fail the shard.
Fix
Introduce
TransientShardResultError(anOSErrorsubclass) as the only"spin me" signal inside
validate_shard_result. Semantic validationfailures keep raising plain
ValueError, whichwait_for_shard_resultnow propagates immediately. Call sites that previously caught
ValueError(run_shardson the initial scheduling pass, the failurebranch after a worker exits, and
merge_results) now catch bothTransientShardResultErrorandValueErrorso transient races andpermanent failures both continue to relaunch cleanly.
Tests
Adds
backend/tests/test_wait_for_shard_permanent_failure.pywith fiveregression tests:
test_wait_for_shard_result_raises_immediately_on_wrong_ids— wrongnumber of question IDs propagates immediately, well under the grace
window.
test_wait_for_shard_result_raises_immediately_on_missing_timestamp—missing
config.finished_atpropagates immediately.test_wait_for_shard_result_retries_transient_missing_file— a missingresult.jsonis retried until it appears (grace period still works).test_validate_shard_result_raises_transient_when_result_missing— themissing-file signal is the new
TransientShardResultError, which isstill catchable as
OSErrorfor back-compat.test_run_shards_treats_transient_and_permanent_during_relaunch—end-to-end check that the scheduler relaunches a shard whose cached
result.jsonis invalid rather than crashing the run.Updates
test_result_publication_grace_retries_transient_missing_artifactto use
TransientShardResultErrorinstead of relying on the old genericValueErrorretry behaviour.Verification
The 8 xfails are pre-existing experimental tests that are pinned to the
r48 leaderboard runner; they are unrelated to this change. All frozen
artifacts and
RESULTS.mdare untouched.Risk
The behavioural change is purely in error handling: permanent validation
failures now raise faster. No production data flow touches
validate_shard_resultoutside the parallel runner itself, and the"happy path" (worker exits cleanly,
result.jsonis valid) isunchanged. The change is deterministic — same inputs, same outputs.