Skip to content

fix(server): record FAILED when a task's terminal save fails - #1400

Open
Sonica-B wants to merge 1 commit into
a2aproject:mainfrom
Sonica-B:fix/task-manager-keep-cache-on-failed-save
Open

Sonica-B wants to merge 1 commit into
a2aproject:mainfrom
Sonica-B:fix/task-manager-keep-cache-on-failed-save

Conversation

@Sonica-B

@Sonica-B Sonica-B commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Description

When the task store rejects the write of a task's terminal status (for example a dropped database connection during the COMPLETED save), DefaultRequestHandler never records FAILED: the task stays TASK_STATE_WORKING in the store, and clients polling tasks/get never see a terminal state.

TaskManager.save_task_event applies a status or artifact event to the cached _current_task in place and only then calls the store. When save() raises, the cache already says COMPLETED. The consumer's failure handler, _write_failed_status in active_task.py, reads that cache through task_manager.get_task() and returns early for a terminal state, so the FAILED write it documents for this case never happens.

Reproducer: an agent emits Task, start_work(), complete(); the store raises ConnectionError on the first COMPLETED write; then tasks/get.

# main (43d1eff)
send raised: ConnectionError database connection lost
tasks/get state: TASK_STATE_WORKING

# this branch
send raised: ConnectionError database connection lost
tasks/get state: TASK_STATE_FAILED
flowchart TD
    E[status or artifact event] --> M{TaskManager.save_task_event}
    M -->|main| A[mutate cached _current_task in place]
    A --> S1[store.save raises]
    S1 --> C1[cache says COMPLETED, store says WORKING]
    C1 --> W1[_write_failed_status: cache is terminal, skip]
    W1 --> R1[task stays WORKING]
    M -->|this PR| B[apply event to a copy]
    B --> S2[store.save raises]
    S2 --> C2[cache still says WORKING, as persisted]
    C2 --> W2[_write_failed_status writes FAILED]
    W2 --> R2[task is FAILED]
Loading

Change, in src/a2a/server/tasks/task_manager.py: save_task_event copies the cached task and applies the event to the copy. _save_task already assigns _current_task only after save() returns, so a failed save leaves the cache as last persisted. The same change fixes prev for a VersionedTaskStore: it was the updated task itself rather than "the task as previously read" that VersionedTaskStore.save documents.

Also:

  • The comment in ActiveTask.cancel that said ordinary status writes still update the shared object in place is updated; Task events and update_with_message still do.
  • Two existing tests in test_default_request_handler.py (legacy handler) relied on the in-place updates:
    • test_on_message_send_non_blocking compared len(result.history) with the stored task after polling. result was the cached object, so it kept growing after it was returned. It is now the task as returned, and the test checks that its history is a prefix of the stored history.
    • test_on_message_send_error_does_not_hang used an AsyncMock(spec=TaskStore) whose get returned an AsyncMock in place of a Task; it now returns None so the task is created and the first save raises as before.

Behaviour notes:

  • RequestContext.current_task is no longer updated in place by later status and artifact events during a request; it is the task as read when the request started. Those in-place updates raced with the consumer anyway (enqueue_event returns before the event is applied).
  • With an unversioned store, a save that raised although it committed (the acknowledgement was lost) is now followed by a FAILED write over the committed state. Before this change the same store left the task WORKING when the write had not committed, which is the usual case. With VersionedDatabaseTaskStore the version check rejects the stale FAILED write.
  • Cost: one Task.CopyFrom per status or artifact event. The default path already copies the whole task per event (_enqueue_to_subscribers, CopyingTaskStoreAdapter.save, DatabaseTaskStore serialisation).

Not in this PR: update_with_message still appends the user message to the cached task before its save (#1392 rewrites those lines in active_task.py).

Tests:

  • tests/server/tasks/test_task_manager.py: test_save_task_event_failed_save_keeps_cached_task (status and artifact events) and test_save_task_event_passes_previous_task_as_prev.
  • tests/server/request_handlers/test_default_request_handler_v2.py: test_failed_terminal_save_records_failed_task, end to end through on_message_send with an InMemoryTaskStore that fails the first COMPLETED write.
  • All four fail on main (assert 3 == 1, artifact left in the cache, assert 2 == 1, assert 2 == 4 for WORKING vs FAILED) and pass with the change.

Checks, run on Windows 11 (CPU, Python 3.10.19); Linux CI will confirm:

  • bash ./scripts/lint.sh: ruff check, ruff format and ty pass (three existing unused type: ignore warnings in files this PR does not touch).

  • uv run pytest tests/server/tasks/test_task_manager.py tests/server/request_handlers/test_default_request_handler.py tests/server/request_handlers/test_default_request_handler_v2.py tests/server/agent_execution: 297 passed.

  • uv run pytest --ignore=tests/integration/cross_version --ignore=tests/integration/test_scenarios.py --cov=src: 2364 passed, 203 skipped, 4 xfailed; total coverage 93%. The 4 setup errors (tests/integration/push_notifications, Can't pickle local object under Windows spawn) happen on main here too. tests/integration/cross_version (uses os.killpg) and tests/integration/test_scenarios.py (hangs on Windows on main) were not run locally, so the Linux CI run is the check for those.

  • Follow the CONTRIBUTING Guide.

  • Make your Pull Request title in the https://www.conventionalcommits.org/ specification.

  • Ensure the tests and linter pass

  • Appropriate docs were updated (if necessary)

TaskManager.save_task_event applied status and artifact events to the
cached task in place and only then called the task store. When the store
raised, the cache was ahead of the store. ActiveTask's failure handler
reads that cache to decide whether to record FAILED, saw the unsaved
COMPLETED, and wrote nothing, so the task stayed WORKING in the store.

Apply the event to a copy instead. _save_task already caches the task
only after the store accepts it, so a failed save leaves the cache as
last persisted, and a VersionedTaskStore now receives the task as
previously read in `prev`, as its contract documents.
@Sonica-B
Sonica-B requested a review from a team as a code owner October 29, 2026 02:30
@github-actions

Copy link
Copy Markdown

🧪 Code Coverage (vs main)

⬇️ Download Full Report

Base PR Delta
src/a2a/server/cluster/database_event_stream.py 92.86% 96.94% 🟢 +4.08%
src/a2a/server/tasks/task_manager.py 98.94% 98.95% 🟢 +0.01%
Total 93.37% 93.41% 🟢 +0.04%

Generated by coverage-comment.yml

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant