Skip to content

fix(workflow-executor): tell the orchestrator why an automated inbox could not read its segment - #1938

Merged
Scra3 merged 3 commits into
feature/prd-1183-runtime-automation-pollerfrom
feature/prd-1396-the-automation-panel-says-running-while-the-agent-refuses
Sep 30, 2026
Merged

Scra3 merged 3 commits into
feature/prd-1183-runtime-automation-pollerfrom
feature/prd-1396-the-automation-panel-says-running-while-the-agent-refuses

Conversation

@Scra3

@Scra3 Scra3 commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

fixes PRD-1396

Targets the integration branch feature/prd-1183-runtime-automation-poller (#1906). Server side: ForestAdmin/forestadmin-server#8542, which must be deployed before #1906 is released.

Problem

When a segment read failed, the poller only logged it, and skipped the sync when both reads failed. The orchestrator never learned it, so the settings panel kept saying "Automation is running." while no run could start.

Change

  • Every sweep sends its sync, even when both reads failed. When a read failed, the sync carries readFailure: { reason, httpStatus? }. The candidate read's failure wins over the membership read's, since that is the read that starts runs.
  • The error caught in tryRead is classified:
error reason
AgentHttpError 401 / 403 agent-forbidden + status
AgentHttpError 502 / 503 / 504 agent-unreachable + status
AgentHttpError with no usable status (outside 100-599) agent-unreachable
any other AgentHttpError segment-read-failed + status
no HTTP answer with a network code (timeout ECONNABORTED, ECONNREFUSED, ENOTFOUND...) agent-unreachable
any other error (unsignable JWT, malformed agent URL, record without id...) segment-read-failed
  • The Automated inbox polled log line names the failure (readFailure).
  • SegmentRead.outcome was only read by the "skip the sync" guard and is removed.

Nothing else relied on "no sync when the agent is unreachable": the runtime watch works from the lease, and a sync with empty lists reconciles and starts nothing.

Tests

  • automation-poller.test.ts: the tests that asserted no sync now expect a sync with the failure. New cases: classification table (401, 403, 502, 503, 504, 500, 400, status 0, timeout, refused connection, unknown host, error of its own, record without id, unsplittable id), candidate failure over membership failure, membership failure alone, no readFailure when every read succeeded, log field.
  • agent-client-segment-reader.test.ts: the port error keeps the AgentHttpError status and the network error code as its cause, which the classification relies on.
  • Mutation check and local replay: below.

Mutation check

Each line broken on its own, automation-poller.test.ts run:

Mutation Failing test
readFailure not sent 21 tests (every read-failure case)
sync skipped when both reads fail should still sync, with the failure, when it could not reach the agent at all
membership failure wins over candidate failure should report the candidate read failure over the membership one
membership failure ignored should report a membership read failure when the candidate read worked
403 not forbidden / 401 not forbidden the 403 / 401 rows of the classification table
503, 504 not unreachable the 503 and 504 rows
any non-HTTP error unreachable the "error of its own before any answer" row
network error not unreachable refused connection row, and the two sync tests using one
timeout not unreachable timeout row, and the sync test using one
status 0 sent the "response without a status" row
status dropped the 500 and 400 rows, and the priority test
reason missing from the log line should name the failure in the poll log line

13 / 13 killed. Sending readFailure: undefined instead of omitting it was the only survivor of a first pass: identical on the wire (JSON.stringify drops it), so the conditional spread was removed.

Local replay

Executor at this PR, server at ForestAdmin/forestadmin-server#8542 (floor lowered locally, never pushed), _example agent, service account with role Operations, sweeps every ~30 s:

  1. Healthy sweep: { configured: true, active: true }.
  2. browseEnabled removed on owner for the role: the next sweep logs readFailure="agent-forbidden", the status becomes { configured: true, active: false, reason: 'agent-forbidden', httpStatus: 403 }, and the server logs automation-inbox-read-failed.
  3. Permission given back: the next sweep clears it, { configured: true, active: true }.
  4. Agent stopped: { configured: true, active: false, reason: 'agent-unreachable' }.

🤖 Generated with Claude Code

Note

Report structured read-failure reasons from automated inbox segment reads

  • Adds a readFailure field to sync requests so the orchestrator learns why a segment read failed. Failures are classified as agent-forbidden (HTTP 401/403), agent-unreachable (502/503/504 and network error codes), or segment-read-failed, with the HTTP status kept when one exists.
  • AutomationPoller no longer skips sync when both reads fail; it always calls automationPort.sync with empty items plus the selected failure reason, and a candidate-read failure takes precedence over a membership-read failure.
  • Removes the skipped/failed internal outcome states from SegmentRead, replacing them with an empty item list plus a structured failure field.
  • Behavioral Change: failed reads now produce a sync request with readFailure instead of no sync at all; see tryRead and the inbox polling workflow in automation-poller.ts.

Macroscope summarized 5b24f3d.

alban bertolini and others added 2 commits September 29, 2026 15:16
…could not read its segment

Every sweep now sends its sync, with readFailure when a segment read failed:
agent-forbidden (401/403), agent-unreachable (no answer, 502-504) or
segment-read-failed. The settings panel shows it instead of "running".

fixes PRD-1396

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…agent

A failure of the executor's own (unsignable JWT, malformed agent URL) is a
segment read failure, and a status the orchestrator would refuse is not sent.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@linear-code

linear-code Bot commented Sep 29, 2026

Copy link
Copy Markdown

PRD-1396

@qltysh

qltysh Bot commented Sep 29, 2026

Copy link
Copy Markdown

4 new issues

Tool Category Rule Count
qlty Structure Function with many returns (count = 6): classifyReadFailure 2
qlty Structure High total complexity (count = 90) 1
qlty Structure Function with high complexity (count = 14): reconcileClosed 1

@qltysh

qltysh Bot commented Sep 29, 2026

Copy link
Copy Markdown

Qlty


Coverage Impact

This PR will not change total coverage.

Modified Files with Diff Coverage (2)

RatingFile% DiffUncovered Line #s
Coverage rating: A Coverage rating: A
packages/workflow-executor/src/automation-poller.ts100.0%
Coverage rating: A Coverage rating: A
packages/workflow-executor/src/adapters/server-types.ts100.0%
Total100.0%
🚦 See full report on Qlty Cloud »

🛟 Help
  • Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

  • Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

  • File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

    • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.

@ShohanRahman
ShohanRahman self-requested a review September 30, 2026 07:37
…ate-chunk membership failure

Cover the six network codes the classification table never exercised
(ECONNRESET, EAI_AGAIN, EHOSTUNREACH, ENETUNREACH, EPIPE, ETIMEDOUT), so
dropping one from UNREACHABLE_AGENT_ERROR_CODES now fails a test instead of
silently reclassifying a real network failure as segment-read-failed. Add a
membership read whose second chunk 503s, locking in that the whole read
reports the failure rather than the first chunk's partial result.

Also reword the isHttpStatus comment, which conflated the customer agent's
status with the orchestrator sync, and drop an unreachable `| null` from the
cause cast.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@ShohanRahman ShohanRahman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after a full multi-perspective review (code, comments, tests, silent-failure, type-design, simplify).

Verified: 110/110 tests pass · lint clean (0 errors) · error classification, instanceof boundary checks, cause extraction, and isHttpStatus boundary all hold up.

No Critical or Important issues found. The always-sync-with-classified-readFailure change is well-tested with exact-value assertions, and the mutation check in the PR body is sound.

Advisory items addressed in 5b24f3d:

  • Reworded the misleading isHttpStatus comment (it conflated the customer agent's status with the orchestrator sync).
  • Dropped an unreachable | null from the cause cast.
  • Pinned all six previously-untested network codes (ECONNRESET, EAI_AGAIN, EHOSTUNREACH, ENETUNREACH, EPIPE, ETIMEDOUT).
  • Added a late-chunk membership-failure test locking in the all-or-nothing read behavior.

Reminder (already noted in the PR body): the server side (ForestAdmin/forestadmin-server#8542) must be deployed before #1906 is released, since the orchestrator has to accept the new readFailure field.

@Scra3
Scra3 merged commit 07a06c1 into feature/prd-1183-runtime-automation-poller Sep 30, 2026
59 of 66 checks passed
@Scra3
Scra3 deleted the feature/prd-1396-the-automation-panel-says-running-while-the-agent-refuses branch September 30, 2026 08:48
Scra3 added a commit that referenced this pull request Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants