feat: DAG-based post-processing pipeline by cedricvidal · Pull Request #1147 · microsoft/scope

Cedric Vidal (cedricvidal) · 2026-07-29T10:27:28Z

Original author: Cedric Vidal (@cedricvidal)

Summary

Introduces a DAG-based post-processing pipeline: post-processors are modelled as nodes in a per-run dependency DAG, the scheduler dispatches each handler when its dependencies are satisfied, tracks per-handler status, and triggers report generation when the DAG drains. This is the foundation that lets the system scale to N independent post-processors per run. (The taxonomy post-processor that consumes this pipeline is split into a separate stacked PR, #1053.)

What's included

DAG-based handler orchestration (`apps/scheduler/`, `apps/workers/post-processor/`, `packages/shared/`)

New HandlerDispatcher (apps/scheduler/src/handler-dispatcher.ts) walks a per-run handler DAG, enqueues each handler when its dependencies are satisfied, and tracks per-handler status in run.handlerStatus[handlerId].
New scheduler notify/registration HTTP server (apps/scheduler/src/notify-routes.ts): POST /notify/run-terminal, POST /notify/handler-complete, and POST /handlers/register, plus GET /health.
Report generation is taken off the DAG. Reports are parameterized (templateId), N-per-run, and trigger-gated, so they don't fit an idempotent per-run DAG node. The scheduler detects when the handler DAG has fully drained (every registered handler reached a terminal state: done/failed/blocked-by-failed-dependency) and calls POST /api/v1/reports/trigger, which evaluates each template's trigger and enqueues { reportId }. Exactly-once is guaranteed by an atomic, attempt-scoped run.reportsTriggeredAt guard with rollback so the poll safety-net can retry.
YAML-driven handler registration: handlers declare themselves via a handler.yaml (apps/workers/post-processor/handler.yaml) registered through the scheduler's POST /handlers/register endpoint at worker-registration time via the generic packages/shared/src/scripts/register-handler.ts script (with cycle check), replacing migration-baked registration and the old per-worker register-version script.
Shared types gain HandlerServiceDocument, HandlerRunStatus, run.handlerStatus, and run.reportsTriggeredAt; the legacy postProcessorStatus/postProcessorVersion fields are retained (deprecated) for migration compatibility. Queue-processor updated to the handler-DAG status model.

Migrations (`packages/db-migrations/`)

018-migrate-services-to-handler-dag — removes the legacy single post-processor service row (registration is now worker-owned via handler.yaml).
019-add-handler-status-index — adds the idx_handler_dispatch_atif dispatch index for handler-DAG polling.
020-backfill-handler-status — backfills run.handlerStatus.pp-atif from the deprecated flat fields.

Deployment (`deploy/`, `docker-compose*.yml`)

Scheduler manifest exposes a Service for /handlers/register + notify endpoints; a register-handler-post-processor Job runs the generic registration script on each deploy. Docker Compose gains scheduler / post-processor / register-handler-post-processor services wired to the new endpoints.

Validation

pnpm build across all 27 workspace projects: green.
npx vitest run packages/db-migrations apps/scheduler apps/portal packages/shared/src/schemas packages/shared/src/scripts apps/api: 59 files, 886 tests passing (scheduler dispatcher drain/exactly-once/rollback, handler registration + cycle check, queue-processor, db-migrations required-list check, OpenAPI snapshot regenerated taxonomy-free).
git diff origin/main...HEAD contains zero taxonomy paths or content.

Notes

Docs updated: docs/architecture/post-processing.md, overview.md, db-migrations.md, ENV_VARIABLES.md.
The taxonomy ("readout") post-processor that consumes this pipeline is reviewed separately in feat: DAG-based post-processing + taxonomy readout post-processor #1053, which will be rebased onto this once merged.

Refactor post-processing from single-handler to DAG orchestration: Phase 1 - Handler Dispatcher: - Add HandlerDispatcher with DAG topology from services collection - Scheduler REST endpoints: POST /notify/run-terminal, /handler-complete - Workers notify scheduler instead of direct queue enqueue (fallback kept) - Remove triggerReportGeneration (reports now DAG node via pp-report) - DB migrations: services → handler entries, run.handlerStatus field Phase 2 - Taxonomy Handler: - Zod schema for 3-layer taxonomy (scorecard, behavior, actions) - taxonomy-handler worker with Copilot SDK + multi-turn validation loop - Docker-compose service with SCHEDULER_URL wiring DAG structure: pp-atif → pp-taxonomy → pp-report Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Add GET /api/v1/requests/:id/taxonomy endpoint - Bundle taxonomy.json in run download tar.gz archive - Add taxonomyUrl to RunState type and request schema - Update taxonomy worker to persist run.taxonomyUrl - Update OpenAPI snapshot and post-processing docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Use multi-stage Dockerfile matching report-generator pattern - Add tsconfig.base.json, proper COPY layers, --prod install in prod stage - Add dev stage for hot-reload with tsx watch - Add taxonomy-handler to docker-compose.dev.yml watch config - Add --profile taxonomy to docker:dev:copilot and docker:dev:all scripts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds a Taxonomy tab to the run detail page that shows: - Scorecard with visual score bars per dimension - Behavior analysis patterns with impact badges - Action items with priority indicators - Collapsible raw JSON view The tab only appears when taxonomy data is available (run.taxonomyUrl set). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…tion jobs Each post-processing handler now owns its own service entry registration via a register-version.ts script (run as K8s Job on deploy): - pp-atif: apps/workers/post-processor/src/register-version.ts - pp-taxonomy: apps/workers/taxonomy/src/register-version.ts - pp-report: apps/workers/report-generator/src/register-version.ts Migration 018 is reduced to only cleaning up the legacy 'post-processor' entry. Each handler defines its own dependencies (dependsOn), queue, version, and selector — decoupling the DAG topology from a single migration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace the three inlined register-version.ts scripts (which wrote handler metadata directly to MongoDB) with a declarative, YAML-driven approach that mirrors the coding-agent registration pattern. - Add per-worker handler.yaml (pp-atif, pp-taxonomy, pp-report) as the single source of truth for handler metadata. - Add generic packages/shared/src/scripts/register-handler.ts that reads HANDLER_YAML_PATH, waits for the scheduler /health, and POSTs to POST /handlers/register with retry. - Add scheduler POST /handlers/register endpoint (notify-routes.ts) + HandlerDispatcher.registerHandler (cycle-checked upsert). Scheduler is now the sole writer of the services collection. - Remove the 3 register-version.ts scripts and the mongodb dep from the taxonomy + report-generator workers. - Wire compose (register-handler-* services) and K8s (scheduler Service, register-handler-post-processor Job, SCHEDULER_URL, /health probes). - Update docs/architecture/post-processing.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Reports were modeled as a DAG handler node (pp-report, dependsOn [pp-atif, pp-taxonomy]), but the report-generator worker is keyed by reportId and requires a templateId, while the DAG dispatches a generic {type, requestId, runId} message — so pp-report could never process a DAG message. Reports are also parameterized, N-per-run, and trigger-gated, so they don't fit the idempotent per-run DAG-node model. Take reports off the DAG. The scheduler now detects when a run's handler DAG has drained (every registered handler terminal: done | failed | blocked-by-failed-dependency) and calls the existing POST /api/v1/reports/trigger endpoint, which evaluates each template's trigger and creates a properly-templated ReportDocument + enqueues {reportId} per match. - HandlerDispatcher: isHandlerTerminal/isDagDrained/maybeTriggerReports/ triggerReports + pollTriggerReports safety net. Exactly-once via an atomic run.reportsTriggeredAt guard, scoped by run._id so a stale notify or concurrent retry can never trigger reports for — or roll back the guard of — a different attempt. Fires on run-terminal (zero-handler / already-drained), both branches of handler-complete (a failure can drain the DAG), and the 30s poll loop. Rolls back the guard on POST failure. - Inject API_URL into the dispatcher; add API_URL to scheduler env in docker-compose.yml and deploy/base/scheduler.yaml. - Remove pp-report as a DAG node: delete report-generator/handler.yaml, its Dockerfile COPY lines, the register-handler-report-generator compose service, the dead idx_handler_dispatch_report index in migration 019, and stale comments. No migration to delete pp-report is needed — nothing is deployed, so a fresh environment never registers it. - Add reportsTriggeredAt to RunState. - Tests: 13 dispatcher tests (drain detection, exactly-once guard, attempt-scoped claim/rollback, stale-runId rejection, POST payload). - Docs: post-processing.md "Report Triggering on DAG Drain" section + ENV_VARIABLES API_URL. Verified E2E: run -> pp-atif(done) -> pp-taxonomy(failed) -> DAG drains -> scheduler POST /reports/trigger -> API matches template -> ReportDocument (templateId) enqueued -> report-generator -> report done. Confirms reports fire even when a handler fails. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…-5.4 Extract the Copilot SDK taxonomy-generation loop into a shared, side-effect-free generateTaxonomy() used by both the worker and a new developer CLI (src/cli.ts, `pnpm taxonomy`). The CLI runs generation against any local or integration run, streams session events, and writes the validated taxonomy to a file — making it easy to troubleshoot and A/B models without the full queue/scheduler pipeline. Switch get_atif_trajectory to fetch ATIF through the run-scoped API endpoint instead of reading blob storage directly, so the CLI needs only an API URL + token (no storage connection string). Change the default TAXONOMY_MODEL from gpt-4.1 to gpt-5.4. A/B testing nine models on a multi-iteration run containing a real defect (agent lost context in iteration 2) showed verdicts ranging 72-100: weaker models (incl. gpt-4.1) missed the critical failure entirely, while gpt-5.4 caught it at the lowest token cost of the models tested. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…/pp-dag-orchestration # Conflicts: # ENV_VARIABLES.md # apps/scheduler/src/index.ts # deploy/base/scheduler.yaml # packages/db-migrations/src/check-migrations.test.ts # packages/db-migrations/src/required-migrations.ts

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…/pp-dag-orchestration

…/pp-dag-orchestration # Conflicts: # packages/db-migrations/src/check-migrations.test.ts

…versions The autoBackfill feature re-processes historical runs when a handler version bumps, enabling backfill of new/updated post-processing features across the whole dataset. However, buildDispatchFilter() was excluding 'done' status universally, preventing the version-check branch ($lt version) from matching any historical runs. This change makes status exclusions conditional: - autoBackfill handlers exclude only ['queued', 'processing'], allowing done runs to qualify for version-based re-dispatch - Non-backfill handlers still exclude 'done' (only dispatch on missing handler status) Fixes regression identified in #1147 code review. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…/pp-dag-orchestration # Conflicts: # packages/db-migrations/src/check-migrations.test.ts

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

loadHandlers() sits on the notify/poll hot path (2-3 reads per notify, one per poll cycle) but the services collection only changes on deploys. Cache the topology in-memory with a 60s TTL (configurable) to eliminate nearly all of those reads — meaningful on CosmosDB where each query costs RUs. registerHandler invalidates the cache after upserting so new topology is picked up immediately; cycle-check validation reads fresh. Addresses jduffney review feedback #4 on #1147. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…v-configurable The scheduler is a singleton (replicas: 1, Recreate) and the sole writer of the services collection, and registerHandler invalidates the cache on every topology change — so cache coherence rests on invalidation, not expiry. A 60s TTL sat barely above the 30s poll, forcing a needless fresh read on roughly every other poll cycle. Raise the default to 5min (well above the poll interval) and expose SCHEDULER_HANDLER_CACHE_TTL_MS so the TTL is only a coarse backstop for out-of-band writes or a future scale-out beyond one replica. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Make explicit that process-local cache invalidation is only correct while the scheduler runs as a single replica, and document the cross-replica mitigations (Redis pub/sub or change-stream watcher, or a shorter TTL) required before any scale-out. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Guards the singleton scheduler HTTP server against unbounded-memory requests. Rejects oversized payloads with 413 Payload Too Large + Connection: close, using a Content-Length fast-reject plus a streaming byte-count cap for chunked/lying clients. maxBodyBytes is threaded through createHttpServer so tests can use a small deterministic cap. Addresses jduffney review feedback #5 on #1147. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The enqueue-before-claim refactor made the scheduler write a best-effort handlerStatus.<id>.status="queued" marker in-process immediately after enqueueing. The worker's claim filter excluded "queued", so once that marker won the (near-certain) race against queue delivery, the worker dequeued its own message, matched nothing on the atomic claim, logged "no longer needed, discarding", and deleted the message — stranding the run at "queued" forever (the 30s backfill poll also excludes "queued", so nothing recovered it). This reintroduced the stuck-dispatch failure mode via a new path. Fix: remove "queued" from the worker claim exclusions so queued -> processing is a permitted transition. Idempotency is preserved: still exclude "processing" (another worker in-flight) and, for non-backfill handlers, "done"; autoBackfill "done" stays gated by the version check. Add a regression test asserting a run in the scheduler's "queued" state is claimable and driven to done, update the existing filter-shape assertion, and document that queued -> processing must be permitted. Verified end-to-end against the docker-compose stack: a synthetic status=done run dispatched via /notify/run-terminal now reaches handlerStatus.pp-atif.status=done + reportsTriggeredAt, with duplicate delivery still no-oping and the report trigger firing exactly once. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…/pp-dag-orchestration

`main` independently claimed 018/019/020 (backfill-criteria-gates, backfill-task-prompt-type, create-codebase-indexes) after this branch had assigned those numbers to the handler-DAG migrations, producing duplicate prefixes once main was merged in. Renumber the three handler migrations to sit after main's highest (024) so prefixes are unique and monotonic: 018-migrate-services-to-handler-dag -> 025-migrate-services-to-handler-dag 019-add-handler-status-index -> 026-add-handler-status-index 020-backfill-handler-status -> 027-backfill-handler-status Class names are unchanged (mongo-migrate-ts tracks applied migrations by className), and these migrations have not been applied to any shared environment (int/prod migrations come from main only), so the renumber is safe. Also updates required-migrations.ts, the internal [NNN] log prefixes, and the db-migrations doc table. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Cedric Vidal (cedricvidal) · 2026-07-29T10:27:31Z

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2012

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
2213	2213	0	0	0	0	0	2m28s

🎉 All tests passed!

Github Test Reporter

Cedric Vidal (cedricvidal) · 2026-07-29T10:27:32Z

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2029

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
2271	2271	0	0	0	0	0	2m25s

🎉 All tests passed!

Github Test Reporter

Cedric Vidal (cedricvidal) · 2026-07-29T10:27:33Z

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2030

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
2271	2271	0	0	0	0	0	2m20s

🎉 All tests passed!

Github Test Reporter

Cedric Vidal (cedricvidal) · 2026-07-29T10:27:34Z

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2103

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
2436	2436	0	0	0	0	0	2m16s

🎉 All tests passed!

Github Test Reporter

Cedric Vidal (cedricvidal) · 2026-07-29T10:27:48Z

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #2192

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
2606	2606	0	0	0	0	0	2m48s

🎉 All tests passed!

Github Test Reporter

Cedric Vidal (cedricvidal) · 2026-07-29T10:27:49Z

Original author: Wassim Chegham (@manekinekko)

Original review: COMMENTED

Cedric Vidal (cedricvidal) · 2026-07-29T10:27:50Z

Original author: Josh Duffney (@duffney)

let me review the comment before posting

Cedric Vidal (cedricvidal) · 2026-07-29T10:27:51Z

Original author: Josh Duffney (@duffney)

Original review: APPROVED

Cedric Vidal (cedricvidal) · 2026-07-29T10:27:52Z

Original author: Wassim Chegham (@manekinekko)

This $nin excludes "done", which breaks autoBackfill version bumps. Any run that has a version is already done, so the { version: { $lt: handler.version } } branch below can never match — and since the poll path is the only backfill mechanism, previously-completed runs are never re-processed when a handler's version is bumped.

This is a regression from the legacy PostProcessorDispatcher, whose filter excluded only ["queued", "processing"] (not done), so version bumps did backfill historical runs. It also contradicts the documented behavior in post-processing.md ("Requests with missing or outdated handler state are re-dispatched ... automatically applied to historical data") and the migration 020 backfill rationale.

Suggest dropping "done" from the $nin when autoBackfill is true (mirroring the legacy filter) so the $lt version branch becomes reachable. Worth adding a test for the done-at-old-version case, which is currently uncovered.

Resolve post-processing telemetry and migration sequence conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4e92f08-5d86-4664-b8de-1b99bcccc1cf

github-actions · 2026-10-11T12:22:09Z

Test Results (Node.js 22)

test: Run #230

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3271	3271	0	0	0	0	0	1m20s

🎉 All tests passed!

Github Test Reporter

…ricvidal-microsoft/pp-dag-orchestration

github-actions · 2026-10-12T15:52:57Z

Test Results (Node.js 22)

test: Run #237

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3287	3287	0	0	0	0	0	1m24s

🎉 All tests passed!

Github Test Reporter

Restore the taxonomy-specific layer after fast-forwarding to PR #1147, reconcile current migrations and OpenAPI artifacts, and adapt the worker to the current Copilot SDK permission contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: addea2af-c951-4473-90bc-0a7356c29862

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 247cbe24-e22c-4f4c-b3e4-e71ce03f4669

github-actions · 2026-10-17T01:00:18Z

Test Results (Node.js 22)

test: Run #277

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3289	3289	0	0	0	0	0	1m31s

🎉 All tests passed!

Github Test Reporter

Wassim Chegham (manekinekko) · 2026-10-19T19:40:50Z

+        currentVersion: currentStatus?.version,
+        targetVersion: handler.version,
+      });
+      await this.safeDeleteMessage(message.messageId, heartbeat.stop());


[P1] Preserve recovery when a processing claim outlives its worker

A failed claim does not necessarily mean the message is obsolete: it can be the redelivery of work whose worker died after setting the handler to processing. The claim filter excludes processing indefinitely, and this branch deletes the recovery message. The scheduler also excludes processing, while StuckRunReaper only scans runs whose overall run.status is processing (these runs are already done). Consequently an ordinary worker crash or forced shutdown can strand this handler, its descendants, and report generation indefinitely.

I reproduced this with the compiled worker and dispatcher: a redelivered message for a processing handler is deleted without invoking the handler, and the subsequent scheduler poll cannot recover it. Please add handler-scoped ownership/liveness or an expiring lease, safely recover stale claims, and re-defer genuinely in-flight duplicates instead of acknowledging every failed claim.

Wassim Chegham (manekinekko) · 2026-10-19T19:40:50Z

+    // This avoids stale/duplicate notifies regressing a newly re-dispatched
+    // handler back to "done" or "failed".
+    const updateResult = await this.collection.updateOne(
+      { _id: requestId, [`run.handlerStatus.${handlerId}.status`]: "processing" } as any,


[P1] Scope handler-completion writes to the notified run attempt

runId is supplied to this method but omitted from the update filter. If a delayed completion for attempt A arrives while attempt B's handler is processing, it changes B's status to A's done/failed. Checking runId later in dispatchIfEligible or maybeTriggerReports cannot undo that write. The current worker's eventual terminal write then fails its own processing guard, and downstream polling can observe a false completion.

I reproduced an old-attempt notification changing a new-attempt handler from processing to done. Please include "run._id": runId in any mutation and fence the execution/version where a handler can be rerun within the same attempt. Alternatively, treat the notification as a wake-up signal and read the worker's already-persisted terminal state without writing it again.

Wassim Chegham (manekinekko) · 2026-10-19T19:40:50Z

+        "run.status": "done",
+        "run.reportsTriggeredAt": { $exists: false },
+      } as any,
+      { $set: { "run.reportsTriggeredAt": new Date() } } as any,


[P1] Make the report-trigger claim recoverable after scheduler termination

The permanent reportsTriggeredAt marker is persisted before the report-trigger POST occurs. If the scheduler exits after this write but before the API processes the request, the catch/rollback cannot run. After restart both maybeTriggerReports and pollTriggerReports exclude the marked run forever, even though no report was created. A rolling deployment or process crash can therefore permanently lose report generation.

I reproduced this by terminating an isolated scheduler child after the guard was set and before any HTTP request; the restarted dispatcher's poll made no trigger call. Please use a durable pending-trigger/outbox or a recoverable claim, with API-side idempotency keyed to the run attempt, and reserve the completed marker for a confirmed trigger. Simply moving the write after the POST would introduce duplicate-trigger races.

Wassim Chegham (manekinekko) · 2026-10-19T19:40:50Z

+  const maxBodyBytes = options.maxBodyBytes ?? MAX_BODY_BYTES;
+  return http.createServer(async (req, res) => {
+    const method = req.method ?? "GET";
+    const url = new URL(req.url ?? "/", "http://localhost");


[P2] Reject malformed request targets without terminating the scheduler

new URL(req.url, ...) executes before the try inside an async HTTP listener. A raw GET // HTTP/1.1 request makes it throw ERR_INVALID_URL; the resulting unhandled rejection terminates Node rather than returning an error. I reproduced this against the actual routes in an isolated Node 22.22.0 child: /health initially returned 200, the malformed request produced no response and exit code 1, while the baseline health server survived the same request.

This is a security-relevant availability issue for any caller that can reach the scheduler, including peers on the checked-in Compose network. Compose does not publish a scheduler host port; Internet/production exposure was not established. Please parse the URL inside error handling and return 400 for invalid request targets so the async listener cannot reject uncaught.

Wassim Chegham (manekinekko) · 2026-10-19T19:40:50Z

+      console.log(
+        `[HandlerDispatcher] Ignored stale/duplicate completion for ${handlerId} on ${requestId}`,
+      );
+      return;


[P2] Re-evaluate the DAG for the worker's already-persisted completion

The post-processor writes run.handlerStatus[handlerId].status = "done" (or "failed") before calling /notify/handler-complete. This method only matches processing, so every normal notification from the implemented worker reaches this early return. No downstream handler or report is triggered through the advertised event-driven path; all progress depends on the polling safety net.

I reproduced a successful worker-style completion notification producing no child queue message, followed by a poll dispatching that same eligible child. Please accept/read the terminal state for the matching run and continue the idempotent DAG/report evaluation, rather than requiring a second processing-to-terminal transition. An integration-style test connecting the worker's terminal write to the scheduler notification would catch the mismatch.

Wassim Chegham (manekinekko) · 2026-10-19T19:40:50Z

+        "run.reportsTriggeredAt": { $exists: false },
+        deletedAt: { $exists: false },
+      } as any)
+      .limit(this.batchSize)


[P2] Advance the report scan past undrained candidates

The batch limit is applied before checking whether a candidate's handler DAG has drained, with no cursor or progress marker. Thirty earlier done runs whose handlers are still queued/processing can occupy every batch, so later fully processed runs are never examined. If any of those handlers are permanently stuck, this defeats the report safety net indefinitely; completion notifications currently do not rescue the ready runs either.

I reproduced two consecutive polls with 30 undrained candidates followed by one drained candidate: neither poll reached or triggered the ready run. Please paginate/rotate the scan across polls, or make terminal eligibility part of a queryable state, so a bounded batch still guarantees progress past non-actionable records.

Wassim Chegham (manekinekko) · 2026-10-19T19:40:50Z

 */
 export class AtifHandler implements PostProcessHandler {
  readonly type = "atif";
+  readonly version = POST_PROCESSOR_VERSION;


[P2] Keep the registered and runtime handler versions in one source of truth

The new registration reads version from handler.yaml, but the worker's claim and success stamp use the independent POST_PROCESSOR_VERSION constant. The documentation explicitly says a YAML-only version bump is sufficient. With a completed v1 run and YAML bumped to v2, the scheduler queues the upgrade, but this worker still requires the stored version to be below its hardcoded v1. It rejects and deletes the message, leaving the scheduler's queued v1 marker stranded indefinitely.

I reproduced that version-bump path with the compiled dispatcher and actual AtifHandler. Please share the version metadata between registration and runtime, or enforce and document synchronization at startup, and cover a metadata version upgrade end-to-end.

Cedric Vidal (cedricvidal) added 25 commits June 9, 2026 02:05

chore: remove taxonomy changes (split to #1053)

07127ae

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

merge: bring in origin/main

d1ea709

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into cedricvidal-microsoft…

e87bb85

…/pp-dag-orchestration

Merge remote-tracking branch 'origin/main' into cedricvidal-microsoft…

8901933

…/pp-dag-orchestration # Conflicts: # packages/db-migrations/src/check-migrations.test.ts

Merge remote-tracking branch 'origin/main' into cedricvidal-microsoft…

d3e5d78

…/pp-dag-orchestration # Conflicts: # packages/db-migrations/src/check-migrations.test.ts

fix(scheduler): ignore stale handler completion notifications

7288b65

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

fix(post-processing): enforce worker-entrypoint idempotency

5c64dea

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

docs(post-processing): clarify enqueue and claim boundaries

fe5b9ee

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into cedricvidal-microsoft…

8b86d2d

…/pp-dag-orchestration

Cedric Vidal (cedricvidal) added the author: cedricvidal label Jul 29, 2026

Cedric Vidal (cedricvidal) mentioned this pull request Jul 29, 2026

feat: observation criteria with taxonomy and run/iteration subject (#1156) #1206

Draft

Merge origin/main into cedricvidal-microsoft/pp-dag-orchestration

90dfca4

Resolve post-processing telemetry and migration sequence conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4e92f08-5d86-4664-b8de-1b99bcccc1cf

Cedric Vidal (cedricvidal) requested a review from Wassim Chegham (manekinekko) as a code owner October 11, 2026 12:19

Merge remote-tracking branch 'origin/feat/portal-copilot-ai' into ced…

2ca370b

…ricvidal-microsoft/pp-dag-orchestration

Cedric Vidal (cedricvidal) changed the base branch from main to feat/portal-copilot-ai October 12, 2026 15:48

Cedric Vidal (cedricvidal) added this pull request to stack #1485 October 12, 2026 15:49

Cedric Vidal (cedricvidal) added a commit that referenced this pull request Oct 12, 2026

merge: restack #1206 onto #1147

a9e9dbd

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 247cbe24-e22c-4f4c-b3e4-e71ce03f4669

Cedric Vidal (cedricvidal) marked this pull request as draft October 14, 2026 16:00

Cedric Vidal (cedricvidal) removed this pull request from stack #1485 October 17, 2026 00:57

Cedric Vidal (cedricvidal) added this pull request to stack #1518 October 17, 2026 00:58

Wassim Chegham (manekinekko) reviewed Oct 19, 2026

View reviewed changes

Conversation

Cedric Vidal (cedricvidal) commented Jul 29, 2026

Summary

What's included

DAG-based handler orchestration (apps/scheduler/, apps/workers/post-processor/, packages/shared/)

Migrations (packages/db-migrations/)

Deployment (deploy/, docker-compose*.yml)

Validation

Notes

Uh oh!

Cedric Vidal (cedricvidal) commented Jul 29, 2026

Test Results (Node.js 22)

test: Run #2012

🎉 All tests passed!

Uh oh!

Cedric Vidal (cedricvidal) commented Jul 29, 2026

Test Results (Node.js 22)

test: Run #2029

🎉 All tests passed!

Uh oh!

Cedric Vidal (cedricvidal) commented Jul 29, 2026

Test Results (Node.js 22)

test: Run #2030

🎉 All tests passed!

Uh oh!

Cedric Vidal (cedricvidal) commented Jul 29, 2026

Test Results (Node.js 22)

test: Run #2103

🎉 All tests passed!

Uh oh!

Cedric Vidal (cedricvidal) commented Jul 29, 2026

Test Results (Node.js 22)

test: Run #2192

🎉 All tests passed!

Uh oh!

Cedric Vidal (cedricvidal) commented Jul 29, 2026

Uh oh!

Cedric Vidal (cedricvidal) commented Jul 29, 2026

Uh oh!

Cedric Vidal (cedricvidal) commented Jul 29, 2026

Uh oh!

Cedric Vidal (cedricvidal) commented Jul 29, 2026

Uh oh!

github-actions Bot commented Oct 11, 2026

Test Results (Node.js 22)

test: Run #230

🎉 All tests passed!

Uh oh!

github-actions Bot commented Oct 12, 2026

Test Results (Node.js 22)

test: Run #237

🎉 All tests passed!

Uh oh!

github-actions Bot commented Oct 17, 2026

Test Results (Node.js 22)

test: Run #277

🎉 All tests passed!

Uh oh!

Wassim Chegham (manekinekko) Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Wassim Chegham (manekinekko) Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Wassim Chegham (manekinekko) Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Wassim Chegham (manekinekko) Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Wassim Chegham (manekinekko) Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Wassim Chegham (manekinekko) Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Wassim Chegham (manekinekko) Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

DAG-based handler orchestration (`apps/scheduler/`, `apps/workers/post-processor/`, `packages/shared/`)

Migrations (`packages/db-migrations/`)

Deployment (`deploy/`, `docker-compose*.yml`)