feat(etl): replay terminal jobs with immutable lineage

[seonghobae]

Stack dependency and current repair state

This draft is locally stacked on PR #133 at exact base ee50859d654b89fdba5b14424fe7b8afc3b6c99d (feat/durable-job-cancellation), and its exact current head is 75339599b18062df9c38ef5f9ddd2f47e74b73b0 (feat/durable-job-replay). The immediate local boundary is intact and GitHub currently reports this PR mergeable and Draft.

The global stack is not eligible for progression. Closed PR #122 has been superseded by replacement PR #138, and the earliest invalid boundary remains exact current PR #121 head 75265fae9fcf5faac6f8b9cab63247ac86f23dd4 -> PR #138. PR #138 does not descend from that predecessor. PR #129 also still targets the closed former-worker branch and is currently non-mergeable. Therefore the intended future integration order is #121 -> repaired/replacement #138 -> repaired/replacement #129 -> #130 -> #131 -> #133 -> #135, with every affected downstream boundary refreshed from its exact predecessor before progression. Old checks, reviews, approvals, statuses, and synthetic-merge evidence do not transfer across any head/base repair.

Closes #134.

Buyer-visible product gap

Failed and deliberately cancelled durable jobs clear their retained payload, so operators previously had no safe resource for retrying the same intended work. This slice creates a new ordinary PENDING job only after the authenticated owner resupplies bounded JSON whose byte-exact SHA-256 digest matches the immutable terminal source.

Implemented boundary

• authenticated POST /api/etl/jobs/{source_job_record_id}/replays with the existing bounded Idempotency-Key profile;
• owner-scoped source selection and row locking;
• replay eligibility limited to FAILED and CANCELLED;
• stable RFC 9457 conflicts for active, succeeded, exhausted, in-progress, mismatched, and reused-key cases;
• replay-specific versioned principal-scoped key and lock domains without retaining raw principals or raw keys;
• same-intent retries return the already-created derived job; conflicting source or payload reuse fails closed;
• immutable immediate-source, first-root, and generation lineage through descriptive multi-word snake_case fields;
• owner-scoped composite source/root foreign keys with ON DELETE RESTRICT and (job_record_id, principal_scope_hash) uniqueness;
• database-trigger enforcement of terminal source state, same-owner root/source identity, exact immediate-source request-digest equality, exact generation continuity, immutable lineage, and immutable referenced terminal evidence;
• one-way locking: child admission locks source/root rows; parent evidence mutation performs an indexed descendant existence read without child locking, avoiding child-to-ancestor lock inversion;
• descendant scans limited to mutations of replay-eligible terminal rows;
• separate V8/V9 CREATE INDEX CONCURRENTLY source/root lookup migrations with nontransactional Flyway configuration;
• exact PostgreSQL catalog verification of index readiness, validity, indexed column, one-key/one-attribute nonunique shape, and IS NOT NULL partial predicate, so a same-named but incorrectly defined index fails closed;
• generation 1 through 100, with generation 101 rejected before insertion;
• ordinary worker claim, lease fencing, retry, success, failure, cancellation, pagination, polling, and conditional-status behavior; no replay-specific execution engine;
• direct PostgreSQL 18 migration rehearsal, rollback/recovery evidence, schema export, rollout, incident, connector-safety limitation, W3C PROV mapping, APA 7 doctoring, ADR, and CHANGELOG.md evidence.

Test-first repair record

The branch preserves fail-first evidence rather than rewriting history:

• 99d5cbc4c5bc70a63777890a7fe2d2f9f8945879: fail-first database digest-continuity contract;
• 3a2fb6dae3b3527357ce6ad8f62b07fb290c8d34: fail-first strict-coverage contract for EtlJobReplayController;
• 41a778cf680e84ad083b8c92f9c206979c775e2a: fail-first CI-diagnostics contract; exact-head CI on 33ba0108b5750b6d0fbc3755e4b63c507f9d269d ran 466 tests with exactly that one failure;
• 946740624632430be733bd407c7ea107eb35397d: replay-controller JaCoCo production coverage;
• bfd705eaefd228d7f274e4d4593da4508fd7e821: CI failure diagnostics aligned to every strict coverage target;
• e249a8d7592aa25de795ba2b46e1953bce40ba0e: fail-first contract requiring terminal-only referenced-evidence descendant scans;
• d41f70490bcd4031cba3f5275ac7f8a06e9e940a: production guard bounded to old FAILED/CANCELLED rows;
• exact-head Windows CI on d41f70490bcd4031cba3f5275ac7f8a06e9e940a ran 467 tests with exactly one failure caused by an over-constrained test regex;
• f33af0227926721d80397773050d2aa86fcc8c90: corrected that test while retaining the intended guard contract;
• ba7fda84c1f0622231859e4b5cf3667288457186: fail-first exact replay-index catalog contract; exact-head CI ran 467 tests with exactly one expected failure;
• fd7c61f3a9167ec89bb8c30c4d56c6db3448a92c: implemented exact replay-index catalog verification;
• b4082961711a6bbc9a4160616433705313f38878: fail-first operations, doctoring, and changelog evidence contract; exact-head CI ran 468 tests with exactly two expected documentation/compatibility failures;
• ba08f863fc2ede7779be0eca3a361b04e9291d9d: restored the retained verifier diagnostics contract;
• 48e9ed9256dc8d31c6a11ddbd59d7389ee32d534: updated the operator runbook and PostgreSQL primary references;
• 94ba0079fcd78806a7b5e77a0346aef0416931cd: updated APA 7 doctoring evidence;
• f5fd3bab3d7e028901011d245c99509aa880dcb0 and 75339599b18062df9c38ef5f9ddd2f47e74b73b0: recorded the changelog contract and corrected an unintended wording drift without rewriting history. The combined delta from 94ba007... is exactly one changelog addition.

RED and intermediate GREEN runs are historical development evidence only. They are not merge evidence for the exact current head.

Current exact-head evidence

For exact head 75339599b18062df9c38ef5f9ddd2f47e74b73b0 on its current stacked target:

Successful exact-head evidence:

• CI run 31176443074: completed success;
• Dependency Review run 31176443105: completed success;
• CycloneDX SBOM run 31176443070: completed success;
• CodeRabbit commit status: success;
• formal reviews: none;
• review threads: none.

Not passing / still blocking:

• global predecessor stack integrity is not satisfied because #121 -> #138 is diverged and feat(etl): list durable jobs with owner-scoped cursors #129 is not a valid next boundary;
• direct SAST Semgrep and Security Scan are absent on this non-default stacked base;
• direct protected-develop PostgreSQL migration integration, provenance, and other release evidence are absent on this stacked target;
• no qualifying independent non-author APPROVED review is anchored to this exact head.

The successful runs above are current-head evidence only for the existing stacked target. They do not authorize stack progression and must not be reused after any head/base repair. Queued, pending, cancelled, skipped-required, neutral-required, absent, failed, stale-head, predecessor-head, old-base, status-only, and synthetic-merge-only evidence are not success.

Reality-based verification contract

The exact direct-develop head must prove all of the following before this PR leaves Draft or merges:

• complete reactor tests on Ubuntu, macOS, and Windows with no project test skipped;
• configured zero-missed production instruction, line, method, and branch coverage;
• Dependency Review and CycloneDX SBOM;
• direct PostgreSQL 18 migration integration, including exact V8/V9 index catalog definitions and replay database-authority tests;
• all applicable SAST, security, secret, provenance, supply-chain, and commit-status gates;
• successful current-head CodeRabbit, OpenCode, Noema, and other configured automated review evidence;
• zero unresolved valid review threads;
• a qualifying independent non-author APPROVED review anchored to the unchanged exact head after retargeting to develop;
• branch protection and expected-head merge semantics without bypass.

Safety and privacy boundaries

The terminal source is never resurrected. Responses and ordinary telemetry exclude payloads, raw principals, raw replay keys, digests, stored hashes, source/root identifiers, lease identifiers, SQL, exception messages, and target identities. Foreign and missing sources remain indistinguishable. Byte-exact digest equality proves replay intent but does not prove that a non-transactional external connector is safe to replay; such connectors require independently tested idempotency, transaction participation, or compensation.

Standards and documentation

Doctoring and operations evidence use APA 7th references to RFC 9110, RFC 9457, PostgreSQL 18 constraint, transaction, trigger, concurrent-index, pg_index, and system-information-function semantics, W3C PROV-O, and NIST SP 800-185 domain-separation methodology. The SHA-256 construction does not claim cSHAKE or TupleHash conformance.

Merge policy

Keep this PR Draft and retain manual-merge. Do not deepen or repair this downstream slice until the earliest invalid boundary #121 -> #138 is eligible and repaired, then repair each downstream boundary in order with fresh exact-head/base evidence. After all predecessors integrate, retarget the valid replay PR to protected develop, regenerate every required direct-base exact-head quality, security, dependency, SBOM/provenance, migration, status, automated-review, thread, and independent-approval gate, and merge only with expected-head semantics. Never bypass stack order, branch protection, tests, coverage, security gates, review independence, or exact-head binding.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e62efc48-f718-4ce7-9d62-36ecf6d2e03a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[opencode-agent]

Queued @cwl-noema-review ; Rejected @opencode-agent: repository is absent from OPENCODE_REPOSITORY_DISPATCH_TARGETS for PR #135 at head 75339599b18062df9c38ef5f9ddd2f47e74b73b0. Central exact-name Actions artifacts are the durable dispatch ledger; existing review workflows remain authoritative for the final verdict and failure evidence.

[opencode-agent]

Queued @cwl-noema-review ; Rejected @opencode-agent: repository is absent from OPENCODE_REPOSITORY_DISPATCH_TARGETS for PR #135 at head 75339599b18062df9c38ef5f9ddd2f47e74b73b0. Central exact-name Actions artifacts are the durable dispatch ledger; existing review workflows remain authoritative for the final verdict and failure evidence.

[seonghobae]

@coderabbitai review

Review exact current head 75339599b18062df9c38ef5f9ddd2f47e74b73b0 against exact predecessor/base ee50859d654b89fdba5b14424fe7b8afc3b6c99d. The prior CodeRabbit findings were bound to obsolete head da035607286e6deab92b14a0014ecdbb64250dde; verify their fixes rather than reusing that evidence. Focus on PostgreSQL owner-scoped composite lineage, immediate-source digest continuity, trigger lock order, terminal-only descendant scans, concurrent V8/V9 index migrations, exact pg_index/pg_get_indexdef/pg_get_expr verification, idempotency and privacy boundaries, configured zero-missed production coverage, rollback/recovery, and APA 7 doctoring. Treat any later head as requiring a new review.

@opencode-agent @cwl-noema-review Please independently inspect the same exact head. Preserve Draft state, manual-merge, stack order, and all exact-head gates. Do not approve unless GitHub counts the reviewer as an eligible independent non-author and no blocker remains.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

repo="ContextualWisdomLab/mightyETL"
pr="135"
head="75339599b18062df9c38ef5f9ddd2f47e74b73b0"
base="ee50859d654b89fdba5b14424fe7b8afc3b6c99d"

echo '== GitHub PR identity =='
gh pr view "$pr" --repo "$repo" \
  --json number,state,isDraft,headRefOid,baseRefOid,baseRefName,headRefName,mergeable,reviewDecision,labels \
  --jq '{number,state,isDraft,headRefOid,baseRefOid,baseRefName,headRefName,mergeable,reviewDecision,labels:[.labels[].name]}'

echo '== Current exact-head checks =='
gh pr checks "$pr" --repo "$repo" --json name,state,workflow,link \
  --jq '.[] | {workflow,name,state,link}' || true

echo '== Local checkout identity =='
git rev-parse HEAD
git status --short

echo '== Requested comparison identity =='
git cat-file -e "${base}^{commit}"
git cat-file -e "${head}^{commit}"
git merge-base --is-ancestor "$base" "$head"
printf 'base=%s\nhead=%s\n' "$base" "$head"

echo '== Changed files from requested base to requested head =='
git diff --name-status "$base" "$head"

echo '== Relevant candidate files =='
fd -HI -t f . | rg -i \
  '(EtlJobReplay|EtlJob(Idempotency|Execution|Lease|Service)|EtlApiProblemHandler|V[789]__.*replay|replay.*migration|postgresql-migration|verify-postgresql|ci\.yml$|pom\.xml$|coverage|durable-job-replay|ARCHITECTURE|CHANGELOG|doctoring|rollback|ADR)' \
  | sort

Length of output: 10392

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

base="ee50859d654b89fdba5b14424fe7b8afc3b6c99d"
head="75339599b18062df9c38ef5f9ddd2f47e74b73b0"

echo '== Production implementation outlines =='
for file in \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobReplayController.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplay.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplayService.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobService.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobIdempotencyService.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobExecutionService.java \
  etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobLeaseRepository.java \
  etl-service/src/main/java/com/xtrmetl/etl/controller/EtlApiProblemHandler.java
do
  echo "---- $file"
  ast-grep outline "$file" --view expanded
done

echo '== V7 lineage migration =='
cat -n etl-service/src/main/resources/db/migration/V7__add_etl_job_replay_lineage.sql

echo '== V8 and V9 index migrations and Flyway configuration =='
for file in \
  etl-service/src/main/resources/db/migration/V8__add_etl_job_replay_source_lookup_index.sql \
  etl-service/src/main/resources/db/migration/V8__add_etl_job_replay_source_lookup_index.sql.conf \
  etl-service/src/main/resources/db/migration/V9__add_etl_job_replay_root_lookup_index.sql \
  etl-service/src/main/resources/db/migration/V9__add_etl_job_replay_root_lookup_index.sql.conf
do
  echo "---- $file"
  cat -n "$file"
done

echo '== PostgreSQL verifier and workflow =='
cat -n scripts/verify-postgresql-migrations.sh
cat -n .github/workflows/postgresql-migration-integration.yml

echo '== Application implementation =='
cat -n etl-service/src/main/java/com/xtrmetl/etl/job/EtlJobReplayService.java
cat -n etl-service/src/main/java/com/xtrmetl/etl/controller/EtlJobReplayController.java
sed -n '1,280p' etl-service/src/main/java/com/xtrmetl/etl/controller/EtlApiProblemHandler.java

echo '== Diff for reviewed production and migration scope =='
git diff --find-renames --unified=30 "$base" "$head" -- \
  etl-service/src/main/java \
  etl-service/src/main/resources/db/migration \
  scripts/verify-postgresql-migrations.sh \
  .github/workflows/postgresql-migration-integration.yml \
  .github/workflows/ci.yml \
  etl-service/pom.xml

Length of output: 128008

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '== Replay migration, trigger, and index tests =='
for file in \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayMigrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayDigestContinuityMigrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLookupIndexMigrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayLineageIntegrityTest.java \
  etl-service/src/test/postgresql/replay_lineage_migration.sql
do
  echo "---- $file"
  cat -n "$file"
done

echo '== Service/controller integration and boundary tests =='
for file in \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayServiceIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayClaimIntegrationTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayBoundaryTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayPayloadBoundaryTest.java \
  etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobReplayControllerTest.java
do
  echo "---- $file"
  ast-grep outline "$file" --view expanded
  rg -n -i -C 3 \
    '(concurr|same.*key|reused|in.progress|foreign|owner|digest|generation|source|root|lease|claim|cancel|succeed|payload|privacy|problem)' \
    "$file"
done

echo '== CI diagnostic and coverage policy =='
cat -n etl-service/src/test/java/com/xtrmetl/etl/documentation/CiCoverageDiagnosticsWorkflowTest.java
cat -n etl-service/src/test/java/com/xtrmetl/etl/job/EtlJobCoveragePolicyTest.java
sed -n '95,205p' etl-service/pom.xml

echo '== Search all replay index and terminal-descendant references =='
rg -n -i -C 3 \
  '(etl_job_replay_(source|root)_lookup_index|indnkeyatts|indnatts|pg_get_indexdef|pg_get_expr|FOR UPDATE|replay_source_job_record_id|replay_root_job_record_id)' \
  etl-service/src/main \
  etl-service/src/test \
  scripts \
  .github

echo '== Rollback, recovery, and APA 7 evidence =='
for file in \
  docs/operations/durable-job-replay.md \
  docs/adr/2026-08-07-immutable-durable-job-replay.md \
  docs/doctoring/durable-job-replay-standards-evidence.md \
  docs/doctoring/durable-job-replay-key-domain-separation.md \
  CHANGELOG.md \
  ARCHITECTURE.md
do
  echo "---- $file"
  rg -n -i -C 3 \
    '(rollback|recovery|postgresql|migration|index|digest|lineage|privacy|rfc 9110|rfc 9457|apa|reference|PROV|NIST)' \
    "$file" || true
done

Length of output: 192431

---

@seonghobae Review complete for exact head 75339599b18062df9c38ef5f9ddd2f47e74b73b0 against base ee50859d654b89fdba5b14424fe7b8afc3b6c99d.

I found two blockers.

1. V8/V9 cannot pass the configured exact index-catalog verification.
   V8__add_etl_job_replay_source_lookup_index.sql defines (replay_source_job_record_id, principal_scope_hash). V9__add_etl_job_replay_root_lookup_index.sql defines (replay_root_job_record_id, principal_scope_hash). Each index therefore has two key attributes.

   scripts/verify-postgresql-migrations.sh requires index_record.indnkeyatts = 1 and index_record.indnatts = 1. The verifier will count neither index. It will then raise "replay lookup indexes are missing or invalid, or have unexpected definitions".

   This also conflicts with the stated one-key/one-attribute contract in the operations and doctoring evidence. Make V8 and V9 create one-column partial indexes, or change the required contract consistently. The stated contract and verifier both require the one-column form.

2. The JaCoCo diagnostic still fails after a Maven failure that produces no XML.
   In .github/workflows/ci.yml, the diagnostic executes only after a failed test step. Its if not reports: branch still runs:

   raise SystemExit("No JaCoCo XML report was produced")

   This produces the second failed step that the diagnostic repair was intended to prevent. If no report exists, print the explanatory message and exit successfully. If a report exists, retain the strict target validation. The Maven JaCoCo check must remain the authoritative failing step.

The earlier digest-continuity finding is fixed. V7__add_etl_job_replay_lineage.sql now selects the immediate source request_digest and rejects a mismatched derived digest. The replay-controller coverage omission is also fixed in both JaCoCo executions and EtlJobCoveragePolicyTest.

The source review found the owner-scoped composite foreign keys, terminal-only descendant scan, source-then-root lock sequence, replay-key domain separation, service-level idempotency behavior, privacy handling, worker reuse, rollback guidance, and APA 7 evidence aligned with the stated design.

I did not approve this Draft. Ubuntu CI, Dependency Review, and CycloneDX SBOM are still queued. The direct PostgreSQL migration workflow is not current-head evidence on this stacked base.

_{You are interacting with an AI system.}

[opencode-agent]

Queued @cwl-noema-review ; Rejected @opencode-agent: repository is absent from OPENCODE_REPOSITORY_DISPATCH_TARGETS for PR #135 at head 75339599b18062df9c38ef5f9ddd2f47e74b73b0. Central exact-name Actions artifacts are the durable dispatch ledger; existing review workflows remain authoritative for the final verdict and failure evidence.