Fixes #29245: Report the lost batch when a paginated read fails

[TeddyCr]

Describe your changes:

Fixes #29245

A failed page read in Data Insights aborted the whole run with an unrelated NullPointerException instead of reporting the reader failure.

Cause. The cursor and keyset read paths of PaginatedEntitiesSource, PaginatedEntityTimeSeriesSource and PaginatedWebAnalyticEventDataSource built their READER IndexingError without submittedCount or failedCount. Only the offset read path set them.

• The NPE. DataAssetsWorkflow, CostAnalysisWorkflow and WebAnalyticsWorkflow all read through readNextKeyset and unbox getFailedCount() into updateStats(int, int). So the null threw an NPE inside the error handler.
• The aborted run. DataInsightsApp.processWorkflow only catches SearchIndexException, so the NPE escaped to the catch-all in startApp. It skipped every later workflow and entity type, and the reader's own error report was lost.

Fix. All six cursor/keyset catch blocks now set submittedCount and failedCount to batchSize, the records the read requested and lost. successCount was already 0.

• PartitionWorker (search reindex) already falls back to currentBatchSize when the count is null. It builds each reader with limit = currentBatchSize, so its numbers are unchanged.
• I didn't add a schema default in indexingError.json. A default of 0 would hide lost pages and change PartitionWorker, which treats null as "use the batch size".

Not in this PR:

• whether a reader failure should be fatal to the run at all (open question 3 in the issue)
• a separate early-stop: when every row of a keyset page fails to deserialize, listAfterKeyset returns a null cursor and the scan stops silently

Type of change:

• Bug fix

High-level design:

N/A — small change: three readers, one line pair per catch block.

Tests:

Use cases covered

• A DB failure during a keyset or cursor page read, in each of the three readers, reports the lost batch: source READER, submitted = failed = batch size, success = 0.
• The workflows' own updateStats(getSuccessCount(), getFailedCount()) call, the line that threw, records the failure without an NPE.

Unit tests

• I added unit tests for the changed logic.
• Files added:
  • openmetadata-service/src/test/java/org/openmetadata/service/workflows/searchIndex/PaginatedSourceReadFailureTest.java: 4 tests, keyset and cursor for PaginatedEntitiesSource and PaginatedEntityTimeSeriesSource.
  • openmetadata-service/src/test/java/org/openmetadata/service/apps/bundles/insights/workflows/webAnalytics/sources/PaginatedWebAnalyticEventDataSourceReadFailureTest.java: 2 tests, keyset and cursor.
• Each test makes the real reader's repository call throw, then asserts the IndexingError and replays the workflow's updateStats call.
• All 6 failed before the fix with expected: <7> but was: <null>, and pass with it.
• Run with PaginatedEntityTimeSeriesSourceStaleRelationshipTest, DataInsightsAppTest and PartitionWorkerTest: 47/47 passed.
• Coverage %: not measured. The change is two builder calls per catch block, and every changed block is exercised by a test.

Backend integration tests

• Not applicable (no backend API changes). The planner still selects ITs because the readers are shared with search reindexing; see the block below.

Local Java test run: PENDING. The planned run (5 unit classes, 90 IT classes across 7 lanes) has not run yet. It needs the local Docker host, which another branch's run is using. This PR stays a draft until the block reads PASSED; it will be filled by make java_affected_run ARGS="--update-pr".

Ingestion integration tests

• Not applicable (no ingestion changes).

Playwright (UI) tests

• Not applicable (no UI changes).

Manual testing performed

None beyond the tests above. Driving a reader into its catch block by hand means cutting the database connection partway through a Data Insights run. The unit tests reproduce that failure directly at the reader boundary.

UI screen recording:

Not applicable — backend-only fix; no UI code changes.

Checklist:

• I have read the CONTRIBUTING document.

• My PR title is Fixes <issue-number>: <short explanation>

• My PR is linked to a GitHub issue via Fixes #<issue-number> above.

• I have commented on my code, particularly in hard-to-understand areas. — Not applicable: the change follows the offset path's existing pattern.

• For JSON Schema changes: I updated the migration scripts or explained why it is not needed. — Not applicable: no schema changes.

• For a UI feature or feature task: … — Not applicable: backend-only.

• I have added tests (unit / integration / Playwright as applicable) and listed them above.

• Decision check — Not applicable: the count reported is the batch size, which matches the offset path and PartitionWorker's existing fallback. No new contract, default or rule.

• I have added a test that covers the exact scenario we are fixing. For complex issues, comment the issue number in the test for future reference.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

🚦 Removed from the merge queue — failed_checks (2026-10-16T01:21:23Z)

Blocked the queue: playwright-summary

• Prune dead merge-queue refs — cancel-stale-runs
• Postgresql PR Playwright E2E Tests — playwright-summary, playwright / playwright-ci (chromium-08), playwright / playwright-ci (chromium-04)

Failed Playwright tests:

• Pages/Entity.spec.ts › Inactive Announcement create & delete
• Pages/ServiceEntity.spec.ts › Inactive Announcement create & delete

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[gitar-bot]

Code Review ✅ Approved

🟡 Medium risk · Adds submitted and failed counts to paginated reader errors; inaccurate batch counts could overstate records lost in failure reports.

Fixes the cursor and keyset read paths in PaginatedEntitiesSource, PaginatedEntityTimeSeriesSource and PaginatedWebAnalyticEventDataSource so a failed page read reports submittedCount and failedCount as the batch size instead of leaving them null, which previously caused an unrelated NPE that aborted the Data Insights run. No issues found.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

_{For more verification details, comment gitar display:verbose.}

Options

Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}