Fixes #34904: Keep the lineage tab responsive with very large lineage responses

[mohityadav766]

Describe your changes:

Fixes #34904

The table Lineage tab downloads the full /lineage/getLineage response, parses it on the main thread and, since #34551, draws it with EntityLineageCanvas. Four parts of that path scaled badly with a large response:

• Parsing was quadratic. parseLineageData copied the edge array once per edge, counted pagination edges for each node by scanning every edge, and deep-compared every pair of nodes. getUpstreamDownstreamNodesEdges scanned all nodes and edges for each visited node. Both are now linear and return the same output.
• ELK got one edge per column mapping. With the column layer on, ELKLayout.layoutGraph received a parallel edge for every mapped column. At 151 tables with 50 mapped columns per link, elkjs exhausted a 4 GB heap and the tab crashed. Smaller graphs still spread layers thousands of pixels apart. ELK now gets one edge per pair of nodes; callers only read node positions.
• The canvas mounted every node. EntityLineageCanvas rendered every node, on screen or not, and each mounted node subscribes to the lineage store. With a 2,651-node response, first paint took 20 s on the Vite dev server. The canvas now sets onlyRenderVisibleElements, as LineageMap already did, and first paint takes 0.9 s.
• A PNG export captures only mounted nodes. The export clones the DOM, so on a virtualized canvas a zoomed-in export would hold only the nodes in view. The lineage export now mounts every node for the capture and goes back to visible-only afterwards.

The first version of this PR also stopped Lineage.tsx from fetching and parsing the same response two or three times. #34551 fixed that on main the same way, by claiming the fetch key before the request and dropping superseded responses. After merging main, this PR keeps only its two integration tests for that behavior.

Type of change:

• Bug fix

High-level design:

• Where the cost was. I reproduced the issue on a local stack with a busy table: one hub feeding 1,000 consumers, an upstream fan-in, 50 columns per table, and column lineage plus SQL on every edge. getLineage?upstreamDepth=2&downstreamDepth=2&size=50 returns 24 MB in 0.8 s, which matches the issue.
  • The upstream walk applies size per node and keeps the last layer's edges to tables it never fetched. A busy table can therefore return about 2,500 nodes and tens of thousands of edges. I used captured responses scaled up to those sizes for the measurements below.
• Changes, all in the UI:
  • utils/EntityLineagePureUtils.ts: pipeline lookup through a map, pagination counts in one pass over each edge map, and deep-equal dedupe only among nodes that share an id. This dedupe is equivalent to uniqWith(nodes, isEqual).
  • utils/EntityLineageNodeUtils.ts: the upstream and downstream walk indexes nodes and edges by FQN once. The DFS order and output are unchanged.
  • utils/Lineage/Layout/ELKUtil/ELKUtil.ts: layoutGraph collapses parallel edges. Both the legacy layout and LineageMap go through it. The knowledge graph calls getElk().layout directly and is unaffected.
  • components/Lineage/EntityLineageCanvas/EntityLineageCanvas.tsx: onlyRenderVisibleElements={!renderAllNodes}. LineageMap.component.tsx, which was already virtualized, reads the same flag.
  • hooks/useLineageStore.ts: a renderAllNodes flag, false by default.
  • utils/Export/ExportUtils.ts and EntityExportModalProvider.interface.ts: ExportData takes an optional beforeCapture, awaited before the PNG capture, and an optional afterCapture, which runs in a finally so a failed capture still restores the canvas. Exports that don't set them are unchanged.
  • components/Lineage/Lineage/Lineage.tsx: the PNG export sets renderAllNodes in beforeCapture, waits two animation frames for React Flow to mount the nodes, and clears the flag in afterCapture.
  • playwright/utils/lineage.ts: connectEdgeBetweenNodes fits the view before it asserts that the new node is visible. A node that lands off-screen is no longer mounted, which broke 15 DataAssetLineage tests.
• Equivalence. Before switching implementations, I compared old and new parseLineageData and getUpstreamDownstreamNodesEdges on 33 graphs: cycles and self-loops, random graphs with missing or duplicate FQNs, duplicated node entries, pipelines whose FQN is missing, dangling edges, both pipeline view modes, the real 24 MB response and a 12.6k-edge one. Parse output was deep-equal and traversal output was reference-identical, in the same order.
• Alternatives considered:
  • Dropping the getLineage fetch from the table tab. Since Fixes #33865: Inline lineage editing via node and column menus #34551, the table tab renders from it.
  • Node/edge thresholds with a fallback layout, the approach of fix(lineage): avoid freezing on very large graphs #34986. It leaves the main-thread parse untouched, and it hides the per-column edges instead of removing them.
  • Moving ELK to a Web Worker. It isn't needed once ELK's input is bounded by node pairs.
  • Virtualizing in a follow-up PR. Without it, the merged canvas alone blocks the main thread for 7.3–9.7 s in this PR's spec, so the spec could not guard the parse fix.
  • Virtualizing without the export hook. A PNG exported while zoomed in would leave out every off-screen node.
• Compatibility. There are no API or schema changes. With the column layer on, layouts become compact: the gap between layers is the configured 280 px instead of growing with the number of mapped columns. Off-screen nodes on the table tab are no longer in the DOM, so a test that looks for one must bring it into view first, as connectEdgeBetweenNodes now does.

Tests:

Use cases covered

• Opening the Lineage tab of a table whose getLineage response has 2,551 nodes and 52,550 edges draws every node and keeps the page responsive: no main-thread block of 5 s or more.
• Parsing a response of that size completes in milliseconds and returns the same nodes and edges, including load-more nodes and pipeline edges.
• Laying out tables that share 50 mapped columns keeps the configured gap between layers.
• A PNG export of a zoomed-in canvas contains every node, and the canvas goes back to visible-only afterwards, also when the capture fails.
• Adding lineage through the canvas still works when the new node lands off-screen.
• Re-running the fetch effect while a request is in flight does not request the same lineage again, and a response that a newer request superseded does not overwrite the newer lineage.

Unit tests

• I added unit tests for the new/changed logic.

• Files added/updated:

  • src/utils/EntityLineagePureUtils.test.ts: dedupe, pipeline routing, load-more counts, and a 2,551-node / 52,550-edge parse.
  • src/utils/EntityLineageNodeUtils.test.ts: a cycle, edges to tables that weren't fetched, and a 52,550-edge walk.
  • src/utils/Lineage/Layout/ELKUtil/ELKUtil.test.ts: elkjs receives one edge per node pair.
  • src/utils/Lineage/Layout/ELKUtil/ELKUtil.integration.test.ts (new): real elkjs, 50 parallel column edges per link, checks the layer gap.
  • src/utils/Export/ExportUtils.test.tsx: the capture starts only after beforeCapture resolves and is followed by afterCapture, and afterCapture also runs when the capture fails.
  • src/components/Lineage/Lineage/Lineage.integration.test.tsx: in-flight dedupe and the superseded response.
  • src/utils/EntityLineageUtils.test.tsx: removed two tests that only asserted uniqWith was called. The new dedupe test covers that behavior with real lodash.

• Each new test fails on the old code:

  test	old code	this PR
  parse of 52,550 edges	30.3 s (budget 3 s)	73 ms
  walk of 52,550 edges	1.26 s (budget 0.3 s)	24–40 ms
  layer gap with column edges	8,480 px (expected 680)	680 px
  elkjs edges	all 5 edges passed	3 edges
  in-flight fetch¹	2 requests	1 request
  superseded response¹	stale.table shown	dropped

  ¹ Measured against main before Fixes #33865: Inline lineage editing via node and column menus #34551 landed. main passes both now.

• Coverage on changed lines: 100%, 137 of 137 executable changed lines in the six source files with logic changes: EntityLineagePureUtils.ts 60, EntityLineageNodeUtils.ts 38, ELKUtil.ts 14, Lineage.tsx 13, ExportUtils.ts 9 and useLineageStore.ts 3. The two canvas components only pass the store flag to a React Flow prop; the Playwright runs below cover them.

• Jest suites run locally: 47 suites, 708 tests passed. These cover src/utils/Export, src/components/Entity/EntityExportModalProvider, src/components/Lineage, src/components/Entity/EntityLineage, src/utils/EntityLineage*, src/utils/Lineage, src/hooks/useLineageStore.test.ts and src/rest/lineageAPI.test.ts.

• tsc --noEmit on src and on the Playwright project: no new errors. The only error in a changed file, playwright/utils/lineage.ts line 645 in connectEdgeBetweenNodesViaAPI, is on main too.

• .github/scripts/tests/test_playwright_ci_planning.py: 123 passed, 3 failed. The same three fail with main's impact maps.

Backend integration tests

• Not applicable (no backend changes).

Ingestion integration tests

• Not applicable (no ingestion changes).

Playwright (UI) tests

• I added Playwright E2E tests under openmetadata-ui/.../ui/playwright/ for UI changes.

• Files added: playwright/e2e/Pages/Lineage/LineageLargeResponse.spec.ts (@basic). It serves a 2,551-node, 52,550-edge getLineage response to a real table's Lineage tab. It first waits until the minimap holds all 2,551 nodes, so a response that fails to parse fails the test. It then fails if the longest gap between 50 ms main-thread heartbeats reaches 5 s.

  • Nightly only, per review. Pull-request and merge-queue runs (PW_ENTITY_MATRIX=representative) do not register the suite. The nightly full suite runs it, and so does the gating run of a pull request that edits the spec, through the isRepresentativeEntityRun exemption that entity-matrix specs use.

  • The spec stays out of playwright/e2e/nightly/ on purpose: no OSS Playwright project picks up specs in that folder, so they never run. --list returns 0 tests for the specs already there.

  • Results on the local stack:

    code	result
    this PR	passed in 10.5 s; the version before the minimap check passed 3 of 3
    this PR, with a response that makes parseLineageData throw (temporary copy of the spec)	failed at the minimap check, as intended: 0 of 2,551 nodes
    merged main canvas, not virtualized	failed: 7.3 s and 9.7 s gaps in two runs
    main before this PR and Fixes #33865: Inline lineage editing via node and column menus #34551	failed: 81.6 s gap

• playwright/utils/lineage.ts: connectEdgeBetweenNodes fits the view before checking the new node. With it, DataAssetLineage passes again: 100 passed, and its 1 failure is the sample-data test below.

• .github/playwright/impact-map.json: the lineage mapping now also covers src/utils/EntityLineage*, src/utils/Lineage/** and src/hooks/useLineageStore.ts. A new export mapping sends EntityExportModalProvider/** and src/utils/Export/** to the six specs that export: ClassificationImportExport, LineageExportPNGSnapshot, SampleDataTableOperations, CSVImportWithQuotesAndCommas, GlossaryImportExport and Users, in the chromium and ImportExport projects. Without these, the files were unmapped, and CI would have escalated them to the full suite.

• impact-map.generated.json was regenerated. --check passes, also on this branch merged with today's main.

Local Playwright run: FAILED (4 failures: 3 need sample data that this local stack does not load, and 1 flake passed on rerun; see below)

• Commit: 323d44dd6904 (base origin/main). The only uncommitted file was the untracked paseo.json, which is not part of this PR.
• Started: 2026-10-16T13:56:06.385Z · Playwright 1.57.0 · 58.7 min · --workers=2
• Totals: 275 passed, 4 failed, 0 flaky, 0 skipped across 20 selected specs

Spec	Reason	Passed	Failed	Flaky	Skipped
playwright/e2e/Features/ClassificationImportExport.spec.ts	impact-mapped	4	0	0	0
playwright/e2e/Features/Dashboards.spec.ts	smoke	3	1	0	0
playwright/e2e/Features/LineageExportPNGSnapshot.spec.ts	impact-mapped	0	1	0	0
playwright/e2e/Features/SampleDataTableOperations.spec.ts	impact-mapped	11	0	0	0
playwright/e2e/Flow/LineageSettings.spec.ts	impact-mapped	3	0	0	0
playwright/e2e/Flow/Navbar.spec.ts	smoke	22	0	0	0
playwright/e2e/Pages/CSVImportWithQuotesAndCommas.spec.ts	impact-mapped	1	0	0	0
playwright/e2e/Pages/DataMarketplace.spec.ts	smoke	5	0	0	0
playwright/e2e/Pages/GlossaryImportExport.spec.ts	impact-mapped	9	0	0	0
playwright/e2e/Pages/Lineage/DataAssetLineage.spec.ts	impact-mapped	100	1	0	0
playwright/e2e/Pages/Lineage/LineageControls.spec.ts	smoke	8	0	0	0
playwright/e2e/Pages/Lineage/LineageFilters.spec.ts	impact-mapped	14	1	0	0
playwright/e2e/Pages/Lineage/LineageInteraction.spec.ts	impact-mapped	23	0	0	0
playwright/e2e/Pages/Lineage/LineageLargeResponse.spec.ts	changed	1	0	0	0
playwright/e2e/Pages/Lineage/LineageNodePagination.spec.ts	impact-mapped	5	0	0	0
playwright/e2e/Pages/Lineage/LineageRightPanel.spec.ts	impact-mapped	11	0	0	0
playwright/e2e/Pages/Lineage/PlatformLineage.spec.ts	impact-mapped	6	0	0	0
playwright/e2e/Pages/Policies.spec.ts	smoke	3	0	0	0
playwright/e2e/Pages/Users.spec.ts	impact-mapped	43	0	0	0
playwright/e2e/auth.setup.ts	setup	1	0	0	0
playwright/e2e/entity-data.setup.ts	setup	1	0	0	0
playwright/e2e/entity-data.teardown.ts	setup	1	0	0	0
playwright/e2e/Flow/Tour.spec.ts	impact-mapped	not run

Failures and gaps:

• DataAssetLineage.spec.ts › Temp lineage table nodes: throws Sample entity 'sample_data.ecommerce_db.shopify.raw_order' not found.
• Dashboards.spec.ts › Data Model › expand / collapse after updating nested fields: waits for a sample_superset data model row.
• LineageExportPNGSnapshot.spec.ts › exported PNG includes edge lines between nodes: its lineage URL points at a sample-data table, so getLineage never fires locally, and the test times out waiting for it. I checked the same export path by hand (Manual testing, step 5). The merge queue loads sample data and runs it.
• LineageFilters.spec.ts › Impact Analysis › downstream count: a flake. It passed on rerun, and the spec passed 15 of 15 in the previous run.
• Flow/Tour.spec.ts shows "not run" because it is quarantined (@quarantine), so no project selects it.

Also run, for canvas specs outside the impacted set: Features/ImpactAnalysis, Features/IncidentManager, Features/DomainIsolation/DomainLineageIsolation and Pages/ExplorePageRightPanel, with 264 passed, 1 failed. The failure is in IncidentManager: its test-suite pipeline never produced a run, because this local stack has no ingestion runner. With these, every spec I found that drives EntityLineageCanvas or LineageMap ran, except the quarantined Tour. The specs that match canvas selectors but were not run use other canvases (three Workflows specs and InputOutputPorts), or create lineage through the API and never open the canvas (OntologyDataExploreRdf, and Pagination, which pages the Impact Analysis list).

After this run, aa3026824d changed only LineageLargeResponse.spec.ts, adding the minimap check. Run alone afterwards, it passed.

Environment: this branch's UI on Vite, against a server built from this branch at c5c83a4ae0, after the first merge of main. main has had no lineage backend changes since.

Command

cd openmetadata-ui/src/main/resources/ui
npx playwright test playwright/e2e/Features/ClassificationImportExport.spec.ts playwright/e2e/Features/Dashboards.spec.ts playwright/e2e/Features/LineageExportPNGSnapshot.spec.ts playwright/e2e/Features/SampleDataTableOperations.spec.ts playwright/e2e/Flow/LineageSettings.spec.ts playwright/e2e/Flow/Navbar.spec.ts playwright/e2e/Flow/Tour.spec.ts playwright/e2e/Pages/CSVImportWithQuotesAndCommas.spec.ts playwright/e2e/Pages/DataMarketplace.spec.ts playwright/e2e/Pages/GlossaryImportExport.spec.ts playwright/e2e/Pages/Lineage/DataAssetLineage.spec.ts playwright/e2e/Pages/Lineage/LineageControls.spec.ts playwright/e2e/Pages/Lineage/LineageFilters.spec.ts playwright/e2e/Pages/Lineage/LineageInteraction.spec.ts playwright/e2e/Pages/Lineage/LineageLargeResponse.spec.ts playwright/e2e/Pages/Lineage/LineageNodePagination.spec.ts playwright/e2e/Pages/Lineage/LineageRightPanel.spec.ts playwright/e2e/Pages/Lineage/PlatformLineage.spec.ts playwright/e2e/Pages/Policies.spec.ts playwright/e2e/Pages/Users.spec.ts --workers=2

Manual testing performed

1. I started an isolated stack: native PostgreSQL 14, native Elasticsearch 9.3, the released openmetadata/server:2.0.5, and a main-equivalent server (build 74a8d70e0d, whose lineage code matched main at the time) on a copy of the same database.
2. I seeded the busy table described above through REST: 5,076 tables and 2,775 lineage edges. getLineage at depth 2/2 with size=50 returned 24 MB (151 nodes, 1,375 edges, 625 of them to tables that weren't fetched).
3. I opened the Lineage tab in headless Chromium with a 50 ms main-thread heartbeat and a CPU profile. Larger shapes were served by intercepting getLineage with scaled copies of the captured response. These runs predate Fixes #33865: Inline lineage editing via node and column menus #34551:

build	response	longest main-thread block	notes
2.0.5	real, 24 MB	1.2 s	graph rendered 1.5 s after the response
2.0.5, column layer on	real, 24 MB	—	tab crashed after 38 s; offline, elkjs passed 4 GB on the same graph
2.0.5	14 MB, 2,651 nodes	7.7 s	8.9 s to first paint
main (production build)	real, 24 MB	0.6 s	getLineage fetched twice
main (production build)	23 MB, 12,650 edges	4.4 s	5.2 s of long tasks
main (production build)	61 MB, 52,650 edges	29.3 s	58 s of long tasks
main (Vite dev), old code	23 MB, 12,650 edges	10.0 s	3 getLineage calls
this PR (Vite dev)	23 MB, 12,650 edges	0.6 s	1 call
main (Vite dev), old code	61 MB, 52,650 edges	35.3 s	3 calls, 96 s of long tasks
this PR (Vite dev)	61 MB, 52,650 edges	0.5 s	1 call

4. After merging main with Fixes #33865: Inline lineage editing via node and column menus #34551, I moved to a server built from this branch at c5c83a4ae0 and served a 2,651-node response to the table tab, with the UI on Vite:

canvas	first paint	longest main-thread block
merged main, not virtualized	20.0 s	24.6 s
this PR	0.9 s	1.7 s

5. PNG export, on the real 151-node graph: I zoomed in until 1 node was mounted, then exported a PNG. All 151 nodes were mounted during the capture, and the count went back to 1 afterwards. The image shows the hub and both its upstream and downstream columns, although the screen showed only the hub. With the beforeCapture hook disabled, only that 1 node was mounted during the capture.

UI screen recording:

Not applicable. This is a performance fix with no visual change. The canvas looks the same; it only leaves off-screen nodes out of the DOM, and a PNG export still holds every node (Manual testing, step 5). The measurements above stand in for a recording.

Checklist:

• I have read the CONTRIBUTING document.

• My PR title is Fixes <issue-number>: <short explanation>

• My PR is linked to a GitHub issue via Fixes #<issue-number> above.

• I have commented on my code, particularly in hard-to-understand areas.

• For JSON Schema changes: not applicable, no schema changes.

• For a UI feature or feature task: not applicable, this is a bug fix (see UI screen recording).

• I have added tests (unit / integration / Playwright as applicable) and listed them above.

• Decision check: no record needed. The new contracts are documented where they are defined: ELKLayout.layoutGraph returns collapsed edges (on the method), and ExportData.beforeCapture / afterCapture (on the interface).

• I have added a test that covers the exact scenario we are fixing. For complex issues, comment the issue number in the test for future reference.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new actionable issues were established.

Summary

This PR reduces work when parsing, walking, laying out, and displaying large lineage graphs.

• Large lineage responses take fewer repeated scans to parse and walk.
• ELK lays out one connection for each pair of lineage nodes.
• Lineage PNG exports include nodes outside the current view.
• Lineage fetch tests cover repeated and outdated responses.
• Playwright selections include the generated specs and fit lineage graphs to view.

_{Reviews (2) · Last reviewed commit: "Fail the large-lineage spec when the gra..." · Reviewed by Greptile}

[github-actions]

⚠️ UI Checkstyle passed — lint findings in changed files

🔍 ESLint findings in this PR's files — 0 error(s), 5 warning(s)

Errors block the build. Warnings do not yet — they are rules whose backlog is still
being worked down, listed so this PR does not add to it. See docs/ui-code-quality-gate.md.

0 error(s), 5 warning(s) across 2 changed file(s).

Count	Rule
4	openmetadata-imports/no-impure-pure-utils
1	react-hooks/exhaustive-deps

All findings

	Location	Rule	Message
🟡	src/components/Lineage/EntityLineageCanvas/EntityLineageCanvas.tsx:132:6	react-hooks/exhaustive-deps	React Hook useEffect has a missing dependency: 'updateEntityData'. Either include it or remove the dependency array.
🟡	src/utils/EntityLineagePureUtils.ts:17:1	openmetadata-imports/no-impure-pure-utils	Pure utilities must not depend on React, UI, state, hooks, pages, or REST clients. Move orchestration/rendering out or move shared types to a lower layer.
🟡	src/utils/EntityLineagePureUtils.ts:19:1	openmetadata-imports/no-impure-pure-utils	Pure utilities must not depend on React, UI, state, hooks, pages, or REST clients. Move orchestration/rendering out or move shared types to a lower layer.
🟡	src/utils/EntityLineagePureUtils.ts:24:1	openmetadata-imports/no-impure-pure-utils	Pure utilities must not depend on React, UI, state, hooks, pages, or REST clients. Move orchestration/rendering out or move shared types to a lower layer.
🟡	src/utils/EntityLineagePureUtils.ts:45:1	openmetadata-imports/no-impure-pure-utils	Pure utilities must not depend on React, UI, state, hooks, pages, or REST clients. Move orchestration/rendering out or move shared types to a lower layer.

---

Fix locally (fast - only checks files changed in this branch):

make ui-checkstyle-changed

[github-actions]

Jest test Coverage

UI tests summary

Lines	Statements	Branches	Functions
	78.19% (128476/164311)	64.86% (81478/125617)	66.14% (27221/41155)

[chirag-madlani]

Let's keep this for nightly only

[mohityadav766]

Done in d630f11: pull-request and merge-queue runs (PW_ENTITY_MATRIX=representative) no longer register this suite. The nightly full suite runs it, and so does the gating run of a PR that edits the spec, through the same isRepresentativeEntityRun exemption entity-matrix specs use.

I kept it out of playwright/e2e/nightly/ on purpose: no OSS Playwright project picks up that folder, and --list returns 0 tests for the specs already there, so it would never run.

[github-actions]

🚦 Removed from the merge queue — manual (2026-10-16T10:00:45Z)

The entry left the queue before it was built, so no checks ran against it.

[github-actions]

🚦 Removed from the merge queue — manual (2026-10-16T15:54:05Z)

The entry left the queue before it was built, so no checks ran against it.

[gitar-bot]

Code Review ✅ Approved

🟡 Medium risk · Changing lineage parsing, canvas virtualization, and ELK edge collapsing could misrepresent graph contents or omit off-screen nodes from PNG exports if handling diverges.

Speeds up the table Lineage tab for large responses by linearizing parsing and graph traversal, giving ELK one edge per node pair, rendering only visible nodes on the canvas, and mounting all nodes during PNG export. No issues found.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

_{For more verification details, comment gitar display:verbose.}

Options

Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ui'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud