Skip to content

docs: redraw the home page architecture diagram as a simple SVG - #6322

Queued
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:diagram-svg-afd8ec6e
Queued

andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:diagram-svg-afd8ec6e

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

No issue; this is documentation only.

Rationale for this change

The architecture diagram on the home page, under "Tight integration with Apache DataFusion", is old and hard to follow. It draws the JVM and native operators as two parallel stacks joined by solid, dashed and dotted arrows that each mean something different, with callouts attached by leader lines. It also doesn't show what the paragraph above it says it shows: the Comet plugin intercepting Spark's physical plan, translating the supported operators to protobuf, and handing them to DataFusion.

This replaces it with a deliberately simple diagram of that flow, in the same style as the executor memory diagram from #6237. It is not meant to be complete. For example, "Anything else keeps running in Spark" reads as falling back one operator at a time, while a query stage that contains an unsupported operator falls back as a whole (see Spark Operator Support). The "How Comet works" page linked under the diagram carries the precise version.

What changes are included in this PR?

  • docs/source/_static/images/comet-overview.svg: a new hand-written SVG (rendered). It has three layers. Apache Spark plans the query as usual, the Comet plugin replaces the operators it supports with native ones, and the Apache DataFusion native engine runs them, reading Parquet files and Iceberg tables directly. The plan goes down the left side, first as Spark's physical plan and then as the protobuf native plan, and the results come back up the right side to Spark as Arrow columnar batches. The colours follow the existing site diagrams, yellow for Spark and green for Comet, and each layer is tagged as JVM or native.
  • docs/source/index.md: the home page figure now uses the SVG, with its dimensions and alt text updated.

The SVG has a transparent background, like the PNG it replaces. The home page figure card is white in both site themes, and the theme dims every image in dark mode, so an opaque white background would show up there as a grey box inside the card. The old comet-overview.png is no longer referenced but is left in the tree; it can be removed in a follow-up.

How are these changes tested?

Documentation only, no code paths touched.

  • Built the site locally with Sphinx from main, with and without this change. Both builds report the same 63 warnings, all of which main already has.
  • npx prettier@latest --check docs/source/index.md passes.
  • Rendered the SVG in headless Chrome, including with a deliberately wide fallback font: an SVG embedded as an image cannot use the site's web fonts, so most readers get a system font. I also screenshotted the built home page in the light and dark themes.

Replace comet-overview.png on the home page with a hand-written SVG in the
style of comet-executor-memory.svg. Spark plans the query, the Comet plugin
replaces the operators it supports with native ones, and the Apache
DataFusion native engine runs them and returns the results to Spark as
Arrow columnar batches.
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 28, 2026

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

  • Prior state and problem: The home page used a raster architecture diagram. This change makes the plan and result flows explicit in a scalable image.
  • Design approach: A standalone SVG shows Spark, the Comet plugin, and DataFusion, with separate arrows for plans, results, and input data.
  • Correctness / compatibility analysis: Checked the architecture against Comet’s implementation and Spark’s extension hooks in versions 3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0. The linked architecture guide explains stage-wide fallback. No execution semantics change.
  • Key design decisions: Transparent background, system fonts, and existing responsive styling keep the asset simple. No new rendering dependency is introduced.
  • Implementation sketch: Adds comet-overview.svg and updates the home-page image reference, dimensions, and alt text.
  • Behavioral changes worth calling out: The displayed asset decreases from 172,624 to 5,374 bytes. Lazy loading remains in place.
  • Suggested improvements: None meeting the P1/P2 threshold. No introduced P1/P2 issues found within this review.

Reviewed the entire diff from 65a0cda1cf62877a37ec1a1f5f9ebc9dd1405ebd to a8f25f63b437a07a25d6248b4afe946eac1a1e61. The PR is not a draft. Read AGENTS.md and applied review-comet-pr. No sibling skill applies to this documentation-only change. Snapshot and live discussion checks found no reviews, comments, or threads.

Exact-head CI: Preflight, CodeQL, Analyze Actions, title validation, and labeling passed. Detect changes was queued at the final check, so CI was not complete. No failures were reported.

Validation: SVG XML, accessibility references, image dimensions, copied build asset, and whitespace checks passed. CairoSVG rendering showed no clipping. A direct Sphinx build succeeded with 63 warnings concerning missing generated content, cross-references, and unavailable mmdc, with none concerning the changed home page or image. The full publishing build, browser light/dark rendering, and JVM/native suites were not run.

@andygrove
andygrove added this pull request to the merge queue Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants