Skip to content

Read FITS inputs directly in xScan and the device pipeline - #64

Open
tpn wants to merge 3 commits into
mainfrom
codex/xscan-fits-pipeline
Open

tpn wants to merge 3 commits into
mainfrom
codex/xscan-fits-pipeline

Conversation

@tpn

@tpn tpn commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

The device pipeline requires prepared NPY inputs. This adds FITS descriptors so workers can decode supported images with xDR and pass device arrays into the existing subtraction, fitting, and classification stages. Shared HDUs are read once, input reading is timed separately, and artifacts record the actual reader. The shared reader and xFit prerequisites are on main; this PR has no pending PR dependency.

xScan also gains predict_fits for explicit candidate stamps from aligned FITS planes. Raw dataset builders preserve named-HDU selection for full images and bounded cutouts. The separate-stage benchmark reads all FITS roles, including variance and masks when the science images are NPY. Inputs must already provide the required alignment and mask semantics.

--fits-reader auto|astropy|xdr is available on data-build-autoscan-raw, data-build-nodiff-raw, data-build-lsstcomcam-smoke, run-pipeline, and benchmark-pipeline. Omitting it preserves manifest selections; an explicit choice overrides every FITS descriptor before constructing work identities and reaches benchmark subprocesses and stage reads. NPY inputs and source manifests are unchanged. Selecting Astropy still requires the pipeline's downstream GPU runtime; xDR can use ordinary I/O with KVIKIO_COMPAT_MODE=ON when native GDS is unavailable.

The independent change passed 326 CLI, FITS, benchmark, executor, and Dragon tests, with four GPU-dependent skips. All lint hooks passed, including type checking. Earlier composed-series validation passed 2,755 CPU tests with 188 skips. Local GPU staged xPois checks with all-FITS and mixed inputs matched NPY outputs exactly under Astropy and automatic selection; auto fell back to Astropy, and native xDR was unavailable in that source environment. The subsequent rebase onto the GPU-check workflow changes ancestry only.

Earlier implementation measurements used registered Rubin inputs and fixed candidates:

Workload Astropy median xDR median
One-GPU pipeline, two image pairs and 32 candidates 15.65 s 10.76 s
Same pipeline including full scientific-array auditing 16.36 s 11.97 s
Two-worker MPI, compact-output batch 13.57 s 10.74 s
Two-worker Dragon, compact-output batch 13.43 s 10.36 s

These results used three warm measured rounds; the distributed checks included one warmup. Retained numerical outputs matched, including 704 distributed decoded-array comparisons. FITS inputs were lossless encodings of already registered arrays. Input conversion and cold startup are excluded; distributed batch timers also exclude readiness and coordinator auditing. A separate native-GDS run recorded cuFile P2P payload reads without POSIX fallback or errors. These GPU, GDS, and distributed measurements predate the current fixes and were not repeated for this rebase.

@tpn
tpn requested a review from melo-gonzo September 26, 2026 22:28
@coderabbitai

coderabbitai Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)
  • ai-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: NVIDIA/cuPhoton/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f9395f65-664c-4b38-8fb7-72e607c91be0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@tpn
tpn force-pushed the codex/xscan-fits-pipeline branch 2 times, most recently from b4b97c7 to 777538f Compare September 26, 2026 23:21
@tpn
tpn marked this pull request as ready for review September 28, 2026 16:07
@tpn
tpn force-pushed the codex/xfit-fits-input branch from 3d51f5f to b0946d0 Compare September 28, 2026 17:25
@tpn
tpn force-pushed the codex/xscan-fits-pipeline branch from 777538f to f0a16d3 Compare September 28, 2026 17:25

@melo-gonzo melo-gonzo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two regressions need fixes: FITS ingestion in the separate-stage benchmark and named-HDU compatibility in raw-data builders. Reproductions and suggested fixes are inline.

Comment thread src/cuphoton/xscan/device_pipeline.py
Comment thread src/cuphoton/xscan/des.py Outdated
@tpn
tpn force-pushed the codex/xfit-fits-input branch from b0946d0 to e3356be Compare September 28, 2026 21:01
@tpn
tpn force-pushed the codex/xscan-fits-pipeline branch from f0a16d3 to dcfc778 Compare September 28, 2026 21:01
@tpn
tpn force-pushed the codex/xfit-fits-input branch from e3356be to 941c8c2 Compare September 28, 2026 21:42
tpn added 3 commits September 28, 2026 15:10
Signed-off-by: Trent Nelson <trentn@nvidia.com>
Signed-off-by: Trent Nelson <trentn@nvidia.com>
Signed-off-by: Trent Nelson <trentn@nvidia.com>
@tpn
tpn force-pushed the codex/xscan-fits-pipeline branch from dcfc778 to e91bbb6 Compare September 28, 2026 22:11
@tpn
tpn changed the base branch from codex/xfit-fits-input to main September 28, 2026 22:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants