Scan Header Ingest and Tomography Handling - #3
Open
RamyaGuru wants to merge 33 commits into
Open
Conversation
PublishToCloudOp consolidated per-batch temp HDF files into a final file, but SinkAndPublishOp.write_scan_file now does the single end-of-scan write directly to the publish folder (commit 2d089e3). The temp files it consumed are no longer produced, so it no-ops on every trigger. Rather than delete it (it may be useful for a future external/temp-file workflow), this just removes it from the live pipeline. That still simplifies the control path for the upcoming flush/header work: ControlOp no longer needs an output port or a processing_end branch. - Keep PublishToCloudOp class in publish.py, documented as not-currently-wired - pipeline.py: drop its construction, import, and both control flows - Stop SinkAndPublishOp emitting processing_end (+ drop its output port) - ControlOp: drop output port and processing_end forwarding Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Foundational plumbing for the header/tomography work. No behavior change for existing single-projection scans (flush still happens at the next start). - PtychoReconstructionOp: emit a one-shot "recon_complete" on a new "complete" output port when the final iteration is reached (reuses the existing is_last predicate, S9); guard + reset via _completed in flush(). - PtychoAccumulatorOp.flush: zero the GPU buffers INSIDE ptycho_state["lock"] so recon can't read half-zeroed buffers (R-3); no-op when nothing has been accumulated since the last flush (R-2 _dirty guard) so the unconditional start-flush is free when clean. - ControlOp: log "recon_complete" only — NO flush here, so a completed single-projection scan keeps its result until the next start/header. The flush-on-completion / per-projection semantics arrive with PR3. - pipeline.py: wire ptycho_recon "complete" -> control_op. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… STXM safety Builds on the flush plumbing so completion drives the flush (Task 3) and every flush is race-safe — a prerequisite for PR2's mid-stream header preemption. - ControlOp: recon_complete now flushes (result is already saved+published before it fires) and marks _flushed; the scan-start flush becomes a safety net that skips when already flushed (no double-flush) and only runs if the previous scan didn't complete. Factored into _do_flush(). - Deferred flush for GatherOp, PtychoAccumulatorOp, PtychoReconstructionOp: flush() sets a flag; the reset happens at the top of the next compute(), so it never mutates caches/GPU buffers/the PIE object concurrently with compute (fixes the GatherOp boolean-index race; safe for mid-stream preemption). recon flush also zeros the shared filled_until so it can't re-process the finished scan before the accumulator's own deferred flush runs. - recon_complete is gated on all_data_arrived, so it only fires when the scan is genuinely complete (not when a low total_iterations exhausts mid-stream). - SinkAndPublishOp: write_scan_file now saves-and-clears (self-contained), and flush() defensively writes any unwritten buffer before clearing so a scan's STXM file is never lost. Completion check uses >= (tolerates overshoot). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On-the-fly scan-geometry reconfiguration from a live JSON header, with
mid-reconstruction preemption, on top of PR1's race-safe deferred flushes.
- header_io.py (NEW): HeaderRxOp — dedicated ZMQ SUB, validates the header
{npoints_h,npoints_v,step_size_h,step_size_v,num_projections}, stages
pending_geometry + requests preemption, updates shared scan_state, emits a
"header" token to ControlOp. Rejects over-capacity grids before staging.
- ptychography_setup.py: split init_ptycho_state into load_ptycho_model
(grid-independent) + configure_scan_geometry (grid-dependent object sizing,
re-points views, needs_gpu_reinit, rejects no_frames > capacity) +
init_ptycho_state (load + allocate GPU buffers ONCE at max capacity [R-6] +
one default configure). New ptycho_state keys: H, W, capacity, scan_state,
preempt_requested/quiesced Events, pending_geometry, needs_gpu_reinit.
- ptychography_ops.py: R-4 quiescence handshake at the top of recon.compute
— on preempt, save the in-flight partial + emit recon_complete, set
quiesced, then apply staged geometry while quiesced and re-init GPU for the
new object. R-6: full-buffer summary sliced to [:no_frames].
- control.py: new "header" case -> flush for the reconfigured scan.
- pipeline.py: always-present scan_state (S11); build HeaderRxOp from the
header_src config; wire header_src -> control_op; pass scan_state to
init_ptycho_state.
- config_test/prod.yaml: remove npoints_*/step_size_*; add header_src block +
max_npoints_h/v (buffer capacity) + default_step_size_h/v.
Validated in-container (A400): launch, back-to-back regression,
header-before-data + full recon on reconfigured geometry, mid-recon
preemption with partial save, post-preempt recovery, oversized-grid reject.
Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… + STXM output
One arm/start wraps all projections; num_projections × no_frames stream
continuously and the projection boundary is segmented by frame count
(series_id shared across projections). Each projection early-stops once its
frames are in and the in-flight PIE iteration finishes, then advances; single
scans still run to total_iterations.
- GatherOp: drain cached frames on series_finished (R-1).
- PtychoAccumulatorOp: projection-boundary split with carry, backpressure once
filled >= no_frames, advance_projection() resets fill level while preserving
carry + centre.
- PtychoReconstructionOp: tomography is_last = all_data_arrived; per-projection
save ({series}_proj{NN}_recon.h5); self-advances on observing filled drop
below no_frames (avoids double-complete); emits projection_complete
(non-final) vs recon_complete (final).
- ControlOp: projection_complete → scoped advance (accumulator + recon only,
NOT GatherOp — its next-projection cache must survive).
- SinkAndPublishOp: per-projection STXM save ({series}_proj{NN}.h5) reading
scan_state; header_io sets scan_state no_frames; pipeline wires scan_state.
- ITER_TIMING per-iteration timing diagnostic retained (sub-ms overhead).
- Fix off-by-one in the dummy_img_index test path (series_frame_count - 1 →
series_frame_count) exposed while testing; test-only path (default False).
Container-tested on the A400 with a 2-projection header over a 2048-frame
daqsim stream: both projections reconstruct and save (STXM + recon), correct
1024-frame counts, and the non-final/final completion signals distinguished.
See DECISIONS_header_tomo.md / TESTING_header_tomo.md (incl. open item O1 on
per-projection post-stream refinement iterations, deferred to PR4).
Assisted-By: Claude Opus 4.8 (1M context)
… from finalize Motivated by moving to a much faster detector: single-buffer backpressures the source across the ~one-iteration finalize window (measured ~490 ms), which throttles a free-running detector and risks dropping positions (PUB/SUB, no flow control). Double-buffering keeps the accumulator draining while the recon finalizes. Two GPU buffer sets (ping-pong). While the recon finalizes projection N on the read buffer, the accumulator fills N+1 into the write buffer. On fill it flips to the free buffer (else clean backpressure fallback when the recon is lapped by a full projection — data is never clobbered). Single-projection scans always use buffer 0 (no flip). - ptychography_setup.py: allocate raw_gpu/positions_full/tilts_full as 2-element lists (once, at max capacity — R-6); add ping-pong state (filled_until[], write_idx, read_idx, buf_free[], num_buffers) reset in configure_scan_geometry. - PtychoAccumulatorOp: _try_flip() on fill (else backpressure + a one-shot "Recon lagging" warning); all writes indexed by write_idx; _perform_flush zeros both buffers + resets the ping-pong idempotently. - PtychoReconstructionOp: reads indexed by read_idx; _flip_read() releases the finished buffer, OWNS current_projection (no ControlOp round-trip → no save-vs-index race), advances to the next buffer, resets the object (probe carried); final projection emits recon_complete and idles. - control.py: dropped the now-dead projection_complete branch; ControlOp keeps recon_complete / header / flush. Container-validated on the A400 (see DESIGN_pr4_double_buffer.md): regression (exact 1024/projection, same iteration budget), overlap (accumulator fills the next projection during a stretched finalize without backpressure), and lapping fallback (clean backpressure + recovery when the recon is lapped). No drops, no deadlock. Open item O1 (per-projection post-stream iterations) is now more visible — pre-filled projections finalize with few refinement iterations. Assisted-By: Claude Opus 4.8 (1M context)
…sing the recon to fail when a new header was sent mid-reconstruction. Plan can be found in fix_header_race.md. Start with PR 1: Control flush split (refactor, behavior-preserving)
…er race. Now PR 2: Transition state primitives + header phase start
…scan reacts well. Now PR 3: Phase-aware ControlOp handling. Header no longer triggers immediate ptycho flush request. recon_complete in preemption path triggers ptycho flush request exactly once.
…sualisation wasn't flushing, it was just accumulating and looking strange. So we added a couple of lines in pipeline/control.py lines 124-126 which send a stxm_flush message to the visualiser
… PR 4: Ptycho flush execution ack + safe barrier release. Release transition only after ptycho flush has actually executed.
…low signaling; Enforce strict branch alignment and bounded blocked buffering. This implements the overflow fail error (ie determine the max number of buffered frames while ptycho state quiesces and flushes).
…ata accumulation.
…th the data accumulation." This reverts commit 013ef6e. All PR 5 needs to be un-done and roll back to PR 4, but using git revert rather than git reset for future clarity.
…st overflow signaling; Enforce strict branch alignment and bounded blocked buffering. This implements the overflow fail error (ie determine the max number of buffered frames while ptycho state quiesces and flushes)." This reverts commit 1eae67f. Fully reverting PR 5.
… to clear/flush visualisation when a new projection comes in.
…n when buffer flips; this will hopefully help with out-of-bound problems at syn_probe_intensity_kernel
…e and probe phase to grayscal rather than twilight.
…nd flux is only calculated if json file value is < 0
…next projection current iteration doesn't do +1. This means that all projections will start from iteration 0. This makes sure they'll calculate flux and do probe normalisation.
…y). Also increased padding zone (this is important with fast scans selun/ck3m because they may overshoot). Stopped printing start message from detector (too long).
… batch - this requires the batch size to be at least 2x the nX, but it's useful for cases where there is a lot of overshoot in x. Also I've included a sign for the calculation of y centre, because it changes in a tomography projection to projection.
…an. In that case, we don't need to re-calculate the flux. Also changed config file for testing at the beamline.
…ppearing in the next projection. The main solution was initialising the pty_model.obj.array_global_kernel with every new projection. Also, simplified the scan_center calculation to be 0 for y and mean(px) for x -- this is not a long term solution but it'll do for now.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Streaming tomography: live headers, dynamic geometry, flush-on-completion, double-buffering
Adds multi-projection tomography to the Holoscan ptycho pipeline, driven by a live
scan-geometry header, with reconstruction that keeps up with a continuous stream.
Six commits (
dcdcbda→761fd35) on top ofmain(a15ab2d).What's in it
PublishToCloudOp(class kept, removed from the live graph; the disk save already lives inSinkAndPublishOp).HeaderRxOp(dedicated ZMQ SUB) sets scan geometry live; grid params removed from config; GPU buffers allocated once at max capacity (never realloced); mid-reconstruction preemption via a quiescence handshake (finish iteration → save → flush → reconfigure).num_projectionsback-to-back projections segmented by frame count; per-projection reconstruction + STXM files ({series}_projNN[_recon].h5); each projection early-stops when its frames are in.Testing
All validated in the
ptycho-holoscan:ptyrexcontainer on the A400, driven by the daqsim simulator (scan 409907, 32×32/projection).PublishToCloudOpunwiredtotal_iterations, idles,recon_complete