nassau: recompute quasi-inverses on demand, with bidegree-major batched lifting

[JoeyBF]

Motivation

On-disk quasi-inverses (QIs) are the sole blocker to larger nassau computations: stem 256 already exceeds 20 TB and stem 300 would blow past 40 TB. The QIs solve the lifting problem so downstream work (products, Massey products, the secondary/d₂ structure) doesn't recompute them — but they are mathematically re-derivable from the (tiny) saved differentials. The catch is that re-deriving one QI costs about as much as computing it did originally, so recomputing per lift is far too slow.

This PR makes recompute-on-demand practical by amortizing each QI recompute across every lift that needs it: sweep output bidegrees, and at each one solve the shared quasi-inverse once for all the maps/homotopies passing through it.

What's here

1. Recompute-on-demand for nassau QIs (nassau.rs)
Regenerate a QI on the fly from differentials[s] alone (plus the module bases and the deterministically-chosen subalgebra) instead of reading it from disk. It streams one signature at a time through the exact byte format write_qi produces, so apply_quasi_inverse reads a recomputed QI through the same code path as a saved one and peak memory matches what resolving that bidegree originally took — never the whole QI, which can reach hundreds of GB at record stems. Opt-in, no default behavior change: EXT_NASSAU_NO_SAVE_QI stops persisting QIs, EXT_NASSAU_RECOMPUTE_QI forces recompute. By default QIs are still saved and read.

2. Bidegree-major batched lifting (resolution_homomorphism.rs)
A Liftable trait (target/source-type-free: prepare(b) -> Option<{inputs, finish}>) and a MultiLift driver that extends many liftables sharing one target complex together, in bidegree-major order via iter_s_t, issuing one batched apply_quasi_inverse per output bidegree. With QIs recomputed on demand this keeps a whole many-map computation at ~1× the on-disk baseline instead of scaling with the number of maps. The single-map lift paths become behavior-preserving prepare_step/finish_step splits.

3. Consumers wired to batch

• Products — ExtAlgebra::extend_all_products builds the product map per generator and extends them all through one MultiLift.
• Massey — massey_iter_c now builds every f_c and null-homotopy up front and extends them in two batched passes (maps, then homotopies) instead of a fresh homotopy per third factor. massey_iter_a already reuses one homotopy, so it's unchanged.
• Secondary — the SecondaryLift homotopy solve gets the same prepare/finish split and a SecondaryLiftable wrapper + batch_extend_secondary; SecondaryExtAlgebra::extend_all_secondary_products batches all secondary products (SecondaryChainHomotopy / secondary Massey is left for a follow-up).

4. A latent MultiLift fix
step_cell skipped the finishers whenever the combined inputs across all liftables at a bidegree were empty. A zero-dimensional step lifts nothing but must still run its finish to register/extend the step, or a later read of that bidegree indexes an unfilled degree and panics. Finishers now always run (with an empty results slice when there's nothing to lift).

Behavior & compatibility

• Default behavior is unchanged — QIs are still saved and read; recompute is opt-in.
• All single-map paths are behavior-preserving refactors (the prepare/finish splits), pinned by the existing tests.

Testing

• test_iter_a_matches_iter_c / test_iter_c_proper_kernel diff batched massey_iter_c against the independent massey_iter_a reference.
• batched_matches_native_d2 pins the secondary wrapper against native compute_homotopies; batched_secondary_products_match pins the multi-lift batched products against the per-lift path.
• Existing products, Massey, secondary, and save/load tests still pass. just lint (nightly rustfmt + clippy) is clean on the touched files, and docs build under -D rustdoc::broken_intra_doc_links.

Illustrative timing (all_products on S₂, QIs recomputed): batched 4.57 s vs. per-map 17.33 s for 516 product maps — the redundancy the batching removes.

Relationship to #260 (ZarrV3)

Deliberately built on master, independent of the ZarrV3 save-format PR, so it can land first — it settles what we store (recompute-on-demand removes the QI files) before that PR finalizes how. Once this lands, #260 can drop its QI stream tier.

🤖 Generated with Claude Code

---

Generated by Claude Code

Summary by CodeRabbit

• New Features
  • Added batch computation for all primary and secondary product maps.
  • Added controls to save previously computed quasi-inverses or recompute them when needed.
  • Added an example benchmark for comparing product-map extension modes.
• Performance
  • Improved large-scale product and Massey product calculations by sharing computations across multiple lifts.
• Bug Fixes
  • Added verification that batched secondary products match individually computed results.

[coderabbitai]

Warning

Review limit reached

@JoeyBF, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 35 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 727f959a-814b-4ef3-addd-ade8365e0f17

📥 Commits

Reviewing files that changed from the base of the PR and between 8e305fc and 971bfba.

📒 Files selected for processing (3)

• ext/src/ext_algebra/massey.rs
• ext/src/resolution_homomorphism.rs
• ext/src/secondary.rs

📝 Walkthrough

Walkthrough

The PR adds shared prepare/finish lifting abstractions and batches quasi-inverse solves across primary, secondary, product, and Massey computations. It also adds configurable quasi-inverse regeneration and an example for comparing per-map and batched product extension.

Changes

Batched lifting pipeline

Layer / File(s)	Summary
Prepare-and-finish lifting infrastructure ext/src/resolution_homomorphism.rs	Resolution homomorphism steps now separate preparation from completion, while MultiLift batches lift requests through a shared quasi-inverse solve.
Chain homotopy batching ext/src/chain_complex/chain_homotopy.rs	Chain homotopies expose prepared lift requests and finalize lifted generators from shared outputs.
Secondary homotopy batching ext/src/secondary.rs, ext/src/ext_algebra/secondary.rs	Secondary lifting uses preparation, batched solving, completion, persistence, and equivalence tests.
Ext algebra consumers ext/src/ext_algebra/mod.rs, ext/src/ext_algebra/massey.rs, ext/src/ext_algebra/secondary.rs	Product, secondary product, and Massey computations use batched lifting paths.
Quasi-inverse controls and benchmark ext/src/nassau.rs, ext/examples/all_products.rs	Environment flags control quasi-inverse persistence and recomputation, and the example compares extension modes.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ExtAlgebra
  participant MultiLift
  participant ChainHomotopy
  participant Nassau
  ExtAlgebra->>MultiLift: extend product and homotopy liftables
  MultiLift->>ChainHomotopy: prepare bidegree requests
  ChainHomotopy-->>MultiLift: lift inputs and finish closures
  MultiLift->>Nassau: apply combined quasi-inverse inputs
  Nassau-->>MultiLift: quasi-inverse outputs
  MultiLift->>ChainHomotopy: finalize prepared steps

Loading

Possibly related PRs

• SpectralSequences/sseq#240: Introduced the Ext algebra product APIs extended by this PR’s batched product processing.
• SpectralSequences/sseq#241: Introduced the secondary product machinery used by the new batched secondary extension.
• SpectralSequences/sseq#272: Modified the Nassau quasi-inverse persistence and loading paths touched by this PR.

Poem

A rabbit hops through lifts in flight,
One solve serves many maps tonight.
Products bloom and brackets rhyme,
Secondary paths align in time.
Quasi-inverses stream anew—
Batch magic makes the burrow stew!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the two main changes: on-demand quasi-inverse recomputation and batched lifting.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@ext/src/ext_algebra/massey.rs`:
- Around line 328-341: Update the batched lifting around MultiLift::extend_all
so each b_hom, f_c, and pending null-homotopy is extended only through its
required tot/profile bound rather than the full overlap. Add or reuse a bounded
MultiLift/profile-lift operation, preserve the per-result tot association from
pending, and skip both batches when no work is required so b_hom is not fully
extended when pending is empty.

In `@ext/src/nassau.rs`:
- Around line 1268-1270: Update the quasi-inverse writer state around
signatures, scratch, and buf so signature blocks are not materialized before
consumption. Retain iterator and pivot-serialization progress across reads, and
change write_qi plus the Read::read implementation to append and return bytes
incrementally for each command while preserving ordering and serialization
behavior.

In `@ext/src/resolution_homomorphism.rs`:
- Around line 408-411: Update the documentation near MultiLift to describe chain
homotopy and secondary lift implementations of Liftable as existing
implementors, removing the stale “future implementors” wording while preserving
the explanation of their shared target quasi-inverse and batching behavior.

In `@ext/src/secondary.rs`:
- Around line 834-848: Update batch_extend_secondary to validate that every
lift’s lift.target() matches the supplied target before calling
prepare_homotopies or constructing MultiLift. Reject or otherwise handle any
mismatch using the existing project convention, while preserving the current
empty-input behavior and batching flow for valid lifts.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 79b864ce-551a-46e7-a990-094736d20d0b

📥 Commits

Reviewing files that changed from the base of the PR and between 4867b30 and 8e305fc.

📒 Files selected for processing (8)

• ext/examples/all_products.rs
• ext/src/chain_complex/chain_homotopy.rs
• ext/src/ext_algebra/massey.rs
• ext/src/ext_algebra/mod.rs
• ext/src/ext_algebra/secondary.rs
• ext/src/nassau.rs
• ext/src/resolution_homomorphism.rs
• ext/src/secondary.rs