Skip to content

refactor: one Constraint class over a store-backed lhs and one netcdf layout (#1019 steps 4a-4c) - #1057

Merged
FabianHofmann merged 8 commits into
masterfrom
store/4-one-constraint
Oct 24, 2026
Merged

FabianHofmann merged 8 commits into
masterfrom
store/4-one-constraint

Conversation

@FabianHofmann

@FabianHofmann FabianHofmann commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1046, closes #1047, closes #1048, closes #1033. Part of #1019 (steps 4a, 4b, 4c).

Note

The following content was generated by AI.

Changes proposed in this Pull Request

One Constraint class, one store behind it, one netcdf layout. Three steps of the #1019 plan in four commits:

4a, f7eb1e1: Constraint over lhs. A constraint holds an lhs: LinearExpression (whose store is a DenseStore or CSRStore) plus sign, rhs, labels, scaling, dual, binary_var, binary_val and blocks as DataArrays on the lhs grid. data is derived. Store.canonical() and Store.positional_csr() join the store protocol; the positional CSR cache lives on CSRStore (keyed on label_to_pos identity and the active rows) and there is none on DenseStore. Every path that replaces the lhs store sets _coef_dirty, so the persistent snapshot and diff shortcuts keep working.

4b, b7f7824: one Constraint class. CSRConstraint, ConstraintBase and the freeze API are deleted. A constraint in a sparse model is a Constraint whose lhs holds a CSRStore. update(), soften, set_blocks and the loc setters keep a sparse lhs sparse (closes #1033). MatrixAccessor, the dual write-back and Model.copy have no isinstance branches on the store. The twin-build test helpers are gone; constraint tests are parametrized over the backend fixture.

4c, d75d04a: one netcdf layout. Constraint lhs, model expressions and the objective are written from the store through Store.to_netcdf_vars() / from_netcdf_vars(), dispatching on a _linopy_store_kind attribute. Every store reads back as the kind it was written with. The _linopy_format="csr" branch, the dense-only reader and the legacy MultiIndex reader are deleted. read_netcdf refuses files written by linopy < 1.0 with a ValueError; a fixture file under test/data/ guards that.

bd8be88: review fixes. Docs no longer promise that pre-1.0 files load; clearer read errors; layout attrs carry the _linopy_ prefix.

Breaking changes (1.0)

  • CSRConstraint, ConstraintBase, Constraint.freeze(), Constraint.mutable(), add_constraints(freeze=) and Constraints.add(freeze=) are removed. Use Model(sparse=True), Constraint.to_sparse() / to_dense() and c.lhs.is_sparse.
  • Constraint(lhs, sign, rhs, model, ...) replaces Constraint(data, model, ...); Constraint.from_dataset builds one from a Dataset. Writing into con.data no longer changes the constraint.
  • One netcdf layout for dense and sparse stores. Files written by linopy < 1.0 are refused by read_netcdf; rebuild and save them again.
  • Non-strict assert_conequal compares the lhs with assert_equivalent, i.e. layout-free.

Review and tests

Two review gates (after 4b, and full-branch before this PR) found no must-fix in the code; the surviving must-fixes were two stale doc statements, fixed in bd8be88. Full suite: 4747 passed, 84 skipped (test/remote excluded, it needs google-cloud-storage). mypy linopy, ruff check . and the doctests pass.

Deferred review findings (nice-to-have, not blocking)
  • The eight constraint field names are spelled out five times in linopy/constraints.py (_FIELDS, __slots__, __init__, _bind, _fields).
  • add_constraints and add_indicator_constraints repeat the same build block.
  • Constraint.to_matrix_with_rhs duplicates the rhs/sense extraction in matrices._row_block.
  • The active-row mask is computed twice per indicator constraint in MatrixAccessor.
  • DenseStore.active_terms is public on one store only.
  • The from_dataset round-trip test is dense-only; the deep-copy test does not cover the field arrays; test_constraint_wrapped_methods asserts nothing.
  • label_range reads back from netcdf as an ndarray, not a tuple (callers unpack it fine; the dense reader behaved the same before).
  • Objective writing is split between objective.py and io.py; read_netcdf reads scalar attrs leniently.
  • Pre-existing, not introduced here: to_netcdf fails for expressions with tz-aware datetime, categorical or MultiIndex coords. Deserves its own issue.

Checklist

  • AI-generated content is marked (see AGENTS.md).
  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

CI follow-ups (AI-generated)

  • 5037b812: mypy over the test suite (CI runs mypy .).
  • d9a03512, f535e29a: CodSpeed flagged a peak-memory regression (35 benchmarks, export up to 7x). Causes: export buffers sized by term slots instead of nonzeros, DenseStore copying broadcast term arrays, per-slice copies in the LP writer. Fixed via Store.nnz and Store.isel (views on the dense store). After the fix all large export benchmarks are at or below master (five improved, up to 7x); two tiny constraint-build ops (test_op[con_le_array], test_op[con_le_scalar]) stay about 34 KB above master, the fixed per-constraint cost of holding sign, rhs, scaling and labels as separate DataArrays. Residual on dense LP export: the int64 cell array from Store.terms, about 8 bytes per nonzero; follow-up in Drop the per-nonzero cell array from Store.terms() to cut transient memory in the dense LP export #1058.
  • codecov/project drops 0.31 points to 88.54 % because about 2900 lines of well-covered code were deleted; patch coverage is 98 %.

…data derived (#1046)

Constraint stores lhs, sign, rhs, labels, scaling, dual, binary fields and blocks;
data is built from them. Store.positional_csr/canonical; CSRStore caches the
positional CSR by weakref. add_constraints, to_constraint, netcdf reader and
Model.copy construct from fields. Step 4a of #1019.
…Base/freeze API removed (#1047)

A constraint in a sparse model is a Constraint whose lhs holds a CSRStore.
update(), soften and set_blocks keep a sparse lhs (closes #1033).
MatrixAccessor, dual write-back and Model.copy lose their isinstance branches.
Twin-build test helpers deleted; tests parametrized over backends. Step 4b of #1019.
… refused (#1048)

Store.to_netcdf_vars/from_netcdf_vars dispatch on a _store_kind attribute for
constraint lhs, model expressions and the objective; the csr/dense legacy
reader branches are deleted. read_netcdf refuses files without the 1.0 layout
attribute. Round trips parametrized over both backends. Step 4c of #1019.
Docs no longer promise that pre-1.0 netcdf files load. read_netcdf tells a
foreign file from a pre-1.0 one; missing store kind raises ValueError; layout
attrs carry the _linopy_ prefix; dead order fallback removed.
@codspeed

codspeed Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 2 benchmarks

1 warning

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

⚡ 5 improved benchmarks
❌ 2 regressed benchmarks
✅ 174 untouched benchmarks
⏩ 181 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ test_op[con_le_array] 188.6 KB 222.7 KB -15.28%
❌ test_op[con_le_scalar] 282.4 KB 316.4 KB -10.76%
⚡ test_to_lp[sparse_network-n=250] 23.6 MB 3.2 MB ×7.4
⚡ test_to_lp[kvl_cycles-severity=100] 27 MB 7.5 MB ×3.6
⚡ test_to_lp[merge_balance-severity=100] 12.7 MB 5.5 MB ×2.3
⚡ test_to_lp[merge_balance-severity=50] 6.6 MB 3.4 MB +94.95%
⚡ test_to_lp[nodal_balance-severity=100] 3.5 MB 2.6 MB +31.74%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing store/4-one-constraint (455f7a1) with master (b45f1ad)

Open in CodSpeed

Footnotes

  1. 181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

…seStore

_row_block sized buffers by term slots instead of nonzeros; DenseStore
has_terms/take/terms copied broadcast term arrays; constraints_to_file held
two slices; constraint construction copied const and coords. Peak memory of
the export benchmarks is back to master level.
…in to_polars

Store.isel returns views on DenseStore; Constraint.iterate_slices uses it
instead of take, which copied coeffs and vars per LP slice.
… adds dims

Reorder an incoming lhs to the constraint grid in Constraint._rebind so
labels stay contiguous for range-based dual write-back; Store.transpose.
Review nits: sizes reuse, dead _fill_value, Sequence[Hashable] annotation,
release-note wording; tests for store swap dirtiness, masked duals on both
backends, dense 1e-10 prune tolerance.
@FabianHofmann
FabianHofmann merged commit 51c4977 into master Oct 24, 2026
20 of 22 checks passed
@FabianHofmann
FabianHofmann deleted the store/4-one-constraint branch October 24, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant