Skip to content

Drop the per-nonzero cell array from Store.terms() to cut transient memory in the dense LP export #1058

Open

Description

@FabianHofmann

Note

This issue was drafted by AI as a follow-up to #1057 (steps 4a-4c of #1019).

Describe the feature you'd like to see

Context

Store.terms() returns three arrays per live term: the flat cell index, the variable label and the coefficient (linopy/store.py, linopy/csr.py). The dense export (Constraint.to_polars in linopy/constraints.py, LinearExpression long-format helpers in linopy/expressions.py, assert_equivalent in linopy/testing.py) only uses the cell array to gather a per-cell value (constraint label or constant) onto the nonzeros. The int64 cell array is therefore a transient of about 8 bytes per nonzero that exists only to be used as an index and then dropped.

#1057 already sizes the export buffers by Store.nnz and slices dense stores by view (commits d9a0351, f535e29), so this cell array is the last per-nonzero residual in the dense LP path that is not part of the output.

Proposed direction

Change the protocol so the store returns the gathered value instead of the index, for example:

  • Store.terms(values: np.ndarray | None = None) returning (gathered, vars, coeffs), where gathered is values repeated per live term in that cell, or
  • Store.terms() returning (counts, vars, coeffs) with one int per cell, so callers expand with np.repeat(values, counts).

Either way the per-nonzero index never materialises. DenseStore can derive counts from the keep mask with keep.reshape(-1, nterm).sum(1); CSRStore already has indptr. Call sites to adapt: Constraint.to_polars, LinearExpression.to_polars/long-format helpers and linear_terms, assert_equivalent.

Separate from this: two CodSpeed memory benchmarks (test_op[con_le_array], test_op[con_le_scalar]) sit about 34 KB above master after #1057. That is the fixed cost of the field DataArrays on Constraint, not the cell array, and is out of scope here.

Acceptance criteria

  • Both backends implement the new signature; the conformance test in test/test_store_conformance.py covers it.
  • to_polars, LP writing and assert_equivalent produce identical output on dense and sparse stores.
  • CodSpeed memory benchmarks benchmarks/drivers/test_to_lp.py and benchmarks/drivers/test_to_solver.py show the peak drop for dense models; no regression for CSR.

Activity

  1. added
    performanceThis improves performance while not (meaningfully) altering behaviour for users
    refactorInternal change with no user-facing behavior change
    on Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceThis improves performance while not (meaningfully) altering behaviour for usersrefactorInternal change with no user-facing behavior changev1

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions