refactor: one Constraint class over a store-backed lhs and one netcdf layout (#1019 steps 4a-4c)

doc/api.rst

@@ -358,8 +358,10 @@ Post-solve access
 Constraint
 ==========
 
-Subclass of ``xarray.DataArray`` carrying labels for a multi-dimensional
-constraint.
+A multi-dimensional constraint ``lhs sign rhs`` with labels. Its lhs holds a
+dense store, or in a sparse model (``Model(sparse=True)``) a memory-efficient
+CSR store; ``to_sparse`` and ``to_dense`` convert it. See the
+:doc:`creating-constraints` and :doc:`sparse-models` guides for usage.
 
 .. autosummary::
    :toctree: generated/
@@ -378,14 +380,13 @@ Structure
    constraints.Constraint.scaling
    constraints.Constraint.coeffs
    constraints.Constraint.vars
+   constraints.Constraint.ncons
+   constraints.Constraint.nterm
 
 Modification
 ------------
 
-``Constraint.update`` is the canonical mutation API. The legacy ``lhs`` /
-``sign`` / ``rhs`` / ``coeffs`` / ``vars`` setters still forward to
-``update`` but emit a ``DeprecationWarning`` and will be removed in a
-future release.
+``Constraint.update`` is the canonical mutation API.
 
 .. autosummary::
    :toctree: generated/
@@ -407,52 +408,8 @@ Conversion
    :toctree: generated/
 
    constraints.Constraint.to_polars
-
-
-CSRConstraint
-=============
-
-Memory-efficient, immutable constraint representation backed by a scipy
-CSR sparse matrix. Opt in via ``Model(sparse=True)`` or
-``Model.add_constraints(..., freeze=True)``. See the
-:doc:`creating-constraints` and :doc:`sparse-models` guides for usage.
-
-.. autosummary::
-   :toctree: generated/
-
-   constraints.CSRConstraint
-
-Structure
----------
-
-.. autosummary::
-   :toctree: generated/
-
-   constraints.CSRConstraint.coeffs
-   constraints.CSRConstraint.vars
-   constraints.CSRConstraint.sign
-   constraints.CSRConstraint.rhs
-   constraints.CSRConstraint.scaling
-   constraints.CSRConstraint.ncons
-   constraints.CSRConstraint.nterm
-
-Post-solve access
------------------
-
-.. autosummary::
-   :toctree: generated/
-
-   constraints.CSRConstraint.dual
-
-Conversion
-----------
-
-.. autosummary::
-   :toctree: generated/
-
-   constraints.CSRConstraint.to_polars
-   constraints.CSRConstraint.to_dense
-   constraints.CSRConstraint.mutable
+   constraints.Constraint.to_sparse
+   constraints.Constraint.to_dense
 
 
 Constraints
@@ -694,8 +651,9 @@ Utilities
 - ``warn_on_densify``: the conversion notice, a ``PerformanceWarning`` when
   a sparse (CSR) backing is converted to dense or a dense operand is
   converted to sparse for a merge with a sparse one. ``None`` (default)
-  warns in a sparse model on implicit conversions only, ``True`` always
-  warns, ``False`` never warns. See :doc:`sparse-models`.
+  warns in a sparse model, ``True`` always warns, ``False`` never warns.
+  Only implicit conversions warn; ``Constraint.to_dense()`` and
+  ``Constraint.to_sparse()`` never do. See :doc:`sparse-models`.
 
 
 Warnings

doc/migrating-to-v1.rst

@@ -107,8 +107,9 @@ Every row is a legacy guess that is now an explicit rule.
      - Rejected; linopy uses a flat dimension with the levels as aux coords, and a
        per-level input must be mapped onto the flat dimension explicitly (no
        implicit projection). Affects PyPSA multi-investment models. Netcdf
-       files written with such a dimension load flat: a positional index with
-       the levels as aux coords.
+       files written with such a dimension by linopy < 1.0 are refused by
+       :func:`~linopy.io.read_netcdf`; rebuild the model and save it again
+       (see `Saved models (netcdf)`_).
      - ``.reset_index(dim)`` to flatten, then project the per-level input by
        its level aux coord.
    * - A multi-key ``groupby(...).sum()`` result
@@ -124,3 +125,11 @@ Every row is a legacy guess that is now an explicit rule.
 
 The full, normative rule list lives in the
 :doc:`arithmetic convention <design/convention>`.
+
+Saved models (netcdf)
+---------------------
+
+The netcdf file layout changed: dense and sparse stores now share one layout.
+:func:`~linopy.io.read_netcdf` refuses files written by linopy < 1.0 and raises a
+``ValueError``. Rebuild the model with linopy 1.0 and save it again with
+:meth:`~linopy.model.Model.to_netcdf`.

doc/release_notes.rst

@@ -12,11 +12,10 @@ Version 1.0.0
 * ``broadcast_to_coords`` no longer accepts ``warn_reorder``.
 * The legacy helpers ``is_v1``, ``warn_legacy``, ``legacy_warnings_muted`` and ``warn_legacy_absence`` are removed from ``linopy.semantics``, and ``check_user_nan`` no longer accepts ``op_kind=``, takes an optional ``context`` message argument and always raises.
 * ``LinearExpressionGroupby.map`` no longer accepts the deprecated ``shortcut`` argument.
-* The deprecated sparse opt-ins are removed: ``Model(freeze_constraints=...)``, the ``Model.freeze_constraints`` property and setter, ``groupby(...).sum(sparse=...)`` and ``linopy.options["sparse_groupby"]``. Use ``Model(sparse=True)`` instead and read ``Model.sparse``. Netcdf files written with a ``freeze_constraints`` attribute still load; the attribute is ignored.
-* ``Model(auto_mask=...)`` and the ``Model.auto_mask`` property are removed: user-supplied NaN no longer masks variables or constraints. NaN in coefficients or RHS values raises; pass ``mask=`` instead. Netcdf files written with an ``auto_mask`` attribute still load; the attribute is ignored.
+* The deprecated sparse opt-ins are removed: ``Model(freeze_constraints=...)``, the ``Model.freeze_constraints`` property and setter, ``groupby(...).sum(sparse=...)`` and ``linopy.options["sparse_groupby"]``. Use ``Model(sparse=True)`` instead and read ``Model.sparse``.
+* ``Model(auto_mask=...)`` and the ``Model.auto_mask`` property are removed: user-supplied NaN no longer masks variables or constraints. NaN in coefficients or RHS values raises; pass ``mask=`` instead.
 * NaN in a user-supplied variable bound (``lower``/``upper`` in ``add_variables``, ``Variable.update``, ``fix``) now raises a ``ValueError``; exclude positions explicitly with ``mask=``.
 * ``MultiIndex`` coordinates are unsupported everywhere: besides ``add_variables``, ``add_constraints`` and ``add_objective``, ``Model.parameters`` and constant operands reject them with the ``reset_index`` hint. ``broadcast_to_coords`` no longer projects inputs indexed by the levels of a ``MultiIndex`` dimension onto it.
-* Netcdf files written with ``MultiIndex`` dimensions load in flat form: the dimension gets a positional index and the former levels become auxiliary coordinates. This includes frozen (CSR) constraints.
 * The ``assign_multiindex_safe`` methods of ``Variable``, ``LinearExpression``, ``QuadraticExpression`` and ``Constraint`` are removed; use ``assign``. The helpers ``linopy.common.assign_multiindex_safe`` and ``linopy.common.assign_coords_multiindex_safe`` are renamed to ``assign_fields`` and ``reassign_coords``.
 * Long-deprecated shims are removed:
 
@@ -36,10 +35,29 @@ Version 1.0.0
 * An operation without a sparse path on a CSR-backed ``LinearExpression``, including reading ``.data``, now works on a dense copy and no longer converts the expression itself, which stays sparse; only assigning ``vars``, ``coeffs`` or an off-grid ``const`` replaces its store. The densify notice names the operation and the case as ``op(case)``, e.g. ``Sparse (CSR) backing densified: reindex(method=...)``, so filters matching the former reason texts need updating. A named sparse expression stays sparse through ``Model.add_expressions`` and ``Model.copy`` and keeps its name through assignments. ``densify_terms``, ``reset_const`` (so also ``Constraint.update(rhs=...)``) and ``Model.assign_coords`` keep a sparse expression sparse. ``LinearExpression.where`` and ``QuadraticExpression.where`` take ``drop`` as their only keyword argument instead of forwarding ``**kwargs`` to ``xarray.Dataset.where``, which accepts no other. (`#1020 <https://github.com/PyPSA/linopy/issues/1020>`__)
 * ``Model.store_cls`` names the store class a model builds expressions on, ``CSRStore`` in a sparse model and ``DenseStore`` otherwise. ``Variable.to_linexpr`` always returns an expression on it; the private ``_dense`` keyword is removed. (`#1045 <https://github.com/PyPSA/linopy/issues/1045>`__)
 * A dense operand in a merge with a CSR-backed one is converted up through the new ``Store.to_sparse(reason)``; a merge that stays sparse emits one ``Dense backing sparsified: merge with a sparse operand`` notice, under the same ``warn_on_densify`` option as the densify notice. A dense operand with coordinates the sparse grid cannot hold, e.g. two-dimensional auxiliary ones, now falls the whole merge back to dense with the reason ``merge(coordinates the Grid cannot hold: <names>)``. A merge of dense operands stays dense, also in a sparse model. (`#1045 <https://github.com/PyPSA/linopy/issues/1045>`__)
-* A named linear expression of a sparse model reads back from netcdf as a CSR-backed expression, unless it carries coordinates the sparse grid cannot hold, e.g. two-dimensional auxiliary ones; then it reads back dense with those coordinates. (`#1045 <https://github.com/PyPSA/linopy/issues/1045>`__)
 * A constant operation that produces NaN in a live cell, e.g. ``0 * inf``, ``0 / 0`` or ``inf - inf`` from multiplying, dividing, adding or subtracting a constant, now raises ``ValueError`` on both backends. Before, the cell kept its terms under a NaN constant. linopy trusts NaN only as absence, so mark cells to drop with ``.where(...)`` or ``mask=`` instead. Cells that were absent before stay absent. A sparse division by zero no longer emits numpy ``RuntimeWarning``\ s. (`#1045 <https://github.com/PyPSA/linopy/issues/1045>`__)
 * Cells with a NaN constant now drop their terms when a ``LinearExpression`` or ``QuadraticExpression`` is built from a dataset or its ``vars`` or ``coeffs`` are assigned, so neither a dataset with terms under a NaN constant nor a ``vars`` assignment can revive an absent cell. (`#1045 <https://github.com/PyPSA/linopy/issues/1045>`__)
 
+
+**Breaking Changes: constraints on stores**
+
+.. note::
+
+   The following content was generated by AI.
+
+* ``Constraint`` holds its left-hand side as a ``LinearExpression`` with a zero constant and its store in canonical form, next to ``sign``, ``rhs``, ``labels``, ``scaling``, ``dual``, ``binary_var``, ``binary_val`` and ``blocks`` as DataArrays on the grid of the lhs. ``Constraint.data`` is derived on each access, so writing into it, e.g. ``con.data["sign"] = ...``, no longer changes the constraint; use ``Constraint.update(...)``. ``Constraint.lhs`` returns a new expression on the held store on each access. (`#1046 <https://github.com/PyPSA/linopy/issues/1046>`__)
+* The ``Constraint`` constructor takes the fields instead of a dataset: ``Constraint(lhs, sign, rhs, model, name="", labels=None, scaling=1.0, dual=None, binary_var=None, binary_val=None, blocks=None, attrs=None)``; an lhs with a nonzero constant raises. Build a constraint from its dataset layout with ``Constraint.from_dataset(ds, model, name)``; the ``skip_broadcast`` argument is removed. A field with dimensions outside the lhs or with coordinates differing from those of the lhs raises, instead of broadcasting the lhs or filling the field with NaN. (`#1046 <https://github.com/PyPSA/linopy/issues/1046>`__)
+* ``Constraint.assign`` and the other wrapped xarray methods reject data variables other than the constraint fields, and ``Constraint.broadcast_like`` no longer broadcasts the fields over ``_term``. After ``Constraint.update(coeffs=...)`` or ``update(variables=...)``, ``coeffs`` and ``vars`` keep the ``_term`` dimension, a value without it is spread over the terms. (`#1046 <https://github.com/PyPSA/linopy/issues/1046>`__)
+* ``Constraints.set_blocks`` stores the block numbers as the ``blocks`` field of each constraint, read as ``con.data.blocks``. (`#1046 <https://github.com/PyPSA/linopy/issues/1046>`__)
+* ``CSRConstraint``, ``ConstraintBase`` and the freeze API are removed without aliases: ``Model.add_constraints(freeze=...)``, ``Constraints.add(freeze=...)``, ``Constraint.freeze()``, ``CSRConstraint.mutable()``, ``CSRConstraint.from_dense``/``from_csr`` and the "frozen constraint" ``AttributeError``. A constraint in a sparse model is a ``Constraint`` whose lhs holds a ``CSRStore``; replace ``isinstance(c, CSRConstraint)`` with ``c.lhs.is_sparse``. It supports the whole ``Constraint`` API, e.g. ``update``, the ``scaling`` setter, ``sanitize_*``, ``set_blocks`` and the wrapped xarray methods; ``update`` and ``soften`` keep a sparse lhs sparse (`#1033 <https://github.com/PyPSA/linopy/issues/1033>`__). Reading ``coeffs`` or ``vars`` of a sparse constraint, ``update(coeffs=..., variables=...)`` and ``set_blocks`` work on a dense copy with a densify notice. (`#1047 <https://github.com/PyPSA/linopy/issues/1047>`__)
+* In a sparse model, ``add_constraints`` and ``add_indicator_constraints`` hold the lhs on a ``CSRStore`` whenever the grid can hold its coordinates, also for a dense lhs, and ``Constraint.from_dataset`` reads into the model's store class. ``Constraint.to_sparse()`` and ``Constraint.to_dense()`` replace the lhs store in place. Explicit conversions emit no notice at all, so ``warn_on_densify=True`` no longer flags them; the ``explicit`` argument of the conversion notice is removed. (`#1047 <https://github.com/PyPSA/linopy/issues/1047>`__)
+* A sparse constraint keeps ``rhs``, ``sign``, ``scaling`` and ``dual`` on its full grid like a dense one, instead of on its active rows; masked rows have label ``-1`` and rows without terms stay labelled but inactive. A masked row with an invalid infinite rhs now raises in a sparse model too. ``Constraint.active_labels``, ``to_matrix`` and ``to_polars`` raise ``ValueError`` on a constraint without labels. A sparse lhs reads its coefficients back as ``float64``. (`#1047 <https://github.com/PyPSA/linopy/issues/1047>`__)
+* ``Constraint.sanitize_zeros`` drops near-zero terms from the lhs store (``Store.prune`` takes a tolerance), so the ``_term`` dimension of a dense constraint can shrink, instead of marking the terms ``-1``/NaN. ``Constraint.iterate_slices`` slices the lhs store with ``Store.isel`` and keeps a sparse lhs sparse. ``MatrixAccessor`` and the dual write-back of ``Model.assign_result`` treat both stores alike. (`#1047 <https://github.com/PyPSA/linopy/issues/1047>`__)
+* Netcdf: one layout for every linear expression, constraint lhs and the objective, written from its store: a ``_linopy_store_kind`` attribute (``"dense"`` or ``"csr"``), ``const`` on the grid dimensions with their coordinates, and either ``coeffs``/``vars`` on the grid dimensions and ``_term`` or the CSR arrays ``data``/``indices``/``indptr``. Constraint fields (``labels``, ``sign``, ``rhs``, ``scaling``, ``dual``, ``binary_var``, ``binary_val``, ``blocks``) lie on the grid dimensions. Each lhs, expression and objective reads back on the store class it was written with, so a dense constraint in a sparse model stays dense and vice versa. The ``_linopy_format``/``_csr_columns`` CSR layout and the active-row and multiindex legacy readers are removed. (`#1048 <https://github.com/PyPSA/linopy/issues/1048>`__)
+* ``read_netcdf`` refuses files written by linopy <1.0 with a ``ValueError``; files carry the layout version in the ``_linopy_netcdf_layout`` attribute next to ``_linopy_version``. Rebuild the model and write it again with linopy >=1.0. (`#1048 <https://github.com/PyPSA/linopy/issues/1048>`__)
+* ``linopy.testing.assert_conequal(strict=False)`` compares the lhs with ``assert_equivalent``, independently of its store and term layout, instead of comparing the dense datasets. (`#1047 <https://github.com/PyPSA/linopy/issues/1047>`__)
+* ``Constraint.update`` raises a ``ValueError`` when the new ``lhs`` or ``rhs`` expression adds dimensions to the constraint; broadcasting a constraint to a larger grid is not supported. Before, the update was accepted and the labels stayed on the old dimensions, which left an inconsistent constraint.
+
 **Deprecations**
 
 * The ``active_fill`` parameter of ``add_piecewise_formulation`` is deprecated and emits a ``FutureWarning``; it is removed in 1.1. Pad a partial gate explicitly instead, e.g. ``active=status.reindex(gen=gens).fillna(1)``, which gives the same model. A partial ``active`` without padding still raises.

doc/user-guide.rst

@@ -25,8 +25,7 @@ you're unsure what a particular operator or argument does.
   quadratic) expressions; arithmetic, broadcasting, ``sum``,
   ``groupby``, ``rolling``, ``where``.
 - :doc:`creating-constraints` — turning expressions into ``≤`` / ``≥``
-  / ``==`` constraints, and the ``CSRConstraint`` memory-efficient
-  alternative.
+  / ``==`` constraints, and their memory-efficient sparse (CSR) storage.
 - :doc:`coordinate-alignment` — how linopy lines up operands that live
   on different coordinates, and how to control it with ``join``.
 
@@ -78,8 +77,9 @@ Sparse models
 Models that aggregate many entities into few rows, such as generators
 into buses, can store their expressions as sparse matrices instead of
 padded dense arrays. Opt in with ``Model(sparse=True)``. ``groupby(...).sum()`` and ``@`` then return sparse-backed
-expressions, and by default every constraint is stored as a frozen
-``CSRConstraint``. Operations without a sparse path convert an
+expressions, and every constraint holds its left-hand side in CSR form
+whenever the grid can hold its coordinates.
+Operations without a sparse path convert an
 expression to dense and emit a ``PerformanceWarning``.
 
 .. code-block:: python