Build and change models via math-spec

.github/workflows/test-notebooks.yml

@@ -30,7 +30,7 @@ jobs:
     - name: Install package and dependencies
       run: |
         python -m pip install uv
-        uv pip install --system -e ".[docs]"
+        uv pip install --system -e ".[docs]" --group spec
 
     - name: Execute notebooks
       run: |

.github/workflows/test.yml

@@ -82,7 +82,7 @@ jobs:
     - name: Install package and dependencies
       run: |
         python -m pip install uv
-        uv pip install --system "$(ls dist/*.whl)[dev,solvers,oetc]"
+        uv pip install --system "$(ls dist/*.whl)[dev,solvers,oetc]" --group spec
 
     - name: Test with pytest
       env:
@@ -120,7 +120,7 @@ jobs:
     - name: Install package and dependencies
       run: |
         python -m pip install uv
-        uv pip install --system "$(ls dist/*.whl)[dev]"
+        uv pip install --system "$(ls dist/*.whl)[dev]" --group spec
 
     - name: Run type checker (mypy)
       run: |

.readthedocs.yaml

@@ -8,6 +8,12 @@ build:
   jobs:
     pre_system_dependencies:
     - git fetch --unshallow # Needed to get version tags
+    post_install:
+    # The spec API is documented via autodoc, which imports linopy.spec and so
+    # needs math-spec. It lives in the `spec` dependency group, not the docs
+    # extra; --group needs pip >= 25.1, hence the upgrade.
+    - python -m pip install --upgrade pip
+    - python -m pip install --group spec
 python:
   install:
     - method: pip

benchmarks/models/__init__.py

@@ -21,5 +21,6 @@
     qp,
     sos,
     sparse_network,
+    spec_pypsa,
     storage,
 )

benchmarks/models/spec_pypsa.py

@@ -0,0 +1,55 @@
+"""
+Model built from math-spec's ``pypsa.yaml`` example (requires math-spec).
+
+The subject is :meth:`linopy.Model.from_spec`: lowering a spec of PyPSA's full
+statement, binding synthetic data to it and building every variable and
+constraint it declares. The example lives outside the wheel, so its directory
+comes from ``MATH_SPEC_EXAMPLES`` and the case skips without it. A sweep
+value is the number of labels per dimension; 40 of them is about 20k
+variables.
+"""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+from benchmarks.registry import BUILD, FROM_NETCDF, TO_NETCDF, BenchSpec, register
+
+if TYPE_CHECKING:
+    import linopy
+
+SIZES = (5, 40)
+
+EXAMPLES = os.environ.get("MATH_SPEC_EXAMPLES")
+EXAMPLE = Path(EXAMPLES, "pypsa.yaml") if EXAMPLES else None
+
+
+def build_spec_pypsa(n: int) -> linopy.Model:
+    """Lower ``pypsa.yaml`` and build it with ``n`` labels per dimension."""
+    import pytest
+
+    if EXAMPLE is None or not EXAMPLE.exists():
+        pytest.skip("set MATH_SPEC_EXAMPLES to a math-spec examples directory")
+    import math_spec
+
+    import linopy
+    from linopy.spec.testing import synthetic_sources
+
+    path = str(EXAMPLE)
+    sources = synthetic_sources(math_spec.to_program(path), n)
+    with linopy.options as options:
+        options["semantics"] = "v1"
+        return linopy.Model.from_spec(path, sources)
+
+
+SPEC = register(
+    BenchSpec(
+        name="spec_pypsa",
+        build=build_spec_pypsa,
+        sweep=SIZES,
+        phases=frozenset({BUILD, TO_NETCDF, FROM_NETCDF}),
+        requires=("math_spec",),
+    )
+)

conftest.py

@@ -0,0 +1,9 @@
+"""Root pytest configuration for ``--doctest-modules`` collection of ``linopy/``."""
+
+from __future__ import annotations
+
+from importlib.util import find_spec
+
+collect_ignore: list[str] = []
+if find_spec("math_spec") is None:
+    collect_ignore.append("linopy/spec")

doc/api.rst

@@ -115,6 +115,26 @@ IO
    model.Model.to_netcdf
    io.read_netcdf
 
+Building from specs
+-------------------
+
+Build a model from a `math-spec
+<https://github.com/energy-models/math-spec>`__ YAML program attached to
+data. Requires the ``spec`` dependency group.
+
+.. autosummary::
+   :toctree: generated/
+
+   model.Model.add_spec
+   model.Model.from_spec
+   model.Model.spec
+   spec.ModelSpec
+   spec.NamedExpressions
+   spec.NamedExpression
+   spec.attach
+   spec.Attached
+   spec.SpecDataError
+
 
 Variable
 ========

doc/building-models-from-specs.nblink

@@ -0,0 +1,3 @@
+{
+    "path": "../examples/building-models-from-specs.ipynb"
+}

doc/contributing.rst

@@ -45,6 +45,9 @@ To run the test suite:
     # Install development dependencies
     uv sync --extra dev --extra solvers
 
+    # Also run the math-spec binder tests (needs Python >= 3.12)
+    uv sync --extra dev --extra solvers --group spec
+
     # Run all tests
     pytest
 

doc/index.rst

@@ -116,6 +116,7 @@ This package is published under MIT license.
    coordinate-alignment
    migrating-to-v1
    manipulating-models
+   building-models-from-specs
 
 .. toctree::
    :hidden:

doc/release_notes.rst

@@ -21,6 +21,15 @@ Upcoming Version
 * Every operation whose result changes under v1 emits a ``LinopySemanticsWarning`` under legacy, naming the fix — so a model can be migrated incrementally before opting in. The full rules are specified in :doc:`the arithmetic convention <design/convention>`.
 
 
+*Build a model from a math-spec program*
+
+* ``Model.from_spec`` / ``model.add_spec`` build a model from a `math-spec <https://github.com/energy-models/math-spec>`__ YAML program attached to data, and ``model.spec`` (a ``linopy.spec.ModelSpec``) reads it back. Requires the ``spec`` dependency group (``uv sync --group spec`` / ``uv pip install --group spec``, Python >= 3.12) and v1 semantics. Data is attached onto the spec's dimensions and parameters with ``linopy.spec.attach``, raising a ``linopy.spec.SpecDataError`` on mismatched or missing data; ``linopy.spec.Attached`` carries the attached result. The parameters the spec retains live in ``model.spec.parameters``, its own dataset; ``model.parameters`` stays the caller's and a build never writes to it. ``retain`` decides what a netcdf file holds, not what a session can read: a parameter it dropped is resolved from the sources the model was built with, and only a model read back from a file can run out of data. The spec API emits an :class:`linopy.EvolvingAPIWarning` once per session while it stabilises. See :doc:`building-models-from-specs` for a worked example.
+
+* ``model.spec.expressions`` (a ``linopy.spec.NamedExpressions`` mapping) returns a ``linopy.spec.NamedExpression`` for each declared name, with three views: ``.node`` (the lowered formula), ``.expression`` (the unsolved linopy expression — a ``LinearExpression``, bare ``Variable``, array or scalar) and ``.solution`` (the expression folded over the solved model). ``model.spec.evaluate(name, sources)`` returns the same object with its parameters attached afresh.
+
+* ``model.spec.to_latex`` / ``.to_markdown`` / ``.to_typst`` typeset the whole model, and ``model.spec.declaration(name)`` returns a ``linopy.spec.Declaration`` whose same three methods typeset one named expression, constraint or variable as a single line (math only, no document); a ``NamedExpression`` carries those methods too. A ``ModelSpec``, a ``Declaration`` and a ``NamedExpression`` all render as Markdown in a notebook.
+
+
 *Numerical scaling*
 
 * Variables, constraints and the objective accept a ``scaling`` factor that rewrites the problem into better-behaved units for the solver, without changing the answer. Variable scaling is column-like, constraint and objective scaling are row-like, and primal values, duals and the objective are transformed back to the original units after solving. See the :doc:`numerical-scaling` tutorial and the *Numerical scaling* section of the :doc:`user-guide`.
@@ -31,6 +40,14 @@ Upcoming Version
 
 *Other*
 
+* ``Model.add_spec`` resolves the parameters it retains before it builds. A ``retain="all"`` build that could not read a parameter no declaration uses raised after the variables and constraints were already added, leaving a model that the "builds into an empty model" guard then refused to build into again.
+
+* ``repr(model.spec)`` no longer raises ``KeyError`` for a dimension the spec declares but nothing reaches, which needs no source and so has no coordinates; it is shown as ``unreached``.
+
+* A grouped sum through a lookup that maps no member at all now holds the empty sum, ``0``, on every declared group, as its documented rule says. It raised xarray's ``ValueError: <dim> must not be empty`` instead.
+
+* ``read_netcdf`` no longer rewrites the coordinates of a container that merely shares a dimension's *name* with a spec-built model's master coordinates. A hand-added variable on its own labels kept them; before, it was silently relabelled onto the master ones, or the read failed outright when the two lengths differed.
+
 * ``add_piecewise_formulation`` gained a ``mask`` parameter declaring which breakpoint slots hold a real breakpoint. It is needed for **ragged** curves — entities with different numbers of breakpoints — which are stored densely with the surplus slots left absent. Under v1 that absence must be declared (``mask=x_pts.notnull()``) rather than read off the NaN padding. (https://github.com/PyPSA/linopy/issues/884)
 
 *Internal*

doc/user-guide.rst

@@ -83,6 +83,20 @@ bound, swap a constraint, or copy it for what-if analysis.
   variables.
 
 
+Building a model from a spec
+-----------------------------
+
+Instead of calling ``add_variables`` / ``add_constraints`` directly,
+you can declare a model as a `math-spec
+<https://github.com/energy-models/math-spec>`__ YAML program attached to
+data, and let linopy build it.
+
+- :doc:`building-models-from-specs` — ``Model.from_spec`` and
+  ``model.add_spec``, attaching data to a spec, and reading named
+  expressions back through ``model.spec`` after solving. Requires the
+  ``spec`` dependency group and v1 semantics.
+
+
 Where to go next
 ----------------