Give the spec its own parameters and one read path

doc/release_notes.rst

@@ -23,7 +23,7 @@ Upcoming Version
 
 *Build a model from a math-spec program*
 
-* ``Model.from_spec`` / ``model.add_spec`` build a model from a `math-spec <https://github.com/energy-models/math-spec>`__ YAML program attached to data, and ``model.spec`` (a ``linopy.spec.ModelSpec``) reads it back. Requires the ``spec`` dependency group (``uv sync --group spec`` / ``uv pip install --group spec``, Python >= 3.12) and v1 semantics. Data is attached onto the spec's dimensions and parameters with ``linopy.spec.attach``, raising a ``linopy.spec.SpecDataError`` on mismatched or missing data; ``linopy.spec.Attached`` carries the attached result. The spec API emits an :class:`linopy.EvolvingAPIWarning` once per session while it stabilises. See :doc:`building-models-from-specs` for a worked example.
+* ``Model.from_spec`` / ``model.add_spec`` build a model from a `math-spec <https://github.com/energy-models/math-spec>`__ YAML program attached to data, and ``model.spec`` (a ``linopy.spec.ModelSpec``) reads it back. Requires the ``spec`` dependency group (``uv sync --group spec`` / ``uv pip install --group spec``, Python >= 3.12) and v1 semantics. Data is attached onto the spec's dimensions and parameters with ``linopy.spec.attach``, raising a ``linopy.spec.SpecDataError`` on mismatched or missing data; ``linopy.spec.Attached`` carries the attached result. The parameters the spec retains live in ``model.spec.parameters``, its own dataset; ``model.parameters`` stays the caller's and a build never writes to it. ``retain`` decides what a netcdf file holds, not what a session can read: a parameter it dropped is resolved from the sources the model was built with, and only a model read back from a file can run out of data. The spec API emits an :class:`linopy.EvolvingAPIWarning` once per session while it stabilises. See :doc:`building-models-from-specs` for a worked example.
 
 * ``model.spec.expressions`` (a ``linopy.spec.NamedExpressions`` mapping) returns a ``linopy.spec.NamedExpression`` for each declared name, with three views: ``.node`` (the lowered formula), ``.expression`` (the unsolved linopy expression — a ``LinearExpression``, bare ``Variable``, array or scalar) and ``.solution`` (the expression folded over the solved model). ``model.spec.evaluate(name, sources)`` returns the same object with its parameters attached afresh.
 

examples/building-models-from-specs.ipynb

@@ -343,7 +343,14 @@
    "id": "18",
    "metadata": {},
    "outputs": [],
-   "source": "print(\"spend.to_latex():        \", spend.to_latex())\nprint(\"power_balance.to_latex():\", m.spec.declaration(\"power_balance\").to_latex())\nprint(\"p.to_latex():            \", m.spec.declaration(\"p\").to_latex())\n\n# each renders as its own formula in a notebook:\nMarkdown(f\"$$\\n{m.spec.declaration('power_balance').to_markdown()}\\n$$\")"
+   "source": [
+    "print(\"spend.to_latex():        \", spend.to_latex())\n",
+    "print(\"power_balance.to_latex():\", m.spec.declaration(\"power_balance\").to_latex())\n",
+    "print(\"p.to_latex():            \", m.spec.declaration(\"p\").to_latex())\n",
+    "\n",
+    "# each renders as its own formula in a notebook:\n",
+    "Markdown(f\"$$\\n{m.spec.declaration('power_balance').to_markdown()}\\n$$\")"
+   ]
   },
   {
    "cell_type": "markdown",
@@ -377,16 +384,25 @@
     "## 5. `retain`: what data stays on the model\n",
     "\n",
     "Folding needs the parameters an expression reads. `retain` controls which\n",
-    "parameters linopy keeps in `model.parameters` after building:\n",
+    "parameters linopy keeps in `model.spec.parameters` after building.\n",
+    "That is the spec's own dataset — `model.parameters` stays yours, and a\n",
+    "build never writes to it:\n",
     "\n",
-    "| `retain`   | keeps in `model.parameters`                     |\n",
+    "| `retain`   | keeps in `model.spec.parameters`                |\n",
     "|------------|-------------------------------------------------|\n",
     "| `\"report\"` | only parameters the named expressions read (default) |\n",
     "| `\"all\"`    | every parameter                                 |\n",
     "| `\"none\"`   | nothing                                         |\n",
     "\n",
     "`spend` reads `cost`, `usage` reads `p_max`, neither reads `load` — so\n",
-    "`\"report\"` keeps `cost` and `p_max` but drops `load`."
+    "`\"report\"` keeps `cost` and `p_max` but drops `load`.\n",
+    "\n",
+    "Dropping is about *storage*, not about what you can read. A parameter\n",
+    "`retain` left out is resolved from the `sources` you built with, which the\n",
+    "model keeps hold of — so every `retain` folds the same in this session.\n",
+    "It is writing the model to netCDF that leaves the sources behind: read that\n",
+    "file back and only what `retain` kept is still there, with\n",
+    "`m.spec.evaluate(name, sources)` as the way in for the rest."
    ]
   },
   {
@@ -398,7 +414,9 @@
    "source": [
     "for retain in [\"report\", \"all\", \"none\"]:\n",
     "    mm = Model.from_spec(DISPATCH, dispatch_data, retain=retain)\n",
-    "    print(f\"retain={retain!r:9} -> parameters kept: {sorted(mm.parameters.data_vars)}\")"
+    "    print(\n",
+    "        f\"retain={retain!r:9} -> parameters kept: {sorted(mm.spec.parameters.data_vars)}\"\n",
+    "    )"
    ]
   },
   {

linopy/io.py

@@ -1160,8 +1160,9 @@ def to_netcdf(m: Model, *args: Any, **kwargs: Any) -> None:
     :func:`linopy.io.read_netcdf`. The insertion order of each container
     is stored as a JSON list in the ``_linopy_<kind>_order`` attribute.
 
-    A model built with :meth:`Model.add_spec` also persists its spec: the
-    YAML text, the master coordinates and the lookups. ``read_netcdf``
+    A model built with :meth:`Model.add_spec` also persists its spec under a
+    ``spec-`` prefix of its own: the YAML text, the master coordinates and the
+    parameters the spec retained, apart from ``m.parameters``. ``read_netcdf``
     lowers the program from the text again, so reading such a file needs
     the ``math-spec`` package; a file without a spec does not.
 
@@ -1209,14 +1210,12 @@ def to_netcdf(m: Model, *args: Any, **kwargs: Any) -> None:
     if m.objective.value is not None:
         objective = objective.assign_attrs(value=m.objective.value)
     obj = [with_prefix(objective, "objective")]
-    parameters = m.parameters
     specs: list[xr.Dataset] = []
     if m._spec is not None:
         from linopy.spec.netcdf import encode
 
-        parameters, spec_ds = encode(m._spec)
-        specs = [spec_ds]
-    params = [with_prefix(record_dtypes(parameters), "parameters")]
+        specs = [encode(m._spec)]
+    params = [with_prefix(record_dtypes(m.parameters), "parameters")]
 
     scalars = {k: getattr(m, k) for k in m.scalar_attrs}
     ds = xr.merge(
@@ -1499,7 +1498,7 @@ def _copy_con_data(con: ConstraintBase) -> xr.Dataset:
 
     new_model._parameters = m._parameters.copy(deep=deep)
     if m._spec is not None:
-        new_model._spec = m._spec._reattach(new_model)
+        new_model._spec = m._spec._reattach(new_model, deep=deep)
     new_model._blocks = m._blocks.copy(deep=deep) if m._blocks is not None else None
 
     for attr in m.scalar_attrs:

linopy/model.py

@@ -482,8 +482,11 @@ def add_spec(
             Data keyed by declared name: dimension labels, parameters and
             lookups. Read by key on demand and never iterated.
         retain : {"report", "all", "none"}
-            Which parameters to keep in ``model.parameters``: those the named
-            expressions read, all of them, or none.
+            Which parameters to keep in ``model.spec.parameters``: those the
+            named expressions read, all of them, or none. ``model.parameters``
+            stays the caller's and is never written to. This decides what a
+            netcdf file holds, not what this session can read: ``model.spec``
+            falls back to ``sources`` for a parameter it did not keep.
 
         Returns
         -------

linopy/spec/accessor.py

@@ -1,10 +1,18 @@
 """
 ``model.spec``: the program a model was built from, and its named expressions as data.
 
-The model owns the data. The spec text, the retained parameters, the lookups
-and the master coordinates all sit on the model, so this accessor holds
-nothing a round trip through a file could lose: it re-lowers the text and
-reads ``model.parameters``.
+The spec owns its data. The spec text, the retained parameters, the lookups
+and the master coordinates sit on the accessor rather than in
+``model.parameters``, which stays the caller's: a spec never overwrites what
+was put there, and nothing reading a spec-built model has to guess which of
+its parameters the spec owns. All of it round trips through a file, written
+under the ``spec-`` prefix.
+
+A parameter is resolved the same way however much of it was retained: from
+the retained dataset, else from the sources the model was built with, which
+the accessor keeps for as long as the model lives. So ``retain`` decides what
+a *file* holds, not what a session can read, and it is only after a round trip
+that a parameter can be out of reach.
 """
 
 from __future__ import annotations
@@ -77,13 +85,18 @@ def attach(
     text, program = _source(spec)
     attached: Attached = attach_data(program, sources, retain=retain)
     build(model, attached)
-    model.parameters = attached.retained().assign_coords(dict(attached.coords))
-    return ModelSpec(model, program, text)
+    parameters = attached.retained().assign_coords(dict(attached.coords))
+    return ModelSpec(model, program, text, parameters, attached)
 
 
-def restore(model: Model, text: str) -> ModelSpec:
-    """The accessor for *model*, with the program lowered afresh from *text*."""
-    return ModelSpec(model, to_program(yaml.safe_load(text)), text)
+def restore(model: Model, text: str, parameters: xr.Dataset) -> ModelSpec:
+    """
+    The accessor for *model*, with the program lowered afresh from *text*.
+
+    Read from a file, so the sources the model was built with are gone and
+    only what ``retain`` kept can be read back.
+    """
+    return ModelSpec(model, to_program(yaml.safe_load(text)), text, parameters, None)
 
 
 def _source(spec: SpecLike) -> tuple[str, ms.Program]:
@@ -123,10 +136,19 @@ class ModelSpec:
         The spec as YAML, verbatim where a file or text was passed.
     """
 
-    def __init__(self, model: Model, program: ms.Program, text: str) -> None:
+    def __init__(
+        self,
+        model: Model,
+        program: ms.Program,
+        text: str,
+        parameters: xr.Dataset,
+        attached: Attached | None,
+    ) -> None:
         self._model = model
         self.program = program
         self.text = text
+        self._parameters = parameters
+        self._attached = attached
 
     def __repr__(self) -> str:
         p = self.program
@@ -143,14 +165,20 @@ def __repr__(self) -> str:
         rows.append(_row("Expressions", list(p.named_expressions)))
         return "\n".join(rows)
 
-    def _reattach(self, model: Model) -> ModelSpec:
-        """The same spec, read off *model*."""
-        return ModelSpec(model, self.program, self.text)
+    def _reattach(self, model: Model, deep: bool = True) -> ModelSpec:
+        """The same spec, read off *model*, holding its own copy of the parameters."""
+        return ModelSpec(
+            model,
+            self.program,
+            self.text,
+            self._parameters.copy(deep=deep),
+            self._attached,
+        )
 
     @property
     def parameters(self) -> xr.Dataset:
-        """The parameters and lookups retained on the model, on the master coordinates."""
-        return self._model.parameters
+        """The parameters and lookups the spec retained, on the master coordinates."""
+        return self._parameters
 
     @property
     def description(self) -> str:
@@ -222,8 +250,9 @@ def evaluate(
         """
         The named expression *name*, with its parameters attached afresh from *sources*.
 
-        For a model built with ``retain="none"``, or an expression reading a
-        parameter ``retain="report"`` did not keep. *sources* is read the way
+        For reading the spec against other data than the model was built with,
+        and for a model read from a file, whose own sources are gone.
+        ``model.spec.expressions`` needs neither. *sources* is read the way
         ``add_spec`` read it, and must describe the coordinates the model was
         built on.
 
@@ -244,14 +273,18 @@ def evaluate(
                 )
         return NamedExpression(self, name, self._context(attached.parameter))
 
-    def _retained(self, name: str) -> xr.DataArray:
-        if name not in self.parameters:
-            raise SpecDataError(
-                f"parameter '{name}' is not retained on the model: retain='report' keeps only what "
-                f"the named expressions read, and retain='none' keeps nothing. Build with "
-                f"retain='all', or read the expression with evaluate(name, sources)."
-            )
-        return self.parameters[name]
+    def _resolve(self, name: str) -> xr.DataArray:
+        """The parameter *name*: retained if it was kept, else read from the sources again."""
+        if name in self.parameters:
+            return self.parameters[name]
+        if self._attached is not None:
+            return self._attached.parameter(name)
+        raise SpecDataError(
+            f"parameter '{name}' was not retained and this model no longer holds the sources "
+            f"it was built with, which is what a model read from a file looks like. Build with "
+            f"retain='all' before writing it out, or read the expression with "
+            f"evaluate(name, sources)."
+        )
 
     def _context(self, resolve: Resolve) -> Context:
         return Context(
@@ -277,7 +310,7 @@ def __getitem__(self, name: str) -> NamedExpression:
                 + did_you_mean(name, self._spec.program.named_expressions)
             )
         return NamedExpression(
-            self._spec, name, self._spec._context(self._spec._retained)
+            self._spec, name, self._spec._context(self._spec._resolve)
         )
 
     def __iter__(self) -> Iterator[str]:
@@ -326,8 +359,9 @@ class NamedExpression(Declaration):
     One named expression, in three views: its math, its linopy fold and its solution.
 
     The object pins the data sources it was made with for its lifetime, so the
-    three views agree. ``expressions[name]`` reads the retained parameters and
-    the solution the model holds; ``evaluate(name, sources)`` attaches fresh data.
+    three views agree. ``expressions[name]`` reads the model's own data --
+    what ``retain`` kept, and the sources behind it for the rest;
+    ``evaluate(name, sources)`` attaches fresh data instead.
 
     Attributes
     ----------
@@ -371,7 +405,8 @@ def solution(self) -> xr.DataArray:
         RuntimeError
             The model reads a variable but holds no solution yet.
         SpecDataError
-            A parameter the body reads was not retained.
+            A parameter the body reads was neither retained nor
+            still reachable through the model's sources.
         """
         return fold(self._name, self._ctx)