Skip to content

Give the spec its own parameters and one read path - #949

Merged
FBumann merged 2 commits into
spec-builderfrom
spec-parameter-ownership
Sep 8, 2026
Merged

Give the spec its own parameters and one read path#949
FBumann merged 2 commits into
spec-builderfrom
spec-parameter-ownership

Conversation

@FBumann

@FBumann FBumann commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Note

The following content was generated by AI.

Stacked on #946. Two review findings from #922, one commit each.

1. The spec owns its parameters, not model.parameters

A build wrote the retained parameters and lookups straight into model.parameters,
by assignment. Two things followed from that:

  • it clobbered whatever the caller had put there, and the emptiness guard on
    add_spec only checks variables and constraints, so nothing caught it;
  • nothing could say which of the model's parameters the spec owned.
    netcdf._coded compensated by scanning every parameter on the model for an
    object dtype, so a caller's own labelled array was factorized into the spec's
    sub-dataset and came back as spec data.

ModelSpec now holds its own dataset. model.parameters is untouched by a build
and by a read, the two can carry the same name over different dimensions, and
_coded sees only what the spec put there. The spec's parameters are written
under the spec- prefix that already carried its coordinates: plainly as
param__<name> over bare dimensions, or as codes where a lookup or an object
array needs them, and put back on the master coordinates on read.
assert_model_equal compares them, and Model.copy hands the copy its own
buffers.

model.spec.parameters was already the documented read path, so nothing that
read the spec through the accessor changes.

2. One read path for a named expression

Reading model.spec.expressions[name] went through the retained parameters and
raised if retain had dropped one; reading the same expression through
evaluate(name, sources) went through the data instead. Two ways to read one
thing, forked by a flag chosen at build time — and retain="none" was a trap in
a live session, where the sources were still right there.

The two paths differed by exactly one argument: Resolve, the callable a
Context resolves a parameter with. So ModelSpec keeps the Attached a build
made and resolves a parameter from the retained dataset first, from the sources
behind it otherwise.

  • retain now decides what a netcdf file holds, not what a session can read.
  • The refusal is left for the one case that cannot be served: a model read back
    from a file, whose sources are gone.
  • evaluate(name, sources) keeps the job that is really its own — reading the
    spec against other data.

Keeping the Attached stores nothing new: a build already built one and threw it
away after build(). It does mean the model holds the caller's sources for its
lifetime; sources is read by key and is often a lazy view, so that is usually
cheap, but it is a change in ownership worth knowing about.

Verification (041ab2d)
uv run --extra dev --extra solvers --group spec pytest test/test_spec_*.py test/test_io.py test/test_model.py test/test_repr.py -q
882 passed, 192 skipped

uv run --extra dev --extra solvers --group spec pytest test/ -q
46 failed, 9112 passed, 1054 skipped, 28 errors
# every failure is in test/remote/ (OETC cloud provider tests), unchanged from the
# base commit and unrelated to this branch

uv run --extra dev --group spec mypy linopy
Found 13 errors in 3 files (checked 49 source files)
# identical to the base commit: missing types-PyYAML and pre-existing
# linopy/expressions.py union-attr errors in this local environment

One unrelated hunk in examples/building-models-from-specs.ipynb: the repo's
ruff-format pre-commit hook normalised a cell that was stored in compact string
form into list form when the file was touched.

Checklist

  • AI-generated content is marked (see AGENTS.md).
  • Code changes are sufficiently documented.
  • Unit tests for new features were added.
  • A note for the release notes doc/release_notes.rst is included.

🤖 Generated with Claude Code

https://claude.ai/code/session_015ZfHfTFENUy6WxnFFri5Td

Base automatically changed from spec/power-children to spec-builder September 8, 2026 17:23
FBumann and others added 2 commits September 8, 2026 19:24
A build wrote the retained parameters and lookups straight into
`model.parameters`, by assignment. That clobbered whatever the caller had
put there, and left nothing able to say which of the model's parameters the
spec owned -- `netcdf._coded` compensated by scanning every parameter on the
model for an object dtype, so a caller's own labelled array was factorized
into the spec's sub-dataset and came back as spec data.

`ModelSpec` now holds its own dataset. `model.parameters` is untouched by a
build and by a read, the two can carry the same name over different
dimensions, and `_coded` sees only what the spec put there. The spec's
parameters are written under the `spec-` prefix that already carried its
coordinates: plainly as `param__<name>` over bare dimensions, or as codes
where a lookup or an object array needs them, and put back on the master
coordinates on read. `assert_model_equal` compares them, and `Model.copy`
hands the copy its own buffers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ZfHfTFENUy6WxnFFri5Td
Reading `model.spec.expressions[name]` went through the retained parameters
and raised if `retain` had dropped one; reading the same expression through
`evaluate(name, sources)` went through the data instead. Two ways to read one
thing, forked by a flag chosen at build time -- and `retain="none"` was a trap
in a live session, where the sources were still right there.

The two paths differed by one argument: `Resolve`, the callable a `Context`
resolves a parameter with. So `ModelSpec` keeps the `Attached` a build made
and resolves a parameter from the retained dataset first, from the sources
behind it otherwise. `retain` now decides what a netcdf file holds, not what
a session can read, and the refusal is left for the one case that cannot be
served: a model read back from a file, whose sources are gone. `evaluate`
keeps the job that is really its own, reading the spec against other data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ZfHfTFENUy6WxnFFri5Td
@FBumann
FBumann force-pushed the spec-parameter-ownership branch from 041ab2d to 4d4a508 Compare September 8, 2026 17:24
@FBumann
FBumann merged commit 48b267d into spec-builder Sep 8, 2026
7 of 8 checks passed
@FBumann
FBumann deleted the spec-parameter-ownership branch September 8, 2026 17:24
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy time — v1 / legacy
peak v1/legacy time v1/legacy
Full table (time + peak, mean)
benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08834 (1.08)   │   15.03 (1.00) 
 (v1)        0.08155 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09382 (1.08)   │   12.04 (1.00) 
 (v1)        0.08709 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03816 (1.08)   │   15.20 (1.0) 
 (v1)        0.03531 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05444 (1.10)   │   44.93 (1.0) 
 (v1)        0.04933 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04156 (1.08)   │   11.51 (1.0) 
 (v1)        0.03835 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09912 (1.07)   │   24.34 (1.06) 
 (v1)        0.09301 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1092 (1.06)   │   16.12 (1.00) 
 (v1)        0.1035 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02361 (1.06)   │   752.18 (1.10) 
 (v1)        0.02222 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                 time (s)        peak (KiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.0231 (1.06)   │   3.12 (1.33) 
 (v1)        0.0219 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                 time (s)          peak (MiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0636 (1.21)   │   126.16 (1.44) 
 (v1)       0.05276 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06198 (1.19)   │   126.16 (1.44) 
 (v1)        0.05228 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06274 (1.20)   │   126.16 (1.44) 
 (v1)        0.05207 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0546 (1.01)   │    715.12 (1.0) 
 (v1)       0.05394 (1.0)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05356 (1.10)   │   4.54 (1.27) 
 (v1)        0.04866 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.3737 (1.04)   │   704.12 (1.09) 
 (v1)        0.3585 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.3935 (1.05)   │   18.34 (1.0) 
 (v1)        0.3741 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.3916 (1.05)   │   9.54 (1.0) 
 (v1)         0.374 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.07491 (1.08)   │   3.77 (1.12) 
 (v1)        0.06933 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.07759 (1.11)   │   216.59 (1.10) 
 (v1)        0.06967 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03791 (1.08)   │   938.49 (1.0) 
 (v1)        0.03515 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03914 (1.07)   │   9.66 (1.0) 
 (v1)        0.03669 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03883 (1.09)   │   5.32 (1.0) 
 (v1)        0.03573 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02112 (1.01)   │   1.47 (1.0) 
 (v1)        0.02099 (1.0)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02098 (1.01)   │   1.47 (1.0) 
 (v1)        0.02083 (1.0)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)    0.02094 (1.0)   │   1.47 (1.0) 
 (v1)       0.02101 (1.00)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                time (s)          peak (KiB) 
 name               mean   │            mean 
─────────────────────────────────────────────
 (legacy)   0.186 (1.04)   │   946.85 (1.06) 
 (v1)        0.179 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1834 (1.05)   │   12.01 (1.00) 
 (v1)        0.1753 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0474 (1.06)   │   147.70 (1.06) 
 (v1)        0.0446 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04699 (1.06)   │   2.60 (1.09) 
 (v1)        0.04429 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03904 (1.08)   │   696.75 (1.03) 
 (v1)        0.03614 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08682 (1.04)   │   137.97 (1.0) 
 (v1)        0.08333 (1.0)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.06001 (1.06)   │   69.22 (1.0) 
 (v1)        0.05657 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04528 (1.10)   │   402.33 (1.00) 
 (v1)        0.04109 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04455 (1.09)   │   3.19 (1.19) 
 (v1)        0.04072 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04918 (1.07)   │   29.00 (1.54) 
 (v1)        0.04577 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05893 (1.11)   │   37.95 (1.43) 
 (v1)        0.05289 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)    0.09366 (1.0)   │    410.93 (1.0) 
 (v1)       0.09511 (1.02)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.09873 (1.0)   │     9.94 (1.0) 
 (v1)       0.1022 (1.04)   │   10.22 (1.03) 

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant