feat(spec): xarray data binder for math-spec programs by FabianHofmann · Pull Request #921 · PyPSA/linopy

FabianHofmann · 2026-09-03T11:48:31Z

Step 2 of the plan in #919: the data layer that Model.from_spec / Model.add_spec will build on. No public model API yet; linopy.spec is importable only when math-spec is installed and import linopy never loads it.

Note

The following content was generated by AI.

Changes proposed in this Pull Request

linopy/spec/binder.py turns a lowered math_spec.Program plus user data into what a builder needs: master coordinates, lookup arrays and on-demand parameter arrays.

bind(program, sources, retain=...) returns a Bound with coords, lookups, parameter(name) and retained(). Parameters are resolved from sources on demand and never cached; retain selects what retained() hands back for persistence: "report" (default, the parameter closure of the named expressions), "all" or "none".
sources is any Mapping pulled by key and never iterated, so a lazy view over PyPSA's n.c.<component>.da accessor is a valid source. A single xr.Dataset is accepted too.
math-spec's three binding rules are enforced: members only from the dimension's own source, source order never sorted, parameters never read for labels. Unknown labels, duplicate rows, wrong rank and wrong dtype raise SpecDataError. A missing row is not an error: it becomes NaN (bool: False) and the builder decides later.
No copies of aligned data: indexes are compared first and reindex runs only on a real mismatch. An aligned array keeps its buffer, asserted with np.shares_memory. A matching index with a different dtype is re-stamped onto the master dtype without copying.
Accepted parameter shapes: scalar, xr.DataArray, pd.Series, tidy or wide pd.DataFrame, dict. Lookups: Series, dict or DataArray over the dimension.
The malformed-data cases of lpspec's test_data_parity.py are ported and give the same verdicts.
math-spec is not on PyPI, so there is no extra yet. Tests skip without it; run them with uv run --with <math-spec checkout> pytest test/test_spec_binder.py.

Left for the model integration step: object-dtype string lookups through netcdf, float promotion of partly mapped lookups into integer dimensions, and derived piecewise parameters.

Verification

uv run --with <math-spec> pytest test/test_spec_binder.py -q
240 passed
coverage linopy/spec/binder.py 100%
uv run --with <math-spec> mypy linopy/spec
Success: no issues found in 3 source files
uv run pytest test/test_spec_binder.py -q
1 skipped

Checklist

AI-generated content is marked (see AGENTS.md).
Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

Turns a lowered math-spec Program plus user data into master coordinates, padded lookups and on-demand parameter arrays under the three binding rules. Missing rows stay NaN for the builder. Sources are pulled by key, never iterated, and aligned arrays keep their buffer.

…s under the right dimension

Validate retain, check a scalar's dtype before casting, bind empty sources as all-NaN, check label-space lookup dtypes, report unknown labels in source order on every path, re-stamp coordinates onto the master dtype without copying, and pin the remaining binder rules.

github-actions · 2026-09-03T11:51:56Z

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                 time (s)           peak (KiB) 
 name                mean   │             mean 
───────────────────────────────────────────────
 (legacy)   0.0456 (1.13)   │      15.03 (1.0) 
 (v1)       0.04051 (1.0)   │   271.00 (18.03) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04919 (1.12)   │   12.04 (1.00) 
 (v1)        0.04405 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.01896 (1.08)   │   15.20 (1.0) 
 (v1)        0.01753 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03098 (1.09)   │   44.93 (1.0) 
 (v1)        0.02841 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02104 (1.08)   │   11.51 (1.0) 
 (v1)        0.01951 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04872 (1.05)   │   24.34 (1.06) 
 (v1)        0.04628 (1.0)   │    23.05 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05482 (1.03)   │   16.12 (1.00) 
 (v1)        0.05328 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.01172 (1.05)   │   752.18 (1.10) 
 (v1)        0.01112 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.01141 (1.05)   │   3.12 (1.33) 
 (v1)        0.01082 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03714 (1.26)   │   126.16 (1.44) 
 (v1)        0.02954 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03615 (1.26)   │   126.16 (1.44) 
 (v1)        0.02874 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03733 (1.28)   │   126.16 (1.44) 
 (v1)        0.02919 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02654 (1.01)   │    715.14 (1.0) 
 (v1)        0.02628 (1.0)   │   787.74 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02575 (1.05)   │   4.55 (1.27) 
 (v1)        0.02449 (1.0)   │    3.59 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.1921 (1.07)   │   704.12 (1.09) 
 (v1)        0.1801 (1.0)   │    647.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.1896 (1.03)   │   18.34 (1.0) 
 (v1)        0.1845 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                time (s)       peak (MiB) 
 name               mean   │         mean 
──────────────────────────────────────────
 (legacy)   0.193 (1.04)   │   9.54 (1.0) 
 (v1)       0.1853 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03661 (1.08)   │   3.77 (1.12) 
 (v1)        0.03401 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03652 (1.06)   │   216.59 (1.10) 
 (v1)        0.03432 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.01893 (1.09)   │   938.49 (1.0) 
 (v1)        0.01738 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02038 (1.10)   │   9.66 (1.0) 
 (v1)        0.01861 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01951 (1.11)   │   5.32 (1.0) 
 (v1)        0.01762 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01057 (1.04)   │   1.41 (1.0) 
 (v1)        0.01012 (1.0)   │   1.41 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.0104 (1.02)   │   1.41 (1.0) 
 (v1)       0.01016 (1.0)   │   1.41 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01044 (1.02)   │   1.41 (1.0) 
 (v1)        0.01018 (1.0)   │   1.41 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.09445 (1.08)   │   946.85 (1.06) 
 (v1)        0.08707 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.08925 (1.03)   │   12.01 (1.0) 
 (v1)        0.08661 (1.0)   │   12.01 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02362 (1.04)   │   147.70 (1.06) 
 (v1)        0.02276 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02445 (1.11)   │   2.60 (1.09) 
 (v1)        0.02205 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.01919 (1.06)   │   696.75 (1.03) 
 (v1)        0.01804 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05377 (1.01)   │   137.97 (1.0) 
 (v1)         0.0535 (1.0)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03534 (1.12)   │   69.22 (1.0) 
 (v1)        0.03166 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02266 (1.14)   │   402.33 (1.00) 
 (v1)        0.01986 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02274 (1.14)   │   3.19 (1.19) 
 (v1)        0.01993 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.02434 (1.06)   │   29.00 (1.54) 
 (v1)        0.02287 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03133 (1.13)   │   37.95 (1.43) 
 (v1)        0.02769 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)    0.04729 (1.0)   │    410.93 (1.0) 
 (v1)       0.04746 (1.00)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.04975 (1.0)   │     9.94 (1.0) 
 (v1)       0.05089 (1.02)   │   10.22 (1.03)

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

codspeed-hq · 2026-09-03T11:52:05Z

Merging this PR will degrade performance by 8.46%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
❌ 2 regressed benchmarks
✅ 170 untouched benchmarks
🆕 6 new benchmarks
⏩ 175 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
❌	`test_to_lp[expression_arithmetic-n=10]`	718.4 KB	1,338.9 KB	-46.34%
❌	`test_to_lp[qp-n=1000]`	2 MB	2.6 MB	-22.48%
⚡	`test_to_lp[storage-n=250]`	37 MB	29.4 MB	+25.68%
⚡	`test_to_lp[nodal_balance-severity=50]`	3.7 MB	3.3 MB	+11.62%
⚡	`test_to_lp[expression_arithmetic-n=250]`	45.3 MB	41.1 MB	+10.15%
🆕	`test_build[nodal_balance_sparse-severity=0]`	N/A	1.5 MB	N/A
🆕	`test_build[nodal_balance_sparse-severity=100]`	N/A	1.5 MB	N/A
🆕	`test_build[nodal_balance_sparse-severity=50]`	N/A	1.5 MB	N/A
🆕	`test_to_lp[nodal_balance_sparse-severity=0]`	N/A	3.3 MB	N/A
🆕	`test_to_lp[nodal_balance_sparse-severity=100]`	N/A	3.7 MB	N/A
🆕	`test_to_lp[nodal_balance_sparse-severity=50]`	N/A	3.1 MB	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing spec-binder (494becf) with master (6b133d5)²}

175 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
No successful run was found on master (b71e9a9) during the generation of this report, so 6b133d5 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

…ype the test spec dicts

… group math-spec is not on PyPI and needs Python >= 3.12. A PEP 735 dependency group keeps the git pin out of the wheel metadata; the 3.12 and 3.13 test jobs install it so the binder tests and their coverage run in CI.

FabianHofmann · 2026-09-07T07:38:38Z

include diff in #922

FabianHofmann added 3 commits September 3, 2026 13:48

refactor(spec): compact binder and tests; report duplicate coordinate…

a4e6dc3

…s under the right dimension

ci(spec): skip doctest collection of linopy/spec without math-spec; t…

5610909

…ype the test spec dicts

This was referenced Sep 3, 2026

Build and change models via math-spec #922

Draft

Round trip spec-built models through netcdf and benchmark from_spec #923

Merged

FabianHofmann added 3 commits September 3, 2026 20:34

doc: mention the spec dependency group in the contributing guide

27c9ab5

Merge branch 'master' into spec-binder

494becf

FabianHofmann closed this Sep 7, 2026

Navigation Menu

Uh oh!

feat(spec): xarray data binder for math-spec programs - #921

feat(spec): xarray data binder for math-spec programs#921
FabianHofmann wants to merge 7 commits into
masterfrom
spec-binder

FabianHofmann commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Sep 3, 2026 •

edited

Loading

Uh oh!

FabianHofmann commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

FabianHofmann commented Sep 3, 2026

Changes proposed in this Pull Request

Checklist

Uh oh!

github-actions Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build cost — v1 vs legacy

Uh oh!

codspeed-hq Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 8.46%

Performance Changes

Footnotes

Uh oh!

FabianHofmann commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Sep 3, 2026 •

edited

Loading

codspeed-hq Bot commented Sep 3, 2026 •

edited

Loading