Build and change models via math-spec - #922
Draft
FabianHofmann wants to merge 39 commits into
Draft
Conversation
Turns a lowered math-spec Program plus user data into master coordinates, padded lookups and on-demand parameter arrays under the three binding rules. Missing rows stay NaN for the builder. Sources are pulled by key, never iterated, and aligned arrays keep their buffer.
…s under the right dimension
Validate retain, check a scalar's dtype before casting, bind empty sources as all-NaN, check label-space lookup dtypes, report unknown labels in source order on every path, re-stamp coordinates onto the master dtype without copying, and pin the remaining binder rules.
…ype the test spec dicts
Build cost — v1 vs legacyv1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).
Full table (time + peak, mean)📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run. Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped. |
4 tasks
FabianHofmann
changed the title
… group math-spec is not on PyPI and needs Python >= 3.12. A PEP 735 dependency group keeps the git pin out of the wheel metadata; the 3.12 and 3.13 test jobs install it so the binder tests and their coverage run in CI.
…ssions Port lpspec's linopy lane onto the binder: builder, where, operators, coverage and curves, wired to Bound and SpecDataError. Add Model.add_spec, Model.from_spec and the model.spec accessor with expressions and evaluate.
…en windows Coverage and the retain closure now descend into a Power's operands; evaluate() refuses sources labelled unlike the model; an all-null window width is a window of nothing; cases fold through the aligned combine.
Persist the spec text, the master coordinates and the lookups alongside the model, re-lowering the program from the text on read; math-spec is imported only for a file that carries a spec. Lookups and arrays of labels are stored as codes into a category table, so partial maps keep their holes and dtypes.
Write the in-memory dtype of every parameter and cast it back on read, and stamp the master coordinates onto every container, so no engine leaves a model disagreeing with itself. assert_model_equal now compares dataset dtypes, and synthetic_sources moves to linopy/spec/testing.py for both users.
FabianHofmann
force-pushed
the
spec-builder
branch
from
September 3, 2026 18:39
7fbfd88 to
e0a29c9
Compare
A missing parameter row was read as a silent zero when it stood as a coefficient, while a bound, constant side or divisor already refused it. Refuse it as a coefficient too, so every position behaves alike and a hole is never filled without the modeller saying so: mask the coordinate out with a where, or fill the value into the data.
A runnable, nbconvert-clean walkthrough of the spec feature: the dispatch program, binding data, folding named expressions, retain and evaluate, the uniform absence rule, lookups and grouped sums, temporal shift, and the netCDF round trip.
m.spec.expressions[name] returns a NamedExpression bundling .node (the lowered formula), .expression (the unsolved linopy expression) and .solution (the fold over the model's solution). evaluate() returns the same object. Add ModelSpec.to_latex/to_markdown/to_typst for whole-model typesetting, rendered as Markdown in a notebook.
building-models-from-specs.ipynb imports math_spec, which the docs CI environment does not install, so the notebook job failed on import. Skip it like the other special-setup notebooks.
5 tasks
FabianHofmann
changed the title
Merging this PR will regress 2 benchmarks
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | test_to_lp[merge_balance-severity=0] |
2.7 MB | 3.3 MB | -18.23% |
| ❌ | test_to_lp[nodal_balance-severity=50] |
3.3 MB | 3.7 MB | -10.82% |
| ⚡ | test_to_lp[sparse_network-n=10] |
1,338.9 KB | 733.8 KB | +82.47% |
| ⚡ | test_to_lp[storage-n=250] |
37 MB | 30.2 MB | +22.52% |
| ⚡ | test_to_lp[rolling-severity=0] |
3.1 MB | 2.8 MB | +12.37% |
| ⚡ | test_to_lp[rolling-severity=50] |
429.9 MB | 390.7 MB | +10.02% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing spec-builder (8b88ac7) with master (718c0c1)
Footnotes
-
181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
# Conflicts: # linopy/io.py
pandas 3 hands strings over as StringDtype, Arrow-backed when pyarrow is installed. xarray keeps the extension array, refuses it in positional indexing and reports no np.dtype, so the netcdf dtype round trip broke.
…rrow in the spec group
…repair moves to io parameters.py owns resolution and derivation, groups.py the axis partition, nodes.amounts_of the parameter-named amounts, Context.lookup the lookups. io records and restores parameter dtypes for every model and owns restamp_coords and the module-level prefix helpers spec/netcdf reuses.
…e walk evaluate.py holds the recursive evaluator, builder.py the declarations. check_coverage collects divisor, constant-side and coefficient obligations in one walk, so cases: masks are evaluated once per declaration. Public docstrings in numpy style so the API pages render.
…shared material in conftest
…in CI with the spec group
…rom_spec and bind warn_evolving_api moves to linopy.constants so piecewise and spec share the once-per-key dedup; the pytest filter silences the spec prefix.
autodoc imports linopy.spec, which needs math-spec. RTD installed only the docs extra, so the Sphinx build failed with ImportError. Add a post_install job installing the spec dependency group (pip >= 25.1).
lpspec reserves "bound" for a variable/constraint limit and calls the data operation "attach". Rename bind()->attach(), Bound->Attached and binder.py->attach.py so "bound" names one thing. Variable-limit names (_bound, check_bounds_cover) stay untouched.
RTD now installs the spec group, so the notebook can import math-spec and solve. Drop `nbsphinx.execute: never` so it runs at build time and its outputs render (nbstripout keeps them out of git).
named_expressions now maps each name to an ExpressionDeclaration that carries the body and whether the math reads it, so read .expression at the five sites that take a body: the evaluator, the builder's coverage check, the report closure and the accessor's node view.
math-spec a75 lets a reported expression hold a dual(constraint) node. Evaluate it as the solved constraint's dual over its own frame, and refuse it before a solve the way an unsolved variable is refused.
Add to_latex/to_markdown/to_typst on NamedExpression via math-spec's typeset_declaration, rendering one expression as a bare line, and fix its _repr_markdown_ to show only itself. Tests, release note and the notebook updated.
Extract a Declaration base carrying to_latex/to_markdown/to_typst (reused by NamedExpression) and add ModelSpec.declaration(name) to typeset a named expression, constraint or variable as one bare line. Tests, release note and the notebook updated.
Show dimensions, variables, constraints, objective and named expressions, one capped line each, instead of only the expression names.
Header says the model is built from a math-spec and prints the spec's description. Spec named expressions are listed with their static dims; spec-owned items are tagged [spec] once hand-added ones exist.
This was referenced Sep 8, 2026
FBumann
reviewed
Sep 8, 2026
| ) | ||
| return self._spec | ||
|
|
||
| def add_spec( |
Collaborator
There was a problem hiding this comment.
@FabianHofmann This method is doing the same as Model.from_spec(), but with a misleading name.
WHat this method is trying to do seems much harder than building a linopy Model from scratch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow the plan in #919; support math-spec for model definition and modification.
Note
The following content was generated by AI.
Changes proposed in this Pull Request
lpspec's linopy lane (fluxopt/lpspec#1518) ported into
linopy/spec/, typed, on top of the data attachment previously drafted in #921 (merged into this PR). math-spec is pinned to a git commit (alpha.75) through the PEP 735 dependency groupspec, Python >= 3.12;import linopynever imports math_spec, the model methods import it lazily.Building
Model.add_spec(spec, sources, retain="report")builds variables, SOS, constraints and objective into an empty v1 model;Model.from_spec(spec, sources, retain=..., **model_kwargs)is sugar over it.specis a path, YAML text, dict ormath_spec.Spec. Both raise under legacy semantics or on a non-empty model, and emitEvolvingAPIWarningonce per session while the API stabilises.linopy.spec.attach(program, sources, retain=...)turns a loweredProgramplus user data into anAttached: master coordinates, lookups and on-demand parameters.sourcesis anyMappingpulled by key and never iterated (a lazy view over PyPSA'sn.c.<component>.daworks) or a singlexr.Dataset. math-spec's binding rules are enforced: members only from the dimension's own source, source order never sorted, parameters never read for labels. Unknown labels, duplicate rows, wrong rank or dtype raiseSpecDataError. Aligned data is never copied.whereor fill the data. Divisor coverage is checked on named-expression bodies, unaligned operands raiseSpecDataError, a constraint with its term on the right is swapped with a flipped sense.attach,builder(declarations),evaluate(recursive evaluator, fold),coverage(one tree walk collecting divisor, constant-side and coefficient obligations),operators(sum_over,grouped_sum,at,shift,sum_back),where,curves(piecewise),terms,parameters(resolution and derivation),groups,context,nodes(node walk that also descends intoPower, which math-spec'schildren()skips; staticdims_of),netcdf,testing(synthetic_sources).Reading back
model.spec(ModelSpec) holds theProgramand the spec text and reads coordinates, lookups and retained parameters offmodel.parameters. Its repr summarises dimensions, variables, constraints, objective and named expressions.model.spec.expressions[name]is aNamedExpressionwith.node(lowered formula),.dims(static, no data bound),.expression(the unsolved linopy expression) and.solution(the fold over the solved model, substitutingVariable.solution, retained parameters and, since alpha.75, a constraint'sdual).model.spec.evaluate(name, sources)returns the same object with parameters attached afresh (forretain="none"), refusing sources labelled unlike the model.model.spec.to_latex/to_markdown/to_typstrender the whole model;model.spec.declaration(name)and aNamedExpressionrender one declaration as a bare line. All three render as Markdown in a notebook.Modelrepr says the model is built from a math-spec, prints the spec description, lists named expressions with their dims and tags spec-owned items[spec]once hand-added variables or constraints exist.Persistence
to_netcdfwrites the spec text as the top-level attribute_linopy_spec;read_netcdfre-lowers theProgramfrom it, and a file without a spec loads without math-spec installed.spec-sub-dataset carries master coordinates and lookups plus object-dtype parameters aspandas.factorizecodes and categories, so partial maps keep holes and dtypes on both netcdf engines. Every parameter records its in-memory dtype and is cast back on read; master coordinates are stamped onto every container.Model.copy()carries the spec;assert_model_equalcompares spec text and parameter datasets including dtypes.Docs, CI, benchmarks
examples/building-models-from-specs.ipynb, wired into the user guide and executed on Read the Docs and in the notebook CI job with thespecgroup installed. API pages forlinopy.spec, release notes added.specgroup; mypy runs with math-spec installed. Tests skip without it.benchmarks/models/spec_pypsa.py:from_specon math-spec'sexamples/pypsa.yamlplus the two netcdf phases; skips without math-spec orMATH_SPEC_EXAMPLES.Not ported / upstream
reads_off_the_solutiongrade predicate (linopy always folds) and the polars loader (replaced byattach). Variable divisors and degree-2 named expressions are refused by math-spec at lowering.program.children()does not descend intoPower, worked around inlinopy/spec/nodes.py.Verification (8b88ac7)
Checklist
AGENTS.md).doc.doc/release_notes.rstof the upcoming release is included.