refac(csr): dedicated csr module, shared Grid, aligned dense/sparse names by FabianHofmann · Pull Request #944 · PyPSA/linopy

FabianHofmann · 2026-09-07T12:28:14Z

We needed a better structure for all the CSR data containers. This improve the module's scopes and reduced duplication of code.

Note

The description below was generated by AI.

Changes proposed in this Pull Request

Structural refactor of the CSR (compressed sparse row) code. No change to public solver behaviour; the sparse groupby/freeze features work as before.

Module and naming

Moved the sparse backing of LinearExpression from linopy/sparse_expression.py to linopy/csr.py.
Renamed CSRExpression to CSRLinearExpression (it encodes A @ x + c; there is no quadratic analog).
Aligned dense/sparse conversion names on both CSR types: from_dense / to_dense (was from_expression / materialize on the expression, from_mutable on the constraint). Constraint.freeze() and CSRConstraint.mutable() are unchanged.
Renamed the term-merge helper to added to match the participle convention (reindexed, renamed, filled, scaled).

Boundaries

linopy/csr.py carries no top-level import of LinearExpression, Constraint or Model. The single exception to the leaf boundary is CSRLinearExpression.to_dense, which wraps its expanded dataset in a LinearExpression (its dense equivalent, mirroring CSRConstraint.to_dense -> Constraint) via a function-local import.
Moved _try_csr_merge next to merge in expressions.py.
Moved the _csr slot and _from_csr off BaseExpression onto LinearExpression, so QuadraticExpression no longer carries an unused slot.

Shared code

Added a small frozen Grid value object (dims, indexes, shape, size, strides, indexer, rename, equality) shared by CSRLinearExpression and CSRConstraint, replacing the two divergent grid representations (grid_dims + indexes dict vs list[pd.Index]).
Extracted csr_to_term_arrays (CSR to padded term arrays) and csr_nterm (widest row), used by both to_dense and the constraint reconstruction.

Fix

csr_to_term_arrays now preserves the coefficient dtype. Reconstructing a frozen constraint previously widened float32 coefficients to float64. Added a regression test.

Checklist

AI-generated content is marked (see AGENTS.md).
Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

…ames - rename CSRExpression -> CSRLinearExpression; sparse_expression.py -> csr.py - make csr.py a pure data module; move _try_csr_merge next to merge - move _csr slot from BaseExpression to LinearExpression - add shared Grid value object backing both CSR types - unify conversions (from_dense/to_dense, added) and share csr_to_term_arrays/csr_nterm - fix: preserve coefficient dtype when reconstructing frozen constraints

github-actions · 2026-09-07T12:31:49Z

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.07754 (1.05)   │   15.03 (1.00) 
 (v1)        0.07385 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08408 (1.09)   │   12.04 (1.00) 
 (v1)        0.07738 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03393 (1.06)   │   15.20 (1.0) 
 (v1)        0.03188 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05103 (1.13)   │   44.93 (1.0) 
 (v1)        0.04508 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03721 (1.08)   │   11.51 (1.0) 
 (v1)        0.03433 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08613 (1.03)   │   24.34 (1.06) 
 (v1)        0.08327 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09611 (1.05)   │   16.12 (1.00) 
 (v1)        0.09144 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02087 (1.06)   │   752.18 (1.10) 
 (v1)        0.01964 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02074 (1.08)   │   3.12 (1.33) 
 (v1)        0.01912 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05888 (1.22)   │   126.16 (1.44) 
 (v1)        0.04819 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05777 (1.22)   │   126.16 (1.44) 
 (v1)        0.04732 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05805 (1.22)   │   126.16 (1.44) 
 (v1)        0.04758 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04845 (1.02)   │    715.12 (1.0) 
 (v1)         0.0473 (1.0)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04635 (1.11)   │   4.54 (1.27) 
 (v1)        0.04194 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.3287 (1.03)   │   704.12 (1.09) 
 (v1)        0.3189 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.3443 (1.02)   │   18.34 (1.0) 
 (v1)        0.3389 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.3393 (1.02)   │   9.54 (1.0) 
 (v1)        0.3339 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.06539 (1.07)   │   3.77 (1.12) 
 (v1)         0.0609 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06516 (1.06)   │   216.59 (1.10) 
 (v1)         0.0615 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03368 (1.07)   │   938.49 (1.0) 
 (v1)        0.03136 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03448 (1.06)   │   9.66 (1.0) 
 (v1)        0.03241 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03451 (1.06)   │   5.32 (1.0) 
 (v1)        0.03244 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01974 (1.02)   │   1.47 (1.0) 
 (v1)        0.01944 (1.0)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)    0.01868 (1.0)   │   1.47 (1.0) 
 (v1)       0.01912 (1.02)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)    0.01912 (1.0)   │   1.47 (1.0) 
 (v1)       0.01915 (1.00)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.1627 (1.04)   │   946.85 (1.06) 
 (v1)        0.1571 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                time (s)         peak (KiB) 
 name               mean   │           mean 
────────────────────────────────────────────
 (legacy)   0.161 (1.03)   │   12.01 (1.00) 
 (v1)       0.1559 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04154 (1.05)   │   147.70 (1.06) 
 (v1)        0.03938 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04159 (1.08)   │   2.60 (1.09) 
 (v1)        0.03868 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03444 (1.08)   │   696.75 (1.03) 
 (v1)        0.03194 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.07833 (1.0)   │   137.97 (1.0) 
 (v1)       0.07936 (1.01)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05525 (1.08)   │   69.22 (1.0) 
 (v1)        0.05134 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03969 (1.09)   │   402.33 (1.00) 
 (v1)        0.03651 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03965 (1.10)   │   3.19 (1.19) 
 (v1)        0.03595 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04306 (1.06)   │   29.00 (1.54) 
 (v1)        0.04067 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05384 (1.13)   │   37.95 (1.43) 
 (v1)        0.04774 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)     0.0814 (1.0)   │    410.93 (1.0) 
 (v1)       0.08433 (1.04)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.08942 (1.0)   │     9.94 (1.0) 
 (v1)       0.09074 (1.01)   │   10.22 (1.03)

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

codspeed-hq · 2026-09-07T12:31:49Z

Merging this PR will degrade performance by 16.87%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 2 regressed benchmarks
✅ 179 untouched benchmarks
⏩ 181 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
❌	`test_to_lp[qp-n=1000]`	2 MB	2.6 MB	-22.88%
❌	`test_to_lp[nodal_balance-severity=50]`	3.3 MB	3.7 MB	-10.39%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing refac/csr-module-and-grid (d76b6c9) with master (c25f779)}

181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Move grid alignment onto Grid (is_unique, reordered, with_indexes, combined) so no module pokes Grid.indexes. CSRLinearExpression.to_dense now returns a LinearExpression, mirroring CSRConstraint.to_dense. Fix transposed-grid merge to build a real Grid via reordered (was passing a dict to reindexed); add a regression test for the default-join path.

Moving _csr onto LinearExpression left merge()/.add() results (typed BaseExpression / the Linear|Quadratic union) without _csr under mypy. Pass cls=LinearExpression on the merge calls and narrow the .add case with isinstance, so `mypy .` passes. Test-only, no runtime change.

…944)

FabianHofmann added 2 commits September 7, 2026 15:15

FabianHofmann merged commit ce6779a into master Sep 7, 2026
24 of 25 checks passed

FabianHofmann deleted the refac/csr-module-and-grid branch September 7, 2026 17:01

FabianHofmann added a commit that referenced this pull request Sep 7, 2026

fix(spec): restamp CSRConstraint grid after the csr module refactor (#…

e6623af

…944)

Navigation Menu

Uh oh!

refac(csr): dedicated csr module, shared Grid, aligned dense/sparse names - #944

refac(csr): dedicated csr module, shared Grid, aligned dense/sparse names#944
FabianHofmann merged 3 commits into
masterfrom
refac/csr-module-and-grid

FabianHofmann commented Sep 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 7, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Sep 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

FabianHofmann commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this Pull Request

Checklist

Uh oh!

github-actions Bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build cost — v1 vs legacy

Uh oh!

codspeed-hq Bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 16.87%

Performance Changes

Footnotes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

FabianHofmann commented Sep 7, 2026 •

edited

Loading

github-actions Bot commented Sep 7, 2026 •

edited

Loading

codspeed-hq Bot commented Sep 7, 2026 •

edited

Loading