Repository navigation
Options for EIA normalisation carried over from pypsa-eur's hydro inflow #29
Description
pypsa-eur has replaced its own hydro inflow with this module (PR #100), which removed four options from its config. Someone found them worth configuring, so they might be useful here:
| pypsa-eur option | What it did |
|---|---|
eia_approximate_missing |
Estimates generation for years without EIA data, from a linear regression of annual EIA generation on annual runoff per country. |
eia_norm_year |
Normalises every weather year with the EIA generation of one fixed year. |
eia_correct_by_capacity |
Scales EIA generation by the ratio of installed capacity in each year to a base year (2019), except for a list of countries. |
clip_min_inflow |
Sets inflow below a threshold (default 1 MW) to zero. |
The first two matter most, because the module needs EIA generation for every modelled year. EIA series start in 1980 at the earliest (Germany 1991, Kosovo 2008), while weather years go back to 1940. Without data the module fails (see #28). pypsa-eur used the median of the available EIA years as a fallback even when neither option was set.
The last two are closer to modelling choices, and the module's cap on annual generation may cover part of what clip_min_inflow did.
Metadata
Metadata
Assignees
Labels
No labels
Activity
irm-codebase commentedon Sep 28, 2026
Thank you @cdgaete!
Here are some notes, I'd be interested to know what you think.
eia_approximate_missingeia_norm_yeareia_correct_by_capacityclip_min_inflowPoints 1 to 3 suggest that being able to replace the EIA annual generation file with an external user provided one would be attractive... I'd rather avoid exploding the amount of options within the module with settings may be too study / model specific.
This way, you could still use the PyPSA-Eur code specific to that project, and it would enable people to use other data sources beyond EIA.
irm-codebase commentedon Sep 28, 2026
I'd also be interested in hearing @brynpickering's opinion on this.
brynpickering commentedon Oct 14, 2026
@irm-codebase this is a tough one. If all the user is going to do is gap-fill the EIA data and pass that, then it seems reasonable to actually move it to configuration within the module. Otherwise they have to repeat the EIA data download step themselves, gap-fill it, then pass that.
irm-codebase commentedon Oct 14, 2026
Might not be @brynpickering ! These statistics are actually a module output (see here, under 'statistics' in the results), which is mostly independent from the timeseries.
What I am proposing is an optional input that lets the user impute the annual generation profile however they want. I'd need to test this out, but I think all this would need is a
branch()decision to either use the module's pre-generated statistics, or (if present) the user generated statistics input.It should be possible to set rules in this order:
<pathvar>location.irm-codebase commentedon Oct 14, 2026
This way the module's code stays lean, and it becomes more powerful by allowing users some degree of flexibility.
brynpickering commentedon Oct 16, 2026
What I mean is that the user will have to then download the full EIA bulk file themselves, do some gap filling / tweaking, then pass it to the module. Since both tasks involve the EIA data, it will involve duplicating that downloading and cleaning code on the user side in every single model builder workflow that wants to gap fill. I understand that the module code should stay lean, but not at the expense of just pushing it to duplicated efforts elsewhere
irm-codebase commentedon Oct 16, 2026
Understood @brynpickering.
What I'm proposing is meant to avoid the double work you describe. Let me explain the proposal a bit better.
In the current approach, EIA statistics are always used internally, and are delivered to the user in
<results>/{shapes}/statistics/generation.parquet. The user has no capacity to modify or configure them.(Inputs are left arrows entering the module, outputs are right arrows exiting it)
Now, say that a user / workflow has a particular case that necessitates modifying these annual totals.
I'm proposing to keep the internal EIA processing as the default and as an independent module output, while giving the users the opportunity to provide adjusted data if they so choose (likely due to a configuration flag like
use_external_stats: True).With this the user can:
They can then tune the data, and feed it back in. Download and cleaning work duplication is avoided.
(this would still comply with Snakemake's DAG requirements, please forgive the "cyclic" look of it).
This still implies that the user would have to develop some imputation code on their side, but it would be pretty small.
I guess we could also bring some of these assumptions / imputations into the module... but this is a bit of an opinionated imputation, which is why I prefer to increase user freedom rather than trying to anticipate all possible cases / features.
irm-codebase commentedon Oct 16, 2026
I began implementing this in https://github.com/modelblocks-org/module_hydropower/tree/feat/flexible-statistics
After tuning some knick-knacks I'll open up a PR for feedback.
brynpickering commentedon Oct 19, 2026
@irm-codebase that makes sense. Then what is needed is some kind of FAQ section of the README on the filename to access to undertake this task. Keep in mind that a user only has access to whatever the INTERFACE.yaml exposes, so this intermediate file is something the user has no reason to be aware of. I guess the modelblocks best practice guide / the template might benefit from specifying an FAQ section for such cases.
irm-codebase commentedon Oct 19, 2026
@brynpickering added some README improvements to #32
I'll likely merge this in the coming days. LMKWYT.
irm-codebase commentedon Oct 19, 2026
One last clarification: the statistics file was never an intermediate file. It was always a module output.
module_hydropower/INTERFACE.yaml
Line 28 in fec2d2d
However, in
mainit only has the year range requested in the configuration (e.g., 2019, 2020, 2021). After #32 it will have the full EIA extent as reported for that country.Examples:
MNE
MEX