diff --git a/.gitignore b/.gitignore index 7d15b16..aad92ba 100644 --- a/.gitignore +++ b/.gitignore @@ -5,6 +5,30 @@ gen_database_updated_test.csv *.lst gens_fixme.csv ATB/inputs/ +atb/output/ +atb/inputs/ +# Raw ATB figures remain local. The smoothing comparisons are intentionally +# versioned so reviewers can see the effect without rerunning the workflow. +atb/figures/* +!atb/figures/smoothing_comparison/ +atb/figures/smoothing_comparison/* +!atb/figures/smoothing_comparison/*.png +# Keep the local ATB comparison utility, but never version generated plots or +# legacy tabular reports. +atb/comparison/* +!atb/comparison/README.md +!atb/comparison/compare_atb_outputs.py +atb/_dev/ +# Raw ATB downloads; reproduced by scripts/scrape_atb_inputs.py from the URLs +# pinned in atb/config.yaml, so they are never committed (see the File Size +# Guidelines in the top-level README). +atb/scraped_input/* +!atb/scraped_input/README.md +# Unpublished ATB 2025 manual inputs; keep local, never commit +atb/manual_input/battery_costs_2025.csv +atb/manual_input/csp_cost_ratios_2025.csv +# Legacy CF source; final historical CF multipliers now live in historical/. +atb/manual_input/historic_capacity_factors.csv /NEMS_database_processing/Outputs/ /AEO_Updates/outputs/ /AEO_Updates/output/AEO_HIGHMACRO_2021_electricity_consumption.csv @@ -12,8 +36,6 @@ ATB/inputs/ /AEO_Updates/output/AEO_LOWMACRO_2021_electricity_consumption.csv /AEO_Updates/output/AEO_REF2021_2021_electricity_consumption.csv /AEO_Updates/output/AEO_REF2021_2021_ng_prices.csv -/ATB/figures/ - # pyc files *.pyc *__pycache__ diff --git a/atb/README.md b/atb/README.md index ff7e80f..6dce026 100644 --- a/atb/README.md +++ b/atb/README.md @@ -1,10 +1,392 @@ -# Overview -This repo includes scripts to plot CAPEX, FOM, and VOM of different versions of the ATB. +# ATB input pipeline -# Required inputs -Users are required to provide Users have the options to provide the ATBe.csv file for the specific ATB version they want to plot. THere are two options to provide ATBe.csv file: -- Save the file in the ATB/inputs folder -- Provide an URL to the ATBe.csv file. If choose URL option, users are prompted to provide the URL. +This directory turns raw NLR Annual Technology Baseline (ATB) data into ReEDS +input files and plots. The workflow has three explicit stages: -# Note -This script was tested on ATB 2024 and 2025 versions only, so they may not work for earlier versions. +1. download and inspect raw data; +2. format the local raw data for ReEDS; +3. plot metrics from the same local raw data. + +## Configure the run + +[`config.yaml`](config.yaml) is the user-facing control file. It shows which +stages will run, the ATB release and source URLs, local filenames, ReEDS path, +technologies, processing choices, and plotting choices. + +Review that file first, especially: + +- `atb.year`, `atb.release_version`, and `atb.dollar_year`; +- the two `raw_data` URLs and filenames; +- the optional observed-cost sources under `historical_cost_sources`; +- `processing.reeds_repo` and selected technologies; +- `historical_data.directory` and its fixed dollar year; +- the metrics and technologies under `plotting`. + +## Run one step at a time + +Run the following commands from the `atb/` directory. + +### Step 1: scrape raw inputs + +```bash +python scripts/scrape_atb_inputs.py +``` + +This downloads or reuses both independent raw inputs: + +- `scraped_input/atb__flat_file.csv`; +- `scraped_input/atb__workbook.xlsx`. + +It then displays a summary of the flat file and the workbook sheets. Existing +files are reused unless `--force` is supplied: + +```bash +python scripts/scrape_atb_inputs.py --force +``` + +Neither raw file is committed, so this step is required after a fresh clone. +See [`scraped_input/README.md`](scraped_input/README.md) for why, and how the +pinned URLs keep a run reproducible without storing the data in Git. + +Downloads normally verify HTTPS certificates. If certificate verification +fails because the active conda environment does not trust an NLR network +inspection certificate, `raw_data.allow_insecure_ssl_fallback: true` permits a +clearly labeled `verify=False` retry. Set it to `false` to prohibit that retry. + +### Optional: scrape observed historical capital costs + +```bash +python scripts/scrape_historical_costs.py +``` + +This separate utility downloads the configured LBNL land-wind and utility-PV, +NLR offshore-wind, and annual EIA generator-cost workbooks. It preserves the +original files under `scraped_input/historical_costs/` and creates +`historical_capital_costs.csv` plus a URL/checksum manifest there. These data +enter ReEDS inputs only through an explicit mapping in `config.yaml`. The +current mappings replace pre-projection UPV and land-based-wind `capcost` with +LBNL national series; their FOM, VOM, and capacity-factor history remain manual. See +[`scraped_input/README.md`](scraped_input/README.md) for the retained unit, +capacity-basis, geography, and dollar-year distinctions. + +### Step 2: format ReEDS inputs + +```bash +python scripts/generate_atb_files.py +``` + +This reads the local raw files plus the versioned history under +`manual_input/historical/`, then writes ReEDS-formatted CSVs to `output/`. It +does **not** download raw data. If a required raw file is missing, run Step 1 +first. + +Before processing any cost files, the formatter checks every selected +technology metric configured as `real`. It stops with a single actionable error +if the normalized historical-cost CSV, a reviewed metric mapping, or matching +source rows are missing. + +A missing history file can be initialized from the matching file in +`processing.reeds_repo` by temporarily setting +`historical_data.seed_missing_from_reeds: true`. Current scraped ATB rows +replace history from the first available projection year onward. That boundary +year is then appended to the fixed-dollar history file, ready for the next +annual ATB release. + +Whether this step generates cost files and financial files, which technologies +it processes, and whether it copies results into ReEDS are all controlled under +`processing:` in `config.yaml`. + +### Selective smoothing logic + +The optional `processing.smooth_cost_curves` block removes short-lived dips, +bumps, and rounded stair steps without replacing each ATB trajectory with one +fully smoothed curve. Each technology has metric-level historical choices and +independent future switches under `smooth_cost_curves.technologies`: + +The default `columns: all` applies future monetary smoothing to every monetary +metric (`capcost*`, `fom*`, `vom`), since a rounding artifact is no more real in +O&M than in capital cost. `capital_costs` narrows it to `capcost*`. Heat rate is +not monetary and is never included by `all`. Historical source choices apply +independently to every metric. By default, metrics with reviewed observations +use `real`, while all others use `broadcast`. Capacity-factor multipliers remain +included automatically for technologies that use them. + +| Historical mode | Effect before each series' first ATB year | +| --- | --- | +| `real` | Use only the reviewed observed-series mapping for that metric. Preserve reported values, linearly interpolate internal missing years, and use the nearest observation for years outside the reported range. An unmapped metric or row variant raises an error. | +| `manual` | Preserve the versioned rows in `manual_input/historical/` exactly. Those rows come from the ReEDS repository, are of mixed origin, and are maintained by hand there year by year rather than derived from any ATB download. | +| `broadcast` | Use the first ATB projection value for every historical year. This generated history does not retain any manual historical values. | + +The switches under `future_smoothing_treatments` apply from +`projection_start_year` onward. ATB's published direction is never overridden: +a cost the source projects to rise, such as the 2023 solar and wind increases, +is passed through unchanged. + +| Future smoothing treatment | Effect | +| --- | --- | +| `smooth_projection_curve` | Bridge compact clusters of slope changes. Major transitions, single ATB milestones, and flat stretches remain explicit. | + +Every technology listed under `smooth_cost_curves.technologies` is processed. +Every modeled metric has an explicit `historical_data` entry. For example, +biopower uses `capcost: real`, while fixed O&M, variable O&M, and heat rate each +explicitly select `broadcast`. + +A metric may instead give one mode per sub-technology, for a technology whose +observed series describes only some of its rows. Two do. Offshore wind CapEx is +a fixed-bottom series, so only that class reads it; EIA reports gas plants only +as "combined cycle" or "combustion turbine", so the H-Frame and aeroderivative +variants have no observation at all. + +```yaml +wind-ofs: + historical_data: + capcost: + fixed: real + floating: broadcast + +gas: + historical_data: + capcost: + Gas-CC: real + Gas-CC_H_1x1: broadcast + Gas-CC_H_2x1: broadcast + Gas-CT: real + Gas-CT_aero: broadcast +``` + +The split form requires the technology to name its sub-technology column +through `history_class_column` in `scripts/settings.yaml` (`turbine` for +offshore wind, `i` for gas), and the entries selecting `real` must match the +mapping's targets exactly; a disagreement raises rather than leaving one +sub-technology on unintended history. The schema offers `real` only where an +observation exists, so a typo is caught in the editor. +Capacity-factor multipliers are included automatically when the technology +output contains `cf_improvement` (utility PV and the two wind technologies). +`real` applies to every technology with an observed series that measures the +same quantity as its ReEDS column: UPV, land-based wind, offshore wind, gas, +and biopower. Metrics without a reviewed observed mapping use broadcast history, +including heat rate and efficiency. + +### Per-technology observed-history appliers + +A `real` mapping supplies one national value per year, but technologies do not +all carry one row per year, so the two halves of the work are separated in +`scripts/generate_atb_files.py`: + +- `_observed_values_by_year` is shared. It filters the normalized observed + series, requires a stated `dollar_year`, and deflates to the ReEDS dollar + year. +- an entry in `REAL_HISTORY_APPLIERS`, keyed by technology, decides which rows + that annual value is allowed to address. A technology selecting a `real` + metric without an entry raises `NameError`. + +Frames at this stage stack all ATB scenarios together, and history is identical +across them, so repetition across `Scenario` is expected. Repetition on any +other dimension is not, and `_assert_one_row_per_year` raises rather than +letting one value silently overwrite several distinct series. + +| Applier | Used by | Behavior | +| --- | --- | --- | +| `apply_real_history_single_series` | `upv`, `wind-ons`, `biopower` | One row per scenario-year; assigns directly. | +| `apply_real_history_by_class` | `wind-ofs`, `gas` | One row per sub-technology, named by `history_class_column`. Each series describes exactly one sub-technology; any other must select its own mode in `historical_data`, otherwise the run raises instead of mixing manual history. | + +Why a given technology targets the rows it does is recorded beside its mapping +in `config.yaml`, where that choice is made. + +Add a technology by writing an applier that owns its row-shape assumption and +registering it, rather than generalizing an existing one. + +### Mapping options + +Mappings are nested as `technology -> metric`. Each metric entry accepts: + +| Key | Default | Effect | +| --- | --- | --- | +| `filters` | required without `series` | Column/value pairs selecting exactly one row per year from the normalized CSV. | +| `turbine_classes` | required for offshore wind | Offshore only: the one turbine class receiving the observed series. Any other class must select its own mode under `historical_data`. | +| `series` | — | A list of sub-series, each with its own `filters` and `technologies`, for technologies whose rows need different observed series. Entries inherit the mapping's other keys. | + +A series measures one thing, so it names exactly one sub-technology. Sharing a +series across several made unrelated rows carry identical history and hid the +fact that no observation existed for the others. + +Rows that all take the same series use `filters` directly; rows needing +different series use `series`, as `gas` does: + +```yaml +gas: + capcost: + series: + - technologies: [Gas-CC, Gas-CC_H_1x1, Gas-CC_H_2x1] + filters: {technology_detail: Natural gas combined cycle, ...} + - technologies: [Gas-CT, Gas-CT_aero] + filters: {technology_detail: Natural gas combustion turbine, ...} +``` + +Missing years never fall back to the manual file when `real` is selected. +Internal gaps are linearly interpolated between the surrounding observations. +For years outside the observed range, where interpolation is impossible, the +nearest observed endpoint is used. The smoothing-comparison plot colors these +derived years separately from directly reported observations. + +The current defaults in `config.yaml` are: + +| Setting | Default | Meaning | +| --- | ---: | --- | +| `projection_start_year` | `2022` | Years before this are historical; this year and later are the current projection | +| `slope_change_threshold` | `0.4` | Detect a normalized slope change of 40% or more | +| `max_kink_years` | `4` | Maximum span of a compact slope-change cluster | +| `major_step_relative_threshold` | `0.1` | Preserve year-to-year changes of 10% or more | +| `minimum_adjustment_relative_threshold` | `0.005` | Keep the original ATB value when a proposed future smoothing adjustment is smaller than 0.5% | + +For example, this preserves manually supplied UPV history and passes the ATB +projection through untouched: + +```yaml +technologies: + upv: + historical_data: + capcost: manual + fom: manual + vom: manual + cf_improvement: manual + future_smoothing_treatments: + smooth_projection_curve: false +``` + +The older flat-history/anchor-to-target behavior remains available as +`method: linear_bridge`. + +### Configuration editor support + +`config.yaml` declares `config.schema.json` on its first line. Editors with YAML +language-server support use that schema for completion lists, hover descriptions, +required-field checks, and invalid-option warnings. Use the editor's completion +command (typically `Ctrl+Space`) to choose valid history modes and other options. + +Historical choices are technology-by-metric. The schema offers `real` only for +metrics with a reviewed observed mapping; other metrics offer `manual` and +`broadcast`. Offshore wind CapEx additionally accepts a per-turbine-class +mapping of modes. Runtime validation remains authoritative when the workflow +runs. + +### File names and column schemas expected by ReEDS + +Generated files must match what ReEDS expects exactly, because ReEDS resolves +these inputs by file name and, for some technologies, reads their columns by +position rather than by name. Two settings in +[`scripts/settings.yaml`](scripts/settings.yaml) handle the cases where the +internal representation and the ReEDS representation differ: + +- `reeds_name` — the file prefix ReEDS uses when it differs from the internal + technology key. Onshore and offshore wind are `wind-ons`/`wind-ofs` + internally but `ons-wind`/`ofs-wind` in ReEDS, so their outputs are written + as `ons-wind_ATB__.csv` and `ofs-wind_ATB__.csv`. + The output file name doubles as the `Scenario` key in the ReEDS + `dollaryear.csv`, so a mismatch here silently leaves ReEDS reading its + previous inputs. +- `output_cols` — an ordered mapping of internal column name to ReEDS header, + applied as the last step before writing. ReEDS reads the two wind files + positionally in `reeds/input_processing/plantcostprep.py`, and detects the + ATB 2024 offshore format by the presence of a `Turbine` column, so those + files must keep the legacy headers and this exact column order: + `Turbine, Year, CF_mult, Overnight Cap Cost $/kW, Fixed O&M $/(kW-yr), + Var O&M $/MWh` (plus `rsc_mult` for offshore). Writing the internal names or + order instead makes ReEDS assign capital cost to the capacity-factor + multiplier without raising an error. + +Technologies without these settings are written using the internal column names +listed under `cols`, which already match their ReEDS files. Everything upstream +of the write step — history files in `manual_input/historical/`, scenario +comparisons, transformations — uses the internal names throughout. + +### Step 3: plot raw ATB data + +```bash +python scripts/atb_plotting.py +``` + +This reads the same local flat file used in Step 2 and saves the configured +figures to `figures/`. It does not scrape data or plot the formatted files from +`output/`. + +## Run the configured pipeline + +To run the enabled stages in order, set the switches under `workflow:` in +`config.yaml`, then run: + +```bash +python scripts/run_pipeline.py +``` + +The runner prints which stages will run before doing any work. A stage can also +be selected explicitly: + +```bash +python scripts/run_pipeline.py --only scrape +python scripts/run_pipeline.py --only format +python scripts/run_pipeline.py --only plot +``` + +## Data flow + +```text +config.yaml + | + +--> scrape_atb_inputs.py + | +--> scraped_input/atb__flat_file.csv + | +--> scraped_input/atb__workbook.xlsx + | +--> terminal summaries/previews of both raw files + | + +--> scrape_historical_costs.py (optional, separate) + | +--> scraped_input/historical_costs/*.xlsx + | +--> historical_capital_costs.csv + source_manifest.csv + | + +--> generate_atb_files.py + | +--> raw flat file (primary ATB data) + | +--> raw workbook (battery power/energy cost split) + | +--> manual_input/historical/ (versioned ReEDS history) + | +--> manual_input/ (CSP ratios and pre-release fallbacks) + | +--> ReEDS deflator and dollaryear tables + | +--> output/*_ATB__.csv + | + +--> atb_plotting.py + +--> the same raw flat file + +--> figures/ +``` + +The flat file and workbook are independent upstream downloads. Neither is +generated from the other. The formatter and plotter do not download data; they +only consume the raw files created by the scraper. + +The normal pipeline validates the formatted pre-smoothing data against the +configured ReEDS repository when `workflow.make_comparison_plots` is enabled. +It holds that data in a temporary directory, prints the validation summary to +the terminal, writes local validation plots under `comparison/plots/`, and +writes versioned before/after plots under +`comparison/smoothing_comparison/`. The +temporary CSVs are deleted when the pipeline exits; no row-level or summary CSV +reports are created. + +For ATB 2024, the URLs are intentionally pinned to corrected release v3. +Changing the release can change technology trajectories; update both source +URLs together when adopting a newer release. + +## Directory layout + +| Path | Purpose | +| --- | --- | +| `config.yaml` | User-facing workflow configuration | +| `config.schema.json` | Editor completion lists and validation for `config.yaml` | +| `scraped_input/` | Visible raw ATB and optional observed-cost downloads (local only; not committed) | +| `manual_input/` | Versioned history and inputs unavailable in ATB downloads | +| `scripts/settings.yaml` | Internal per-technology ReEDS formatting rules | +| `scripts/scrape_atb_inputs.py` | Raw-data download and inspection | +| `scripts/scrape_historical_costs.py` | Observed capital-cost download, normalization, and manifest | +| `scripts/generate_atb_files.py` | ReEDS input formatter | +| `scripts/atb_plotting.py` | Raw ATB plotting | +| `output/` | Generated ReEDS-formatted CSVs | +| `figures/` | Generated ATB plots | +| `comparison/smoothing_comparison/` | Versioned before/after smoothing plots | + +See [`scripts/README.md`](scripts/README.md) for the scripts folder structure. diff --git a/atb/atb_plotting.py b/atb/atb_plotting.py deleted file mode 100644 index 5ca66ba..0000000 --- a/atb/atb_plotting.py +++ /dev/null @@ -1,313 +0,0 @@ -#%% Imports -import pandas as pd -import matplotlib as mpl -import matplotlib.pyplot as plt -import os -import sys -import math -import json -import argparse - -# Get reedsplots from ReEDS repo -reeds_path = os.path.expanduser('~/Documents/Github/ReEDS/ReEDS') -sys.path.append(reeds_path) -from reeds import plots -plots.plotparams() - -############################################################################################################# -def main(): - - ########################################## USER-DEFINED INPUTS ########################################## - parser = argparse.ArgumentParser(description="Plotting ATB") - parser.add_argument('--version', '-v', type=int, required=True, - help='version of ATB to plot.') - parser.add_argument('--metric', '-m', type=str, default='CAPEX', - help='metric to plot, default is CAPEX, other options are Fixed O&M and Variable O&M') - parser.add_argument('--crpyears', '-y', type=int, default=20, - help='crpyears, default is 20') - parser.add_argument('--inputs', '-i', type=str, default='url', - help='where to look for the ATBe csv input file to plot, options are url and local') - parser.add_argument('--format', '-f', type=str, default='png', - help='format of output plots, options are .png and .pdf') - parser.add_argument('--save', '-s', action="store_true", - help='save cleaned ATB data used for plotting to csv.') - args = parser.parse_args() - - atb_version = args.version # ATB version - core_metric_parameter = args.metric # Metric to plot: 'CAPEX', 'Fixed O&M','Variable O&M' - crpyears = args.crpyears # crpyears - atb_inputs = args.inputs # Read in ATBe.csv input file from 'url' or 'local' inputs folder - # (if a version of ATB is not yet available online, read from 'inputs') - # Link to past ATBe versions: https://data.openei.org/s3_viewer?bucket=oedi-data-lake&prefix=ATB%2Felectricity%2Fcsv%2F&limit=50 - # 2024 atb_path = 'https://oedi-data-lake.s3.amazonaws.com/ATB/electricity/csv/2024/v3.0.0/ATBe.csv' - # 2025 atb is not yet available online, so only option is reading it from inputs folder. - figure_format = args.format # 'png' or 'pdf' - save_data = args.save # Save the version of ATBe that has been cleaned and ready for plotting to csv - - ######################################### FOR TESTING/DEBUGGING ######################################### - # atb_version = 2024 # ATB version - # core_metric_parameter = 'CAPEX' # Metric to plot: 'CAPEX', 'Fixed O&M','Variable O&M' - # crpyears = 20 - # atb_inputs = 'url' # Read in ATBe.csv input file from 'url' or 'local' inputs folder - # (if a version of ATB is not yet available online, read from 'inputs') - # Link to past ATBe versions: https://data.openei.org/s3_viewer?bucket=oedi-data-lake&prefix=ATB%2Felectricity%2Fcsv%2F&limit=50 - # 2024 atb_path = 'https://oedi-data-lake.s3.amazonaws.com/ATB/electricity/csv/2024/v3.0.0/ATBe.csv' - # 2025 atb is not yet available online, so only option is reading it from inputs folder. - # figure_format = 'png' # 'png' or 'pdf' - # save_data = False # Print the version of ATBe that has been cleaned and ready for plotting to csv - ######################################################################################################### - - # Specify dollar year: - if atb_version == 2024: - dollar_year = 2022 - elif atb_version == 2025: - dollar_year = 2023 - - # Get the path for ATB data and clean the data: - (inputs_path, atb_path, figures_path) = read_path(atb_inputs, atb_version) - (technologies, dfplot) = clean_atb_data(atb_inputs, atb_path, atb_version, core_metric_parameter, crpyears) - if save_data: - print("Saving the version of ATBe that has been cleaned and ready for plotting to csv file...") - dfplot.to_csv(os.path.join(inputs_path, "ATB_"+str(atb_version)+"_cleaned.csv")) - - # Define plot attributes: - (traces, colors, tracelabels, legendtitle, plottitle) = plot_attributes(inputs_path, atb_version) - - # Plot ATB: - plot_atb(figures_path, atb_version, technologies, dfplot, traces, colors, legendtitle, plottitle, - tracelabels, core_metric_parameter, figure_format, dollar_year) - -###################################################################################################### -#%% FUNCTIONS ### -def read_path(atb_inputs, atb_version): - # Iputs path: - inputs_path = os.path.join(os.getcwd(),'inputs') - if not os.path.exists(inputs_path): - os.makedirs(inputs_path) - - # ATB path: - if atb_inputs == 'url': - print('\nYou choose to read ATBe '+str(atb_version) + ' file from an URL.') - user_input = input('\nPlease enter the URL to the ATBe '+str(atb_version) + ' file here to continue: ') - atb_path = user_input - elif atb_inputs == 'local': - print('\nYou choose to read ATBe_'+str(atb_version)+ '.csv file from local "inputs" folder.') - atb_path = os.path.join(inputs_path,'ATBe_'+str(atb_version)+'.csv') - - # Figures path: - figures_path = os.path.join(os.getcwd(),'figures') - if not os.path.exists(figures_path): - os.makedirs(figures_path) - - return (inputs_path, atb_path, figures_path) - -def clean_atb_data(atb_inputs, atb_path, atb_version, core_metric_parameter, crpyears): - # Read ATB data - try: - atb = pd.read_csv(atb_path, index_col=0, low_memory=False) - except Exception: - if atb_inputs == 'url': - raise FileNotFoundError('URL is incorrect') - elif atb_inputs == 'local': - raise FileNotFoundError('ATBe file is missing or incorrect. Please provide data file with name ATBe_' + str(atb_version) + '.csv in the "inputs" folder.') - atb.technology.unique() - - # Use 'Market' case for 2024 version, which is equivalent to 'Exp + TC' in 2025 version - if atb_version == 2024: - core_metric_case = 'Market' - elif atb_version == 2025: - core_metric_case = 'Exp + TC' - - # Technology to plot - technologies = [ - 'Biopower', - 'Coal_FE', - 'CSP', - 'Geothermal', - 'LandbasedWind', - 'NaturalGas_FE', - 'Nuclear', - 'OffShoreWind', - 'Utility-Scale Battery Storage', - 'UtilityPV', - # 'CommPV', - # 'Hydropower', - # 'Pumped Storage Hydropower', - # 'ResPV', - # 'Utility-Scale PV-Plus-Battery', - # 'Commercial Battery Storage', - # 'Residential Battery Storage', - ] - - # Clean the raw ATB data in the right format for plotting - dictplot = {} - keepcols = ['technology','techdetail','scenario','core_metric_variable','value'] - for technology in technologies: - # In ATB 2025 version, some technologies do not have 'Exp + TC' case, so use 'R&D' for them - if atb_version == 2025: - if technology in [ - 'Biopower', - 'Coal_FE', - 'CSP', - 'Geothermal', - 'NaturalGas_FE', - 'Nuclear', - 'OffShoreWind', - ]: - core_metric_case = 'R&D' - else: - core_metric_case = 'Exp + TC' - - dictplot[technology] = atb.loc[ - (atb.technology==technology) - & (atb.core_metric_parameter==core_metric_parameter) - & (atb.core_metric_case==core_metric_case) - & (atb.crpyears==crpyears) - # & (atb.techdetail==techdetail) - # & (atb.core_metric_variable==2050) - , keepcols - ] - dfplot = pd.concat(dictplot, ignore_index=True).set_index('technology') - - return (technologies, dfplot) - -def plot_attributes(inputs_path, atb_version): - cm = plt.cm.coolwarm - cm = plt.cm.RdBu - d = 0.06 - # traces setting - with open(os.path.join(inputs_path,"traces_"+str(atb_version)+".json")) as file: - traces = json.load(file) - # tracelabels setting - with open(os.path.join(inputs_path,"tracelabels.json")) as file: - tracelabels = json.load(file) - - # colors setting - with open(os.path.join(inputs_path,"colors.json")) as file: - colors = json.load(file) - # add colors for offshore wind classes - colors["OffShoreWind"] = { - 'Class1': cm(0.), - 'Class2': cm(d), - 'Class3': cm(2*d), - 'Class4': cm(3*d), - 'Class5': cm(4*d), - 'Class6': cm(5*d), - 'Class7': cm(6*d), - 'Class8': cm(1.), - 'Class9': cm(1-d), - 'Class10': cm(1-d*2), - 'Class11': cm(1-d*3), - 'Class12': cm(1-d*4), - 'Class13': cm(1-d*5), - 'Class14': cm(1-d*6), - } - - legendtitle = { - 'UtilityPV':None, - 'Nuclear':None, - 'LandbasedWind':None, - 'Biopower':None, - 'CSP':None, - 'Utility-Scale Battery Storage': 'Duration\n[hours]', - 'OffShoreWind': 'Class', - 'NaturalGas_FE': None, - 'Coal_FE': None, - 'Geothermal': None, - } - - plottitle = { - 'UtilityPV':'Utility PV', - 'Nuclear':'Nuclear', - 'LandbasedWind':'Land-based wind', - 'Biopower':'Biopower', - 'CSP':'CSP', - 'Utility-Scale Battery Storage': 'Utility battery', - 'OffShoreWind': 'Offshore wind', - 'NaturalGas_FE': 'Natural gas', - 'Coal_FE': 'Coal', - 'Geothermal': 'Geothermal', - } - return (traces, colors, tracelabels, legendtitle, plottitle) - -def plot_atb(figures_path, atb_version, technologies, dfplot, traces, colors, legendtitle, plottitle, - tracelabels, core_metric_parameter, figure_format, dollar_year): - - # Set up number of panels for figure - alpha = 0.4 - nrows, ncols = 2, 5 - coords = dict(zip( - technologies, - [(row, col) for row in range(nrows) for col in range(ncols)] - )) - - # Plot the figure - plt.close() - f,ax = plt.subplots( - nrows, ncols, figsize=(13,8), sharex=True, sharey=True, - # gridspec_kw={'wspace':1.0, 'hspace':0.2}, - gridspec_kw={'hspace':0.25}, - ) - - for technology in technologies: - df = {} - #scenarios = ['Moderate'] if technology == 'Nuclear' else ['Moderate', 'Advanced', 'Conservative'] - scenarios = ['Moderate', 'Advanced', 'Conservative'] - for scenario in scenarios: - df[scenario] = ( - dfplot - .loc[dfplot.scenario==scenario] - .loc[technology] - .pivot(columns='techdetail',index='core_metric_variable',values='value') - )[traces[technology]] - ### Central - for trace in df['Moderate']: - ax[coords[technology]].plot( - df['Moderate'].index, df['Moderate'][trace].values, - color=colors[technology][trace], label=tracelabels[technology][trace], - ) - if len(traces[technology]) != 1: - ax[coords[technology]].legend( - fontsize=8, #'small', - loc = ('upper right'), - #loc=('center left' if coords[technology][1]==2 else 'lower left'), - #bbox_to_anchor=((1,0.5) if coords[technology][1]==2 else (0.,0.)), - ncol=(2 if technology == 'OffShoreWind' else 1), - handlelength=0.7, handletextpad=0.3, columnspacing=0.5, - title=legendtitle[technology], - frameon=False, - ) - ### Formatting - ax[coords[technology]].set_title(plottitle[technology], weight='bold') - ax[coords[technology]].grid(which='major',axis='y',ls=':',lw=0.5,c='0.5') - ### Range - #if technology == 'Nuclear': - # continue - for trace in df['Advanced']: - ax[coords[technology]].fill_between( - df['Advanced'].index, df['Conservative'][trace].values, df['Advanced'][trace].values, - color=colors[technology][trace], alpha=alpha, label='_nolabel_', lw=0, - ) - - ### Format the figure - for col in range(ncols): - ax[-1,col].set_xlabel(None) - ax[0,0].xaxis.set_major_locator(mpl.ticker.MultipleLocator(10)) - ax[0,0].xaxis.set_minor_locator(mpl.ticker.AutoMinorLocator(2)) - if core_metric_parameter == 'CAPEX': - ax[0,0].yaxis.set_major_locator(mpl.ticker.MultipleLocator(2000)) - else: - ax[0,0].yaxis.set_major_locator(mpl.ticker.MultipleLocator(40)) - ax[0,0].yaxis.set_minor_locator(mpl.ticker.AutoMinorLocator(2)) - ax[0,0].set_ylim(0,math.ceil(dfplot.value.max()/100)*100) - ax[1,0].set_ylabel(core_metric_parameter + ' cost [' + str(dollar_year) + '$/kW]') - ax[0,0].set_ylabel(core_metric_parameter + ' cost [' + str(dollar_year) + '$/kW]') - plots.despine(ax) - - # Save the figure - if figure_format == 'png': - plt.savefig(os.path.join(figures_path,'ATB'+str(atb_version)+'-{}.png'.format(core_metric_parameter))) - elif figure_format == 'pdf': - plt.savefig(os.path.join(figures_path,'ATB'+str(atb_version)+'-{}.pdf'.format(core_metric_parameter))) - -main() diff --git a/atb/comparison/README.md b/atb/comparison/README.md new file mode 100644 index 0000000..e3d0f8f --- /dev/null +++ b/atb/comparison/README.md @@ -0,0 +1,53 @@ +# Local ATB output comparison + +The comparison utility is versioned. Generated/ReEDS validation plots remain +local and ignored, while before/after smoothing plots are versioned for review. + +For validation plus before/after smoothing figures, run formatting and +comparison together from `atb/`: + +```bash +python scripts/run_pipeline.py --only format compare +``` + +When the full pipeline runs formatting and comparison together, it captures +fully processed files just before smoothing in a temporary directory. The +comparison script validates those files against the matching source files in +the ReEDS repository configured in `../config.yaml`, then the temporary files +are deleted automatically. Thus smoothing differences do not obscure scraper +and formatting validation. The script normalizes legacy wind filenames and +column names, then checks: + +- missing files and columns; +- duplicate keys; +- rows present on only one side; +- changed values using configurable numeric tolerances. + +Only plots are written, under `comparison/plots/`: + +- `comparison_overview.png` summarizes file-level validation status; +- one time-series plot is written for each generated/ReEDS file pair. + +Detailed comparison statuses and reverse-coverage gaps are printed to the +terminal instead of being written as CSV reports. In these local plots, solid +lines are the unsmoothed generated baseline and dashed lines are ReEDS. + +Running `python comparison/compare_atb_outputs.py` by itself compares the final +files already in `output/` with ReEDS. It cannot regenerate before/after plots +because the temporary pre-smoothing data is intentionally not retained. + +The full pipeline also compares the temporary unsmoothed data with the final +smoothed outputs. Those plots are written to +`smoothing_comparison/` and are versioned so branch users can review +the smoothing effect without retaining duplicate CSV outputs. Final processed +values are lines, while solid dots identify input data values. Manual history +and directly observed real history receive dots; broadcast history and filled +real-history years do not. Future dots show the raw ATB values, while the line +shows the final raw or smoothed trajectory. A selected manual or broadcast +history uses one color for its entire historical curve. Real history uses green +for observations and orange for years filled from that same real series; an +internal gap is linearly interpolated. Gray and gold distinguish raw and +smoothed ATB projections. A dotted vertical line marks the configured +projection start year; line styles distinguish technology series. Each interval +uses the source color of its starting year, so a new source color never extends +backward into the preceding year. diff --git a/atb/comparison/compare_atb_outputs.py b/atb/comparison/compare_atb_outputs.py new file mode 100644 index 0000000..bf3fca5 --- /dev/null +++ b/atb/comparison/compare_atb_outputs.py @@ -0,0 +1,1175 @@ +"""Compare generated ATB 2024 CSVs with the current ReEDS input files.""" + +from __future__ import annotations + +import argparse +import math +from pathlib import Path +import sys + +import matplotlib.pyplot as plt +import numpy as np +import pandas as pd + + +ATB_DIR = Path(__file__).resolve().parents[1] +SCRIPT_DIR = ATB_DIR / "scripts" +sys.path.insert(0, str(SCRIPT_DIR)) + +from atb_config import ( + DEFAULT_CONFIG_PATH, + load_processing_settings, + raw_file_path, +) + + +DEFAULT_PLOT_DIR = Path(__file__).resolve().parent / "plots" + +FILENAME_PREFIX_MAP = { + "wind-ons_": "ons-wind_", + "wind-ofs_": "ofs-wind_", +} + +REEDS_COLUMN_MAP = { + "Turbine": "turbine", + "Year": "t", + "CF_mult": "cf_improvement", + "Overnight Cap Cost $/kW": "capcost", + "Fixed O&M $/(kW-yr)": "fom", + "Var O&M $/MWh": "vom", +} + +PREFERRED_KEYS = ["i", "type", "turbine", "t", "rsc_mult"] + +METRIC_LABELS = { + "capcost": "Capital cost ($/kW)", + "capcost_energy": "Energy capital cost ($/kWh)", + "fom": "Fixed O&M ($/kW-year)", + "fom_energy": "Energy fixed O&M ($/kWh-year)", + "vom": "Variable O&M ($/MWh)", + "heatrate": "Heat rate (MMBtu/MWh)", + "cf_improvement": "Capacity-factor multiplier", + "rte": "Round-trip efficiency", + "rsc_mult": "Resource-supply-curve multiplier", +} + +PROVENANCE_COLORS = { + "Manual history": "#0072B2", + "Observed history (real)": "#009E73", + "Filled real history": "#D55E00", + "Broadcast history": "#CC79A7", + "ATB projection (raw)": "#6E6E6E", + "ATB projection (smoothed)": "#E69F00", +} + +SERIES_LINESTYLES = [ + "-", + "--", + "-.", + ":", + (0, (5, 1)), + (0, (3, 1, 1, 1)), + (0, (1, 1)), + (0, (5, 2, 1, 2)), +] + + +def resolve_atb_config_path(value: str | Path) -> Path: + """Resolve a configured path relative to the ATB workflow directory.""" + path = Path(value) + return path.resolve() if path.is_absolute() else (ATB_DIR / path).resolve() + + +def reeds_filename(generated_name: str) -> str: + """Map a generated filename to its ReEDS filename.""" + for generated_prefix, reeds_prefix in FILENAME_PREFIX_MAP.items(): + if generated_name.startswith(generated_prefix): + return reeds_prefix + generated_name[len(generated_prefix):] + return generated_name + + +def normalize_frame(frame: pd.DataFrame) -> pd.DataFrame: + """Normalize known legacy ReEDS column names and cell dtypes.""" + frame = frame.rename(columns=REEDS_COLUMN_MAP).copy() + for column in frame.columns: + numeric = pd.to_numeric(frame[column], errors="coerce") + nonempty = frame[column].notna().sum() + if nonempty and numeric.notna().sum() == nonempty: + frame[column] = numeric + elif frame[column].dtype == object: + frame[column] = frame[column].astype("string").str.strip() + return frame + + +def choose_keys(generated: pd.DataFrame, reeds: pd.DataFrame) -> list[str]: + """Choose stable identifier columns shared by both frames.""" + shared = set(generated.columns) & set(reeds.columns) + keys = [column for column in PREFERRED_KEYS if column in shared] + if "t" not in keys and "t" in shared: + keys.append("t") + return keys + + +def compare_file( + generated_path: Path, + reeds_path: Path, + absolute_tolerance: float, + relative_tolerance: float, +) -> dict: + """Compare one generated/ReEDS file pair in memory.""" + result = { + "generated_file": generated_path.name, + "reeds_file": reeds_path.name, + "generated_rows": 0, + "reeds_rows": 0, + "key_columns": "", + "columns_only_generated": "", + "columns_only_reeds": "", + "duplicate_keys_generated": 0, + "duplicate_keys_reeds": 0, + "rows_only_generated": 0, + "rows_only_reeds": 0, + "shared_rows": 0, + "changed_cells": 0, + "max_absolute_difference": 0.0, + "max_relative_difference": 0.0, + "status": "", + } + + if not reeds_path.exists(): + result["status"] = "MISSING_REEDS_FILE" + return result + + generated = normalize_frame(pd.read_csv(generated_path)) + reeds = normalize_frame(pd.read_csv(reeds_path)) + result["generated_rows"] = len(generated) + result["reeds_rows"] = len(reeds) + + generated_columns = set(generated.columns) + reeds_columns = set(reeds.columns) + only_generated = sorted(generated_columns - reeds_columns) + only_reeds = sorted(reeds_columns - generated_columns) + result["columns_only_generated"] = ";".join(only_generated) + result["columns_only_reeds"] = ";".join(only_reeds) + + keys = choose_keys(generated, reeds) + result["key_columns"] = ";".join(keys) + if not keys: + result["status"] = "NO_SHARED_KEY" + return result + + result["duplicate_keys_generated"] = int(generated.duplicated(keys).sum()) + result["duplicate_keys_reeds"] = int(reeds.duplicated(keys).sum()) + if result["duplicate_keys_generated"] or result["duplicate_keys_reeds"]: + result["status"] = "DUPLICATE_KEYS" + return result + + value_columns = sorted((generated_columns & reeds_columns) - set(keys)) + merged = generated.merge( + reeds, + on=keys, + how="outer", + suffixes=("__generated", "__reeds"), + indicator=True, + ) + only_generated_rows = merged.loc[merged["_merge"] == "left_only", keys] + only_reeds_rows = merged.loc[merged["_merge"] == "right_only", keys] + shared = merged.loc[merged["_merge"] == "both"].copy() + result["rows_only_generated"] = len(only_generated_rows) + result["rows_only_reeds"] = len(only_reeds_rows) + result["shared_rows"] = len(shared) + + differences = [] + max_abs = 0.0 + max_rel = 0.0 + for column in value_columns: + generated_column = f"{column}__generated" + reeds_column = f"{column}__reeds" + generated_values = shared[generated_column] + reeds_values = shared[reeds_column] + if pd.api.types.is_numeric_dtype(generated_values) and pd.api.types.is_numeric_dtype(reeds_values): + generated_numeric = pd.to_numeric(generated_values, errors="coerce") + reeds_numeric = pd.to_numeric(reeds_values, errors="coerce") + equal = np.isclose( + generated_numeric, + reeds_numeric, + atol=absolute_tolerance, + rtol=relative_tolerance, + equal_nan=True, + ) + changed = shared.loc[~equal, keys].copy() + if not changed.empty: + changed["column"] = column + changed["generated_value"] = generated_numeric.loc[~equal].values + changed["reeds_value"] = reeds_numeric.loc[~equal].values + changed["absolute_difference"] = np.abs( + changed["generated_value"] - changed["reeds_value"] + ) + denominator = np.maximum(np.abs(changed["reeds_value"]), absolute_tolerance) + changed["relative_difference"] = changed["absolute_difference"] / denominator + max_abs = max(max_abs, float(changed["absolute_difference"].max())) + max_rel = max(max_rel, float(changed["relative_difference"].max())) + differences.append(changed) + else: + equal = generated_values.fillna("").eq(reeds_values.fillna("")) + changed = shared.loc[~equal, keys].copy() + if not changed.empty: + changed["column"] = column + changed["generated_value"] = generated_values.loc[~equal].values + changed["reeds_value"] = reeds_values.loc[~equal].values + changed["absolute_difference"] = np.nan + changed["relative_difference"] = np.nan + differences.append(changed) + + if differences: + difference_frame = pd.concat(differences, ignore_index=True) + result["changed_cells"] = len(difference_frame) + result["max_absolute_difference"] = max_abs + result["max_relative_difference"] = max_rel + + has_structure_difference = bool( + only_generated + or only_reeds + or result["rows_only_generated"] + or result["rows_only_reeds"] + ) + if has_structure_difference: + result["status"] = "STRUCTURE_DIFFERENCE" + elif result["changed_cells"]: + result["status"] = "VALUE_DIFFERENCE" + else: + result["status"] = "MATCH" + return result + + +def series_columns(frame: pd.DataFrame) -> list[str]: + """Return categorical columns that identify separate plotted series.""" + excluded = {"t", "rsc_mult"} + return [ + column + for column in frame.columns + if column not in excluded and not pd.api.types.is_numeric_dtype(frame[column]) + ] + + +def metric_columns(generated: pd.DataFrame, reeds: pd.DataFrame) -> list[str]: + """Return shared numeric output columns, with familiar metrics first.""" + excluded = {"t"} + shared = set(generated.columns) & set(reeds.columns) + numeric = [ + column + for column in shared + if column not in excluded + and pd.api.types.is_numeric_dtype(generated[column]) + and pd.api.types.is_numeric_dtype(reeds[column]) + ] + preferred = [column for column in METRIC_LABELS if column in numeric] + return preferred + sorted(set(numeric) - set(preferred)) + + +def series_label(group_values: tuple, columns: list[str]) -> str: + """Build a compact label for a plotted technology series.""" + if not columns: + return "all" + return ", ".join( + f"{column}={value}" for column, value in zip(columns, group_values) + ) + + +def grouped_series(frame: pd.DataFrame, columns: list[str]): + """Split a frame into time-series groups and labels.""" + if not columns: + return [("all", frame.sort_values("t"))] + groups = [] + grouper = columns[0] if len(columns) == 1 else columns + for values, group in frame.groupby(grouper, dropna=False, sort=True): + values_tuple = values if isinstance(values, tuple) else (values,) + groups.append((series_label(values_tuple, columns), group.sort_values("t"))) + return groups + + +def output_technology(filename: str, settings: dict) -> str: + """Return the internal technology key represented by an output filename.""" + matches = [] + for tech, tech_settings in settings["techs"].items(): + filename_root = tech_settings.get("reeds_name", tech) + if filename.startswith(f"{filename_root}_ATB_"): + matches.append(tech) + if len(matches) != 1: + raise ValueError( + f"Could not identify one technology for smoothing plot {filename}: " + f"{matches}" + ) + return matches[0] + + +_ATB_START_CACHE: dict = {} + + +def atb_start_year( + settings: dict, technology: str, default: int, series_label=None +) -> int: + """Return the first year ATB publishes one technology. + + ATB starts some technologies later than the release year, and a series is + historical until its own data begins. + """ + if not _ATB_START_CACHE: + flat = raw_file_path(settings["config"], "flat_file") + if not flat.is_file(): + _ATB_START_CACHE["_missing"] = True + else: + raw = pd.read_csv( + flat, + usecols=[ + "technology", + "display_name", + "core_metric_parameter", + "core_metric_variable", + ], + low_memory=False, + ) + # financial parameters span every year even where the cost and + # performance metrics start later, so only the mapped metrics + # decide when a series' ATB data begins + raw = raw.loc[ + raw["core_metric_parameter"].isin(settings["param_names"]) + ] + for tech, tech_settings in settings["techs"].items(): + names = tech_settings["DisplayName"] + labels = names if isinstance(names, dict) else {names: None} + rows = raw.loc[ + (raw["technology"] == tech_settings["Technology"]) + & (raw["display_name"].isin(list(labels))) + ] + if rows.empty: + continue + starts = {None: int(rows["core_metric_variable"].min())} + for display_name, label in labels.items(): + if label is None: + continue + series = rows.loc[rows["display_name"] == display_name] + if not series.empty: + starts[label] = int(series["core_metric_variable"].min()) + _ATB_START_CACHE[tech] = starts + starts = _ATB_START_CACHE.get(technology) + if not isinstance(starts, dict): + return default + return starts.get(series_label, starts.get(None, default)) + + +def smoothing_provenance(settings: dict, generated_path: Path) -> dict: + """Describe the source/treatment rules for one smoothing comparison.""" + technology = output_technology(generated_path.name, settings) + smoothing = settings["config"]["processing"].get("smooth_cost_curves", {}) + technology_settings = smoothing.get("technologies", {}).get(technology, {}) + if not isinstance(technology_settings, dict): + technology_settings = {} + historical_data = technology_settings.get( + "historical_data", {} + ) + mappings = ( + settings["config"] + .get("historical_cost_sources", {}) + .get("reeds_mappings", {}) + .get(technology, {}) + ) + source_settings = settings["config"].get("historical_cost_sources", {}) + source_path = resolve_atb_config_path( + Path(source_settings.get("directory", "")) + / source_settings.get("normalized_filename", "") + ) + observed = pd.read_csv(source_path) if source_path.is_file() else pd.DataFrame() + observed_series = {} + boundary = atb_start_year( + settings, technology, int(smoothing.get("projection_start_year", 2022)) + ) + default_boundary = int(smoothing.get("projection_start_year", 2022)) + for metric, mapping in mappings.items(): + entries = mapping.get("series") or [mapping] + inherited = { + key: value for key, value in mapping.items() if key != "series" + } + metric_series = [] + for entry in entries: + merged = {**inherited, **entry} + selected = observed + for column, value in merged.get("filters", {}).items(): + if column not in selected.columns: + selected = selected.iloc[0:0] + break + selected = selected.loc[selected[column] == value] + years = ( + set( + pd.to_numeric(selected["year"], errors="coerce") + .dropna() + .astype(int) + .tolist() + ) + if "year" in selected.columns + else set() + ) + target_column = None + targets = None + if merged.get("turbine_classes"): + target_column = "turbine" + targets = set(merged["turbine_classes"]) + elif merged.get("technologies"): + target_column = "i" + targets = set(merged["technologies"]) + metric_series.append({ + "years": years, + "target_column": target_column, + "targets": targets, + }) + observed_series[metric] = metric_series + return { + "technology": technology, + "projection_start_year": boundary, + "historical_data": historical_data, + "observed_series": observed_series, + "history_class_column": ( + settings["techs"].get(technology, {}).get("history_class_column") + ), + "settings": settings, + "default_boundary": default_boundary, + } + + +def resolve_historical_mode( + metric: str, + final_group: pd.DataFrame, + provenance: dict, +) -> str: + """Return one metric's history mode for the plotted sub-technology class. + + A metric may give one mode per class, so the mode depends on which class + this series plots rather than on the technology alone. + """ + historical_data = provenance["historical_data"] + if metric not in historical_data: + raise KeyError( + f"{provenance['technology']} has no historical_data entry for " + f"metric {metric!r}." + ) + configured = historical_data[metric] + if not isinstance(configured, dict): + return configured + class_column = provenance["history_class_column"] + if not class_column or class_column not in final_group.columns: + raise KeyError( + f"{provenance['technology']}.{metric} gives one history mode per " + f"sub-technology class, but the plotted series has no " + f"{class_column!r} column identifying which class it is." + ) + classes = set(final_group[class_column].dropna().unique()) + modes = {configured[c] for c in classes if c in configured} + if len(modes) != 1: + raise KeyError( + f"{provenance['technology']}.{metric} resolves to {sorted(modes)} " + f"for plotted classes {sorted(classes)}; one series must plot one " + "class so it has a single history mode." + ) + return modes.pop() + + +def is_observed_history_point( + final_group: pd.DataFrame, + metric: str, + year: int, + provenance: dict, +) -> bool: + """Return whether one plotted point was populated from observed history.""" + for series in provenance["observed_series"].get(metric, []): + if year not in series["years"]: + continue + target_column = series["target_column"] + if target_column is None: + return True + if target_column not in final_group.columns: + continue + group_targets = set(final_group[target_column].dropna().unique()) + if group_targets & series["targets"]: + return True + return False + + +def is_real_history_target( + final_group: pd.DataFrame, + metric: str, + provenance: dict, +) -> bool: + """Return whether a plotted series is covered by a selected real mapping.""" + for series in provenance["observed_series"].get(metric, []): + target_column = series["target_column"] + if target_column is None: + return True + if target_column not in final_group.columns: + continue + group_targets = set(final_group[target_column].dropna().unique()) + if group_targets & series["targets"]: + return True + return False + + +def series_boundary(final_group: pd.DataFrame, provenance: dict) -> int: + """Return the first ATB year for the series being plotted.""" + label = None + class_column = provenance["history_class_column"] + if class_column and class_column in final_group.columns: + values = set(final_group[class_column].dropna().unique()) + if len(values) == 1: + label = values.pop() + return atb_start_year( + provenance["settings"], + provenance["technology"], + provenance["default_boundary"], + label, + ) + + +def provenance_categories( + final_group: pd.DataFrame, + baseline_group: pd.DataFrame | None, + metric: str, + provenance: dict, +) -> list[str]: + """Label each final point by its exclusive data source or treatment.""" + years = pd.to_numeric(final_group["t"], errors="raise").astype(int) + final_values = pd.to_numeric(final_group[metric], errors="coerce") + if baseline_group is None: + baseline_values = pd.Series(np.nan, index=final_group.index) + else: + baseline_by_year = ( + baseline_group[["t", metric]] + .drop_duplicates("t", keep="last") + .set_index("t")[metric] + ) + baseline_values = years.map(baseline_by_year) + baseline_values.index = final_group.index + baseline_values = pd.to_numeric(baseline_values, errors="coerce") + + changed = ~np.isclose( + final_values, + baseline_values, + rtol=1e-9, + atol=1e-9, + equal_nan=True, + ) + boundary = series_boundary(final_group, provenance) + historical_mode = resolve_historical_mode(metric, final_group, provenance) + final_categories = [] + for year, was_changed in zip(years, changed): + if year < boundary: + if historical_mode == "manual": + final_categories.append("Manual history") + elif historical_mode == "broadcast": + final_categories.append("Broadcast history") + elif historical_mode == "real": + if not is_real_history_target(final_group, metric, provenance): + raise KeyError( + f"{provenance['technology']}.{metric} selects real " + "history, but this plotted series has no real mapping." + ) + final_categories.append( + "Observed history (real)" + if is_observed_history_point( + final_group, metric, int(year), provenance + ) + else "Filled real history" + ) + else: + raise KeyError( + f"Unknown historical mode for " + f"{provenance['technology']}.{metric}: {historical_mode!r}" + ) + else: + final_categories.append( + "ATB projection (smoothed)" + if was_changed + else "ATB projection (raw)" + ) + return final_categories + + +def input_point_categories( + final_group: pd.DataFrame, + metric: str, + provenance: dict, +) -> list[str | None]: + """Label input points; derived history and broadcast history have no dot.""" + years = pd.to_numeric(final_group["t"], errors="raise").astype(int) + boundary = series_boundary(final_group, provenance) + historical_mode = resolve_historical_mode(metric, final_group, provenance) + categories = [] + for year in years: + if year >= boundary: + categories.append("ATB projection (raw)") + elif historical_mode == "manual": + categories.append("Manual history") + elif historical_mode == "real" and is_observed_history_point( + final_group, metric, int(year), provenance + ): + categories.append("Observed history (real)") + else: + categories.append(None) + return categories + + +def plot_colored_segments( + axis, + years: np.ndarray, + values: np.ndarray, + categories: list[str], + linestyle, +) -> None: + """Plot intervals using the provenance color of their starting point.""" + if not len(years): + return + if len(years) == 1: + axis.plot( + years, + values, + color=PROVENANCE_COLORS[categories[0]], + linestyle=linestyle, + linewidth=2.0, + alpha=0.95, + zorder=2, + ) + return + + interval_categories = categories[:-1] + run_start = 0 + for position in range(1, len(interval_categories) + 1): + if ( + position < len(interval_categories) + and interval_categories[position] == interval_categories[run_start] + ): + continue + axis.plot( + years[run_start:position + 1], + values[run_start:position + 1], + color=PROVENANCE_COLORS[interval_categories[run_start]], + linestyle=linestyle, + linewidth=2.0, + alpha=0.95, + zorder=2, + ) + run_start = position + + +def plot_file_with_provenance( + generated_path: Path, + reeds_path: Path, + plot_dir: Path, + settings: dict, +) -> bool: + """Plot a smoothing comparison with color encoding source and treatment.""" + generated = normalize_frame(pd.read_csv(generated_path)) + baseline = normalize_frame(pd.read_csv(reeds_path)) + metrics = metric_columns(generated, baseline) + if not metrics or "t" not in generated.columns or "t" not in baseline.columns: + return False + + provenance = smoothing_provenance(settings, generated_path) + identifiers = sorted( + set(series_columns(generated)) | set(series_columns(baseline)) + ) + generated_groups = grouped_series(generated, identifiers) + baseline_groups = dict(grouped_series(baseline, identifiers)) + labels = sorted( + {label for label, _ in generated_groups} + | set(baseline_groups) + ) + series_styles = { + label: SERIES_LINESTYLES[index % len(SERIES_LINESTYLES)] + for index, label in enumerate(labels) + } + + columns = min(3, len(metrics)) + rows = math.ceil(len(metrics) / columns) + figure, axes = plt.subplots( + rows, + columns, + figsize=(5.4 * columns, 3.7 * rows), + squeeze=False, + constrained_layout=True, + ) + figure.suptitle( + generated_path.stem, + y=1.03, + fontsize=12, + fontweight="bold", + ) + + present_categories = set() + points_present = False + for axis, metric in zip(axes.flat, metrics): + for label, final_group in generated_groups: + baseline_group = baseline_groups.get(label) + final_categories = provenance_categories( + final_group, + baseline_group, + metric, + provenance, + ) + present_categories.update(final_categories) + years = final_group["t"].to_numpy() + values = final_group[metric].to_numpy() + plot_colored_segments( + axis, + years, + values, + final_categories, + series_styles[label], + ) + if baseline_group is not None: + point_categories = input_point_categories( + final_group, + metric, + provenance, + ) + baseline_by_year = ( + baseline_group[["t", metric]] + .drop_duplicates("t", keep="last") + .set_index("t")[metric] + ) + point_years = pd.to_numeric( + final_group["t"], errors="raise" + ).to_numpy() + point_values = pd.to_numeric( + pd.Series(point_years).map(baseline_by_year), + errors="coerce", + ).to_numpy() + for category in PROVENANCE_COLORS: + point_mask = np.asarray([ + item == category for item in point_categories + ]) & np.isfinite(point_values) + if not point_mask.any(): + continue + points_present = True + present_categories.add(category) + axis.scatter( + point_years[point_mask], + point_values[point_mask], + s=18, + marker="o", + facecolors=PROVENANCE_COLORS[category], + edgecolors="none", + alpha=0.95, + zorder=3, + ) + axis.axvline( + provenance["projection_start_year"], + color="0.35", + linestyle=":", + linewidth=1.0, + alpha=0.7, + ) + axis.set_title(METRIC_LABELS.get(metric, metric)) + axis.set_xlabel("Year") + axis.set_ylabel(METRIC_LABELS.get(metric, metric)) + axis.grid(True, alpha=0.25) + + for axis in axes.flat[len(metrics):]: + axis.remove() + + category_handles = [ + plt.Line2D( + [0], + [0], + color=color, + linewidth=2.5, + label=category, + ) + for category, color in PROVENANCE_COLORS.items() + if category in present_categories + ] + series_handles = [] + if labels != ["all"]: + series_handles = [ + plt.Line2D( + [0], + [0], + color="0.25", + linewidth=2, + linestyle=series_styles[label], + label=label, + ) + for label in labels + ] + point_handles = [] + if points_present: + point_handles.append( + plt.Line2D( + [0], + [0], + color="0.25", + linewidth=0, + marker="o", + markerfacecolor="0.25", + markeredgewidth=0, + label="Input data values", + ) + ) + handles = category_handles + point_handles + series_handles + figure.legend( + handles=handles, + loc="outside lower center", + ncol=min(4, len(handles)), + fontsize=8, + frameon=False, + ) + figure.savefig( + plot_dir / f"{generated_path.stem}.png", + dpi=160, + bbox_inches="tight", + ) + plt.close(figure) + return True + + +def plot_file( + generated_path: Path, + reeds_path: Path, + plot_dir: Path, + solid_label: str = "Generated", + dashed_label: str = "ReEDS", + reference_linestyle: str = "--", + reference_marker: str | None = None, +) -> bool: + """Plot all shared numeric metrics for one generated/ReEDS file pair.""" + generated = normalize_frame(pd.read_csv(generated_path)) + reeds = normalize_frame(pd.read_csv(reeds_path)) + metrics = metric_columns(generated, reeds) + if not metrics or "t" not in generated.columns or "t" not in reeds.columns: + return False + + identifiers = sorted(set(series_columns(generated)) | set(series_columns(reeds))) + generated_groups = grouped_series(generated, identifiers) + reeds_groups = grouped_series(reeds, identifiers) + labels = sorted({label for label, _ in generated_groups + reeds_groups}) + colors = {label: plt.cm.tab10(index % 10) for index, label in enumerate(labels)} + + columns = min(3, len(metrics)) + rows = math.ceil(len(metrics) / columns) + figure, axes = plt.subplots( + rows, + columns, + figsize=(5.4 * columns, 3.7 * rows), + squeeze=False, + constrained_layout=True, + ) + figure.suptitle( + generated_path.stem, + y=1.03, + fontsize=12, + fontweight="bold", + ) + + for axis, metric in zip(axes.flat, metrics): + for label, group in reeds_groups: + axis.plot( + group["t"], + group[metric], + color=colors[label], + linestyle=reference_linestyle, + marker=reference_marker, + markersize=3.2 if reference_marker else None, + linewidth=2.0, + alpha=0.9, + ) + for label, group in generated_groups: + axis.plot( + group["t"], + group[metric], + color=colors[label], + linestyle="-", + linewidth=1.6, + alpha=0.9, + ) + axis.set_title(METRIC_LABELS.get(metric, metric)) + axis.set_xlabel("Year") + axis.set_ylabel(METRIC_LABELS.get(metric, metric)) + axis.grid(True, alpha=0.25) + + for axis in axes.flat[len(metrics):]: + axis.remove() + + source_handles = [ + plt.Line2D( + [0], [0], color="0.25", linewidth=2, linestyle="-", label=solid_label + ), + plt.Line2D( + [0], + [0], + color="0.25", + linewidth=2, + linestyle=reference_linestyle, + marker=reference_marker, + markersize=4 if reference_marker else None, + label=dashed_label, + ), + ] + series_handles = [ + plt.Line2D([0], [0], color=colors[label], linewidth=2, label=label) + for label in labels + ] + figure.legend( + handles=source_handles + series_handles, + loc="outside lower center", + ncol=min(4, len(source_handles + series_handles)), + fontsize=8, + frameon=False, + ) + figure.savefig( + plot_dir / f"{generated_path.stem}.png", + dpi=160, + bbox_inches="tight", + ) + plt.close(figure) + return True + + +def plot_overview(summary: pd.DataFrame, plot_dir: Path, atb_year: int) -> None: + """Plot file-level validation status and difference counts.""" + plot_data = summary.sort_values( + ["status", "changed_cells"], ascending=[True, True] + ).reset_index(drop=True) + labels = plot_data["generated_file"].str.replace( + "_ATB_2024_", " | ", regex=False + ) + y = np.arange(len(plot_data)) + + figure, (axis_status, axis_counts) = plt.subplots( + 1, + 2, + figsize=(18, 14), + gridspec_kw={"width_ratios": [0.8, 2.2]}, + constrained_layout=True, + ) + figure.suptitle( + f"ATB {atb_year} output comparison overview", + fontsize=16, + fontweight="bold", + ) + status_colors = { + "MATCH": "#2a9d8f", + "VALUE_DIFFERENCE": "#e9c46a", + "STRUCTURE_DIFFERENCE": "#e76f51", + "MISSING_REEDS_FILE": "#9b2226", + } + for index, row in plot_data.iterrows(): + axis_status.barh( + index, 1, color=status_colors.get(row["status"], "0.6") + ) + axis_status.text( + 0.5, + index, + row["status"].replace("_", " "), + ha="center", + va="center", + fontsize=8, + ) + axis_status.set_yticks(y, labels) + axis_status.set_xlim(0, 1) + axis_status.set_xticks([]) + axis_status.set_title("Validation status") + axis_status.invert_yaxis() + + axis_counts.barh( + y, + plot_data["changed_cells"], + label="Changed cells", + color="#457b9d", + ) + axis_counts.barh( + y, + plot_data["rows_only_reeds"], + left=plot_data["changed_cells"], + label="Rows only in ReEDS", + color="#e76f51", + ) + axis_counts.set_yticks([]) + axis_counts.set_xlabel("Difference count") + axis_counts.set_title("Changed cells and missing generated rows") + axis_counts.grid(True, axis="x", alpha=0.25) + axis_counts.invert_yaxis() + axis_counts.legend(frameon=False, loc="lower right") + figure.savefig(plot_dir / "comparison_overview.png", dpi=160) + plt.close(figure) + + +def write_plots( + summary: pd.DataFrame | None, + generated_files: list[Path], + reeds_dir: Path, + plot_dir: Path, + atb_year: int, + solid_label: str = "Generated", + dashed_label: str = "ReEDS", + include_overview: bool = True, + reference_linestyle: str = "--", + reference_marker: str | None = None, + provenance_settings: dict | None = None, +) -> int: + """Generate the overview and all available file-level comparison plots.""" + plot_dir.mkdir(parents=True, exist_ok=True) + for old_plot in plot_dir.glob("*.png"): + old_plot.unlink() + if include_overview: + if summary is None: + raise ValueError("A comparison summary is required for the overview plot.") + plot_overview(summary, plot_dir, atb_year) + + plotted = 0 + skipped = [] + for generated_path in generated_files: + reeds_path = reeds_dir / reeds_filename(generated_path.name) + if reeds_path.exists(): + if provenance_settings is None: + wrote_plot = plot_file( + generated_path, + reeds_path, + plot_dir, + solid_label=solid_label, + dashed_label=dashed_label, + reference_linestyle=reference_linestyle, + reference_marker=reference_marker, + ) + else: + wrote_plot = plot_file_with_provenance( + generated_path, + reeds_path, + plot_dir, + provenance_settings, + ) + if wrote_plot: + plotted += 1 + else: + skipped.append(generated_path.name) + if skipped: + raise RuntimeError( + "Could not plot generated/ReEDS file pairs because they have no " + f"shared year-based numeric metrics: {skipped}" + ) + return plotted + + +def main() -> None: + parser = argparse.ArgumentParser( + description="Compare generated ATB outputs with current ReEDS inputs." + ) + parser.add_argument("--config", type=Path, default=DEFAULT_CONFIG_PATH) + parser.add_argument("--generated-dir", type=Path) + parser.add_argument( + "--unsmoothed-dir", + type=Path, + help="temporary pre-smoothing outputs supplied by the full pipeline", + ) + parser.add_argument("--reeds-dir", type=Path) + parser.add_argument("--plot-dir", type=Path, default=DEFAULT_PLOT_DIR) + parser.add_argument( + "--absolute-tolerance", + type=float, + default=2e-4, + help=( + "Absolute numeric tolerance (default: 2e-4, reflecting the " + "four-decimal precision of some raw ATB values)" + ), + ) + parser.add_argument("--relative-tolerance", type=float, default=1e-9) + args = parser.parse_args() + + settings = load_processing_settings(args.config) + smoothing = settings['config']['processing'].get('smooth_cost_curves', {}) + baseline_dir = args.unsmoothed_dir.resolve() if args.unsmoothed_dir else None + if args.generated_dir: + generated_dir = args.generated_dir.resolve() + else: + generated_dir = Path(settings['output_dir']).resolve() + reeds_dir = ( + args.reeds_dir.resolve() + if args.reeds_dir + else Path(settings['reedspath']).resolve() / "inputs" / "plant_characteristics" + ) + plot_dir = args.plot_dir.resolve() + atb_year = int(settings['atbyear']) + + generated_files = sorted(generated_dir.glob(f"*_ATB_{atb_year}_*.csv")) + if not generated_files: + raise FileNotFoundError( + f"No ATB {atb_year} outputs found in {generated_dir}" + ) + if not reeds_dir.is_dir(): + raise FileNotFoundError(f"ReEDS plant-characteristics directory not found: {reeds_dir}") + + results = [] + for generated_path in generated_files: + reeds_path = reeds_dir / reeds_filename(generated_path.name) + result = compare_file( + generated_path, + reeds_path, + args.absolute_tolerance, + args.relative_tolerance, + ) + results.append(result) + print( + f"{result['status']:>20} {generated_path.name} -> {reeds_path.name}" + ) + + summary = pd.DataFrame(results) + expected_reeds_names = {reeds_filename(path.name) for path in generated_files} + reeds_only_files = sorted( + path.name + for path in reeds_dir.glob(f"*_ATB_{atb_year}_*.csv") + if path.name not in expected_reeds_names + ) + plotted = write_plots( + summary, + generated_files, + reeds_dir, + plot_dir, + atb_year, + solid_label="Generated", + dashed_label="ReEDS", + ) + print(f"\nWrote overview and {plotted} file-level plots to {plot_dir}") + print(summary["status"].value_counts().to_string()) + print( + f"ReEDS files without a generated counterpart: {len(reeds_only_files)}" + ) + for filename in reeds_only_files: + print(f" {filename}") + + if smoothing.get('enabled', False) and baseline_dir is not None: + smoothed_dir = Path(settings['output_dir']).resolve() + smoothed_files = sorted( + smoothed_dir.glob(f"*_ATB_{atb_year}_*.csv") + ) + smoothing_plot_setting = settings['config']['plotting'].get( + 'smoothing_comparison_directory', + 'comparison/smoothing_comparison', + ) + smoothing_plot_dir = resolve_atb_config_path(smoothing_plot_setting) + smoothing_plotted = write_plots( + None, + smoothed_files, + baseline_dir, + smoothing_plot_dir, + atb_year, + solid_label="After smoothing", + dashed_label="Before smoothing", + include_overview=False, + reference_linestyle="None", + reference_marker="o", + provenance_settings=settings, + ) + print( + f"Wrote {smoothing_plotted} before/after smoothing plots to " + f"{smoothing_plot_dir}" + ) + + +if __name__ == "__main__": + main() diff --git a/atb/config.schema.json b/atb/config.schema.json new file mode 100644 index 0000000..fc829d0 --- /dev/null +++ b/atb/config.schema.json @@ -0,0 +1,556 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$id": "https://github.com/NREL/ReEDS-2.0/atb/config.schema.json", + "title": "ReEDS ATB workflow configuration", + "description": "Editor validation and option lists for atb/config.yaml.", + "type": "object", + "additionalProperties": false, + "required": [ + "workflow", + "atb", + "raw_data", + "historical_cost_sources", + "processing", + "historical_data", + "plotting" + ], + "properties": { + "workflow": { + "type": "object", + "additionalProperties": false, + "required": [ + "scrape_raw_data", + "format_reeds_inputs", + "make_plots", + "make_comparison_plots" + ], + "properties": { + "scrape_raw_data": {"type": "boolean"}, + "format_reeds_inputs": {"type": "boolean"}, + "make_plots": {"type": "boolean"}, + "make_comparison_plots": {"type": "boolean"} + } + }, + "atb": { + "type": "object", + "additionalProperties": false, + "required": ["year", "release_version", "dollar_year"], + "properties": { + "year": {"type": "integer", "minimum": 2015}, + "release_version": {"type": "integer", "minimum": 1}, + "dollar_year": {"type": "integer", "minimum": 1900} + } + }, + "raw_data": { + "type": "object", + "additionalProperties": false, + "required": ["directory", "flat_file", "workbook"], + "properties": { + "directory": {"type": "string", "minLength": 1}, + "allow_insecure_ssl_fallback": {"type": "boolean"}, + "flat_file": {"$ref": "#/definitions/downloadArtifact"}, + "workbook": {"$ref": "#/definitions/downloadArtifact"} + } + }, + "historical_cost_sources": { + "type": "object", + "additionalProperties": false, + "required": [ + "directory", + "normalized_filename", + "manifest_filename", + "reeds_mappings", + "sources" + ], + "properties": { + "directory": {"type": "string", "minLength": 1}, + "normalized_filename": {"type": "string", "minLength": 1}, + "manifest_filename": {"type": "string", "minLength": 1}, + "allow_insecure_ssl_fallback": {"type": "boolean"}, + "reeds_mappings": {"$ref": "#/definitions/reedsMappings"}, + "sources": { + "type": "object", + "additionalProperties": {"$ref": "#/definitions/historicalSource"} + } + } + }, + "processing": { + "type": "object", + "additionalProperties": false, + "required": [ + "reeds_repo", + "output_directory", + "copy_to_reeds", + "technologies", + "update_costs", + "update_financials", + "reeds_start_year", + "decimals", + "smooth_cost_curves" + ], + "properties": { + "reeds_repo": {"type": "string", "minLength": 1}, + "output_directory": {"type": "string", "minLength": 1}, + "copy_to_reeds": {"type": "boolean"}, + "technologies": { + "description": "Technologies formatted by the workflow.", + "oneOf": [ + {"const": "all"}, + { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": {"$ref": "#/definitions/technologyName"} + } + ] + }, + "update_costs": {"type": "boolean"}, + "update_financials": {"type": "boolean"}, + "sensitivity_name": {"type": ["string", "null"]}, + "reeds_start_year": {"type": "integer", "minimum": 1900}, + "decimals": {"type": "integer", "minimum": 0, "maximum": 12}, + "smooth_cost_curves": {"$ref": "#/definitions/smoothing"} + } + }, + "historical_data": { + "type": "object", + "additionalProperties": false, + "required": ["directory", "dollar_year", "seed_missing_from_reeds"], + "properties": { + "directory": {"type": "string", "minLength": 1}, + "dollar_year": {"type": "integer", "minimum": 1900}, + "seed_missing_from_reeds": {"type": "boolean"} + } + }, + "plotting": { + "type": "object", + "additionalProperties": false, + "required": [ + "metrics", + "case", + "crp_years", + "figure_format", + "output_directory", + "smoothing_comparison_directory", + "technologies" + ], + "properties": { + "metrics": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "enum": ["CAPEX", "OCC", "Fixed O&M", "Variable O&M"] + } + }, + "case": {"enum": ["Market", "R&D"]}, + "crp_years": {"type": "integer", "minimum": 1}, + "figure_format": {"enum": ["png", "pdf", "svg"]}, + "output_directory": {"type": "string", "minLength": 1}, + "smoothing_comparison_directory": {"type": "string", "minLength": 1}, + "technologies": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "enum": [ + "Biopower", + "Coal_FE", + "CSP", + "LandbasedWind", + "NaturalGas_FE", + "Nuclear", + "OffShoreWind", + "Utility-Scale Battery Storage", + "UtilityPV" + ] + } + } + } + } + }, + "definitions": { + "downloadArtifact": { + "type": "object", + "additionalProperties": false, + "required": ["filename", "url"], + "properties": { + "filename": {"type": "string", "minLength": 1}, + "url": {"type": "string", "minLength": 1} + } + }, + "historicalSource": { + "type": "object", + "additionalProperties": false, + "required": ["enabled", "page_url", "filename"], + "properties": { + "enabled": {"type": "boolean"}, + "report_year": {"type": "integer", "minimum": 1900}, + "first_year": {"type": "integer", "minimum": 1900}, + "last_year": {"type": "integer", "minimum": 1900}, + "last_historical_year": {"type": "integer", "minimum": 1900}, + "page_url": {"type": "string", "minLength": 1}, + "data_url": {"type": "string", "minLength": 1}, + "current_data_url": {"type": "string", "minLength": 1}, + "archive_data_url": {"type": "string", "minLength": 1}, + "filename": {"type": "string", "minLength": 1} + }, + "anyOf": [ + {"required": ["data_url"]}, + {"required": ["current_data_url", "archive_data_url"]} + ] + }, + "filterValue": { + "type": ["string", "number", "integer", "boolean"] + }, + "filters": { + "type": "object", + "minProperties": 1, + "additionalProperties": {"$ref": "#/definitions/filterValue"} + }, + "mappingSeries": { + "type": "object", + "additionalProperties": false, + "required": ["filters"], + "properties": { + "filters": {"$ref": "#/definitions/filters"}, + "technologies": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": {"type": "string"} + }, + "turbine_classes": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": {"type": "string"} + } + } + }, + "observedMapping": { + "type": "object", + "additionalProperties": false, + "properties": { + "filters": {"$ref": "#/definitions/filters"}, + "turbine_classes": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": {"type": "string"} + }, + "series": { + "type": "array", + "minItems": 1, + "items": {"$ref": "#/definitions/mappingSeries"} + } + }, + "oneOf": [ + {"required": ["filters"]}, + {"required": ["series"]} + ] + }, + "metricMappings": { + "type": "object", + "additionalProperties": false, + "required": ["capcost"], + "properties": { + "capcost": {"$ref": "#/definitions/observedMapping"} + } + }, + "reedsMappings": { + "type": "object", + "additionalProperties": false, + "properties": { + "upv": {"$ref": "#/definitions/metricMappings"}, + "wind-ons": {"$ref": "#/definitions/metricMappings"}, + "wind-ofs": {"$ref": "#/definitions/metricMappings"}, + "gas": {"$ref": "#/definitions/metricMappings"}, + "biopower": {"$ref": "#/definitions/metricMappings"} + } + }, + "historyMode": { + "enum": ["real", "manual", "broadcast"] + }, + "historyModeWithoutReal": { + "enum": ["manual", "broadcast"] + }, + "historyModeByTurbineClass": { + "type": "object", + "additionalProperties": false, + "minProperties": 1, + "description": "One history mode per turbine class. Use when an observed series describes only some classes; each class selecting real must match the mapping's turbine_classes. Only fixed offers real: NREL's Figure 31 CapEx series is fixed-bottom, and no observed floating cost history exists.", + "properties": { + "fixed": {"$ref": "#/definitions/historyMode"}, + "floating": {"$ref": "#/definitions/historyModeWithoutReal"} + } + }, + "historyModeOrByTurbineClass": { + "oneOf": [ + {"$ref": "#/definitions/historyMode"}, + {"$ref": "#/definitions/historyModeByTurbineClass"} + ] + }, + "historyModeByGasPlant": { + "type": "object", + "additionalProperties": false, + "minProperties": 1, + "description": "One history mode per gas plant configuration. EIA reports only 'combined cycle' and 'combustion turbine', so just Gas-CC and Gas-CT offer real; the H-Frame and aeroderivative variants have no observation and must generate their own history.", + "properties": { + "Gas-CC": {"$ref": "#/definitions/historyMode"}, + "Gas-CT": {"$ref": "#/definitions/historyMode"}, + "Gas-CC_H_1x1": {"$ref": "#/definitions/historyModeWithoutReal"}, + "Gas-CC_H_2x1": {"$ref": "#/definitions/historyModeWithoutReal"}, + "Gas-CT_aero": {"$ref": "#/definitions/historyModeWithoutReal"} + } + }, + "historyModeOrByGasPlant": { + "oneOf": [ + {"$ref": "#/definitions/historyMode"}, + {"$ref": "#/definitions/historyModeByGasPlant"} + ] + }, + "historyObservedCapcostWithCf": { + "type": "object", + "additionalProperties": false, + "required": ["capcost", "fom", "vom", "cf_improvement"], + "properties": { + "capcost": {"$ref": "#/definitions/historyMode"}, + "fom": {"$ref": "#/definitions/historyModeWithoutReal"}, + "vom": {"$ref": "#/definitions/historyModeWithoutReal"}, + "cf_improvement": {"$ref": "#/definitions/historyModeWithoutReal"} + } + }, + "historyObservedCapcostWithCfAndRsc": { + "type": "object", + "additionalProperties": false, + "required": [ + "capcost", + "fom", + "vom", + "cf_improvement", + "rsc_mult" + ], + "properties": { + "capcost": {"$ref": "#/definitions/historyModeOrByTurbineClass"}, + "fom": {"$ref": "#/definitions/historyModeWithoutReal"}, + "vom": {"$ref": "#/definitions/historyModeWithoutReal"}, + "cf_improvement": {"$ref": "#/definitions/historyModeWithoutReal"}, + "rsc_mult": {"$ref": "#/definitions/historyModeWithoutReal"} + } + }, + "historyObservedCapcost": { + "type": "object", + "additionalProperties": false, + "required": ["capcost", "fom", "vom", "heatrate"], + "properties": { + "capcost": {"$ref": "#/definitions/historyModeOrByGasPlant"}, + "fom": {"$ref": "#/definitions/historyModeWithoutReal"}, + "vom": {"$ref": "#/definitions/historyModeWithoutReal"}, + "heatrate": {"$ref": "#/definitions/historyModeWithoutReal"} + } + }, + "historyBattery": { + "type": "object", + "additionalProperties": false, + "required": [ + "capcost", + "capcost_energy", + "fom", + "fom_energy", + "vom", + "rte" + ], + "properties": { + "capcost": {"$ref": "#/definitions/historyModeWithoutReal"}, + "capcost_energy": {"$ref": "#/definitions/historyModeWithoutReal"}, + "fom": {"$ref": "#/definitions/historyModeWithoutReal"}, + "fom_energy": {"$ref": "#/definitions/historyModeWithoutReal"}, + "vom": {"$ref": "#/definitions/historyModeWithoutReal"}, + "rte": {"$ref": "#/definitions/historyModeWithoutReal"} + } + }, + "historyCosts": { + "type": "object", + "additionalProperties": false, + "required": ["capcost", "fom", "vom"], + "properties": { + "capcost": {"$ref": "#/definitions/historyModeWithoutReal"}, + "fom": {"$ref": "#/definitions/historyModeWithoutReal"}, + "vom": {"$ref": "#/definitions/historyModeWithoutReal"} + } + }, + "historyCostsWithHeatrate": { + "type": "object", + "additionalProperties": false, + "required": ["capcost", "fom", "vom", "heatrate"], + "properties": { + "capcost": {"$ref": "#/definitions/historyModeWithoutReal"}, + "fom": {"$ref": "#/definitions/historyModeWithoutReal"}, + "vom": {"$ref": "#/definitions/historyModeWithoutReal"}, + "heatrate": {"$ref": "#/definitions/historyModeWithoutReal"} + } + }, + "futureTreatments": { + "type": "object", + "additionalProperties": false, + "required": ["smooth_projection_curve"], + "properties": { + "smooth_projection_curve": {"type": "boolean"} + } + }, + "technologyConfigBase": { + "type": "object", + "additionalProperties": false, + "required": ["historical_data", "future_smoothing_treatments"], + "properties": { + "historical_data": {"type": "object"}, + "future_smoothing_treatments": {"$ref": "#/definitions/futureTreatments"} + } + }, + "technologyObservedCapcost": { + "allOf": [ + {"$ref": "#/definitions/technologyConfigBase"}, + { + "properties": { + "historical_data": {"$ref": "#/definitions/historyObservedCapcost"} + } + } + ] + }, + "technologyObservedCapcostWithCf": { + "allOf": [ + {"$ref": "#/definitions/technologyConfigBase"}, + { + "properties": { + "historical_data": {"$ref": "#/definitions/historyObservedCapcostWithCf"} + } + } + ] + }, + "technologyObservedCapcostWithCfAndRsc": { + "allOf": [ + {"$ref": "#/definitions/technologyConfigBase"}, + { + "properties": { + "historical_data": {"$ref": "#/definitions/historyObservedCapcostWithCfAndRsc"} + } + } + ] + }, + "technologyBattery": { + "allOf": [ + {"$ref": "#/definitions/technologyConfigBase"}, + { + "properties": { + "historical_data": {"$ref": "#/definitions/historyBattery"} + } + } + ] + }, + "technologyCosts": { + "allOf": [ + {"$ref": "#/definitions/technologyConfigBase"}, + { + "properties": { + "historical_data": {"$ref": "#/definitions/historyCosts"} + } + } + ] + }, + "technologyCostsWithHeatrate": { + "allOf": [ + {"$ref": "#/definitions/technologyConfigBase"}, + { + "properties": { + "historical_data": {"$ref": "#/definitions/historyCostsWithHeatrate"} + } + } + ] + }, + "technologyName": { + "enum": [ + "upv", + "wind-ons", + "wind-ofs", + "battery", + "csp", + "coal", + "coal-ccs", + "gas", + "gas-ccs", + "fuelcell", + "nuclear", + "nuclear-smr", + "biopower" + ] + }, + "smoothing": { + "type": "object", + "additionalProperties": false, + "required": [ + "enabled", + "method", + "columns", + "projection_start_year", + "slope_change_threshold", + "max_kink_years", + "major_step_relative_threshold", + "minimum_adjustment_relative_threshold", + "technologies" + ], + "properties": { + "enabled": {"type": "boolean"}, + "method": {"enum": ["selective", "linear_bridge"]}, + "columns": { + "oneOf": [ + {"const": "all"}, + {"const": "capital_costs"}, + { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "enum": [ + "capcost", + "capcost_energy", + "fom", + "fom_energy", + "vom", + "cf_improvement" + ] + } + } + ] + }, + "projection_start_year": {"type": "integer", "minimum": 1900}, + "anchor_year": {"type": "integer", "minimum": 1900}, + "target_year": {"type": "integer", "minimum": 1900}, + "slope_change_threshold": {"type": "number", "exclusiveMinimum": 0}, + "max_kink_years": {"type": "integer", "minimum": 1}, + "major_step_relative_threshold": {"type": "number", "exclusiveMinimum": 0}, + "minimum_adjustment_relative_threshold": {"type": "number", "minimum": 0}, + "future_smoothing_treatments": {"$ref": "#/definitions/futureTreatments"}, + "technologies": { + "type": "object", + "additionalProperties": false, + "properties": { + "upv": {"$ref": "#/definitions/technologyObservedCapcostWithCf"}, + "wind-ons": {"$ref": "#/definitions/technologyObservedCapcostWithCf"}, + "wind-ofs": {"$ref": "#/definitions/technologyObservedCapcostWithCfAndRsc"}, + "battery": {"$ref": "#/definitions/technologyBattery"}, + "csp": {"$ref": "#/definitions/technologyCosts"}, + "coal": {"$ref": "#/definitions/technologyCostsWithHeatrate"}, + "coal-ccs": {"$ref": "#/definitions/technologyCostsWithHeatrate"}, + "gas": {"$ref": "#/definitions/technologyObservedCapcost"}, + "gas-ccs": {"$ref": "#/definitions/technologyCostsWithHeatrate"}, + "fuelcell": {"$ref": "#/definitions/technologyCostsWithHeatrate"}, + "nuclear": {"$ref": "#/definitions/technologyCostsWithHeatrate"}, + "nuclear-smr": {"$ref": "#/definitions/technologyCostsWithHeatrate"}, + "biopower": {"$ref": "#/definitions/technologyObservedCapcost"} + } + } + } + } + } +} diff --git a/atb/config.yaml b/atb/config.yaml new file mode 100644 index 0000000..c77b42a --- /dev/null +++ b/atb/config.yaml @@ -0,0 +1,343 @@ +# yaml-language-server: $schema=./config.schema.json +# User-facing configuration for the ATB workflow. +# Set the workflow switches, review the paths/URLs, then run: +# python scripts/run_pipeline.py + +workflow: + scrape_raw_data: true + format_reeds_inputs: true + make_plots: true + make_comparison_plots: true + +atb: + year: 2024 + release_version: 3 + dollar_year: 2022 + +raw_data: + directory: scraped_input + # Some NLR networks inspect HTTPS with an internal certificate that may not + # be installed in the active conda environment. Normal certificate checking + # is always attempted first; this permits a warning + verify=False retry only + # after that check fails. + allow_insecure_ssl_fallback: true + flat_file: + filename: atb_2024_flat_file.csv + url: https://oedi-data-lake.s3.amazonaws.com/ATB/electricity/csv/2024/v3.0.0/ATBe.csv + workbook: + filename: atb_2024_workbook.xlsx + url: https://data.openei.org/files/6006/2024_v3_Workbook.xlsx + +# Observed project costs are downloaded and normalized by the separate +# scripts/scrape_historical_costs.py utility. Only explicitly reviewed entries +# under reeds_mappings are merged into ReEDS curves; all others remain review +# inputs. +historical_cost_sources: + directory: scraped_input/historical_costs + normalized_filename: historical_capital_costs.csv + manifest_filename: source_manifest.csv + allow_insecure_ssl_fallback: true + # Reviewed mappings from normalized observed series into ReEDS columns. + # Values before processing.smooth_cost_curves.projection_start_year replace + # the corresponding manual value when that metric selects `real`. Every + # historical metric declares its own mode below. + reeds_mappings: + upv: + capcost: + filters: + source_id: utility_pv + metric: capital_cost + unit: USD/kW + capacity_basis: AC + statistic: capacity_weighted_mean + geography: United States + # EIA reported no biomass installations in 2018, so that internal gap is + # linearly interpolated from 2017 and 2019. Years before EIA begins in 2013 + # use the nearest real observation; no manual values are mixed into `real`. + # Annual biomass additions are small and idiosyncratic, so the observed + # capacity-weighted mean swings widely (1,889-4,910 $/kW over 2013-2021). + biopower: + capcost: + filters: + source_id: eia_generator_costs + metric: capital_cost + unit: USD/kW + technology_detail: Biomass + source_table: major_energy_source + # EIA reports gas plants only as "combined cycle" or "combustion turbine"; + # Form EIA-860 has no field for turbine frame class or plant configuration, + # so there is no observation for the H-Frame or aeroderivative variants. + # Each series therefore describes the one variant it measures, and the + # others generate their own history under smooth_cost_curves below. EIA + # starts in 2013, so 2010-2012 use the nearest real observation. + gas: + capcost: + series: + - technologies: + - Gas-CC + filters: + source_id: eia_generator_costs + metric: capital_cost + unit: USD/kW + technology_detail: Natural gas combined cycle + source_table: natural_gas_technology + - technologies: + - Gas-CT + filters: + source_id: eia_generator_costs + metric: capital_cost + unit: USD/kW + technology_detail: Natural gas combustion turbine + source_table: natural_gas_technology + # Figure 31 reports one capacity-weighted mean over all offshore projects + # with no substructure split, but by 2023 only 231 MW of the ~75 GW built + # worldwide was floating (report Figure 18), so the series describes + # fixed-bottom costs. It is mapped to `fixed` only; `floating` broadcasts + # its own first ATB projection value under smooth_cost_curves below. + # scraped_input/README.md covers the "Global" and "Asia" alternatives. + wind-ofs: + capcost: + turbine_classes: + - fixed + filters: + source_id: offshore_wind + metric: capital_cost + unit: USD/kW + capacity_basis: nameplate + statistic: capacity_weighted_mean + geography: Europe and United States + wind-ons: + capcost: + filters: + source_id: land_based_wind + metric: capital_cost + unit: USD/kW + capacity_basis: nameplate + statistic: capacity_weighted_mean + geography: United States + sources: + land_based_wind: + enabled: true + report_year: 2025 + page_url: https://emp.lbl.gov/publications/land-based-wind-energy-technology + data_url: https://eta-publications.lbl.gov/sites/default/files/2025-08/data_file_land-based_wind_energy_technology_update_2025_edition.xlsx + filename: lbnl_land_based_wind_2025.xlsx + utility_pv: + enabled: true + report_year: 2025 + page_url: https://emp.lbl.gov/publications/us-utility-scale-solar-2025-data + data_url: https://eta-publications.lbl.gov/sites/default/files/2025-10/2025_utility-scale_solar_data_update.xlsx + filename: lbnl_utility_scale_solar_2025.xlsx + offshore_wind: + enabled: true + report_year: 2024 + # Projects after 2023 in this workbook are pipeline/projection data, not + # completed historical observations, so the normalized CSV stops at 2023. + last_historical_year: 2023 + page_url: https://www.nlr.gov/wind/offshore-market-assessment + data_url: https://www.nlr.gov/docs/fy24osti/90525data.xlsx + filename: nlr_offshore_wind_market_2024.xlsx + eia_generator_costs: + enabled: true + first_year: 2013 + last_year: 2024 + page_url: https://www.eia.gov/electricity/generatorcosts/ + current_data_url: https://www.eia.gov/electricity/generatorcosts/xls/generator_costs.xlsx + archive_data_url: https://www.eia.gov/electricity/generatorcosts/archive/{year}/xls/generator_costs_{year}.xlsx + filename: eia_generator_costs_{year}.xlsx + +processing: + reeds_repo: C:/ReEDS/ReEDS + output_directory: output + copy_to_reeds: false + technologies: all + update_costs: true + update_financials: false + sensitivity_name: null + reeds_start_year: 2010 + decimals: 6 + # Optional treatment of the historical/projection boundary and future ATB + # trajectory. Each technology explicitly selects a historical mode for every + # modeled metric and its future-year smoothing switches. + smooth_cost_curves: + enabled: true + method: selective + # "all" smooths every monetary column (capcost*, fom*, vom); rounding + # artifacts are no more real in O&M than in capital cost. Historical source + # choices below apply to every metric independently of this selector. + # Capacity-factor multipliers are still included automatically where used. + # Heat rate is not monetary and is never smoothed by "all". + columns: all + # Years before this boundary are historical; this year and later are the + # current ATB projection. Change this once here rather than in the code. + projection_start_year: 2022 + # A relative change in adjacent annual slopes at or above this value is a + # possible kink. Only clusters of 2+ changes are smoothed. + slope_change_threshold: 0.4 + max_kink_years: 4 + # Year-to-year transitions of 10% or more are published milestones and are + # never smoothed. Runs of equal values are left alone too: a flat stretch + # is how ATB states that a cost stops improving. + major_step_relative_threshold: 0.1 + # Keep the original ATB value when a proposed future adjustment is smaller + # than 0.5%. This prevents visually insignificant smoothing changes. + minimum_adjustment_relative_threshold: 0.005 + # historical_data values (editor dropdowns are supplied by the schema): + # real use observed history; interpolate internal missing years + # manual preserve manual_input/historical exactly + # broadcast use the first ATB projection value for all history years + # `real` is offered only where a reviewed mapping exists. The defaults use + # `real` where reviewed observations exist and `broadcast` everywhere else. + # Every technology listed below is processed. Capacity-factor multipliers + # are included automatically when the technology output contains one. + technologies: + upv: + historical_data: + capcost: real # options: real / manual / broadcast + fom: manual # options: manual / broadcast + vom: manual # options: manual / broadcast + cf_improvement: manual # options: manual / broadcast + future_smoothing_treatments: + smooth_projection_curve: true # options: true / false + wind-ons: + historical_data: + capcost: real # options: real / manual / broadcast + fom: manual # options: manual / broadcast + vom: manual # options: manual / broadcast + cf_improvement: manual # options: manual / broadcast + future_smoothing_treatments: + smooth_projection_curve: true # options: true / false + wind-ofs: + historical_data: + # Observed offshore CapEx is a fixed-bottom series, so only that + # class reads it. No floating cost history exists to observe. + capcost: + fixed: real # options: real / manual / broadcast + floating: manual # options: manual / broadcast + fom: manual # options: manual / broadcast + vom: manual # options: manual / broadcast + cf_improvement: manual # options: manual / broadcast + rsc_mult: manual # options: manual / broadcast + future_smoothing_treatments: + smooth_projection_curve: true # options: true / false + battery: + historical_data: + capcost: broadcast # options: manual / broadcast + capcost_energy: broadcast # options: manual / broadcast + fom: broadcast # options: manual / broadcast + fom_energy: broadcast # options: manual / broadcast + vom: broadcast # options: manual / broadcast + rte: broadcast # options: manual / broadcast + future_smoothing_treatments: + smooth_projection_curve: true # options: true / false + csp: + historical_data: + capcost: manual # options: manual / broadcast + fom: manual # options: manual / broadcast + vom: manual # options: manual / broadcast + future_smoothing_treatments: + smooth_projection_curve: true # options: true / false + coal: + historical_data: + capcost: broadcast # options: manual / broadcast + fom: broadcast # options: manual / broadcast + vom: broadcast # options: manual / broadcast + heatrate: broadcast # options: manual / broadcast + future_smoothing_treatments: + smooth_projection_curve: true # options: true / false + coal-ccs: + historical_data: + capcost: broadcast # options: manual / broadcast + fom: broadcast # options: manual / broadcast + vom: broadcast # options: manual / broadcast + heatrate: broadcast # options: manual / broadcast + future_smoothing_treatments: + smooth_projection_curve: true # options: true / false + gas: + historical_data: + # Only the plain combined-cycle and combustion-turbine plants have an + # EIA observation. The H-Frame and aeroderivative variants have none, + # so they generate their own history instead of borrowing one. + capcost: + Gas-CC: real # options: real / manual / broadcast + Gas-CC_H_1x1: broadcast # options: manual / broadcast + Gas-CC_H_2x1: broadcast # options: manual / broadcast + Gas-CT: real # options: real / manual / broadcast + Gas-CT_aero: broadcast # options: manual / broadcast + fom: broadcast # options: manual / broadcast + vom: broadcast # options: manual / broadcast + heatrate: broadcast # options: manual / broadcast + future_smoothing_treatments: + smooth_projection_curve: true # options: true / false + gas-ccs: + historical_data: + capcost: broadcast # options: manual / broadcast + fom: broadcast # options: manual / broadcast + vom: broadcast # options: manual / broadcast + heatrate: broadcast # options: manual / broadcast + future_smoothing_treatments: + smooth_projection_curve: true # options: true / false + fuelcell: + historical_data: + capcost: manual # options: manual / broadcast + fom: manual # options: manual / broadcast + vom: manual # options: manual / broadcast + heatrate: manual # options: manual / broadcast + future_smoothing_treatments: + smooth_projection_curve: true # options: true / false + nuclear: + historical_data: + capcost: manual # options: manual / broadcast + fom: manual # options: manual / broadcast + vom: manual # options: manual / broadcast + heatrate: manual # options: manual / broadcast + future_smoothing_treatments: + smooth_projection_curve: true # options: true / false + nuclear-smr: + historical_data: + capcost: manual # options: manual / broadcast + fom: manual # options: manual / broadcast + vom: manual # options: manual / broadcast + heatrate: manual # options: manual / broadcast + future_smoothing_treatments: + smooth_projection_curve: true # options: true / false + biopower: + historical_data: + capcost: real # options: real / manual / broadcast + fom: broadcast # options: manual / broadcast + vom: broadcast # options: manual / broadcast + heatrate: broadcast # options: manual / broadcast + future_smoothing_treatments: + smooth_projection_curve: true # options: true / false + +# Versioned ReEDS history used before each ATB projection series begins. +# Set seed_missing_from_reeds to true only when initializing a missing file from +# processing.reeds_repo. Each run appends the first available projection year, +# so a later ATB release can move that year into history without a gap. +historical_data: + directory: manual_input/historical + dollar_year: 2022 + seed_missing_from_reeds: false + +plotting: + metrics: + - CAPEX + - Fixed O&M + - Variable O&M + case: Market + crp_years: 20 + figure_format: png + output_directory: figures + # Before/after plots are versioned; their unsmoothed inputs are held only in + # a temporary directory while the full pipeline is running. + smoothing_comparison_directory: comparison/smoothing_comparison + technologies: + - Biopower + - Coal_FE + - CSP + - LandbasedWind + - NaturalGas_FE + - Nuclear + - OffShoreWind + - Utility-Scale Battery Storage + - UtilityPV diff --git a/atb/manual_input/README.md b/atb/manual_input/README.md new file mode 100644 index 0000000..09327b9 --- /dev/null +++ b/atb/manual_input/README.md @@ -0,0 +1,39 @@ +# manual_input/ + +Inputs that are not available from the raw ATB downloads. They are consumed by +`generate_atb_files.py`. + +- `historical/` — versioned ReEDS rows that precede each scraped ATB projection + series. These come directly from the ReEDS repository, not from ATB. Their + upstream sources are mixed and are not recorded per row: they have been + maintained by hand in ReEDS and carried forward year by year, with each new + ATB release adding one more year. Treat them as ReEDS's own record rather + than a derived product, and update them in ReEDS rather than here. + + Two things live in these files. Every technology stores the years before its + ATB data begins, which is why a technology ATB publishes late (nuclear from + 2030, fuel cell from 2035, floating offshore wind from 2030) stores more + years than one starting at the release year. Separately, technologies ATB + never publishes at all — cofired and existing coal, the aeroderivative + combustion turbine — store their complete series here, so those rows are the + only source for them. + + The formatter rolls the first current projection year into these files + automatically after every run, so any run rewrites one row per series. +- `csp_cost_ratios_.csv` — CSP configuration cost multipliers (csp1–csp4) + relative to the base configuration (csp2). They come from the separate ReEDS + CSP thermal-storage sizing/SAM analysis rather than either ATB download. The + 2024 multipliers reproduce the published ReEDS ATB 2024 configurations. +- `offshore_cost_multipliers_.csv` — ReEDS fixed-bottom and floating + configuration adjustments applied to the corresponding ATB offshore class + proxies. These account for ReEDS configuration assumptions outside the raw + ATB flat file. +- `coal_projection_overrides_.csv` — published ReEDS coal projection + values retained where a later ATB errata release changed intermediate years. + For 2024 this pins only 2033–2034; all other coal years come from ATB v3. +- `battery_costs_.csv` — power/energy capital-cost split for an ATB year + whose Excel workbook has not yet been published. It is a temporary fallback + for `format_continuous_battery` and can be removed after publication. + +When a new ATB year is released, review the year-specific CSP or battery inputs. +The files under `historical/` update automatically. diff --git a/atb/manual_input/coal_projection_overrides_2024.csv b/atb/manual_input/coal_projection_overrides_2024.csv new file mode 100644 index 0000000..8f5083f --- /dev/null +++ b/atb/manual_input/coal_projection_overrides_2024.csv @@ -0,0 +1,11 @@ +tech,Scenario,i,t,capcost,fom,vom,heatrate +coal,Moderate,Coal-IGCC,2033,5349.3,160.4,15.79,8.308 +coal,Moderate,Coal-IGCC,2034,5328.1,159.8,15.74,8.308 +coal,Moderate,Coal-new,2033,2991.2,81.7,8.83,8.205 +coal,Moderate,Coal-new,2034,2973.7,81.2,8.79,8.182 +coal-ccs,Advanced,coal-CCS_mod,2033,4399.4,116.3,13.23,9.588 +coal-ccs,Advanced,coal-CCS_mod,2034,4323.3,114.5,12.99,9.476 +coal-ccs,Conservative,coal-CCS_mod,2033,4723.8,123.6,15.49,10.938 +coal-ccs,Conservative,coal-CCS_mod,2034,4674.7,122.4,15.44,10.938 +coal-ccs,Moderate,coal-CCS_mod,2033,4561.6,119.9,14.36,10.319 +coal-ccs,Moderate,coal-CCS_mod,2034,4499.0,118.5,14.22,10.263 diff --git a/atb/manual_input/csp_cost_ratios_2024.csv b/atb/manual_input/csp_cost_ratios_2024.csv new file mode 100644 index 0000000..0efeb29 --- /dev/null +++ b/atb/manual_input/csp_cost_ratios_2024.csv @@ -0,0 +1,5 @@ +type,duration,solar_multiple,ratio,base_tech +csp1,14,2.7,1.120642878353,0 +csp2,10,2.4,1,1 +csp3,8,1.3,0.709285122138,0 +csp4,6,1,0.612585771590,0 diff --git a/atb/manual_input/historical/README.md b/atb/manual_input/historical/README.md new file mode 100644 index 0000000..132f62b --- /dev/null +++ b/atb/manual_input/historical/README.md @@ -0,0 +1,26 @@ +# ATB historical baseline + +These CSVs provide the ReEDS years that are not present in the current scraped +ATB release. They were initially seeded from +`C:/ReEDS/ReEDS/inputs/plant_characteristics/*_ATB_2024_*.csv` and are stored in +constant 2022 dollars, as configured by `historical_data.dollar_year`. + +For each technology series and scenario, `generate_atb_files.py`: + +1. reads these rows before the first year available in the current raw ATB; +2. concatenates the current scraped ATB projection from that year onward; +3. preserves complete historical series that are no longer published by ATB; +4. appends the first current projection year to this baseline if it is missing. + +Step 4 makes the update idempotent and prepares continuity for the next annual +release. For example, ATB 2024 uses history through 2021 and projections from +2022. After the run, 2022 is stored here; if the next release begins in 2023, +the formatter uses history through 2022 and the new projection from 2023. + +The formatter validates every technology/scenario series from +`processing.reeds_start_year` through its final year and stops with an error if +any year is missing. + +Missing files may be initialized from the configured ReEDS repository when +`historical_data.seed_missing_from_reeds` is `true`. Once seeded, these files +are the versioned source of truth and should be committed with the pipeline. diff --git a/atb/manual_input/historical/battery_ATB_historical_advanced.csv b/atb/manual_input/historical/battery_ATB_historical_advanced.csv new file mode 100644 index 0000000..e422391 --- /dev/null +++ b/atb/manual_input/historical/battery_ATB_historical_advanced.csv @@ -0,0 +1,14 @@ +i,t,capcost,capcost_energy,fom,fom_energy,vom,rte +battery_li,2010,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2011,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2012,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2013,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2014,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2015,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2016,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2017,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2018,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2019,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2020,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2021,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2022,363.01,390.93,9.08,9.77,0.0,0.85 diff --git a/atb/manual_input/historical/battery_ATB_historical_conservative.csv b/atb/manual_input/historical/battery_ATB_historical_conservative.csv new file mode 100644 index 0000000..e422391 --- /dev/null +++ b/atb/manual_input/historical/battery_ATB_historical_conservative.csv @@ -0,0 +1,14 @@ +i,t,capcost,capcost_energy,fom,fom_energy,vom,rte +battery_li,2010,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2011,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2012,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2013,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2014,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2015,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2016,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2017,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2018,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2019,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2020,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2021,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2022,363.01,390.93,9.08,9.77,0.0,0.85 diff --git a/atb/manual_input/historical/battery_ATB_historical_moderate.csv b/atb/manual_input/historical/battery_ATB_historical_moderate.csv new file mode 100644 index 0000000..e422391 --- /dev/null +++ b/atb/manual_input/historical/battery_ATB_historical_moderate.csv @@ -0,0 +1,14 @@ +i,t,capcost,capcost_energy,fom,fom_energy,vom,rte +battery_li,2010,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2011,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2012,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2013,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2014,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2015,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2016,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2017,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2018,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2019,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2020,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2021,363.01,390.93,9.08,9.77,0.0,0.85 +battery_li,2022,363.01,390.93,9.08,9.77,0.0,0.85 diff --git a/atb/manual_input/historical/biopower_ATB_historical_advanced.csv b/atb/manual_input/historical/biopower_ATB_historical_advanced.csv new file mode 100644 index 0000000..6d0a43f --- /dev/null +++ b/atb/manual_input/historical/biopower_ATB_historical_advanced.csv @@ -0,0 +1,14 @@ +i,t,capcost,fom,vom,heatrate +biopower,2010,4726.476766,142.9276928,5.485907804,13.5 +biopower,2011,4712.415325,142.9276928,5.485907804,13.5 +biopower,2012,4698.353884,142.9276928,5.485907804,13.5 +biopower,2013,4684.292442,142.9276928,5.485907804,13.5 +biopower,2014,4670.231002,142.9276928,5.485907804,13.5 +biopower,2015,4656.169559,142.9276928,5.485907804,13.5 +biopower,2016,4642.108118,142.9276928,5.485907804,13.5 +biopower,2017,4628.046676,142.9276928,5.485907804,13.5 +biopower,2018,4613.985236,142.9276928,5.485907804,13.5 +biopower,2019,4599.923794,142.9276928,5.485907804,13.5 +biopower,2020,4617.272559,170.5719249,5.419053793,13.5 +biopower,2021,5183.685345,169.7976,5.4432,13.5 +biopower,2022,4829.860171,163.66602,5.24664,13.5 diff --git a/atb/manual_input/historical/biopower_ATB_historical_conservative.csv b/atb/manual_input/historical/biopower_ATB_historical_conservative.csv new file mode 100644 index 0000000..6d0a43f --- /dev/null +++ b/atb/manual_input/historical/biopower_ATB_historical_conservative.csv @@ -0,0 +1,14 @@ +i,t,capcost,fom,vom,heatrate +biopower,2010,4726.476766,142.9276928,5.485907804,13.5 +biopower,2011,4712.415325,142.9276928,5.485907804,13.5 +biopower,2012,4698.353884,142.9276928,5.485907804,13.5 +biopower,2013,4684.292442,142.9276928,5.485907804,13.5 +biopower,2014,4670.231002,142.9276928,5.485907804,13.5 +biopower,2015,4656.169559,142.9276928,5.485907804,13.5 +biopower,2016,4642.108118,142.9276928,5.485907804,13.5 +biopower,2017,4628.046676,142.9276928,5.485907804,13.5 +biopower,2018,4613.985236,142.9276928,5.485907804,13.5 +biopower,2019,4599.923794,142.9276928,5.485907804,13.5 +biopower,2020,4617.272559,170.5719249,5.419053793,13.5 +biopower,2021,5183.685345,169.7976,5.4432,13.5 +biopower,2022,4829.860171,163.66602,5.24664,13.5 diff --git a/atb/manual_input/historical/biopower_ATB_historical_moderate.csv b/atb/manual_input/historical/biopower_ATB_historical_moderate.csv new file mode 100644 index 0000000..6d0a43f --- /dev/null +++ b/atb/manual_input/historical/biopower_ATB_historical_moderate.csv @@ -0,0 +1,14 @@ +i,t,capcost,fom,vom,heatrate +biopower,2010,4726.476766,142.9276928,5.485907804,13.5 +biopower,2011,4712.415325,142.9276928,5.485907804,13.5 +biopower,2012,4698.353884,142.9276928,5.485907804,13.5 +biopower,2013,4684.292442,142.9276928,5.485907804,13.5 +biopower,2014,4670.231002,142.9276928,5.485907804,13.5 +biopower,2015,4656.169559,142.9276928,5.485907804,13.5 +biopower,2016,4642.108118,142.9276928,5.485907804,13.5 +biopower,2017,4628.046676,142.9276928,5.485907804,13.5 +biopower,2018,4613.985236,142.9276928,5.485907804,13.5 +biopower,2019,4599.923794,142.9276928,5.485907804,13.5 +biopower,2020,4617.272559,170.5719249,5.419053793,13.5 +biopower,2021,5183.685345,169.7976,5.4432,13.5 +biopower,2022,4829.860171,163.66602,5.24664,13.5 diff --git a/atb/manual_input/historical/coal-ccs_ATB_historical_advanced.csv b/atb/manual_input/historical/coal-ccs_ATB_historical_advanced.csv new file mode 100644 index 0000000..312e56b --- /dev/null +++ b/atb/manual_input/historical/coal-ccs_ATB_historical_advanced.csv @@ -0,0 +1,14 @@ +i,t,capcost,fom,vom,heatrate +coal-CCS_mod,2010,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2011,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2012,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2013,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2014,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2015,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2016,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2017,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2018,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2019,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2020,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2021,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2022,5312.5,137.2,16.12,10.938 diff --git a/atb/manual_input/historical/coal-ccs_ATB_historical_conservative.csv b/atb/manual_input/historical/coal-ccs_ATB_historical_conservative.csv new file mode 100644 index 0000000..312e56b --- /dev/null +++ b/atb/manual_input/historical/coal-ccs_ATB_historical_conservative.csv @@ -0,0 +1,14 @@ +i,t,capcost,fom,vom,heatrate +coal-CCS_mod,2010,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2011,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2012,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2013,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2014,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2015,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2016,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2017,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2018,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2019,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2020,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2021,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2022,5312.5,137.2,16.12,10.938 diff --git a/atb/manual_input/historical/coal-ccs_ATB_historical_moderate.csv b/atb/manual_input/historical/coal-ccs_ATB_historical_moderate.csv new file mode 100644 index 0000000..312e56b --- /dev/null +++ b/atb/manual_input/historical/coal-ccs_ATB_historical_moderate.csv @@ -0,0 +1,14 @@ +i,t,capcost,fom,vom,heatrate +coal-CCS_mod,2010,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2011,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2012,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2013,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2014,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2015,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2016,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2017,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2018,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2019,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2020,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2021,5125.032,131.868,15.8652,10.938 +coal-CCS_mod,2022,5312.5,137.2,16.12,10.938 diff --git a/atb/manual_input/historical/coal_ATB_historical_moderate.csv b/atb/manual_input/historical/coal_ATB_historical_moderate.csv new file mode 100644 index 0000000..8e77235 --- /dev/null +++ b/atb/manual_input/historical/coal_ATB_historical_moderate.csv @@ -0,0 +1,247 @@ +i,t,capcost,fom,vom,heatrate +Coal-IGCC,2010,5851.647055,163.5759711,16.43772529,9.325 +Coal-IGCC,2011,5803.570143,163.5759711,16.43772529,9.239333333 +Coal-IGCC,2012,5755.493231,163.5759711,16.43772529,9.153666667 +Coal-IGCC,2013,5707.416318,163.5759711,16.43772529,9.068 +Coal-IGCC,2014,5659.339405,163.5759711,16.43772529,8.982333333 +Coal-IGCC,2015,5611.262494,163.5759711,16.43772529,8.896666667 +Coal-IGCC,2016,5563.185581,163.5759711,16.43772529,8.811 +Coal-IGCC,2017,5515.108668,163.5759711,16.43772529,8.725333333 +Coal-IGCC,2018,5467.031756,163.5759711,16.43772529,8.639666667 +Coal-IGCC,2019,5418.954843,163.5759711,16.43772529,8.554 +Coal-IGCC,2020,5030.656241,158.9140095,16.23740698,8.45 +Coal-IGCC,2021,5402.592,162.108,15.9516,8.308 +Coal-IGCC,2022,5603.2,167.5,16.44,8.308 +Coal-IGCC,2023,5582.1,166.9,16.39,8.308 +Coal-IGCC,2024,5560.9,166.3,16.33,8.308 +Coal-IGCC,2025,5539.7,165.7,16.28,8.308 +Coal-IGCC,2026,5518.6,165.2,16.22,8.308 +Coal-IGCC,2027,5497.4,164.6,16.17,8.308 +Coal-IGCC,2028,5476.3,164.0,16.12,8.308 +Coal-IGCC,2029,5455.1,163.4,16.06,8.308 +Coal-IGCC,2030,5433.9,162.8,16.01,8.308 +Coal-IGCC,2031,5412.8,162.2,15.96,8.308 +Coal-IGCC,2032,5391.6,161.6,15.9,8.308 +Coal-IGCC,2033,5349.3,160.4,15.79,8.308 +Coal-IGCC,2034,5328.1,159.8,15.74,8.308 +Coal-IGCC,2035,5328.1,159.8,15.74,8.308 +Coal-IGCC,2036,5306.9,159.2,15.69,8.308 +Coal-IGCC,2037,5285.8,158.7,15.63,8.308 +Coal-IGCC,2038,5264.6,158.1,15.58,8.308 +Coal-IGCC,2039,5243.5,157.5,15.53,8.308 +Coal-IGCC,2040,5222.3,156.9,15.47,8.308 +Coal-IGCC,2041,5201.1,156.3,15.42,8.308 +Coal-IGCC,2042,5180.0,155.7,15.36,8.308 +Coal-IGCC,2043,5158.8,155.1,15.31,8.308 +Coal-IGCC,2044,5137.6,154.5,15.26,8.308 +Coal-IGCC,2045,5116.5,153.9,15.2,8.308 +Coal-IGCC,2046,5095.3,153.3,15.15,8.308 +Coal-IGCC,2047,5074.1,152.7,15.1,8.308 +Coal-IGCC,2048,5053.0,152.2,15.04,8.308 +Coal-IGCC,2049,5031.8,151.6,14.99,8.308 +Coal-IGCC,2050,5010.7,151.0,14.93,8.308 +Coal-new,2010,3066.6202,82.18862648,8.997250754,8.83 +Coal-new,2011,3057.462693,82.18862648,8.997250754,8.790333333 +Coal-new,2012,3048.305185,82.18862648,8.997250754,8.750666667 +Coal-new,2013,3039.147678,82.18862648,8.997250754,8.711 +Coal-new,2014,3029.990171,82.18862648,8.997250754,8.671333333 +Coal-new,2015,3020.832665,82.18862648,8.997250754,8.631666667 +Coal-new,2016,3011.675157,82.18862648,8.997250754,8.592 +Coal-new,2017,3002.51765,82.18862648,8.997250754,8.552333333 +Coal-new,2018,2993.360143,82.18862648,8.997250754,8.512666667 +Coal-new,2019,2984.202635,82.18862648,8.997250754,8.473 +Coal-new,2020,2903.736846,83.67465988,8.887605766,8.473 +Coal-new,2021,3085.236,83.808,9.1368,8.49 +Coal-new,2022,3201.2,86.5,9.3,8.49 +Coal-new,2023,3183.7,86.1,9.26,8.466 +Coal-new,2024,3166.2,85.7,9.22,8.442 +Coal-new,2025,3148.7,85.3,9.18,8.419 +Coal-new,2026,3131.2,84.9,9.14,8.395 +Coal-new,2027,3113.7,84.5,9.1,8.371 +Coal-new,2028,3096.2,84.1,9.06,8.347 +Coal-new,2029,3078.7,83.7,9.03,8.324 +Coal-new,2030,3061.2,83.3,8.99,8.3 +Coal-new,2031,3043.7,82.9,8.95,8.276 +Coal-new,2032,3026.2,82.5,8.91,8.253 +Coal-new,2033,2991.2,81.7,8.83,8.205 +Coal-new,2034,2973.7,81.2,8.79,8.182 +Coal-new,2035,2973.7,81.2,8.79,8.182 +Coal-new,2036,2945.2,80.6,8.76,8.182 +Coal-new,2037,2916.6,79.9,8.73,8.182 +Coal-new,2038,2888.1,79.3,8.7,8.182 +Coal-new,2039,2859.5,78.6,8.67,8.182 +Coal-new,2040,2830.9,77.9,8.64,8.182 +Coal-new,2041,2802.4,77.3,8.61,8.182 +Coal-new,2042,2773.8,76.6,8.58,8.182 +Coal-new,2043,2745.3,75.9,8.55,8.182 +Coal-new,2044,2716.7,75.3,8.52,8.182 +Coal-new,2045,2688.1,74.6,8.49,8.182 +Coal-new,2046,2659.6,74.0,8.46,8.182 +Coal-new,2047,2631.0,73.3,8.43,8.182 +Coal-new,2048,2602.5,72.6,8.4,8.182 +Coal-new,2049,2573.9,72.0,8.37,8.182 +Coal-new,2050,2545.4,71.3,8.34,8.182 +CoalOldScr,2010,3029.248836,41.49039589,7.345591952,9.984242986 +CoalOldScr,2011,3020.202926,41.49039589,7.498498976,9.984242986 +CoalOldScr,2012,3011.157017,41.49039589,7.651405997,9.984242986 +CoalOldScr,2013,3002.111109,41.49039589,7.804313019,9.984242986 +CoalOldScr,2014,2993.065199,41.49039589,7.957220041,9.984242986 +CoalOldScr,2015,2984.01929,41.49039589,8.110127063,9.984242986 +CoalOldScr,2016,2974.973381,41.49039589,8.27894877,9.984242986 +CoalOldScr,2017,2965.927471,41.49039589,8.447770478,9.984242986 +CoalOldScr,2018,2956.881562,41.49039589,8.616592186,9.984242986 +CoalOldScr,2019,2947.835654,41.49039589,8.785413895,9.984242986 +CoalOldScr,2020,2938.789744,41.49039589,8.954235602,9.984242986 +CoalOldScr,2021,2929.743835,41.49039589,9.140628409,9.984242986 +CoalOldScr,2022,2920.697925,41.49039589,9.327021215,9.984242986 +CoalOldScr,2023,2901.475368,41.49039589,9.513414022,9.984242986 +CoalOldScr,2024,2901.475368,41.49039589,9.699806829,9.984242986 +CoalOldScr,2025,2884.514289,41.49039589,9.886199635,9.984242986 +CoalOldScr,2026,2853.984346,41.49039589,10.09199235,9.984242986 +CoalOldScr,2027,2823.454401,41.49039589,10.29778508,9.984242986 +CoalOldScr,2028,2791.79372,41.49039589,10.50357779,9.984242986 +CoalOldScr,2029,2760.133038,41.49039589,10.70937051,9.984242986 +CoalOldScr,2030,2725.080139,41.49039589,10.91516323,9.984242986 +CoalOldScr,2031,2694.550196,41.49039589,11.14237502,9.984242986 +CoalOldScr,2032,2658.366559,41.49039589,11.36958682,9.984242986 +CoalOldScr,2033,2621.052184,41.49039589,11.59679861,9.984242986 +CoalOldScr,2034,2602.960366,41.49039589,11.8240104,9.984242986 +CoalOldScr,2035,2584.868547,41.49039589,12.05122218,9.984242986 +CoalOldScr,2036,2565.64599,41.49039589,12.30208237,9.984242986 +CoalOldScr,2037,2547.554172,41.49039589,12.55294255,9.984242986 +CoalOldScr,2038,2529.462354,41.49039589,12.80380272,9.984242986 +CoalOldScr,2039,2513.632013,41.49039589,13.0546629,9.984242986 +CoalOldScr,2040,2495.540194,41.49039589,13.30552308,9.984242986 +CoalOldScr,2041,2475.186899,41.49039589,13.58249298,9.984242986 +CoalOldScr,2042,2458.225818,41.49039589,13.85946288,9.984242986 +CoalOldScr,2043,2441.264739,41.49039589,14.13643279,9.984242986 +CoalOldScr,2044,2424.30366,41.49039589,14.4134027,9.984242986 +CoalOldScr,2045,2409.604057,41.49039589,14.6903726,9.984242986 +CoalOldScr,2046,2392.642977,41.49039589,14.99616976,9.984242986 +CoalOldScr,2047,2373.42042,41.49039589,15.30196691,9.984242986 +CoalOldScr,2048,2354.197863,41.49039589,15.60776407,9.984242986 +CoalOldScr,2049,2336.106045,41.49039589,15.91356123,9.984242986 +CoalOldScr,2050,2295.399453,41.49039589,16.21935838,9.984242986 +CoalOldUns,2010,2361.107915,41.49039589,7.345591952,10.26026846 +CoalOldUns,2011,2352.062005,41.49039589,7.498498976,10.26026846 +CoalOldUns,2012,2343.016096,41.49039589,7.651405997,10.26026846 +CoalOldUns,2013,2333.970188,41.49039589,7.804313019,10.26026846 +CoalOldUns,2014,2324.924278,41.49039589,7.957220041,10.26026846 +CoalOldUns,2015,2315.878369,41.49039589,8.110127063,10.26026846 +CoalOldUns,2016,2306.832459,41.49039589,8.27894877,10.26026846 +CoalOldUns,2017,2297.78655,41.49039589,8.447770478,10.26026846 +CoalOldUns,2018,2288.740641,41.49039589,8.616592186,10.26026846 +CoalOldUns,2019,2279.694732,41.49039589,8.785413895,10.26026846 +CoalOldUns,2020,2270.648823,41.49039589,8.954235602,10.26026846 +CoalOldUns,2021,2261.602914,41.49039589,9.140628409,10.26026846 +CoalOldUns,2022,2252.557004,41.49039589,9.327021215,10.26026846 +CoalOldUns,2023,2233.334447,41.49039589,9.513414022,10.26026846 +CoalOldUns,2024,2233.334447,41.49039589,9.699806829,10.26026846 +CoalOldUns,2025,2216.373368,41.49039589,9.886199635,10.26026846 +CoalOldUns,2026,2185.843425,41.49039589,10.09199235,10.26026846 +CoalOldUns,2027,2155.31348,41.49039589,10.29778508,10.26026846 +CoalOldUns,2028,2123.652799,41.49039589,10.50357779,10.26026846 +CoalOldUns,2029,2091.992117,41.49039589,10.70937051,10.26026846 +CoalOldUns,2030,2056.939218,41.49039589,10.91516323,10.26026846 +CoalOldUns,2031,2026.409275,41.49039589,11.14237502,10.26026846 +CoalOldUns,2032,1990.225638,41.49039589,11.36958682,10.26026846 +CoalOldUns,2033,1952.911263,41.49039589,11.59679861,10.26026846 +CoalOldUns,2034,1934.819445,41.49039589,11.8240104,10.26026846 +CoalOldUns,2035,1916.727626,41.49039589,12.05122218,10.26026846 +CoalOldUns,2036,1897.505069,41.49039589,12.30208237,10.26026846 +CoalOldUns,2037,1879.413251,41.49039589,12.55294255,10.26026846 +CoalOldUns,2038,1861.321433,41.49039589,12.80380272,10.26026846 +CoalOldUns,2039,1845.491092,41.49039589,13.0546629,10.26026846 +CoalOldUns,2040,1827.399273,41.49039589,13.30552308,10.26026846 +CoalOldUns,2041,1807.045978,41.49039589,13.58249298,10.26026846 +CoalOldUns,2042,1790.084897,41.49039589,13.85946288,10.26026846 +CoalOldUns,2043,1773.123818,41.49039589,14.13643279,10.26026846 +CoalOldUns,2044,1756.162739,41.49039589,14.4134027,10.26026846 +CoalOldUns,2045,1741.463136,41.49039589,14.6903726,10.26026846 +CoalOldUns,2046,1724.502056,41.49039589,14.99616976,10.26026846 +CoalOldUns,2047,1705.279499,41.49039589,15.30196691,10.26026846 +CoalOldUns,2048,1686.056942,41.49039589,15.60776407,10.26026846 +CoalOldUns,2049,1667.965124,41.49039589,15.91356123,10.26026846 +CoalOldUns,2050,1627.258533,41.49039589,16.21935838,10.26026846 +CofireNew,2010,3389.016969,81.18703486,8.887605766,8.83 +CofireNew,2011,3379.97106,81.18703486,8.887605766,8.790333333 +CofireNew,2012,3370.92515,81.18703486,8.887605766,8.750666667 +CofireNew,2013,3361.879241,81.18703486,8.887605766,8.711 +CofireNew,2014,3352.833332,81.18703486,8.887605766,8.671333333 +CofireNew,2015,3343.787423,81.18703486,8.887605766,8.631666667 +CofireNew,2016,3334.741514,81.18703486,8.887605766,8.592 +CofireNew,2017,3325.695605,81.18703486,8.887605766,8.552333333 +CofireNew,2018,3316.649695,81.18703486,8.887605766,8.512666667 +CofireNew,2019,3307.603786,81.18703486,8.887605766,8.473 +CofireNew,2020,3298.557876,81.18703486,8.887605766,8.473 +CofireNew,2021,3289.511968,81.18703486,8.887605766,8.473 +CofireNew,2022,3280.466059,81.18703486,8.887605766,8.473 +CofireNew,2023,3261.243502,81.18703486,8.887605766,8.473 +CofireNew,2024,3261.243502,81.18703486,8.887605766,8.473 +CofireNew,2025,3244.282421,80.73473941,8.819761448,8.473 +CofireNew,2026,3213.752478,80.28244395,8.763224515,8.386 +CofireNew,2027,3183.222535,79.83014848,8.695380197,8.3 +CofireNew,2028,3151.561853,79.37785302,8.627535878,8.213 +CofireNew,2029,3119.90117,78.92555757,8.570998945,8.126 +CofireNew,2030,3084.848272,78.47326211,8.503154627,8.04 +CofireNew,2031,3054.318329,78.02096665,8.435310308,7.953 +CofireNew,2032,3018.134693,77.5686712,8.378773376,7.866 +CofireNew,2033,2980.820317,77.11637574,8.310929057,7.78 +CofireNew,2034,2962.728498,77.11637574,8.310929057,7.78 +CofireNew,2035,2944.636681,77.11637574,8.310929057,7.78 +CofireNew,2036,2925.414123,77.11637574,8.310929057,7.78 +CofireNew,2037,2907.322305,77.11637574,8.310929057,7.78 +CofireNew,2038,2889.230487,77.11637574,8.310929057,7.78 +CofireNew,2039,2873.400145,77.11637574,8.310929057,7.78 +CofireNew,2040,2855.308328,77.11637574,8.310929057,7.78 +CofireNew,2041,2834.955032,77.11637574,8.310929057,7.78 +CofireNew,2042,2817.993952,77.11637574,8.310929057,7.78 +CofireNew,2043,2801.032872,77.11637574,8.310929057,7.78 +CofireNew,2044,2784.071792,77.11637574,8.310929057,7.78 +CofireNew,2045,2769.37219,77.11637574,8.310929057,7.78 +CofireNew,2046,2752.41111,77.11637574,8.310929057,7.78 +CofireNew,2047,2733.188553,77.11637574,8.310929057,7.78 +CofireNew,2048,2713.965996,77.11637574,8.310929057,7.78 +CofireNew,2049,2695.874178,77.11637574,8.310929057,7.78 +CofireNew,2050,2655.167586,77.11637574,8.310929057,7.78 +CofireOld,2010,3389.016969,41.49039589,7.345591952,10.74020434 +CofireOld,2011,3379.97106,41.49039589,7.498498976,10.74020434 +CofireOld,2012,3370.92515,41.49039589,7.651405997,10.74020434 +CofireOld,2013,3361.879241,41.49039589,7.804313019,10.74020434 +CofireOld,2014,3352.833332,41.49039589,7.957220041,10.74020434 +CofireOld,2015,3343.787423,41.49039589,8.110127063,10.74020434 +CofireOld,2016,3334.741514,41.49039589,8.27894877,10.74020434 +CofireOld,2017,3325.695605,41.49039589,8.447770478,10.74020434 +CofireOld,2018,3316.649695,41.49039589,8.616592186,10.74020434 +CofireOld,2019,3307.603786,41.49039589,8.785413895,10.74020434 +CofireOld,2020,3298.557876,41.49039589,8.954235602,10.74020434 +CofireOld,2021,3289.511968,41.49039589,9.140628409,10.74020434 +CofireOld,2022,3280.466059,41.49039589,9.327021215,10.74020434 +CofireOld,2023,3261.243502,41.49039589,9.513414022,10.74020434 +CofireOld,2024,3261.243502,41.49039589,9.699806829,10.74020434 +CofireOld,2025,3244.282421,41.49039589,9.886199635,10.74020434 +CofireOld,2026,3213.752478,41.49039589,10.09199235,10.74020434 +CofireOld,2027,3183.222535,41.49039589,10.29778508,10.74020434 +CofireOld,2028,3151.561853,41.49039589,10.50357779,10.74020434 +CofireOld,2029,3119.90117,41.49039589,10.70937051,10.74020434 +CofireOld,2030,3084.848272,41.49039589,10.91516323,10.74020434 +CofireOld,2031,3054.318329,41.49039589,11.14237502,10.74020434 +CofireOld,2032,3018.134693,41.49039589,11.36958682,10.74020434 +CofireOld,2033,2980.820317,41.49039589,11.59679861,10.74020434 +CofireOld,2034,2962.728498,41.49039589,11.8240104,10.74020434 +CofireOld,2035,2944.636681,41.49039589,12.05122218,10.74020434 +CofireOld,2036,2925.414123,41.49039589,12.30208237,10.74020434 +CofireOld,2037,2907.322305,41.49039589,12.55294255,10.74020434 +CofireOld,2038,2889.230487,41.49039589,12.80380272,10.74020434 +CofireOld,2039,2873.400145,41.49039589,13.0546629,10.74020434 +CofireOld,2040,2855.308328,41.49039589,13.30552308,10.74020434 +CofireOld,2041,2834.955032,41.49039589,13.58249298,10.74020434 +CofireOld,2042,2817.993952,41.49039589,13.85946288,10.74020434 +CofireOld,2043,2801.032872,41.49039589,14.13643279,10.74020434 +CofireOld,2044,2784.071792,41.49039589,14.4134027,10.74020434 +CofireOld,2045,2769.37219,41.49039589,14.6903726,10.74020434 +CofireOld,2046,2752.41111,41.49039589,14.99616976,10.74020434 +CofireOld,2047,2733.188553,41.49039589,15.30196691,10.74020434 +CofireOld,2048,2713.965996,41.49039589,15.60776407,10.74020434 +CofireOld,2049,2695.874178,41.49039589,15.91356123,10.74020434 +CofireOld,2050,2655.167586,41.49039589,16.21935838,10.74020434 diff --git a/atb/manual_input/historical/csp_ATB_historical_advanced.csv b/atb/manual_input/historical/csp_ATB_historical_advanced.csv new file mode 100644 index 0000000..fe14d08 --- /dev/null +++ b/atb/manual_input/historical/csp_ATB_historical_advanced.csv @@ -0,0 +1,53 @@ +type,t,capcost,fom,vom +csp1,2010,10118.39247,83.63396315,4.892457604 +csp1,2011,10118.39247,83.63396315,4.892457604 +csp1,2012,10118.39247,83.63396315,4.892457604 +csp1,2013,10118.39247,83.63396315,4.892457604 +csp1,2014,10118.39247,83.63396315,4.892457604 +csp1,2015,10118.39247,83.63396315,4.892457604 +csp1,2016,10118.39247,83.63396315,4.892457604 +csp1,2017,10118.39247,83.63396315,4.892457604 +csp1,2018,9655.056558,83.63396315,4.807945152 +csp1,2019,9205.528937,83.63396315,4.807945152 +csp1,2020,8731.09044,83.63396315,4.621539781 +csp1,2021,8256.651943,83.63396315,4.43513441 +csp1,2022,8233.260436,83.633757,4.435123 +csp2,2010,9022.519272,78.75663459,4.892457604 +csp2,2011,9022.519272,78.75663459,4.892457604 +csp2,2012,9022.519272,78.75663459,4.892457604 +csp2,2013,9022.519272,78.75663459,4.892457604 +csp2,2014,9022.519272,78.75663459,4.892457604 +csp2,2015,9022.519272,78.75663459,4.892457604 +csp2,2016,9022.519272,78.75663459,4.892457604 +csp2,2017,9022.519272,78.75663459,4.892457604 +csp2,2018,8605.929835,77.39619026,4.807945152 +csp2,2019,7448.628019,75.10677024,4.006648147 +csp2,2020,7408.195726,74.86846512,3.982154074 +csp2,2021,7367.763433,74.63016,3.95766 +csp2,2022,7346.908275,74.63016,3.95766 +csp3,2010,6378.213448,78.75663459,4.892457604 +csp3,2011,6378.213448,78.75663459,4.892457604 +csp3,2012,6378.213448,78.75663459,4.892457604 +csp3,2013,6378.213448,78.75663459,4.892457604 +csp3,2014,6378.213448,78.75663459,4.892457604 +csp3,2015,6378.213448,78.75663459,4.892457604 +csp3,2016,6378.213448,78.75663459,4.892457604 +csp3,2017,6378.213448,78.75663459,4.892457604 +csp3,2018,6290.932719,78.78932168,4.894488165 +csp3,2019,6036.47607,78.78932168,4.894488165 +csp3,2020,5631.600376,65.86614726,3.851035029 +csp3,2021,5226.724683,52.94297284,2.807581893 +csp3,2022,5211.052733,52.934062,2.807109 +csp4,2010,5499.241675,78.75663459,4.892457604 +csp4,2011,5499.241675,78.75663459,4.892457604 +csp4,2012,5499.241675,78.75663459,4.892457604 +csp4,2013,5499.241675,78.75663459,4.892457604 +csp4,2014,5499.241675,78.75663459,4.892457604 +csp4,2015,5499.241675,78.75663459,4.892457604 +csp4,2016,5499.241675,78.75663459,4.892457604 +csp4,2017,5499.241675,78.75663459,4.892457604 +csp4,2018,5411.623189,78.78932168,4.894488165 +csp4,2019,5195.786083,78.78932168,4.894488165 +csp4,2020,4855.091586,62.2584634,3.65971846 +csp4,2021,4514.397087,45.72760513,2.424948757 +csp4,2022,4500.611474,45.717374,2.424406 diff --git a/atb/manual_input/historical/csp_ATB_historical_conservative.csv b/atb/manual_input/historical/csp_ATB_historical_conservative.csv new file mode 100644 index 0000000..fe14d08 --- /dev/null +++ b/atb/manual_input/historical/csp_ATB_historical_conservative.csv @@ -0,0 +1,53 @@ +type,t,capcost,fom,vom +csp1,2010,10118.39247,83.63396315,4.892457604 +csp1,2011,10118.39247,83.63396315,4.892457604 +csp1,2012,10118.39247,83.63396315,4.892457604 +csp1,2013,10118.39247,83.63396315,4.892457604 +csp1,2014,10118.39247,83.63396315,4.892457604 +csp1,2015,10118.39247,83.63396315,4.892457604 +csp1,2016,10118.39247,83.63396315,4.892457604 +csp1,2017,10118.39247,83.63396315,4.892457604 +csp1,2018,9655.056558,83.63396315,4.807945152 +csp1,2019,9205.528937,83.63396315,4.807945152 +csp1,2020,8731.09044,83.63396315,4.621539781 +csp1,2021,8256.651943,83.63396315,4.43513441 +csp1,2022,8233.260436,83.633757,4.435123 +csp2,2010,9022.519272,78.75663459,4.892457604 +csp2,2011,9022.519272,78.75663459,4.892457604 +csp2,2012,9022.519272,78.75663459,4.892457604 +csp2,2013,9022.519272,78.75663459,4.892457604 +csp2,2014,9022.519272,78.75663459,4.892457604 +csp2,2015,9022.519272,78.75663459,4.892457604 +csp2,2016,9022.519272,78.75663459,4.892457604 +csp2,2017,9022.519272,78.75663459,4.892457604 +csp2,2018,8605.929835,77.39619026,4.807945152 +csp2,2019,7448.628019,75.10677024,4.006648147 +csp2,2020,7408.195726,74.86846512,3.982154074 +csp2,2021,7367.763433,74.63016,3.95766 +csp2,2022,7346.908275,74.63016,3.95766 +csp3,2010,6378.213448,78.75663459,4.892457604 +csp3,2011,6378.213448,78.75663459,4.892457604 +csp3,2012,6378.213448,78.75663459,4.892457604 +csp3,2013,6378.213448,78.75663459,4.892457604 +csp3,2014,6378.213448,78.75663459,4.892457604 +csp3,2015,6378.213448,78.75663459,4.892457604 +csp3,2016,6378.213448,78.75663459,4.892457604 +csp3,2017,6378.213448,78.75663459,4.892457604 +csp3,2018,6290.932719,78.78932168,4.894488165 +csp3,2019,6036.47607,78.78932168,4.894488165 +csp3,2020,5631.600376,65.86614726,3.851035029 +csp3,2021,5226.724683,52.94297284,2.807581893 +csp3,2022,5211.052733,52.934062,2.807109 +csp4,2010,5499.241675,78.75663459,4.892457604 +csp4,2011,5499.241675,78.75663459,4.892457604 +csp4,2012,5499.241675,78.75663459,4.892457604 +csp4,2013,5499.241675,78.75663459,4.892457604 +csp4,2014,5499.241675,78.75663459,4.892457604 +csp4,2015,5499.241675,78.75663459,4.892457604 +csp4,2016,5499.241675,78.75663459,4.892457604 +csp4,2017,5499.241675,78.75663459,4.892457604 +csp4,2018,5411.623189,78.78932168,4.894488165 +csp4,2019,5195.786083,78.78932168,4.894488165 +csp4,2020,4855.091586,62.2584634,3.65971846 +csp4,2021,4514.397087,45.72760513,2.424948757 +csp4,2022,4500.611474,45.717374,2.424406 diff --git a/atb/manual_input/historical/csp_ATB_historical_moderate.csv b/atb/manual_input/historical/csp_ATB_historical_moderate.csv new file mode 100644 index 0000000..fe14d08 --- /dev/null +++ b/atb/manual_input/historical/csp_ATB_historical_moderate.csv @@ -0,0 +1,53 @@ +type,t,capcost,fom,vom +csp1,2010,10118.39247,83.63396315,4.892457604 +csp1,2011,10118.39247,83.63396315,4.892457604 +csp1,2012,10118.39247,83.63396315,4.892457604 +csp1,2013,10118.39247,83.63396315,4.892457604 +csp1,2014,10118.39247,83.63396315,4.892457604 +csp1,2015,10118.39247,83.63396315,4.892457604 +csp1,2016,10118.39247,83.63396315,4.892457604 +csp1,2017,10118.39247,83.63396315,4.892457604 +csp1,2018,9655.056558,83.63396315,4.807945152 +csp1,2019,9205.528937,83.63396315,4.807945152 +csp1,2020,8731.09044,83.63396315,4.621539781 +csp1,2021,8256.651943,83.63396315,4.43513441 +csp1,2022,8233.260436,83.633757,4.435123 +csp2,2010,9022.519272,78.75663459,4.892457604 +csp2,2011,9022.519272,78.75663459,4.892457604 +csp2,2012,9022.519272,78.75663459,4.892457604 +csp2,2013,9022.519272,78.75663459,4.892457604 +csp2,2014,9022.519272,78.75663459,4.892457604 +csp2,2015,9022.519272,78.75663459,4.892457604 +csp2,2016,9022.519272,78.75663459,4.892457604 +csp2,2017,9022.519272,78.75663459,4.892457604 +csp2,2018,8605.929835,77.39619026,4.807945152 +csp2,2019,7448.628019,75.10677024,4.006648147 +csp2,2020,7408.195726,74.86846512,3.982154074 +csp2,2021,7367.763433,74.63016,3.95766 +csp2,2022,7346.908275,74.63016,3.95766 +csp3,2010,6378.213448,78.75663459,4.892457604 +csp3,2011,6378.213448,78.75663459,4.892457604 +csp3,2012,6378.213448,78.75663459,4.892457604 +csp3,2013,6378.213448,78.75663459,4.892457604 +csp3,2014,6378.213448,78.75663459,4.892457604 +csp3,2015,6378.213448,78.75663459,4.892457604 +csp3,2016,6378.213448,78.75663459,4.892457604 +csp3,2017,6378.213448,78.75663459,4.892457604 +csp3,2018,6290.932719,78.78932168,4.894488165 +csp3,2019,6036.47607,78.78932168,4.894488165 +csp3,2020,5631.600376,65.86614726,3.851035029 +csp3,2021,5226.724683,52.94297284,2.807581893 +csp3,2022,5211.052733,52.934062,2.807109 +csp4,2010,5499.241675,78.75663459,4.892457604 +csp4,2011,5499.241675,78.75663459,4.892457604 +csp4,2012,5499.241675,78.75663459,4.892457604 +csp4,2013,5499.241675,78.75663459,4.892457604 +csp4,2014,5499.241675,78.75663459,4.892457604 +csp4,2015,5499.241675,78.75663459,4.892457604 +csp4,2016,5499.241675,78.75663459,4.892457604 +csp4,2017,5499.241675,78.75663459,4.892457604 +csp4,2018,5411.623189,78.78932168,4.894488165 +csp4,2019,5195.786083,78.78932168,4.894488165 +csp4,2020,4855.091586,62.2584634,3.65971846 +csp4,2021,4514.397087,45.72760513,2.424948757 +csp4,2022,4500.611474,45.717374,2.424406 diff --git a/atb/manual_input/historical/fuelcell_ATB_historical_advanced.csv b/atb/manual_input/historical/fuelcell_ATB_historical_advanced.csv new file mode 100644 index 0000000..ddcabcb --- /dev/null +++ b/atb/manual_input/historical/fuelcell_ATB_historical_advanced.csv @@ -0,0 +1,27 @@ +i,t,capcost,fom,vom,heatrate +ng-fuel-cell,2010,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2011,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2012,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2013,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2014,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2015,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2016,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2017,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2018,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2019,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2020,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2021,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2022,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2023,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2024,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2025,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2026,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2027,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2028,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2029,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2030,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2031,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2032,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2033,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2034,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2035,1444.6,40.7,7.79,5.543 diff --git a/atb/manual_input/historical/fuelcell_ATB_historical_moderate.csv b/atb/manual_input/historical/fuelcell_ATB_historical_moderate.csv new file mode 100644 index 0000000..ddcabcb --- /dev/null +++ b/atb/manual_input/historical/fuelcell_ATB_historical_moderate.csv @@ -0,0 +1,27 @@ +i,t,capcost,fom,vom,heatrate +ng-fuel-cell,2010,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2011,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2012,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2013,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2014,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2015,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2016,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2017,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2018,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2019,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2020,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2021,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2022,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2023,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2024,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2025,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2026,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2027,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2028,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2029,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2030,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2031,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2032,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2033,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2034,9999.0,40.7,7.79,5.543 +ng-fuel-cell,2035,1444.6,40.7,7.79,5.543 diff --git a/atb/manual_input/historical/gas-ccs_ATB_historical_advanced.csv b/atb/manual_input/historical/gas-ccs_ATB_historical_advanced.csv new file mode 100644 index 0000000..f67ffa8 --- /dev/null +++ b/atb/manual_input/historical/gas-ccs_ATB_historical_advanced.csv @@ -0,0 +1,40 @@ +i,t,capcost,fom,vom,heatrate +Gas-CC-CCS_mod,2010,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2011,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2012,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2013,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2014,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2015,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2016,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2017,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2018,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2019,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2020,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2021,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2022,2656.6,70.7,5.11,7.22 +Gas-CC_H_1x1-CCS_mod,2010,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2011,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2012,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2013,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2014,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2015,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2016,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2017,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2018,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2019,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2020,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2021,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2022,2968.7,78.0,5.3,6.848 +Gas-CC_H_2x1-CCS_mod,2010,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2011,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2012,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2013,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2014,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2015,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2016,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2017,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2018,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2019,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2020,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2021,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2022,2524.1,65.8,4.86,7.007 diff --git a/atb/manual_input/historical/gas-ccs_ATB_historical_conservative.csv b/atb/manual_input/historical/gas-ccs_ATB_historical_conservative.csv new file mode 100644 index 0000000..f67ffa8 --- /dev/null +++ b/atb/manual_input/historical/gas-ccs_ATB_historical_conservative.csv @@ -0,0 +1,40 @@ +i,t,capcost,fom,vom,heatrate +Gas-CC-CCS_mod,2010,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2011,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2012,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2013,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2014,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2015,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2016,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2017,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2018,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2019,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2020,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2021,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2022,2656.6,70.7,5.11,7.22 +Gas-CC_H_1x1-CCS_mod,2010,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2011,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2012,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2013,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2014,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2015,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2016,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2017,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2018,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2019,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2020,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2021,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2022,2968.7,78.0,5.3,6.848 +Gas-CC_H_2x1-CCS_mod,2010,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2011,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2012,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2013,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2014,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2015,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2016,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2017,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2018,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2019,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2020,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2021,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2022,2524.1,65.8,4.86,7.007 diff --git a/atb/manual_input/historical/gas-ccs_ATB_historical_moderate.csv b/atb/manual_input/historical/gas-ccs_ATB_historical_moderate.csv new file mode 100644 index 0000000..f67ffa8 --- /dev/null +++ b/atb/manual_input/historical/gas-ccs_ATB_historical_moderate.csv @@ -0,0 +1,40 @@ +i,t,capcost,fom,vom,heatrate +Gas-CC-CCS_mod,2010,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2011,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2012,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2013,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2014,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2015,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2016,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2017,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2018,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2019,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2020,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2021,2572.02,68.796,5.0004,7.22 +Gas-CC-CCS_mod,2022,2656.6,70.7,5.11,7.22 +Gas-CC_H_1x1-CCS_mod,2010,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2011,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2012,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2013,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2014,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2015,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2016,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2017,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2018,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2019,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2020,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2021,2874.2,75.9,5.19,6.848 +Gas-CC_H_1x1-CCS_mod,2022,2968.7,78.0,5.3,6.848 +Gas-CC_H_2x1-CCS_mod,2010,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2011,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2012,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2013,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2014,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2015,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2016,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2017,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2018,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2019,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2020,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2021,2443.7,64.0,4.76,7.007 +Gas-CC_H_2x1-CCS_mod,2022,2524.1,65.8,4.86,7.007 diff --git a/atb/manual_input/historical/gas_ATB_historical_moderate.csv b/atb/manual_input/historical/gas_ATB_historical_moderate.csv new file mode 100644 index 0000000..76e6de1 --- /dev/null +++ b/atb/manual_input/historical/gas_ATB_historical_moderate.csv @@ -0,0 +1,94 @@ +i,t,capcost,fom,vom,heatrate +Gas-CC,2010,1141.254325,31.24999307,1.991757801,6.5375 +Gas-CC,2011,1136.675573,31.24999307,1.991757801,6.518111111 +Gas-CC,2012,1132.096819,31.24999307,1.991757801,6.498722222 +Gas-CC,2013,1127.518066,31.24999307,1.991757801,6.479333333 +Gas-CC,2014,1122.939312,31.24999307,1.991757801,6.459944444 +Gas-CC,2015,1118.360558,31.24999307,1.991757801,6.440555556 +Gas-CC,2016,1113.781805,31.24999307,1.991757801,6.421166667 +Gas-CC,2017,1109.203051,31.24999307,1.991757801,6.401777778 +Gas-CC,2018,1104.624297,31.24999307,1.991757801,6.382388889 +Gas-CC,2019,1100.045544,31.24999307,1.991757801,6.363 +Gas-CC,2020,1070.809499,31.66068212,1.967485246,6.363 +Gas-CC,2021,1205.712,33.588,2.1168,6.363 +Gas-CC,2022,1245.4,34.5,2.17,6.363 +Gas-CC_H_1x1,2010,1482.1,39.8,2.53,6.068 +Gas-CC_H_1x1,2011,1482.1,39.8,2.53,6.068 +Gas-CC_H_1x1,2012,1482.1,39.8,2.53,6.068 +Gas-CC_H_1x1,2013,1482.1,39.8,2.53,6.068 +Gas-CC_H_1x1,2014,1482.1,39.8,2.53,6.068 +Gas-CC_H_1x1,2015,1482.1,39.8,2.53,6.068 +Gas-CC_H_1x1,2016,1482.1,39.8,2.53,6.068 +Gas-CC_H_1x1,2017,1482.1,39.8,2.53,6.068 +Gas-CC_H_1x1,2018,1482.1,39.8,2.53,6.068 +Gas-CC_H_1x1,2019,1482.1,39.8,2.53,6.068 +Gas-CC_H_1x1,2020,1482.1,39.8,2.53,6.068 +Gas-CC_H_1x1,2021,1482.1,39.8,2.53,6.068 +Gas-CC_H_1x1,2022,1482.1,39.8,2.53,6.068 +Gas-CC_H_2x1,2010,1278.4,34.1,2.16,6.196 +Gas-CC_H_2x1,2011,1278.4,34.1,2.16,6.196 +Gas-CC_H_2x1,2012,1278.4,34.1,2.16,6.196 +Gas-CC_H_2x1,2013,1278.4,34.1,2.16,6.196 +Gas-CC_H_2x1,2014,1278.4,34.1,2.16,6.196 +Gas-CC_H_2x1,2015,1278.4,34.1,2.16,6.196 +Gas-CC_H_2x1,2016,1278.4,34.1,2.16,6.196 +Gas-CC_H_2x1,2017,1278.4,34.1,2.16,6.196 +Gas-CC_H_2x1,2018,1278.4,34.1,2.16,6.196 +Gas-CC_H_2x1,2019,1278.4,34.1,2.16,6.196 +Gas-CC_H_2x1,2020,1278.4,34.1,2.16,6.196 +Gas-CC_H_2x1,2021,1278.4,34.1,2.16,6.196 +Gas-CC_H_2x1,2022,1278.4,34.1,2.16,6.196 +Gas-CT,2010,1059.98145,23.92398737,5.654760652,10.2225 +Gas-CT,2011,1050.823943,23.92398737,5.654760652,10.16633333 +Gas-CT,2012,1041.666436,23.92398737,5.654760652,10.11016667 +Gas-CT,2013,1032.508929,23.92398737,5.654760652,10.054 +Gas-CT,2014,1023.351422,23.92398737,5.654760652,9.997833333 +Gas-CT,2015,1014.193915,23.92398737,5.654760652,9.941666667 +Gas-CT,2016,1005.036407,23.92398737,5.654760652,9.8855 +Gas-CT,2017,995.8789002,23.92398737,5.654760652,9.829333333 +Gas-CT,2018,986.7213932,23.92398737,5.654760652,9.773166667 +Gas-CT,2019,977.563886,23.92398737,5.654760652,9.717 +Gas-CT,2020,950.9512022,23.74551159,5.585848916,9.717 +Gas-CT,2021,1082.808,25.92,6.9552,9.717 +Gas-CT,2022,1110.4,26.1,6.94,9.717 +Gas-CT_aero,2010,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2011,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2012,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2013,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2014,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2015,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2016,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2017,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2018,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2019,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2020,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2021,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2022,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2023,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2024,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2025,1540.246,11.02,5.476,9.447 +Gas-CT_aero,2026,1521.614,10.977,5.476,9.447 +Gas-CT_aero,2027,1506.757,10.891,5.476,9.447 +Gas-CT_aero,2028,1478.9,10.848,5.476,9.447 +Gas-CT_aero,2029,1455.704,10.763,5.476,9.447 +Gas-CT_aero,2030,1436.601,10.72,5.476,9.447 +Gas-CT_aero,2031,1421.585,10.634,5.476,9.447 +Gas-CT_aero,2032,1408.486,10.591,5.476,9.447 +Gas-CT_aero,2033,1395.738,10.505,5.476,9.447 +Gas-CT_aero,2034,1368.85,10.463,5.476,9.447 +Gas-CT_aero,2035,1359.531,10.377,5.476,9.447 +Gas-CT_aero,2036,1349.629,10.334,5.476,9.447 +Gas-CT_aero,2037,1341.603,10.248,5.476,9.447 +Gas-CT_aero,2038,1337.263,10.205,5.476,9.447 +Gas-CT_aero,2039,1332.502,10.162,5.476,9.447 +Gas-CT_aero,2040,1326.853,10.077,5.476,9.447 +Gas-CT_aero,2041,1318.519,10.034,5.476,9.447 +Gas-CT_aero,2042,1314.485,9.948,5.476,9.447 +Gas-CT_aero,2043,1310.826,9.905,5.476,9.447 +Gas-CT_aero,2044,1306.131,9.819,5.476,9.447 +Gas-CT_aero,2045,1302.451,9.776,5.476,9.447 +Gas-CT_aero,2046,1297.241,9.691,5.476,9.447 +Gas-CT_aero,2047,1291.484,9.648,5.476,9.447 +Gas-CT_aero,2048,1287.003,9.562,5.476,9.447 +Gas-CT_aero,2049,1283.086,9.519,5.476,9.447 +Gas-CT_aero,2050,1275.174,9.433,5.476,9.447 diff --git a/atb/manual_input/historical/nuclear-smr_ATB_historical_advanced.csv b/atb/manual_input/historical/nuclear-smr_ATB_historical_advanced.csv new file mode 100644 index 0000000..b05ab7b --- /dev/null +++ b/atb/manual_input/historical/nuclear-smr_ATB_historical_advanced.csv @@ -0,0 +1,22 @@ +i,t,capcost,fom,vom,heatrate +Nuclear-SMR,2010,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2011,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2012,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2013,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2014,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2015,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2016,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2017,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2018,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2019,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2020,7699.675484,128.9042058,3.37433865,10.455 +Nuclear-SMR,2021,8658.389123,128.304,3.3804,10.455 +Nuclear-SMR,2022,5500.0,118.0,2.2,9.18 +Nuclear-SMR,2023,5500.0,118.0,2.2,9.18 +Nuclear-SMR,2024,5500.0,118.0,2.2,9.18 +Nuclear-SMR,2025,5500.0,118.0,2.2,9.18 +Nuclear-SMR,2026,5500.0,118.0,2.2,9.18 +Nuclear-SMR,2027,5500.0,118.0,2.2,9.18 +Nuclear-SMR,2028,5500.0,118.0,2.2,9.18 +Nuclear-SMR,2029,5500.0,118.0,2.2,9.18 +Nuclear-SMR,2030,5500.0,118.0,2.2,9.18 diff --git a/atb/manual_input/historical/nuclear-smr_ATB_historical_conservative.csv b/atb/manual_input/historical/nuclear-smr_ATB_historical_conservative.csv new file mode 100644 index 0000000..9845331 --- /dev/null +++ b/atb/manual_input/historical/nuclear-smr_ATB_historical_conservative.csv @@ -0,0 +1,22 @@ +i,t,capcost,fom,vom,heatrate +Nuclear-SMR,2010,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2011,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2012,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2013,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2014,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2015,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2016,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2017,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2018,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2019,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2020,7699.675484,128.9042058,3.37433865,10.455 +Nuclear-SMR,2021,8658.389123,128.304,3.3804,10.455 +Nuclear-SMR,2022,10000.0,216.0,2.8,9.18 +Nuclear-SMR,2023,10000.0,216.0,2.8,9.18 +Nuclear-SMR,2024,10000.0,216.0,2.8,9.18 +Nuclear-SMR,2025,10000.0,216.0,2.8,9.18 +Nuclear-SMR,2026,10000.0,216.0,2.8,9.18 +Nuclear-SMR,2027,10000.0,216.0,2.8,9.18 +Nuclear-SMR,2028,10000.0,216.0,2.8,9.18 +Nuclear-SMR,2029,10000.0,216.0,2.8,9.18 +Nuclear-SMR,2030,10000.0,216.0,2.8,9.18 diff --git a/atb/manual_input/historical/nuclear-smr_ATB_historical_moderate.csv b/atb/manual_input/historical/nuclear-smr_ATB_historical_moderate.csv new file mode 100644 index 0000000..2b970df --- /dev/null +++ b/atb/manual_input/historical/nuclear-smr_ATB_historical_moderate.csv @@ -0,0 +1,22 @@ +i,t,capcost,fom,vom,heatrate +Nuclear-SMR,2010,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2011,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2012,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2013,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2014,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2015,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2016,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2017,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2018,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2019,7624.567702,129.5986335,3.415967332,10.455 +Nuclear-SMR,2020,7699.675484,128.9042058,3.37433865,10.455 +Nuclear-SMR,2021,8658.389123,128.304,3.3804,10.455 +Nuclear-SMR,2022,8000.0,136.0,2.6,9.18 +Nuclear-SMR,2023,8000.0,136.0,2.6,9.18 +Nuclear-SMR,2024,8000.0,136.0,2.6,9.18 +Nuclear-SMR,2025,8000.0,136.0,2.6,9.18 +Nuclear-SMR,2026,8000.0,136.0,2.6,9.18 +Nuclear-SMR,2027,8000.0,136.0,2.6,9.18 +Nuclear-SMR,2028,8000.0,136.0,2.6,9.18 +Nuclear-SMR,2029,8000.0,136.0,2.6,9.18 +Nuclear-SMR,2030,8000.0,136.0,2.6,9.18 diff --git a/atb/manual_input/historical/nuclear_ATB_historical_advanced.csv b/atb/manual_input/historical/nuclear_ATB_historical_advanced.csv new file mode 100644 index 0000000..8731f56 --- /dev/null +++ b/atb/manual_input/historical/nuclear_ATB_historical_advanced.csv @@ -0,0 +1,22 @@ +i,t,capcost,fom,vom,heatrate +Nuclear,2010,7686.711275,165.9481454,2.692053726,10.455 +Nuclear,2011,7633.565731,165.9481454,2.692053726,10.455 +Nuclear,2012,7580.420186,165.9481454,2.692053726,10.455 +Nuclear,2013,7527.27464,165.9481454,2.692053726,10.455 +Nuclear,2014,7474.129095,165.9481454,2.692053726,10.455 +Nuclear,2015,7420.983549,165.9481454,2.692053726,10.455 +Nuclear,2016,7367.838005,165.9481454,2.692053726,10.455 +Nuclear,2017,7314.692459,165.9481454,2.692053726,10.455 +Nuclear,2018,7261.546913,165.9481454,2.692053726,10.455 +Nuclear,2019,7208.401368,165.9481454,2.692053726,10.455 +Nuclear,2020,7172.069171,165.0426129,2.659247015,10.455 +Nuclear,2021,8065.093644,164.2896,2.6676,10.455 +Nuclear,2022,5250.0,126.0,1.9,10.497 +Nuclear,2023,5250.0,126.0,1.9,10.497 +Nuclear,2024,5250.0,126.0,1.9,10.497 +Nuclear,2025,5250.0,126.0,1.9,10.497 +Nuclear,2026,5250.0,126.0,1.9,10.497 +Nuclear,2027,5250.0,126.0,1.9,10.497 +Nuclear,2028,5250.0,126.0,1.9,10.497 +Nuclear,2029,5250.0,126.0,1.9,10.497 +Nuclear,2030,5250.0,126.0,1.9,10.497 diff --git a/atb/manual_input/historical/nuclear_ATB_historical_conservative.csv b/atb/manual_input/historical/nuclear_ATB_historical_conservative.csv new file mode 100644 index 0000000..f44626f --- /dev/null +++ b/atb/manual_input/historical/nuclear_ATB_historical_conservative.csv @@ -0,0 +1,22 @@ +i,t,capcost,fom,vom,heatrate +Nuclear,2010,7686.711275,165.9481454,2.692053726,10.455 +Nuclear,2011,7633.565731,165.9481454,2.692053726,10.455 +Nuclear,2012,7580.420186,165.9481454,2.692053726,10.455 +Nuclear,2013,7527.27464,165.9481454,2.692053726,10.455 +Nuclear,2014,7474.129095,165.9481454,2.692053726,10.455 +Nuclear,2015,7420.983549,165.9481454,2.692053726,10.455 +Nuclear,2016,7367.838005,165.9481454,2.692053726,10.455 +Nuclear,2017,7314.692459,165.9481454,2.692053726,10.455 +Nuclear,2018,7261.546913,165.9481454,2.692053726,10.455 +Nuclear,2019,7208.401368,165.9481454,2.692053726,10.455 +Nuclear,2020,7172.069171,165.0426129,2.659247015,10.455 +Nuclear,2021,8065.093644,164.2896,2.6676,10.455 +Nuclear,2022,7750.0,204.0,3.4,10.497 +Nuclear,2023,7750.0,204.0,3.4,10.497 +Nuclear,2024,7750.0,204.0,3.4,10.497 +Nuclear,2025,7750.0,204.0,3.4,10.497 +Nuclear,2026,7750.0,204.0,3.4,10.497 +Nuclear,2027,7750.0,204.0,3.4,10.497 +Nuclear,2028,7750.0,204.0,3.4,10.497 +Nuclear,2029,7750.0,204.0,3.4,10.497 +Nuclear,2030,7750.0,204.0,3.4,10.497 diff --git a/atb/manual_input/historical/nuclear_ATB_historical_moderate.csv b/atb/manual_input/historical/nuclear_ATB_historical_moderate.csv new file mode 100644 index 0000000..088942b --- /dev/null +++ b/atb/manual_input/historical/nuclear_ATB_historical_moderate.csv @@ -0,0 +1,22 @@ +i,t,capcost,fom,vom,heatrate +Nuclear,2010,7686.711275,165.9481454,2.692053726,10.455 +Nuclear,2011,7633.565731,165.9481454,2.692053726,10.455 +Nuclear,2012,7580.420186,165.9481454,2.692053726,10.455 +Nuclear,2013,7527.27464,165.9481454,2.692053726,10.455 +Nuclear,2014,7474.129095,165.9481454,2.692053726,10.455 +Nuclear,2015,7420.983549,165.9481454,2.692053726,10.455 +Nuclear,2016,7367.838005,165.9481454,2.692053726,10.455 +Nuclear,2017,7314.692459,165.9481454,2.692053726,10.455 +Nuclear,2018,7261.546913,165.9481454,2.692053726,10.455 +Nuclear,2019,7208.401368,165.9481454,2.692053726,10.455 +Nuclear,2020,7172.069171,165.0426129,2.659247015,10.455 +Nuclear,2021,8065.093644,164.2896,2.6676,10.455 +Nuclear,2022,5750.0,175.0,2.8,10.497 +Nuclear,2023,5750.0,175.0,2.8,10.497 +Nuclear,2024,5750.0,175.0,2.8,10.497 +Nuclear,2025,5750.0,175.0,2.8,10.497 +Nuclear,2026,5750.0,175.0,2.8,10.497 +Nuclear,2027,5750.0,175.0,2.8,10.497 +Nuclear,2028,5750.0,175.0,2.8,10.497 +Nuclear,2029,5750.0,175.0,2.8,10.497 +Nuclear,2030,5750.0,175.0,2.8,10.497 diff --git a/atb/manual_input/historical/upv_ATB_historical_advanced.csv b/atb/manual_input/historical/upv_ATB_historical_advanced.csv new file mode 100644 index 0000000..621f0e1 --- /dev/null +++ b/atb/manual_input/historical/upv_ATB_historical_advanced.csv @@ -0,0 +1,14 @@ +t,capcost,fom,vom,cf_improvement +2010,7134.47322,35.6723661,0.0,0.930346357 +2011,6626.226717,35.06566801,0.0,0.930346357 +2012,6117.98021,34.45896991,0.0,0.930346357 +2013,4246.109494,33.85227179,0.0,0.930346357 +2014,3195.163027,33.2455737,0.0,0.930346357 +2015,3190.13065,32.6388756,0.0,0.930346357 +2016,2777.886445,32.0321775,0.0,0.930346357 +2017,1699.682398,30.81878129,0.0,0.930346357 +2018,1629.132468,28.50901493,0.0,0.930346357 +2019,1558.582537,26.19924857,0.0,0.930346357 +2020,1473.096512,25.58148089,0.0,0.930346357 +2021,1345.480289,24.33122164,0.0,0.930346357 +2022,1366.598204,23.765603,0.0,0.930346 diff --git a/atb/manual_input/historical/upv_ATB_historical_conservative.csv b/atb/manual_input/historical/upv_ATB_historical_conservative.csv new file mode 100644 index 0000000..2742518 --- /dev/null +++ b/atb/manual_input/historical/upv_ATB_historical_conservative.csv @@ -0,0 +1,14 @@ +t,capcost,fom,vom,cf_improvement +2010,7134.47322,35.6723661,0.0,0.920465813 +2011,6626.226717,35.06566801,0.0,0.920465813 +2012,6117.98021,34.45896991,0.0,0.920465813 +2013,4246.109494,33.85227179,0.0,0.920465813 +2014,3195.163027,33.2455737,0.0,0.920465813 +2015,3190.13065,32.6388756,0.0,0.920465813 +2016,2777.886445,32.0321775,0.0,0.920465813 +2017,1699.682398,30.81878129,0.0,0.920465813 +2018,1629.132468,28.50901493,0.0,0.920465813 +2019,1558.582537,26.19924857,0.0,0.920465813 +2020,1473.096512,25.58148089,0.0,0.920465813 +2021,1345.480289,24.33122164,0.0,0.920465813 +2022,1366.598204,23.765603,0.0,0.920466 diff --git a/atb/manual_input/historical/upv_ATB_historical_moderate.csv b/atb/manual_input/historical/upv_ATB_historical_moderate.csv new file mode 100644 index 0000000..201561f --- /dev/null +++ b/atb/manual_input/historical/upv_ATB_historical_moderate.csv @@ -0,0 +1,14 @@ +t,capcost,fom,vom,cf_improvement +2010,7134.47322,35.6723661,0.0,0.926146827 +2011,6626.226717,35.06566801,0.0,0.926146827 +2012,6117.98021,34.45896991,0.0,0.926146827 +2013,4246.109494,33.85227179,0.0,0.926146827 +2014,3195.163027,33.2455737,0.0,0.926146827 +2015,3190.13065,32.6388756,0.0,0.926146827 +2016,2777.886445,32.0321775,0.0,0.926146827 +2017,1699.682398,30.81878129,0.0,0.926146827 +2018,1629.132468,28.50901493,0.0,0.926146827 +2019,1558.582537,26.19924857,0.0,0.926146827 +2020,1473.096512,25.58148089,0.0,0.926146827 +2021,1345.480289,24.33122164,0.0,0.926146827 +2022,1366.598204,23.765603,0.0,0.926147 diff --git a/atb/manual_input/historical/wind-ofs_ATB_historical_advanced.csv b/atb/manual_input/historical/wind-ofs_ATB_historical_advanced.csv new file mode 100644 index 0000000..81842cd --- /dev/null +++ b/atb/manual_input/historical/wind-ofs_ATB_historical_advanced.csv @@ -0,0 +1,35 @@ +turbine,t,capcost,fom,vom,cf_improvement,rsc_mult +fixed,2010,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2011,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2012,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2013,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2014,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2015,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2016,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2017,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2018,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2019,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2020,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2021,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2022,4601.48313,90.91066,0.0,0.94615,1.0 +floating,2010,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2011,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2012,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2013,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2014,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2015,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2016,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2017,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2018,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2019,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2020,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2021,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2022,14229.47981,81.5295,0.0,0.94619,1.0 +floating,2023,9577.6067,68.07725,0.0,0.97388,1.0 +floating,2024,8500.13714,64.96144,0.0,0.99021,1.0 +floating,2025,7910.7477,63.25705,0.0,1.00162,1.0 +floating,2026,7114.39406,62.11205,0.0,1.01029,1.0 +floating,2027,6421.40359,61.26249,0.0,1.01724,1.0 +floating,2028,5790.91499,60.59367,0.0,1.02301,1.0 +floating,2029,5601.50773,60.04594,0.0,1.02793,1.0 +floating,2030,5441.95175,59.58449,0.0,1.03202,1.0 diff --git a/atb/manual_input/historical/wind-ofs_ATB_historical_conservative.csv b/atb/manual_input/historical/wind-ofs_ATB_historical_conservative.csv new file mode 100644 index 0000000..c107e60 --- /dev/null +++ b/atb/manual_input/historical/wind-ofs_ATB_historical_conservative.csv @@ -0,0 +1,35 @@ +turbine,t,capcost,fom,vom,cf_improvement,rsc_mult +fixed,2010,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2011,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2012,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2013,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2014,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2015,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2016,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2017,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2018,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2019,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2020,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2021,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2022,6359.77349,90.91066,0.0,0.94615,1.0 +floating,2010,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2011,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2012,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2013,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2014,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2015,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2016,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2017,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2018,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2019,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2020,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2021,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2022,16679.24373,81.5295,0.0,0.94619,1.0 +floating,2023,15201.41551,77.74446,0.0,0.94742,1.0 +floating,2024,14509.81355,75.97312,0.0,0.94854,1.0 +floating,2025,14064.92972,74.83367,0.0,0.94957,1.0 +floating,2026,13740.20065,74.00197,0.0,0.95052,1.0 +floating,2027,13486.1424,73.35127,0.0,0.9514,1.0 +floating,2028,13278.40941,72.81922,0.0,0.95223,1.0 +floating,2029,13103.27806,72.37067,0.0,0.953,1.0 +floating,2030,12344.1771,71.9839,0.0,0.95372,1.0 diff --git a/atb/manual_input/historical/wind-ofs_ATB_historical_moderate.csv b/atb/manual_input/historical/wind-ofs_ATB_historical_moderate.csv new file mode 100644 index 0000000..ebaa24e --- /dev/null +++ b/atb/manual_input/historical/wind-ofs_ATB_historical_moderate.csv @@ -0,0 +1,35 @@ +turbine,t,capcost,fom,vom,cf_improvement,rsc_mult +fixed,2010,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2011,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2012,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2013,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2014,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2015,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2016,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2017,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2018,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2019,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2020,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2021,5237.4605,90.9107,0.0,0.94619,1.0 +fixed,2022,5237.46054,90.91066,0.0,0.94615,1.0 +floating,2010,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2011,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2012,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2013,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2014,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2015,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2016,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2017,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2018,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2019,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2020,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2021,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2022,15115.56463,81.5295,0.0,0.94619,1.0 +floating,2023,11918.12957,72.69266,0.0,0.95632,1.0 +floating,2024,10952.52993,70.02401,0.0,0.96351,1.0 +floating,2025,10396.79349,68.4881,0.0,0.96904,1.0 +floating,2026,10013.32394,67.4283,0.0,0.97351,1.0 +floating,2027,9723.70334,66.62786,0.0,0.97726,1.0 +floating,2028,9075.70416,65.98939,0.0,0.98046,1.0 +floating,2029,8467.56141,65.46107,0.0,0.98327,1.0 +floating,2030,7888.16784,65.01223,0.0,0.98574,1.0 diff --git a/atb/manual_input/historical/wind-ons_ATB_historical_advanced.csv b/atb/manual_input/historical/wind-ons_ATB_historical_advanced.csv new file mode 100644 index 0000000..7cabef1 --- /dev/null +++ b/atb/manual_input/historical/wind-ons_ATB_historical_advanced.csv @@ -0,0 +1,14 @@ +turbine,t,capcost,fom,vom,cf_improvement +115hh_170rd,2010,2639.12325,47.01292,0.0,0.61277 +115hh_170rd,2011,2466.78385,47.01292,0.0,0.60304 +115hh_170rd,2012,2294.44446,47.01292,0.0,0.69848 +115hh_170rd,2013,2157.98645,47.01292,0.0,0.79294 +115hh_170rd,2014,2021.52845,47.01292,0.0,0.87144 +115hh_170rd,2015,1909.53503,47.01292,0.0,0.82324 +115hh_170rd,2016,1797.54161,47.01292,0.0,0.89312 +115hh_170rd,2017,1795.05224,47.01292,0.0,0.85843 +115hh_170rd,2018,1649.67472,47.01292,0.0,0.92017 +115hh_170rd,2019,1504.29721,47.01292,0.0,0.85963 +115hh_170rd,2020,1517.57908,46.44,0.0,0.83469 +115hh_170rd,2021,1386.59368,32.724,0.0,0.76975 +115hh_170rd,2022,1471.55594,32.44305,0.0,0.94599 diff --git a/atb/manual_input/historical/wind-ons_ATB_historical_conservative.csv b/atb/manual_input/historical/wind-ons_ATB_historical_conservative.csv new file mode 100644 index 0000000..c6f1967 --- /dev/null +++ b/atb/manual_input/historical/wind-ons_ATB_historical_conservative.csv @@ -0,0 +1,14 @@ +turbine,t,capcost,fom,vom,cf_improvement +115hh_170rd,2010,2639.12325,47.01292,0.0,0.61277 +115hh_170rd,2011,2466.78385,47.01292,0.0,0.60304 +115hh_170rd,2012,2294.44446,47.01292,0.0,0.69848 +115hh_170rd,2013,2157.98645,47.01292,0.0,0.79294 +115hh_170rd,2014,2021.52845,47.01292,0.0,0.87144 +115hh_170rd,2015,1909.53503,47.01292,0.0,0.82324 +115hh_170rd,2016,1797.54161,47.01292,0.0,0.89312 +115hh_170rd,2017,1795.05224,47.01292,0.0,0.85843 +115hh_170rd,2018,1649.67472,47.01292,0.0,0.92017 +115hh_170rd,2019,1504.29721,47.01292,0.0,0.85963 +115hh_170rd,2020,1517.57908,46.44,0.0,0.83469 +115hh_170rd,2021,1386.59368,32.724,0.0,0.76975 +115hh_170rd,2022,1471.55594,32.44305,0.0,0.93285 diff --git a/atb/manual_input/historical/wind-ons_ATB_historical_moderate.csv b/atb/manual_input/historical/wind-ons_ATB_historical_moderate.csv new file mode 100644 index 0000000..21ba64c --- /dev/null +++ b/atb/manual_input/historical/wind-ons_ATB_historical_moderate.csv @@ -0,0 +1,14 @@ +turbine,t,capcost,fom,vom,cf_improvement +115hh_170rd,2010,2639.12325,47.01292,0.0,0.61277 +115hh_170rd,2011,2466.78385,47.01292,0.0,0.60304 +115hh_170rd,2012,2294.44446,47.01292,0.0,0.69848 +115hh_170rd,2013,2157.98645,47.01292,0.0,0.79294 +115hh_170rd,2014,2021.52845,47.01292,0.0,0.87144 +115hh_170rd,2015,1909.53503,47.01292,0.0,0.82324 +115hh_170rd,2016,1797.54161,47.01292,0.0,0.89312 +115hh_170rd,2017,1795.05224,47.01292,0.0,0.85843 +115hh_170rd,2018,1649.67472,47.01292,0.0,0.92017 +115hh_170rd,2019,1504.29721,47.01292,0.0,0.85963 +115hh_170rd,2020,1517.57908,46.44,0.0,0.83469 +115hh_170rd,2021,1386.59368,32.724,0.0,0.76975 +115hh_170rd,2022,1471.55594,32.44305,0.0,0.93873 diff --git a/atb/manual_input/offshore_cost_multipliers_2024.csv b/atb/manual_input/offshore_cost_multipliers_2024.csv new file mode 100644 index 0000000..af02848 --- /dev/null +++ b/atb/manual_input/offshore_cost_multipliers_2024.csv @@ -0,0 +1,3 @@ +turbine,capcost,fom +fixed,1.190671038860,1.002655373133 +floating,1.372107839750,0.978659140911 diff --git a/atb/manual_input/offshore_cost_multipliers_2025.csv b/atb/manual_input/offshore_cost_multipliers_2025.csv new file mode 100644 index 0000000..af02848 --- /dev/null +++ b/atb/manual_input/offshore_cost_multipliers_2025.csv @@ -0,0 +1,3 @@ +turbine,capcost,fom +fixed,1.190671038860,1.002655373133 +floating,1.372107839750,0.978659140911 diff --git a/atb/scraped_input/README.md b/atb/scraped_input/README.md new file mode 100644 index 0000000..430da78 --- /dev/null +++ b/atb/scraped_input/README.md @@ -0,0 +1,104 @@ +# scraped_input/ + +This directory exposes the independent raw downloads used by the NLR ATB +workflow. None of the source files is generated from another. + +| File | Upstream source | Used for | +| --- | --- | --- | +| `atb__flat_file.csv` | ATB summary flat file (`ATBe.csv`) on OEDI | Primary cost, performance, and financial data | +| `atb__workbook.xlsx` | ATB Excel data workbook | Battery power ($/kW) and energy ($/kWh) capital-cost components | + +## Observed historical capital costs + +The optional `historical_costs/` subdirectory contains original research +workbooks from LBNL, NLR, and EIA together with two generated raw-data aids: + +- `historical_capital_costs.csv`: long-form observed capital costs with units, + capacity basis, statistic, geography, dollar year, and source provenance. +- `source_manifest.csv`: upstream URLs, local filenames, byte counts, and + SHA-256 checksums for every downloaded workbook. + +Download and extract them separately from the ATB inputs: + +```bash +python ../scripts/scrape_historical_costs.py +``` + +Use `--only wind`, `--only solar`, `--only offshore`, or `--only eia` to limit +the sources. Use `--no-download` to rebuild the two CSVs from existing local +workbooks. The formatter uses only explicitly reviewed mappings under +`historical_cost_sources.reeds_mappings` in `config.yaml`. Currently, the LBNL +AC-based UPV and land-based-wind series and the NLR offshore fixed-bottom series +replace their respective `capcost` columns before the configured projection +boundary; other series remain review inputs. + +Important distinctions retained in the CSV are: + +- LBNL utility-PV costs are included on both AC and DC capacity bases. +- NLR offshore costs remain separate by geography (`Global`, `Europe and United + States`, `Asia`). None is U.S.-only. Pipeline years after 2023 are excluded. + The workbook states no units, but the report's Figure 31 axis reads + `USD2023/kW` and its section 1.2.2 normalizes every cost to real 2023 USD by + currency conversion followed by U.S. CPI inflation, so the rows are recorded + as `dollar_year: 2023`, `price_basis: real`. +- EIA costs are nominal in the year of installation, and its categories follow + EIA definitions rather than ATB technology definitions. + +### Which EIA tables are used + +Each EIA workbook holds roughly a dozen tables. `source_table` records which one +a row came from, because the same label means different things in different +tables: + +| `source_table` | EIA table | Adds | +| --- | --- | --- | +| `major_energy_source` | Generators installed by major energy source | The broad fuel categories: gas, wind, solar, battery, biomass, hydro, geothermal, petroleum | +| `prime_mover` | Generators installed by prime mover | Equipment-level detail; the only table reporting fuel cells, and it isolates onshore wind and photovoltaics from their broader categories | +| `natural_gas_technology` | Natural gas generators installed by technology (and the 2013-only "by plant type") | Whole-plant combined cycle, rather than the cost split across its turbine halves | + +The remaining tables break the same capacity down by Census region, state, plant +size, wind class, or PV panel type. None can become a national ReEDS series, so +they are skipped deliberately rather than by omission. + +Some labels are intentionally left unmapped: `Steam turbine` in the prime-mover +table has an ambiguous fuel, `... (as part of combined cycle)` rows are one half +of a plant reported whole elsewhere, and `Internal combustion engine` in that +table spans both gas and oil units. + +Coverage gaps are real absences, not extraction failures: EIA reports only +categories with installations in a given year, which is why biomass skips 2018 +and geothermal appears only in 2013. The series also cannot start before 2013 — +`generatorcosts/archive/` has no editions before that year. + +The URLs and local filenames are declared under `raw_data:` in +[`../config.yaml`](../config.yaml). Download and inspect both files with: + +```bash +python ../scripts/scrape_atb_inputs.py +``` + +Useful variants: + +```bash +python ../scripts/scrape_atb_inputs.py --only flat +python ../scripts/scrape_atb_inputs.py --only workbook +python ../scripts/scrape_atb_inputs.py --force +``` + +Without `--force`, existing files are reused and displayed. These files remain +raw and unmodified: derived ReEDS inputs go in `../output/`, and plots go in +`../figures/`. + +HTTPS certificates are checked first. The +`raw_data.allow_insecure_ssl_fallback` setting controls whether a failed check +may be retried with `verify=False`, which can be needed when the active conda +environment does not trust an NLR network inspection certificate. + +All raw files and generated raw-data CSVs are local only and are never +committed. The ATB flat file is far +above the 50 MB limit in the top-level [`README.md`](../../README.md) File Size +Guidelines, and both files can be pulled directly from their upstream source, so +this directory is ignored except for this README. Reproducibility comes from the +URLs instead: they are pinned to a specific ATB release in +[`../config.yaml`](../config.yaml) (ATB 2024 is pinned to corrected release v3), +so re-running the scraper restores the exact inputs used for a run. diff --git a/atb/scripts/README.md b/atb/scripts/README.md new file mode 100644 index 0000000..de5c5ab --- /dev/null +++ b/atb/scripts/README.md @@ -0,0 +1,37 @@ +# scripts/ + +All scripts share the user configuration in [`../config.yaml`](../config.yaml). +User-facing run instructions are in the main [`../README.md`](../README.md). + +| File | Role | +| --- | --- | +| `run_pipeline.py` | Runs the config-enabled scrape, format, and plot stages in order | +| `scrape_atb_inputs.py` | Downloads and summarizes both raw ATB inputs | +| `scrape_historical_costs.py` | Separately downloads and normalizes observed capital-cost workbooks | +| `generate_atb_files.py` | Converts local raw and manual inputs into ReEDS-formatted CSVs | +| `atb_plotting.py` | Plots configured metrics from the local raw flat file | +| `atb_config.py` | Shared config and path-loading functions | +| `settings.yaml` | Internal per-technology mappings and transformations | +| `battery_workbook.py` | Extracts battery cost components from the downloaded workbook; imported by the formatter | + +Normal workflow choices belong in `../config.yaml`; `settings.yaml` is only for +technology-specific formatting rules. + +## settings.yaml keys that control the ReEDS output format + +Most keys describe how to pull a technology out of the ATB flat file. These two +instead describe how the result must be written so ReEDS can read it, and are +only needed where the ReEDS representation differs from the internal one: + +| Key | Effect | +| --- | --- | +| `reeds_name` | File prefix used in ReEDS when it differs from the technology key (`wind-ons` -> `ons-wind`, `wind-ofs` -> `ofs-wind`). Also becomes the `Scenario` key written to the ReEDS `dollaryear.csv`. | +| `output_cols` | Ordered mapping of internal column name -> ReEDS header, applied as the last step before writing. Required for the wind files because ReEDS assigns their columns by position, not by name. | + +Both are optional. A technology without them is written as +`_ATB__.csv` using the internal column names listed under +`cols`. `output_cols` must cover every column in `cols`; the formatter raises an +error rather than silently dropping or reordering one. + +See "File names and column schemas expected by ReEDS" in the main +[`../README.md`](../README.md) for why the wind files need this. diff --git a/atb/scripts/atb_config.py b/atb/scripts/atb_config.py new file mode 100644 index 0000000..cc80688 --- /dev/null +++ b/atb/scripts/atb_config.py @@ -0,0 +1,77 @@ +"""Shared configuration and path helpers for the ATB workflow.""" + +from pathlib import Path + +import yaml + + +SCRIPT_DIR = Path(__file__).resolve().parent +ATB_DIR = SCRIPT_DIR.parent +DEFAULT_CONFIG_PATH = ATB_DIR / "config.yaml" +TECH_SETTINGS_PATH = SCRIPT_DIR / "settings.yaml" + +ATBE_COLUMN_MAPPING = { + "core_metric_parameter": "Parameter", + "core_metric_case": "Case", + "tax_credit_case": "TaxCreditCase", + "crpyears": "CRPYears", + "technology": "Technology", + "technology_alias": "TechnologyAlias", + "techdetail": "TechDetail", + "techdetail2": "TechDetail2", + "resourcedetail": "ResourceDetail", + "display_name": "DisplayName", + "scenario": "Scenario", + "core_metric_variable": "variable", +} + + +def load_config(config_path=None): + """Load the user-facing workflow configuration.""" + path = Path(config_path).resolve() if config_path else DEFAULT_CONFIG_PATH + with path.open(encoding="utf-8") as stream: + config = yaml.safe_load(stream) + config["_config_path"] = str(path) + return config + + +def resolve_atb_path(value): + """Resolve an absolute path or a path relative to the atb directory.""" + path = Path(value) + return path if path.is_absolute() else ATB_DIR / path + + +def raw_file_path(config, kind): + """Return the configured local path for ``flat_file`` or ``workbook``.""" + raw = config["raw_data"] + return resolve_atb_path(raw["directory"]) / raw[kind]["filename"] + + +def load_processing_settings(config_path=None): + """Merge user config with the internal per-technology settings.""" + config = load_config(config_path) + with TECH_SETTINGS_PATH.open(encoding="utf-8") as stream: + settings = yaml.safe_load(stream) + + processing = config["processing"] + history = config["historical_data"] + settings.update( + { + "atbyear": config["atb"]["year"], + "dollaryear": config["atb"]["dollar_year"], + "reedspath": str(Path(processing["reeds_repo"])), + "copy_to_reeds": bool(processing["copy_to_reeds"]), + "reeds_start_year": int(processing["reeds_start_year"]), + "decimals": int(processing["decimals"]), + "output_dir": str(resolve_atb_path(processing["output_directory"])), + "flat_file_path": str(raw_file_path(config, "flat_file")), + "workbook_path": str(raw_file_path(config, "workbook")), + "history_dir": str(resolve_atb_path(history["directory"])), + "history_dollar_year": int(history["dollar_year"]), + "seed_missing_history": bool( + history.get("seed_missing_from_reeds", False) + ), + "config": config, + } + ) + return settings diff --git a/atb/scripts/atb_plotting.py b/atb/scripts/atb_plotting.py new file mode 100644 index 0000000..9f7c2f8 --- /dev/null +++ b/atb/scripts/atb_plotting.py @@ -0,0 +1,200 @@ +"""Plot raw NLR ATB metrics using the shared workflow configuration.""" + +import argparse +import math + +import matplotlib.pyplot as plt +import pandas as pd + +from atb_config import ATBE_COLUMN_MAPPING, load_config, raw_file_path, resolve_atb_path + + +SCENARIOS = ("Moderate", "Advanced", "Conservative") +UNITS = { + "CAPEX": "USD/kW", + "OCC": "USD/kW", + "Fixed O&M": "USD/kW-yr", + "Variable O&M": "USD/MWh", +} + + +def select_year_ticks(years, max_ticks=5): + """Return at most ``max_ticks`` evenly spaced years present in the data.""" + years = sorted({int(year) for year in years if pd.notna(year)}) + if len(years) <= max_ticks: + return years + indices = { + round(position * (len(years) - 1) / (max_ticks - 1)) + for position in range(max_ticks) + } + return [years[index] for index in sorted(indices)] + + +def load_raw_atb(config): + """Load the same local flat file used by the ReEDS formatter.""" + path = raw_file_path(config, "flat_file") + if not path.is_file(): + raise FileNotFoundError( + f"Raw ATB flat file not found: {path}\n" + "Run 'python scrape_atb_inputs.py --only flat' first." + ) + print(f"Loading raw ATB flat file: {path}") + return pd.read_csv(path, low_memory=False).rename(columns=ATBE_COLUMN_MAPPING) + + +def clean_metric_data(raw, metric, case, crp_years, technologies): + """Select and aggregate one metric into a plotting-friendly table.""" + required = { + "Technology", "DisplayName", "Scenario", "Parameter", "Case", + "CRPYears", "variable", "value", + } + missing = sorted(required - set(raw.columns)) + if missing: + raise ValueError(f"Raw ATB flat file is missing columns: {missing}") + + data = raw.loc[ + (raw["Parameter"] == metric) + & (raw["Case"] == case) + & (raw["CRPYears"] == crp_years) + & (raw["Technology"].isin(technologies)) + & (raw["Scenario"].isin(SCENARIOS)), + ["Technology", "DisplayName", "Scenario", "variable", "value"], + ].copy() + data["variable"] = pd.to_numeric(data["variable"], errors="coerce") + data["value"] = pd.to_numeric(data["value"], errors="coerce") + data = data.dropna(subset=["variable", "value"]) + # Some raw ATBe rows differ in metadata not needed for plotting. Average + # exact display/scenario/year duplicates into one visible series. + return ( + data.groupby( + ["Technology", "DisplayName", "Scenario", "variable"], + as_index=False, + )["value"] + .mean() + .sort_values(["Technology", "DisplayName", "Scenario", "variable"]) + ) + + +def plot_metric(data, metric, config, figure_format, output_dir): + """Write one multi-panel figure and return its path.""" + technologies = config["plotting"]["technologies"] + technologies = [tech for tech in technologies if tech in set(data["Technology"])] + if not technologies: + raise ValueError( + f"No data found for metric={metric!r}, case={config['plotting']['case']!r}, " + f"CRPYears={config['plotting']['crp_years']}." + ) + + ncols = min(3, len(technologies)) + nrows = math.ceil(len(technologies) / ncols) + figure, axes = plt.subplots( + nrows, ncols, figsize=(5.2 * ncols, 3.6 * nrows), squeeze=False + ) + colors = plt.get_cmap("tab20") + + for panel, technology in enumerate(technologies): + axis = axes.flat[panel] + tech_data = data.loc[data["Technology"] == technology] + display_names = sorted(tech_data["DisplayName"].dropna().unique()) + for index, display_name in enumerate(display_names): + series = tech_data.loc[tech_data["DisplayName"] == display_name] + pivot = series.pivot_table( + index="variable", columns="Scenario", values="value", aggfunc="mean" + ).sort_index() + if "Moderate" not in pivot: + continue + color = colors(index % colors.N) + axis.plot(pivot.index, pivot["Moderate"], color=color, label=display_name) + if {"Advanced", "Conservative"}.issubset(pivot.columns): + band = pivot[["Advanced", "Conservative"]].dropna() + axis.fill_between( + band.index, + band.min(axis=1), + band.max(axis=1), + color=color, + alpha=0.18, + linewidth=0, + ) + axis.set_title(technology) + axis.set_xlabel("Projection year") + year_ticks = select_year_ticks(tech_data["variable"]) + axis.set_xticks(year_ticks, labels=[str(year) for year in year_ticks]) + axis.tick_params(axis="x", labelsize=8, pad=3) + axis.margins(x=0.03) + unit = UNITS.get(metric, "value") + axis.set_ylabel(f"{metric} ({config['atb']['dollar_year']} {unit})") + axis.grid(axis="y", linestyle=":", alpha=0.5) + if len(display_names) > 1: + axis.legend(fontsize=7, frameon=False) + + for panel in range(len(technologies), nrows * ncols): + axes.flat[panel].set_visible(False) + + figure.suptitle( + f"NLR ATB {config['atb']['year']} — {metric} " + f"({config['plotting']['case']}, CRP {config['plotting']['crp_years']})" + ) + figure.tight_layout() + output_dir.mkdir(parents=True, exist_ok=True) + safe_metric = metric.lower().replace(" ", "_").replace("&", "and") + output = output_dir / f"atb_{config['atb']['year']}_{safe_metric}.{figure_format}" + figure.savefig(output, dpi=200, bbox_inches="tight") + plt.close(figure) + print(f"Saved plot: {output}") + return output + + +def make_plots(config, metrics=None, show=False, save_cleaned=False): + """Create all configured plots from the local raw flat file.""" + plot_config = config["plotting"] + metrics = metrics or plot_config["metrics"] + raw = load_raw_atb(config) + output_dir = resolve_atb_path(plot_config["output_directory"]) + outputs = [] + for metric in metrics: + data = clean_metric_data( + raw, + metric, + plot_config["case"], + plot_config["crp_years"], + plot_config["technologies"], + ) + if save_cleaned: + output_dir.mkdir(parents=True, exist_ok=True) + clean_path = output_dir / ( + f"atb_{config['atb']['year']}_{metric.lower().replace(' ', '_')}_cleaned.csv" + ) + # LF line endings on every platform, matching the formatter's output + data.to_csv(clean_path, index=False, lineterminator="\n") + print(f"Saved cleaned plotting data: {clean_path}") + outputs.append( + plot_metric(data, metric, config, plot_config["figure_format"], output_dir) + ) + if show: + print("Plots were saved; open the files listed above to view them.") + return outputs + + +def main(): + parser = argparse.ArgumentParser(description="Plot configured raw NLR ATB metrics.") + parser.add_argument("--config", help="Path to config.yaml (default: ../config.yaml).") + parser.add_argument( + "--metric", "-m", nargs="+", + help="One or more ATB parameters; defaults to plotting.metrics in config.yaml.", + ) + parser.add_argument( + "--save-cleaned", action="store_true", + help="Also save the cleaned data behind each plot.", + ) + parser.add_argument("--show", action="store_true", help="Print a viewing reminder.") + args = parser.parse_args() + make_plots( + load_config(args.config), + metrics=args.metric, + show=args.show, + save_cleaned=args.save_cleaned, + ) + + +if __name__ == "__main__": + main() diff --git a/atb/scripts/battery_workbook.py b/atb/scripts/battery_workbook.py new file mode 100644 index 0000000..d697a1c --- /dev/null +++ b/atb/scripts/battery_workbook.py @@ -0,0 +1,86 @@ +"""Extract ReEDS battery cost components from a downloaded ATB workbook.""" + +from pathlib import Path + +import openpyxl +import pandas as pd + + +BATTERY_SHEET = "Utility-Scale Battery Storage" +BLOCKS = { + "Capital Cost ($/kWh)": "capcost_energy", + "Battery Power Capital Cost ($/kW)": "capcost", +} +SCENARIOS = ["Advanced", "Moderate", "Conservative"] + + +def extract_battery_costs(xlsx_path): + """Return battery power/energy capital costs by scenario and year.""" + workbook_path = Path(xlsx_path) + workbook = openpyxl.load_workbook( + workbook_path, read_only=True, data_only=True + ) + try: + if BATTERY_SHEET not in workbook.sheetnames: + raise ValueError( + f"Sheet {BATTERY_SHEET!r} not found in {workbook_path}." + ) + rows = list(workbook[BATTERY_SHEET].iter_rows(values_only=True)) + finally: + workbook.close() + + def find_label_row(label): + for index, row in enumerate(rows): + if any( + isinstance(value, str) and value.strip() == label + for value in row[:6] + ): + return index + raise ValueError(f"Label {label!r} not found in {BATTERY_SHEET}.") + + records = [] + for label, cost_name in BLOCKS.items(): + label_row = find_label_row(label) + year_header = rows[label_row + 1] + year_columns = [ + (column, int(value)) + for column, value in enumerate(year_header) + if isinstance(value, (int, float)) and 2018 <= value <= 2061 + ] + if not year_columns: + raise ValueError(f"No year header found below {label!r}.") + + for offset, scenario in enumerate(SCENARIOS): + data_row = rows[label_row + 2 + offset] + labels = [ + str(value).strip() + for value in data_row[:5] + if isinstance(value, str) + ] + if scenario not in labels: + raise ValueError( + f"Expected scenario {scenario!r} below {label!r}, " + f"got {labels!r}." + ) + record = {"cost": cost_name, "Scenario": scenario} + record.update( + {year: data_row[column] for column, year in year_columns} + ) + records.append(record) + + frame = pd.DataFrame(records) + year_columns = sorted( + column for column in frame.columns if isinstance(column, int) + ) + cost_order = {"capcost": 0, "capcost_energy": 1} + scenario_order = {scenario: index for index, scenario in enumerate(SCENARIOS)} + frame = ( + frame.assign( + _cost_order=frame["cost"].map(cost_order), + _scenario_order=frame["Scenario"].map(scenario_order), + ) + .sort_values(["_cost_order", "_scenario_order"]) + .drop(columns=["_cost_order", "_scenario_order"]) + .reset_index(drop=True) + ) + return frame[["cost", "Scenario", *year_columns]] diff --git a/atb/scripts/generate_atb_files.py b/atb/scripts/generate_atb_files.py new file mode 100644 index 0000000..326bb86 --- /dev/null +++ b/atb/scripts/generate_atb_files.py @@ -0,0 +1,2273 @@ +""" +This script processes and produces ATB costs input files for ReEDS. +Workflow choices are defined in ../config.yaml; technology-specific formatting +rules are defined in settings.yaml. This formatter reads local raw inputs only. +The script runs in the standard ReEDS environment (reeds2). +""" + +#%% =========================================================================== +### --- IMPORTS --- +### =========================================================================== +import argparse +import os +import numpy as np +import pandas as pd +import shutil +from pandas.errors import EmptyDataError +import sys + +from atb_config import ATBE_COLUMN_MAPPING, load_processing_settings + +# directory containing this script (atb/scripts); settings.yaml lives here. +THISDIR = os.path.dirname(os.path.abspath(__file__)) +# parent atb/ directory; used to build paths to input/, manual_input/, output/. +ATBDIR = os.path.dirname(THISDIR) + +# define zero marginal cost technologies (to assign zero 'vom' if input data is missing) +zero_vom_techs = {'upv', 'wind-ons', 'wind-ofs', 'battery'} + +# write LF line endings on every platform so repeated runs on Windows do not +# rewrite every versioned CSV with different line endings than the stored ones +CSV_LINE_TERMINATOR = "\n" + +#%% =========================================================================== +### --- FUNCTIONS --- +### =========================================================================== + +def subset_atb_rows(subset_rows, atb_data_in): + """ + Subset the ATB flat file data to desired Case | CRPYears rows + + Parameters + ---------- + subset_rows: dict + Default {Case: "Exp + TC", CRPYears: 30} for most techs; some techs can have other values for Case, e.g., Market, R&D etc.) + atb_data_in: pd.DataFrame + ATB flat file dataframe to subset + """ + atb_data = atb_data_in.copy() + atb_col_vals = {} + # subset to relevant cases (typically Market case and 30-year CRP, but specified in settings.yaml) + for sr in subset_rows: + print(f"Subsetting to {sr}={subset_rows[sr]}") + if sr not in atb_data.columns: + raise ValueError(f"Column {sr} not in ATB file.") + else: + atb_col_vals[sr] = atb_data_in[sr].unique().tolist() + atb_data = atb_data.loc[atb_data[sr] == subset_rows[sr]] + return atb_data, atb_col_vals + +def load_atb_flat_file(settings, args, techs_to_run): + """ + load ATB flat file with all inputs for specified techs + + Parameters + ---------- + settings: dict + Parameter settings from settings.yaml + args: argparse.Namespace + Used for debug mode + techs_to_run: list[str] + Ordered list of tech keys (matching keys in settings['techs']) + """ + filepath = settings['flat_file_path'] + if not os.path.isfile(filepath): + raise FileNotFoundError( + f"Raw ATB flat file not found: {filepath}\n" + "Run 'python scrape_atb_inputs.py' before formatting ReEDS inputs." + ) + print(f"Loading raw ATB flat file: {filepath}") + atb_data_in = pd.read_csv(filepath, low_memory=False) + # Apply the same mapping for downloaded ATBe files and canonical flat files. + atb_data_in = atb_data_in.rename(columns=ATBE_COLUMN_MAPPING) + + if args.debug: + breakpoint() + + # store validated/subsetted data for all requested techs here + validated_data_list = [] + + # iterate over requested techs to validate and subset immediately + for tech in techs_to_run: + tech_settings = settings['techs'][tech] + + # 1. Get tech-specific subset settings + if 'subset_rows' not in tech_settings: + raise ValueError(f"Missing 'subset_rows' for {tech} in settings.yaml") + + # 2. Filter by Technology + tech_rows_full = atb_data_in[atb_data_in['Technology'] == tech_settings['Technology']] + + if tech_rows_full.empty: + print(f"Warning: No data found for Technology '{tech_settings['Technology']}' ({tech}).") + continue + + # 3. Apply specific subset_rows (Case, CRPYears, etc.) + # subset_atb_rows returns (subset_df, col_vals) + tech_subset, _ = subset_atb_rows(tech_settings['subset_rows'], tech_rows_full) + + # 4. Filter for specific DisplayName(s) if applicable + if 'DisplayName' in tech_settings: + atb_techs = tech_settings['DisplayName'] + atb_techs_list = list(atb_techs.keys()) if isinstance(atb_techs, dict) else [atb_techs] + + # append all subtechs present + found_data = tech_subset[tech_subset['DisplayName'].isin(atb_techs_list)] + missing_subtechs = [t for t in atb_techs_list if t not in found_data['DisplayName'].values] + + if found_data is not None and not found_data.empty: + validated_data_list.append(found_data) + + # fallback: handle any subtechs that were missing from the subset + for atb_tech_name in missing_subtechs: + # Fallback logic: check if it exists in the wider tech data (e.g., for a different Case) + temp_df = tech_rows_full[tech_rows_full['DisplayName'] == atb_tech_name] + + if temp_df.empty: + print(f"Warning: DisplayName '{atb_tech_name}' not found in source data for {tech}.") + continue + print(f"Could not subset {atb_tech_name} data to {tech_settings['subset_rows']}.") + print(f"Available Case options: {temp_df.Case.unique()}") + + new_case = str(input("Select an alternative Case to proceed, or (q) to quit: ")) + + if new_case.lower() == "q": + sys.exit("Exiting now.") + elif new_case not in temp_df.Case.unique(): + print(f"'{new_case}' not found in available Case options. Exiting.") + sys.exit(1) + else: + # Grab the data for the alternative case + new_subset_rows = tech_settings['subset_rows'].copy() + new_subset_rows['Case'] = new_case + + # We only need this specific subtech with the new case + subtech_data, _ = subset_atb_rows(new_subset_rows, temp_df) + validated_data_list.append(subtech_data) + else: + # If no DisplayName filtering is needed, just keep the successful subset + if not tech_subset.empty: + validated_data_list.append(tech_subset) + + # Combine all valid chunks + if not validated_data_list: + raise EmptyDataError(f"\n\nNo valid data found for any of the requested technologies: {techs_to_run}") + + atb_data_final = pd.concat(validated_data_list, ignore_index=True) + return atb_data_final + +def get_atb_file_path(tech, atbyear, scenario, settings): + """ + function to construct full filepath and per-tech ATB filename + + Parameters + ---------- + tech: str + Tech identifier used as the file prefix (e.g., 'gas', 'upv' etc.) + atbyear: int + ATB year (e.g., 2025) to process + scenario: str + ATB scenario (Moderate/Advanced) from settings.yaml + settings: dict + Loaded settings from settings.yaml + """ + # allow per-tech override of the filename in existing ReEDS input files + # e.g. settings['techs']['wind-ons']['reeds_name'] = "ons-wind" + filename_root = settings['techs'].get(tech, {}).get('reeds_name', tech) + filename = f"{filename_root}_ATB_{atbyear}_{scenario.lower()}" + filepath = os.path.join( + settings['reedspath'], 'inputs', 'plant_characteristics', filename + ".csv" + ) + return filename, filepath + +def check_columns(df, col_list, setting, tech): + """ + function to check that all columns specified in settings.yml exist in the data + + Parameters + ---------- + col_list: list[str] + Sequence of required column names based on tech settings + setting: str + Settings key being validated (used in the error message, e.g. 'indexcols' or 'cols') + tech: str + Technology identifier + """ + missing_cols = [c for c in col_list if c not in df.columns] + if len(missing_cols) > 0: + raise ValueError( + f"The following columns are specified as '{setting}' but are not in the data: {missing_cols}. " + f"Please update your 'settings.yaml' for {tech}." + ) + +def _history_file_path(tech, scenario, settings): + """Return the stable, year-independent history path for one scenario.""" + return os.path.join( + settings['history_dir'], + f"{tech}_ATB_historical_{str(scenario).lower()}.csv", + ) + + +def _cost_columns(df): + """Return every monetary output column, including battery energy costs.""" + return [ + col for col in df.columns + if col == 'vom' or col.startswith('capcost') or col.startswith('fom') + ] + + +def _select_pre_projection_history(history, current, idcols): + """Keep history before each current series starts and all retired series.""" + if not idcols: + return history.loc[history['t'] < current['t'].min()].copy() + + starts = ( + current.groupby(idcols, dropna=False, as_index=False)['t'] + .min() + .rename(columns={'t': '_projection_start'}) + ) + selected = history.merge(starts, on=idcols, how='left') + selected = selected.loc[ + selected['_projection_start'].isna() + | (selected['t'] < selected['_projection_start']) + ] + return selected.drop(columns='_projection_start') + + +def _projection_boundary_rows(current, idcols): + """Select the first current-ATB year for every active technology series.""" + if not idcols: + return current.loc[current['t'] == current['t'].min()].copy() + starts = current.groupby(idcols, dropna=False)['t'].transform('min') + return current.loc[current['t'] == starts].copy() + + +def _validate_year_continuity(frame, tech, settings): + """Fail if any output series has a missing year or starts after ReEDS.""" + groupcols = [ + col for col in settings['techs'][tech]['indexcols'] if col != 't' + ] + failures = [] + grouper = groupcols[0] if len(groupcols) == 1 else groupcols + for group_values, group in frame.groupby(grouper, dropna=False, sort=False): + years = sorted(pd.to_numeric(group['t'], errors='raise').astype(int).unique()) + expected = set(range(settings['reeds_start_year'], max(years) + 1)) + missing = sorted(expected - set(years)) + if years[0] != settings['reeds_start_year'] or missing: + values = group_values if isinstance(group_values, tuple) else (group_values,) + label = dict(zip(groupcols, values)) + failures.append( + f"{label}: starts {years[0]}, missing {missing[:10]}" + ) + if failures: + raise ValueError( + f"Historical continuity check failed for {tech}: " + + "; ".join(failures[:10]) + ) + + +def _normalize_reeds_history(frame, tech, settings): + """Normalize a ReEDS file to the current technology's output schema.""" + tech_settings = settings['techs'][tech] + frame = frame.rename(columns=tech_settings.get('renamecols', {})).copy() + required = tech_settings['cols'] + missing = [col for col in required if col not in frame.columns] + if missing: + raise KeyError( + f"Historical ReEDS file for {tech} is missing columns {missing}. " + f"Available columns: {list(frame.columns)}" + ) + frame = frame[required] + frame['t'] = pd.to_numeric(frame['t'], errors='raise').astype(int) + return frame + + +def _seed_history_from_reeds( + tech, scenario, current, history_path, settings, dollaryear, deflator +): + """Create a local historical baseline from the current ReEDS ATB file.""" + reeds_key, reeds_path = get_atb_file_path( + tech, settings['atbyear'], scenario, settings + ) + if not os.path.isfile(reeds_path): + moderate_key, moderate_path = get_atb_file_path( + tech, settings['atbyear'], 'moderate', settings + ) + if os.path.isfile(moderate_path): + print( + f"...no {str(scenario).lower()} ReEDS history for {tech}; " + "using moderate history" + ) + reeds_key, reeds_path = moderate_key, moderate_path + else: + raise FileNotFoundError( + f"Historical baseline is missing: {history_path}\n" + f"Could not seed it because neither the matching nor moderate " + f"ReEDS file exists: {reeds_path}" + ) + + history = _normalize_reeds_history(pd.read_csv(reeds_path), tech, settings) + idcols = [ + col for col in settings['techs'][tech]['indexcols'] + if col not in ['Scenario', 't'] + ] + history = _select_pre_projection_history(history, current, idcols) + + source_dollar_year = int(dollaryear[reeds_key]) + history_dollar_year = settings['history_dollar_year'] + adjustment = ( + deflator[source_dollar_year] / deflator[history_dollar_year] + ) + costcols = _cost_columns(history) + history[costcols] = history[costcols] * adjustment + history = history.sort_values(settings['techs'][tech]['cols']).reset_index(drop=True) + os.makedirs(settings['history_dir'], exist_ok=True) + history.to_csv(history_path, index=False, lineterminator=CSV_LINE_TERMINATOR) + print( + f"...seeded {os.path.basename(history_path)} from " + f"{os.path.basename(reeds_path)}" + ) + return history + + +def _observed_history_source_path(settings): + """Return the configured normalized observed-history CSV path.""" + source_settings = settings['config'].get('historical_cost_sources', {}) + return os.path.join( + ATBDIR, + source_settings['directory'], + source_settings['normalized_filename'], + ) + + +def _observed_values_by_year(tech, mapping, settings, deflator): + """Read, filter, and deflate one observed series into {year: value}.""" + source_path = _observed_history_source_path(settings) + if not os.path.isfile(source_path): + raise FileNotFoundError( + f"Observed historical-cost file is missing: {source_path}. " + "Run scripts/scrape_historical_costs.py first." + ) + observed = pd.read_csv(source_path) + for column, value in mapping.get('filters', {}).items(): + if column not in observed.columns: + raise KeyError( + f"Observed historical-cost file is missing filter column {column}." + ) + observed = observed.loc[observed[column] == value] + if observed.empty: + raise KeyError( + f"No observed historical rows match the configured mapping for {tech}." + ) + if observed['year'].duplicated().any(): + duplicate_years = sorted( + observed.loc[observed['year'].duplicated(False), 'year'].unique() + ) + raise ValueError( + f"Observed historical mapping for {tech} has duplicate years: " + f"{duplicate_years}" + ) + if observed['dollar_year'].isna().any(): + raise ValueError( + f"Observed historical mapping for {tech} requires a dollar_year." + ) + + current_dollar_year = int(settings['dollaryear']) + converted_values = {} + for row in observed.itertuples(index=False): + source_dollar_year = int(row.dollar_year) + if source_dollar_year not in deflator.index: + raise KeyError( + f"Deflator table has no value for observed dollar year " + f"{source_dollar_year}." + ) + converted_values[int(row.year)] = ( + float(row.value) + * float(deflator[source_dollar_year]) + / float(deflator[current_dollar_year]) + ) + return converted_values + + +def _complete_observed_years( + values, required_years, tech, label=None, report=True +): + """Fill a real-history series without falling back to manual history. + + Missing years between observations are linearly interpolated. Years before + the first or after the last observation use the nearest observed endpoint, + because linear interpolation requires an observation on each side. + """ + if not values or not required_years: + return values + observed_years = np.asarray(sorted(values), dtype=float) + if len(observed_years) < 2 and not required_years.issubset(values): + raise ValueError( + f"Observed historical mapping for {label or tech} needs at least " + "two observations to fill missing years." + ) + observed_values = np.asarray( + [values[int(year)] for year in observed_years], dtype=float + ) + missing_years = sorted(required_years - set(values)) + if not missing_years: + return values + + completed = dict(values) + completed.update({ + year: float(np.interp(year, observed_years, observed_values)) + for year in missing_years + }) + if report: + internal = [ + year for year in missing_years + if observed_years[0] < year < observed_years[-1] + ] + endpoints = sorted(set(missing_years) - set(internal)) + if internal: + print( + f" {label or tech}: linearly interpolated missing real-history " + f"years {internal}" + ) + if endpoints: + print( + f" {label or tech}: filled endpoint real-history years " + f"{endpoints} from the nearest observation" + ) + return completed + + +def _assign_observed_values(frame, mask, output_column, values): + """Write the observed series into the masked rows and report the span.""" + result = frame.copy() + result.loc[mask, output_column] = result.loc[mask, 't'].map(values) + return result, sorted(result.loc[mask, 't'].astype(int).unique()) + + +def _year_key(frame): + """Columns identifying one value slot per year. + + Frames stack every ATB scenario, and historical years are identical across + them, so repetition across `Scenario` is expected and anything else is not. + """ + return [column for column in ('Scenario', 't') if column in frame.columns] + + +def _assert_one_row_per_year(frame, mask, tech, applier_name): + """Fail loudly when one observed value would address several rows.""" + candidates = frame.loc[mask] + duplicated = candidates.duplicated(subset=_year_key(frame), keep=False) + if duplicated.any(): + crowded_years = sorted( + candidates.loc[duplicated, 't'].astype(int).unique() + ) + raise ValueError( + f"{tech} uses the {applier_name} observed-history applier, but " + f"more than one row shares a scenario and year in {crowded_years}. " + f"One observed value cannot address them all. Define a " + f"technology-specific applier for {tech} and register it in " + f"REAL_HISTORY_APPLIERS." + ) + + +def _only_series(resolved, tech, applier_name): + """Unpack a mapping that must resolve to exactly one observed series.""" + if len(resolved) != 1: + raise ValueError( + f"{tech} uses the {applier_name} observed-history applier, which " + f"takes one series, but its mapping resolved {len(resolved)}. " + f"Use an applier that assigns each series to its own rows." + ) + return resolved[0] + + +def _series_target(series, tech): + """Return the one sub-technology a mapping series describes. + + A series measures one thing, so it names one sub-technology. Sharing a + series across several made unrelated rows carry identical history and hid + the fact that no observation existed for the others. + """ + targets = series.get('turbine_classes') or series.get('technologies') or [] + if len(targets) != 1: + raise ValueError( + f"{tech} observed-history mapping series names {targets or 'nothing'}. " + "Each series describes exactly one sub-technology; give the others " + "their own series or their own mode in historical_data." + ) + return targets[0] + + +def _assign_series_to_target( + frame, tech, mapping, values, historical_mask, key_column, target +): + """Assign one observed series to the rows of one sub-technology.""" + if key_column not in frame.columns: + raise KeyError( + f"{tech} observed-history applier expects a {key_column!r} column " + "identifying the sub-technology." + ) + present = set(frame[key_column].unique()) + if target not in present: + raise ValueError( + f"{tech} observed-history mapping targets {target!r}, which is " + f"absent from the frame. Present: {sorted(present)}." + ) + mask = ( + historical_mask + & frame['t'].isin(values) + & (frame[key_column] == target) + ) + _assert_one_row_per_year(frame, mask, tech, key_column) + return _assign_observed_values( + frame, mask, mapping['output_column'], values + ) + + +def apply_real_history_single_series( + frame, tech, mapping, resolved, historical_mask, historical_data, class_column +): + """Applier for technologies carrying one row per scenario-year. + + upv and wind-ons each select a single ATB DisplayName. The guard catches a + technology that splits a year into several series being routed here. + """ + _series, values = _only_series(resolved, tech, 'single-series') + mask = historical_mask & frame['t'].isin(values) + _assert_one_row_per_year(frame, mask, tech, 'single-series') + return _assign_observed_values(frame, mask, mapping['output_column'], values) + + +def apply_real_history_by_class( + frame, tech, mapping, resolved, historical_mask, historical_data, class_column +): + """Applier for technologies carrying one row per sub-technology. + + Offshore wind splits by turbine class and gas by plant configuration; both + assign each series to the one sub-technology it measures. A sub-technology + no series describes must select its own mode in historical_data, so none + falls back to manual values by accident. + """ + if not class_column: + raise KeyError( + f"{tech} uses the per-class observed-history applier but declares " + "no history_class_column in settings.yaml." + ) + present = ( + set(frame[class_column].unique()) if class_column in frame.columns else set() + ) + result = frame + replaced = set() + targeted = set() + for series, values in resolved: + target = _series_target(series, tech) + result, years = _assign_series_to_target( + result, tech, mapping, values, historical_mask, class_column, target + ) + replaced.update(years) + targeted.add(target) + metric = mapping['output_column'] + configured = set(_metric_modes(historical_data, metric)) - {None} + untouched = sorted(present - targeted - configured) + if untouched: + raise KeyError( + f"{tech}.{metric} selects real history, but {untouched} have no " + "observed series. Give each one its own mode in historical_data." + ) + return result, sorted(replaced) + + +# Each applier owns which rows its series may address, since that depends on +# the technology's row shape. Every metric selecting `real` requires an entry. +# Each receives the metric's historical_data modes so it can tell a row left +# unclaimed by accident from one that explicitly generates its own history. +REAL_HISTORY_APPLIERS = { + 'biopower': apply_real_history_single_series, + 'upv': apply_real_history_single_series, + 'wind-ons': apply_real_history_single_series, + 'wind-ofs': apply_real_history_by_class, + 'gas': apply_real_history_by_class, +} + + +def _resolve_observed_series( + tech, + mapping, + settings, + deflator, + required_years, + report_fill=True, +): + """Resolve a mapping into [(series_config, {year: value}), ...]. + + A mapping describes one series directly or several under `series`. Entries + inherit the mapping's other keys, so shared settings are written once. + """ + entries = mapping.get('series') or [mapping] + inherited = {key: value for key, value in mapping.items() if key != 'series'} + resolved = [] + for entry in entries: + merged = {**inherited, **entry} + values = _observed_values_by_year(tech, merged, settings, deflator) + targets = merged.get('technologies') or merged.get('turbine_classes') + label = f"{tech} {', '.join(targets)}" if targets else tech + values = _complete_observed_years( + values, + required_years, + tech, + label, + report=report_fill, + ) + resolved.append((merged, values)) + return resolved + + +def _apply_real_historical_costs(frame, tech, settings, deflator): + """Replace mapped pre-projection metrics with normalized observed values.""" + technology_config = _technology_smoothing_config(tech, settings) + if technology_config is None: + return frame + real_metrics = _real_historical_metrics(technology_config) + if not real_metrics: + return frame + smoothing = settings['config']['processing']['smooth_cost_curves'] + source_settings = settings['config'].get('historical_cost_sources', {}) + technology_mappings = source_settings.get('reeds_mappings', {}).get(tech, {}) + applier = REAL_HISTORY_APPLIERS.get(tech) + if applier is None: + raise NameError( + f"{tech} selects a real historical metric but has no observed-history " + "applier. Define one and add it to REAL_HISTORY_APPLIERS." + ) + projection_start_year = int(smoothing.get('projection_start_year', 2022)) + historical_mask = frame['t'] < projection_start_year + required_years = set(frame.loc[historical_mask, 't'].astype(int).unique()) + result = frame + for metric in real_metrics: + mapping = technology_mappings.get(metric) + if mapping is None: + raise ValueError( + f"{tech}.{metric} selects historical_data: real but has no " + "reviewed mapping under historical_cost_sources.reeds_mappings." + ) + if metric not in result.columns: + raise KeyError( + f"Observed-history metric {metric!r} is absent for {tech}." + ) + metric_mapping = {**mapping, 'output_column': metric} + resolved = _resolve_observed_series( + tech, + metric_mapping, + settings, + deflator, + required_years, + ) + result, replaced_years = applier( + result, + tech, + metric_mapping, + resolved, + historical_mask, + technology_config['historical_data'], + settings['techs'][tech].get('history_class_column'), + ) + if replaced_years: + current_dollar_year = int(settings['dollaryear']) + print( + f"Applied real historical {metric} for {tech}: " + f"{replaced_years[0]}-{replaced_years[-1]} " + f"({current_dollar_year}$)" + ) + return result + + +def validate_real_historical_data(settings, techs, deflator): + """Fail early when a selected ``real`` technology lacks observed data. + + The normal per-technology checks still run while values are applied. This + preflight check gathers configuration and source-data problems before any + technology output is processed, and gives the user one recovery command. + """ + smoothing = settings['config']['processing'].get('smooth_cost_curves', {}) + if not smoothing.get('enabled', False): + return + real_metrics = [] + for tech in techs: + technology_config = _technology_smoothing_config(tech, settings) + if technology_config is not None: + real_metrics.extend( + (tech, metric) + for metric in _real_historical_metrics(technology_config) + ) + if not real_metrics: + return + real_labels = [f"{tech}.{metric}" for tech, metric in real_metrics] + + source_settings = settings['config'].get('historical_cost_sources', {}) + missing_source_settings = [ + key for key in ('directory', 'normalized_filename') + if not source_settings.get(key) + ] + if missing_source_settings: + raise ValueError( + "Historical data validation failed for technologies configured " + f"with historical_data: real: {real_labels}. Missing settings under " + f"historical_cost_sources: {missing_source_settings}." + ) + + source_path = _observed_history_source_path(settings) + if not os.path.isfile(source_path): + raise FileNotFoundError( + "Historical data validation failed for technologies configured " + f"with historical_data: real: {real_labels}.\n" + f"The normalized historical-cost file does not exist: {source_path}\n" + "Run 'python scripts/scrape_historical_costs.py' from the atb " + "directory, then run the formatter again." + ) + + mappings = source_settings.get('reeds_mappings', {}) + if not isinstance(mappings, dict): + raise TypeError("historical_cost_sources.reeds_mappings must be a mapping.") + projection_start_year = int(smoothing.get('projection_start_year', 2022)) + required_years = set(range( + int(settings['reeds_start_year']), projection_start_year + )) + missing_choices = [] + failures = [] + for tech, metric in real_metrics: + mapping = mappings.get(tech, {}).get(metric) + if mapping is None: + missing_choices.append( + f"{tech}.{metric}: no mapping under " + "historical_cost_sources.reeds_mappings" + ) + continue + if REAL_HISTORY_APPLIERS.get(tech) is None: + missing_choices.append( + f"{tech}.{metric}: no observed-history applier in " + "REAL_HISTORY_APPLIERS" + ) + continue + if metric not in settings['techs'][tech]['cols']: + missing_choices.append( + f"{tech}.{metric}: metric is not in its ReEDS schema" + ) + continue + try: + _resolve_observed_series( + tech, + {**mapping, 'output_column': metric}, + settings, + deflator, + required_years, + report_fill=False, + ) + except KeyError as error: + missing_choices.append(f"{tech}.{metric}: {error}") + except (FileNotFoundError, TypeError, ValueError) as error: + failures.append(f"{tech}.{metric}: {error}") + + if missing_choices: + details = "\n".join(f" - {failure}" for failure in missing_choices) + raise KeyError( + "A selected real historical metric is unavailable:\n" + f"{details}\n" + "Choose an available option in config.yaml or add and review the " + "required source mapping." + ) + if failures: + details = "\n".join(f" - {failure}" for failure in failures) + raise ValueError( + "Historical data validation failed for one or more technologies " + "configured with historical_data: real:\n" + f"{details}\n" + "The combined historical CSV may be missing a source. Run " + "'python scripts/scrape_historical_costs.py --no-download' from " + "the atb directory to rebuild it from all local source files, or " + "review the mappings in config.yaml." + ) + + print( + "Validated observed historical data for metrics configured as real: " + f"{real_labels}" + ) + + +def merge_historical_atb_data( + tech_data, tech, settings, dollaryear, deflator +): + """Combine versioned local history with current scraped ATB projections. + + The local history is stored in a fixed dollar year. For each active series, + only rows before the first current scraped year are used. Series no longer + published by ATB are retained from history. The first current projection + year is appended to the stored history for the next annual update. + """ + tech_settings = settings['techs'][tech] + idcols = [ + col for col in tech_settings['indexcols'] + if col not in ['Scenario', 't'] + ] + combined = [] + series_starts = {} + current_dollar_year = settings['dollaryear'] + history_dollar_year = settings['history_dollar_year'] + + for scenario in tech_data['Scenario'].unique(): + current = tech_data.loc[tech_data['Scenario'] == scenario].copy() + history_path = _history_file_path(tech, scenario, settings) + if os.path.isfile(history_path): + history_stored = _normalize_reeds_history( + pd.read_csv(history_path), tech, settings + ) + elif settings['seed_missing_history']: + history_stored = _seed_history_from_reeds( + tech, scenario, current, history_path, settings, dollaryear, deflator + ) + else: + raise FileNotFoundError( + f"Historical baseline is missing: {history_path}. " + "Enable historical_data.seed_missing_from_reeds or add the file." + ) + + history_for_output = _select_pre_projection_history( + history_stored, current, idcols + ) + costcols = _cost_columns(history_for_output) + adjustment = ( + deflator[history_dollar_year] / deflator[current_dollar_year] + ) + history_for_output[costcols] = history_for_output[costcols] * adjustment + history_for_output['Scenario'] = scenario + combined.extend([history_for_output, current]) + + for _, row in _projection_boundary_rows(current, idcols).iterrows(): + series_starts[(scenario, *(row[column] for column in idcols))] = int(row['t']) + boundary = _projection_boundary_rows(current, idcols)[tech_settings['cols']] + boundary = boundary.copy().round( + tech_settings.get('decimals', settings['decimals']) + ) + boundary_costcols = _cost_columns(boundary) + boundary[boundary_costcols] = boundary[boundary_costcols] / adjustment + updated_history = pd.concat( + [history_stored, boundary], ignore_index=True + ).drop_duplicates(subset=[*idcols, 't'], keep='last') + updated_history = updated_history.sort_values( + tech_settings['cols'] + ).reset_index(drop=True) + updated_history.to_csv( + history_path, index=False, lineterminator=CSV_LINE_TERMINATOR + ) + + output = pd.concat(combined, ignore_index=True) + output = output.drop_duplicates( + subset=tech_settings['indexcols'], keep='last' + ) + output = output.reset_index(drop=True) + settings.setdefault('atb_series_start', {})[tech] = series_starts + output = _apply_real_historical_costs(output, tech, settings, deflator) + _validate_year_continuity(output, tech, settings) + return output + +def normalize_cf(tech, settings, df): + """ + function to normalize capacity factor to relevant base year and scenario (specified by 'cfbase') + + Parameters + ---------- + tech: str + Technology key in settings['techs'] for which cf_improvement values will be normalized + settings: dict + Parsed settings.yaml values. Expects settings['techs'][tech]['cfbase'] to define the base + selection. Example cfbase entries: + - keys with exact match values to select the base row(s) + - optional 'keepcols' list to indicate merge keys when multiple columns define the base + - 't' may be used to indicate the base year + """ + print("Normalizing cf_improvement") + tech_settings = settings['techs'][tech] + cf_base = df.copy() + for k,v in tech_settings['cfbase'].items(): + if k in df.columns: + print(f"...base {k}={v}") + cf_base = cf_base.loc[(cf_base[k] == v)] + if 'keepcols' in tech_settings['cfbase']: + cf_base = cf_base[tech_settings['cfbase']['keepcols'] + ['cf_improvement']].rename(columns={"cf_improvement":"cf_base"}) + df = df.merge(cf_base, on=tech_settings['cfbase']['keepcols'], how='left') + df['cf_improvement'] = df['cf_improvement'] / df['cf_base'] + df = df.drop(columns='cf_base') + else: + if len(cf_base) > 1: + raise ValueError("Error: more than one value found for cf_improvement without specifying which columns to keep.") + else: + df['cf_improvement'] = df['cf_improvement'] / cf_base['cf_improvement'].values[0] + + # check that base level is properly normalized to 1 + assert (df.loc[cf_base.index, 'cf_improvement'] == 1).all(), "Check cf_improvement normalization." + + return df + +def apply_offshore_cost_multipliers(tech, settings, df): + """Apply ReEDS offshore configuration adjustments to ATB class proxies.""" + multiplier_path = os.path.join( + ATBDIR, + "manual_input", + f"offshore_cost_multipliers_{settings['atbyear']}.csv", + ) + multipliers = pd.read_csv(multiplier_path).set_index("turbine") + missing = set(df["turbine"].unique()) - set(multipliers.index) + if missing: + raise ValueError( + f"Missing offshore cost multiplier(s) for turbine(s): {sorted(missing)}" + ) + for metric in ("capcost", "fom"): + df[metric] *= df["turbine"].map(multipliers[metric]) + return df + + +def apply_coal_projection_overrides(tech, settings, df): + """Preserve the published ReEDS 2024 coal values changed by ATB v3.""" + override_path = os.path.join( + ATBDIR, + "manual_input", + f"coal_projection_overrides_{settings['atbyear']}.csv", + ) + overrides = pd.read_csv(override_path) + overrides = overrides.loc[overrides["tech"] == tech].drop(columns="tech") + keys = ["Scenario", "i", "t"] + values = [column for column in overrides if column not in keys] + indexed = df.set_index(keys) + override_indexed = overrides.set_index(keys) + missing = override_indexed.index.difference(indexed.index) + if not missing.empty: + raise ValueError(f"Coal override rows not found in ATB data: {missing.tolist()}") + indexed.update(override_indexed[values]) + return indexed.reset_index() + +def smooth_hist_cf(tech, settings, df): + """ + function to interpolate historical capacity factor values through 2035 + + Parameters + ---------- + tech: str + Technology key in settings['techs'] to use when looking up cfbase/base year + settings: dict + Expects settings['techs'][tech]['cfbase']['t'] to specify the base year for interpolation + df: pd.DataFrame + Input dataframe containing ['Scenario', 't', 'cf_improvement'] columns + """ + # sort by scenario and year + df = df.sort_values(by=['Scenario', 't']) + # remove values between 2022 (ATB starts in 2023) and atb base year (usually 2035) + tech_settings = settings['techs'][tech] + baseyear = tech_settings['cfbase']['t'] + df['cf_improvement'] = np.where((df['t'] > 2022) & (df['t'] < baseyear), np.nan, df['cf_improvement']) + # interpolate to fill dropped values + df['cf_improvement'] = df.groupby('Scenario')['cf_improvement'].transform(lambda x: x.interpolate(method='linear')) + + return df + + +# Divide-by-zero floor when expressing a change relative to its own value. +RELATIVE_SCALE_FLOOR = 1e-9 + + +def _relative_change(left, right, epsilon): + """Return a scale-independent change between two finite values.""" + return abs(right - left) / max(abs(left), abs(right), epsilon) + + +def _smooth_curve_segment( + years, + values, + slope_change_threshold, + max_kink_years, + major_step_threshold, +): + """Smooth compact kink clusters in one time segment. + + Runs of equal values are left alone. A flat stretch is how ATB states that + a cost stops improving, so bridging one would replace a published + trajectory with a slope ATB never gave. + """ + years = np.asarray(years, dtype=float) + smoothed = np.asarray(values, dtype=float).copy() + if len(smoothed) < 3: + return smoothed + + year_deltas = np.diff(years) + if np.any(year_deltas <= 0): + raise ValueError("Cost smoothing requires unique, increasing years.") + + year_span = years[-1] - years[0] + slopes = np.diff(smoothed) / year_deltas + baseline_slope = abs((smoothed[-1] - smoothed[0]) / year_span) + epsilon = max(np.max(np.abs(smoothed)) * 1e-12, 1e-12) + candidate_knots = [] + for knot in range(1, len(smoothed) - 1): + left_slope = slopes[knot - 1] + right_slope = slopes[knot] + scale = max(abs(left_slope), abs(right_slope), baseline_slope, epsilon) + if abs(right_slope - left_slope) / scale >= slope_change_threshold: + candidate_knots.append(knot) + + clusters = [] + for knot in candidate_knots: + if clusters and knot == clusters[-1][-1] + 1: + clusters[-1].append(knot) + else: + clusters.append([knot]) + for cluster in clusters: + first, last = cluster[0], cluster[-1] + if len(cluster) < 2 or years[last] - years[first] > max_kink_years: + continue + left = max(0, first - 1) + right = min(len(smoothed) - 1, last + 2) + step_changes = [ + _relative_change(smoothed[position], smoothed[position + 1], epsilon) + for position in range(left, right) + ] + if any(change >= major_step_threshold for change in step_changes): + continue + interpolation_index = np.arange(left, right + 1) + smoothed[interpolation_index] = np.interp( + years[interpolation_index], + [years[left], years[right]], + [smoothed[left], smoothed[right]], + ) + return smoothed + + +def _selective_smooth_cost_values( + years, + values, + projection_start_year, + slope_change_threshold, + max_kink_years, + major_step_threshold, + historical_data_mode, + smooth_projection_curve, +): + """Apply one historical mode and independent future smoothing treatments.""" + years = np.asarray(years, dtype=float) + smoothed = np.asarray(values, dtype=float).copy() + if len(smoothed) < 3: + return smoothed + + anchor_candidates = np.flatnonzero(years == projection_start_year) + if len(anchor_candidates) != 1: + raise ValueError( + "Selective cost smoothing requires exactly one row for " + f"projection_start_year={projection_start_year}." + ) + anchor = int(anchor_candidates[0]) + + # Broadcast the first ATB projection value backward across the complete + # historical period. This makes `broadcast` an exclusive generated history + # rather than a selective treatment mixed with manual input values. + if historical_data_mode == 'broadcast': + smoothed[:anchor] = smoothed[anchor] + if not smooth_projection_curve: + return smoothed + + smoothed[anchor:] = _smooth_curve_segment( + years[anchor:], + smoothed[anchor:], + slope_change_threshold, + max_kink_years, + major_step_threshold, + ) + return smoothed + + +FUTURE_SMOOTHING_TREATMENTS = ( + 'smooth_projection_curve', +) +HISTORICAL_DATA_MODES = ('real', 'manual', 'broadcast') +NON_HISTORY_COLUMNS = {'Scenario', 'i', 't', 'turbine', 'type'} + + +def _metric_modes(historical_data, metric): + """Return one metric's configured modes keyed by sub-technology class. + + A metric configured as a plain mode applies to every row and is returned + under the ``None`` key. A metric split by class returns one entry per + class, which lets an observed series describe some rows of a technology + while the rest generate their own history. + """ + try: + configured = historical_data[metric] + except KeyError as error: + raise KeyError( + f"Historical data has no configured entry for metric {metric!r}." + ) from error + if isinstance(configured, dict): + return dict(configured) + return {None: configured} + + +def _historical_mode_for_metric(historical_data, metric, class_value=None): + """Return one metric history mode, resolved for one sub-technology class.""" + modes = _metric_modes(historical_data, metric) + if None in modes: + return modes[None] + try: + return modes[class_value] + except KeyError as error: + raise KeyError( + f"Historical data for metric {metric!r} has no entry for " + f"{class_value!r}. Configured classes: {sorted(modes)}." + ) from error + + +def _real_historical_classes(historical_data, metric): + """Return the sub-technology classes one metric reads from observations.""" + return sorted( + class_value + for class_value, mode in _metric_modes(historical_data, metric).items() + if mode == 'real' and class_value is not None + ) + + +def _real_historical_metrics(technology_config): + """Return explicitly configured observed metrics for one technology. + + A metric split by sub-technology class qualifies when any class selects + `real`; its applier then assigns observations to only those classes. + """ + historical_data = technology_config['historical_data'] + return sorted( + metric + for metric in historical_data + if 'real' in _metric_modes(historical_data, metric).values() + ) + + +def _mapping_target_classes(mapping): + """Return the sub-technologies a reviewed mapping claims, one per series.""" + entries = mapping.get('series') or [mapping] + inherited = {key: value for key, value in mapping.items() if key != 'series'} + targets = set() + for entry in entries: + merged = {**inherited, **entry} + targets.update( + merged.get('turbine_classes') or merged.get('technologies') or [] + ) + return targets + + +def _validate_historical_data_config(tech, historical_data, settings): + """Validate one technology's explicit metric-level history choices.""" + label = ( + f"processing.smooth_cost_curves.technologies.{tech}.historical_data" + ) + if not isinstance(historical_data, dict): + raise TypeError( + f"{label} must be a mapping with one entry for every modeled metric." + ) + valid_metrics = set(settings['techs'][tech]['cols']) - NON_HISTORY_COLUMNS + unknown = sorted(set(historical_data) - valid_metrics) + if unknown: + raise KeyError(f"Unknown historical metrics in {label}: {unknown}") + missing = sorted(valid_metrics - set(historical_data)) + if missing: + raise KeyError(f"Missing historical metrics in {label}: {missing}") + class_column = settings['techs'][tech].get('history_class_column') + split_metrics = sorted( + metric for metric in historical_data + if isinstance(historical_data[metric], dict) + ) + if split_metrics and not class_column: + raise KeyError( + f"Metrics {split_metrics} in {label} are split by sub-technology " + f"class, but {tech} declares no history_class_column in " + "settings.yaml naming the column that identifies the class." + ) + invalid = { + f"{metric}.{class_value}" if class_value else metric: mode + for metric in historical_data + for class_value, mode in _metric_modes(historical_data, metric).items() + if mode not in HISTORICAL_DATA_MODES + } + if invalid: + raise KeyError( + f"Unavailable historical metric modes in {label}; choose from " + f"{list(HISTORICAL_DATA_MODES)}: {invalid}" + ) + mappings = ( + settings['config'] + .get('historical_cost_sources', {}) + .get('reeds_mappings', {}) + .get(tech, {}) + ) + real_metrics = sorted( + metric for metric in historical_data + if 'real' in _metric_modes(historical_data, metric).values() + ) + unavailable_real = [ + metric for metric in real_metrics if metric not in mappings + ] + if unavailable_real: + raise KeyError( + f"Observed history is unavailable for metrics in {label}: " + f"{unavailable_real}" + ) + # A class-split metric states twice which rows observations describe: here + # and in the mapping that selects them. Disagreement would silently leave + # one class on the wrong history, so require the two to name the same set. + for metric in real_metrics: + real_classes = set(_real_historical_classes(historical_data, metric)) + if not real_classes: + continue + targeted = _mapping_target_classes(mappings[metric]) + if targeted != real_classes: + raise ValueError( + f"{label}.{metric} selects real history for classes " + f"{sorted(real_classes)}, but its mapping under " + f"historical_cost_sources.reeds_mappings.{tech}.{metric} " + f"targets {sorted(targeted)}. Make the two agree." + ) + return dict(historical_data) + + +def _technology_smoothing_config(tech, settings): + """Merge common smoothing parameters with one technology's switches. + + ``technologies`` maps a technology name to its switches; every technology + present in the mapping is processed and every other one is skipped. + """ + smoothing = settings['config']['processing'].get('smooth_cost_curves', {}) + if not smoothing.get('enabled', False): + return None + + configured_techs = smoothing.get('technologies', {}) + if not isinstance(configured_techs, dict): + raise TypeError( + "processing.smooth_cost_curves.technologies must map each " + "technology name to its switches." + ) + if tech not in configured_techs: + return None + technology_overrides = configured_techs[tech] or {} + if not isinstance(technology_overrides, dict): + raise TypeError( + "Each processing.smooth_cost_curves.technologies entry must " + "be a mapping." + ) + + config = { + key: value for key, value in smoothing.items() + if key not in {'technologies', 'future_smoothing_treatments'} + } + config.update({ + key: value for key, value in technology_overrides.items() + if key != 'future_smoothing_treatments' + }) + + future_treatments = {name: True for name in FUTURE_SMOOTHING_TREATMENTS} + common_treatments = smoothing.get('future_smoothing_treatments', {}) + technology_treatments = technology_overrides.get( + 'future_smoothing_treatments', {} + ) + for label, overrides in ( + ('processing.smooth_cost_curves.future_smoothing_treatments', + common_treatments), + (f'processing.smooth_cost_curves.technologies.{tech}.' + 'future_smoothing_treatments', + technology_treatments), + ): + if not isinstance(overrides, dict): + raise TypeError(f"{label} must be a mapping.") + unknown = sorted(set(overrides) - set(FUTURE_SMOOTHING_TREATMENTS)) + if unknown: + raise ValueError( + f"Unknown future smoothing treatments in {label}: {unknown}" + ) + nonboolean = sorted( + name for name, enabled in overrides.items() + if not isinstance(enabled, bool) + ) + if nonboolean: + raise TypeError( + f"Smoothing treatment switches in {label} must be boolean: " + f"{nonboolean}" + ) + future_treatments.update(overrides) + config['future_smoothing_treatments'] = future_treatments + + config['historical_data'] = _validate_historical_data_config( + tech, + config.get('historical_data', {}), + settings, + ) + return config + + +def _smoothing_columns(tech, df, smoothing): + """Return the columns one technology smooths. + + Capacity-factor multipliers ride along with whatever else is selected, + since a rounded multiplier shows the same artifacts a rounded cost does. + """ + configured = smoothing.get('columns', 'all') + if configured in ('all', ['all']): + columns = _cost_columns(df) + elif configured == 'capital_costs': + columns = [c for c in df.columns if c.startswith('capcost')] + elif isinstance(configured, list): + columns = list(configured) + else: + raise TypeError( + "processing.smooth_cost_curves.columns must be 'all', " + "'capital_costs', or a list." + ) + if 'cf_improvement' in df.columns and 'cf_improvement' not in columns: + columns.append('cf_improvement') + if not columns: + raise ValueError( + f"Cannot smooth {tech}; no monetary cost columns were selected." + ) + missing = [column for column in columns if column not in df.columns] + if missing: + raise ValueError( + f"Cannot smooth {tech}; configured cost columns are missing: " + f"{missing}." + ) + return columns + + +def _smoothing_parameters(smoothing): + """Parse and range-check the numeric smoothing switches.""" + method = smoothing.get('method', 'selective') + if method not in ('selective', 'linear_bridge'): + raise ValueError( + "processing.smooth_cost_curves.method must be 'selective' " + "or 'linear_bridge'." + ) + anchor_year = int(smoothing.get('anchor_year', 2022)) + target_year = int(smoothing.get('target_year', 2035)) + if method == 'linear_bridge' and target_year <= anchor_year: + raise ValueError( + "processing.smooth_cost_curves.target_year must be later than " + "anchor_year." + ) + parameters = { + 'method': method, + 'anchor_year': anchor_year, + 'target_year': target_year, + 'projection_start_year': int( + smoothing.get('projection_start_year', anchor_year) + ), + 'slope_change_threshold': float( + smoothing.get('slope_change_threshold', 0.5) + ), + 'max_kink_years': int(smoothing.get('max_kink_years', 4)), + 'major_step_threshold': float( + smoothing.get('major_step_relative_threshold', 0.1) + ), + 'minimum_adjustment_threshold': float( + smoothing.get('minimum_adjustment_relative_threshold', 0.005) + ), + } + if parameters['slope_change_threshold'] <= 0: + raise ValueError("slope_change_threshold must be greater than zero.") + if parameters['max_kink_years'] < 1: + raise ValueError("max_kink_years must be at least one.") + if parameters['major_step_threshold'] <= 0: + raise ValueError( + "major_step_relative_threshold must be greater than zero." + ) + if parameters['minimum_adjustment_threshold'] < 0: + raise ValueError( + "minimum_adjustment_relative_threshold must be nonnegative." + ) + return parameters + + +def smooth_cost_curve(tech, settings, df): + """Smooth monetary costs without erasing normal ATB trajectory changes. + + ``selective`` mode fills the historical period from each metric's + configured source, then bridges clusters of adjacent slope changes in the + projection. A single slope change, a major step, and a flat stretch are + each retained as a published statement. ``linear_bridge`` preserves the + earlier flat-history/anchor-to-target behavior. + """ + smoothing = _technology_smoothing_config(tech, settings) + if smoothing is None: + return df + + columns = _smoothing_columns(tech, df, smoothing) + parameters = _smoothing_parameters(smoothing) + method = parameters['method'] + anchor_year = parameters['anchor_year'] + target_year = parameters['target_year'] + projection_start_year = parameters['projection_start_year'] + slope_change_threshold = parameters['slope_change_threshold'] + max_kink_years = parameters['max_kink_years'] + major_step_threshold = parameters['major_step_threshold'] + minimum_adjustment_threshold = parameters['minimum_adjustment_threshold'] + + series_starts = settings.get('atb_series_start', {}).get(tech, {}) + future_treatments = smoothing['future_smoothing_treatments'] + historical_data = smoothing['historical_data'] + history_columns = [ + column for column in historical_data if column in df.columns + ] + # Modes are resolved per group below, because a metric may select one mode + # per sub-technology class and each group is one class. + mode_columns = list(dict.fromkeys([*history_columns, *columns])) + configured_modes = { + column: historical_data[column] for column in mode_columns + } + + tech_settings = settings['techs'][tech] + idcols = [ + column for column in tech_settings['indexcols'] + if column not in ['Scenario', 't'] + ] + groupcols = ['Scenario', *idcols] + class_column = tech_settings.get('history_class_column') + output = df.sort_values([*groupcols, 't']).copy() + # Interpolation generally creates fractional values even when an input + # frame happened to infer an integer dtype. + output[columns] = output[columns].astype(float) + original_values = output[columns].copy() + groups = output.groupby(groupcols, dropna=False, sort=False).groups + + row_series_start = pd.Series(projection_start_year, index=output.index) + enabled_future_treatments = [ + name for name, enabled in future_treatments.items() if enabled + ] + print( + f"Smoothing {tech} cost curves: method={method}, columns={columns}, " + f"historical={configured_modes}, " + f"minimum_adjustment={minimum_adjustment_threshold:.3%}, " + f"future_smoothing_treatments={enabled_future_treatments}" + ) + for group_values, index in groups.items(): + group = output.loc[index].sort_values('t') + label_values = ( + group_values + if isinstance(group_values, tuple) + else (group_values,) + ) + label = dict(zip(groupcols, label_values)) + # Each group is one sub-technology class, so a metric split by class + # resolves to a single mode here. + group_modes = { + column: _historical_mode_for_metric( + historical_data, column, label.get(class_column) + ) + for column in mode_columns + } + # ATB publishes some technologies later than the release year, so a + # series stays historical until its own data begins rather than until + # the release's first year. + series_start = series_starts.get( + (label['Scenario'], *(label[column] for column in idcols)), + projection_start_year, + ) + anchor_rows = group.loc[group['t'] == series_start] + if len(anchor_rows) != 1: + raise ValueError( + f"Cannot apply historical data modes for {tech} series " + f"{label}: expected exactly one row for projection start " + f"year {series_start}." + ) + row_series_start.loc[index] = series_start + anchor_index = anchor_rows.index[0] + historical_index = group.index[group['t'] < series_start] + broadcast_columns = [ + column for column in history_columns + if group_modes[column] == 'broadcast' + ] + if broadcast_columns: + output.loc[historical_index, broadcast_columns] = output.loc[ + anchor_index, broadcast_columns + ].to_numpy() + group = output.loc[index].sort_values('t') + if method == 'selective': + for column in columns: + output.loc[group.index, column] = _selective_smooth_cost_values( + group['t'], + group[column], + series_start, + slope_change_threshold, + max_kink_years, + major_step_threshold, + group_modes[column], + future_treatments['smooth_projection_curve'], + ) + continue + + anchor_rows = group.loc[group['t'] == anchor_year] + target_rows = group.loc[group['t'] == target_year] + if len(anchor_rows) != 1 or len(target_rows) != 1: + raise ValueError( + f"Cannot smooth {tech} series {label}: expected exactly one row " + f"for anchor year {anchor_year} and target year {target_year}." + ) + + anchor_index = anchor_rows.index[0] + target_index = target_rows.index[0] + broadcast_columns = [ + column for column in columns + if group_modes[column] == 'broadcast' + ] + if broadcast_columns: + historical_index = group.index[group['t'] <= anchor_year] + output.loc[historical_index, broadcast_columns] = output.loc[ + anchor_index, broadcast_columns + ].to_numpy() + bridge_index = group.index[ + (group['t'] > anchor_year) & (group['t'] < target_year) + ] + for column in columns: + anchor_value = float(output.at[anchor_index, column]) + target_value = float(output.at[target_index, column]) + bridge_years = output.loc[bridge_index, 't'].astype(float) + output.loc[bridge_index, column] = anchor_value + ( + (target_value - anchor_value) + * (bridge_years - anchor_year) + / (target_year - anchor_year) + ) + + # Respect the published ATB trajectory when smoothing proposes only a + # negligible future adjustment. Historical mode changes (including a full + # broadcast) are intentionally outside this filter. + future_mask = output['t'] >= row_series_start + candidate_values = output.loc[future_mask, columns] + source_values = original_values.loc[future_mask, columns] + relative_scale = source_values.abs().clip(lower=RELATIVE_SCALE_FLOOR) + relative_adjustment = ( + (candidate_values - source_values).abs() / relative_scale + ) + keep_original = relative_adjustment < minimum_adjustment_threshold + output.loc[future_mask, columns] = candidate_values.mask( + keep_original, source_values + ) + + return output.sort_index() + +def add_beccs_techs(tech, settings, df, techcol='i'): + """ + function to copy costs for beccs_mod to beccs_max + + Parameters + ---------- + tech: str + Technology key in settings['techs'] + df: pd.DataFrame + Must include column `techcol` for beccs technologies + techcol: str, optional + Column name identifying the technology label in `df` (default 'i') + """ + # copy beccs_max from beccs_mod + df_add = df.copy() + df_add['i'] = "beccs_max" + df_out = pd.concat([df, df_add]) + + return df_out + +def format_continuous_battery(tech, settings, df): + """ + function to format battery technology data + + Parameters + ---------- + tech: str + Technology key from settings (used for logging) + settings: dict + Parsed settings.yaml values (for expected column names) + df: pd.DataFrame + Input dataframe with columns that will be replaced by the battery cost fields (the function drops ['capcost', 'fom'] before merging) + """ + # Battery power/energy capital costs are extracted from the raw workbook + # downloaded by scrape_atb_inputs.py. The formatter itself never downloads + # raw data. A manual CSV remains available for pre-release ATB years. + atbyear = settings['atbyear'] + from battery_workbook import extract_battery_costs + workbook = settings['workbook_path'] + if os.path.isfile(workbook): + battery_costs = extract_battery_costs(workbook) + # round to 2 decimals to match the historical battery_costs_.csv + yearcols = [c for c in battery_costs.columns if c not in ('cost', 'Scenario')] + battery_costs[yearcols] = battery_costs[yearcols].round(2) + battery_costs.columns = [str(c) for c in battery_costs.columns] + else: + fallback = os.path.join(ATBDIR, 'manual_input', f"battery_costs_{atbyear}.csv") + print(f"Raw ATB workbook not found for {atbyear}; using manual fallback {fallback}") + if not os.path.isfile(fallback): + raise FileNotFoundError( + f"Neither raw workbook nor manual battery fallback exists.\n" + f"Expected workbook: {workbook}\nExpected fallback: {fallback}\n" + "Run 'python scrape_atb_inputs.py --only workbook'." + ) + battery_costs = pd.read_csv(fallback) + # reshape and format + battery_costs = pd.melt(battery_costs, id_vars=['cost','Scenario'], var_name='t') + battery_costs = battery_costs.pivot(index=['Scenario', 't'], columns='cost', values='value').reset_index().rename_axis(None, axis=1) + battery_costs['t'] = pd.to_numeric(battery_costs['t'], errors='coerce').astype('Int64') + # assign FOM (assumed to be 2.5% of capital costs) + fom_mult = 0.025 + battery_costs['fom'] = battery_costs['capcost'] * fom_mult + battery_costs['fom_energy'] = battery_costs['capcost_energy'] * fom_mult + battery_costs[['fom', 'fom_energy']] = battery_costs[ + ['fom', 'fom_energy'] + ].round(2) + # merge with placeholder data and return + df = df.drop(['capcost', 'fom'], axis=1).merge(battery_costs) + # if rte is missing from the ATB data, assign 0.85 as the default value + if 'rte' not in df.columns: + print("Warning: 'Round-Trip Efficiency' not found in ATB data for battery. Assigning default rte = 0.85.") + df['rte'] = 0.85 + return df + +def add_csp_techs(tech, settings, df, techcol='i'): + """ + function to create and append CSP tech types by scaling base tech values + + Parameters + ---------- + tech: str + Technology key in settings['techs'] + settings: dict + Parsed settings.yaml values to determine index columns and cost column names + df: pd.DataFrame + Must include `techcol` and cost columns listed in settings['cost_cols']) + techcol: str, optional + Column name identifying the technology label in `df` (default 'i'). + """ + # load cost ratios for csp techs + csp_ratios = pd.read_csv( + os.path.join(ATBDIR, "manual_input", f"csp_cost_ratios_{settings['atbyear']}.csv") + ).dropna(subset=['type', 'ratio', 'base_tech']) + print("updating csp tech costs using the following ratios:") + print(csp_ratios[['type','ratio']]) + + # merge with data + base_csp_tech = csp_ratios.loc[csp_ratios.base_tech==1, "type"].squeeze() + if isinstance(base_csp_tech, pd.Series): + raise ValueError("Multiple base csp techs specified in 'csp_cost_ratios.csv") + + # loop through non-base csp techs + df_add_all = [] + for _,row in csp_ratios.loc[csp_ratios.base_tech==0].iterrows(): + print(f"...updating cost data for {row['type']} using multiplier of {row['ratio']}") + # get data for baseline csp tech + df_add = df.loc[df.type == base_csp_tech].copy() + # update tech name + df_add['type'] = row['type'] + # apply multiplier to cost fields + df_add[settings['cost_cols']] *= row['ratio'] + # add to list + df_add_all.append(df_add) + + # combine entries and merge with original data + df_add_all = pd.concat(df_add_all) + df_out = pd.concat([df, df_add_all]) + # drop duplicates. keep values added here since previous ones + # come from the historic files and we want to preserve the newer ATB when available. + tech_settings = settings['techs'][tech] + df_out = df_out.drop_duplicates(subset=tech_settings['indexcols'], keep="last") + + return df_out + +def format_reeds_output(df, tech_settings): + """ + function to apply the ReEDS output schema to a formatted scenario table + + ReEDS reads some plant characteristic files by column position rather than by + column name, so those technologies declare an 'output_cols' mapping of + internal column name -> ReEDS header in settings.yaml. Technologies without + that mapping are written using the internal names in 'cols'. + + Parameters + ---------- + df: pd.DataFrame + Formatted single-scenario data using the internal column names + tech_settings: dict + Settings for the technology being written (settings['techs'][tech]) + """ + output_cols = tech_settings.get('output_cols') + if not output_cols: + return df + missing_cols = [c for c in output_cols if c not in df.columns] + if missing_cols: + raise ValueError( + f"The following columns are specified as 'output_cols' but are not in " + f"the data: {missing_cols}. Please update your 'settings.yaml'." + ) + dropped_cols = [c for c in df.columns if c not in output_cols] + if dropped_cols: + raise ValueError( + f"'output_cols' does not cover every output column; missing " + f"{dropped_cols}. Please update your 'settings.yaml'." + ) + return df[list(output_cols)].rename(columns=output_cols) + + +def write_technology_outputs( + tech_data, + tech, + settings, + tech_settings, + sensitivity_name, + outfolder, + filenames=None, + output_label="", +): + """Write one technology's scenario files using the ReEDS output schema.""" + tech_data = tech_data.copy() + tech_data['Scenario'] = tech_data['Scenario'].str.lower() + scenarios = list(tech_data['Scenario'].unique()) + if 'moderate' in scenarios: + scenarios.insert(0, scenarios.pop(scenarios.index('moderate'))) + + filename_root = tech_settings.get('reeds_name', tech) + baseline = None + written = [] + os.makedirs(outfolder, exist_ok=True) + + for scenario in scenarios: + if sensitivity_name is not None: + filename = ( + f"{filename_root}_ATB_{settings['atbyear']}_{scenario}_" + f"{sensitivity_name}.csv" + ) + else: + filename = f"{filename_root}_ATB_{settings['atbyear']}_{scenario}.csv" + + check_columns(tech_data, tech_settings['cols'], 'cols', tech) + scendata = tech_data.loc[ + tech_data.Scenario == scenario, tech_settings['cols'] + ] + scendata = scendata.round( + tech_settings.get('decimals', settings['decimals']) + ) + scendata = scendata.sort_values( + by=scendata.columns.to_list() + ).reset_index(drop=True) + + if baseline is None: + baseline = scendata.copy() + elif scendata.equals(baseline): + print( + f"...{scenario} is identical to the {scenarios[0]} scenario, " + "skipping." + ) + continue + + scendata_out = format_reeds_output(scendata, tech_settings) + label = f" {output_label}" if output_label else "" + print(f"Saving{label} {filename}") + scendata_out.to_csv( + os.path.join(outfolder, filename), + index=False, + lineterminator=CSV_LINE_TERMINATOR, + ) + written.append(filename) + if filenames is not None: + filenames.append(filename) + + return written + + +def process_tech_file(atb_data, tech, settings, filenames, dollaryear, deflator, sensitivity_name, outfolder, args): + """ + function to format per-technology output files + + Parameters + ---------- + atb_data: pd.DataFrame + Technology-level data from ATB flat-file + tech: str + Technology-specific key in settings['techs'] to process + settings: dict + Parsed settings.yaml values + filenames: list + List to which produced filenames will be appended. + dollaryear: pd.Series + Mapping of scenario -> dollar year + deflator: pd.Series + Mapping of dollar year -> deflator multiplier + sensitivity_name: str or None + Optional suffix added to output filenames for sensitivity runs + outfolder: str + Directory path where output CSVs will be written + args: argparse.Namespace + CLI arguments for debug mode and run options + """ + # subset to relevant technology + print(f"Processing {tech} file.") + tech_settings = settings['techs'][tech] + tech_data = atb_data.loc[atb_data['Technology'] == tech_settings['Technology']] + # apply tech-specific subset_rows (Case, CRPYears) + if 'subset_rows' in tech_settings: + tech_data, _ = subset_atb_rows(tech_settings['subset_rows'], tech_data) + else: + raise ValueError(f"Missing 'subset_rows' for {tech} in settings.yaml") + # get subtech(s), specified either as a dictionary or a single string + if 'DisplayName' in tech_settings: + if isinstance(tech_settings['DisplayName'], dict): + tech_data = tech_data.loc[tech_data['DisplayName'].isin(tech_settings['DisplayName'].keys())] + missing_subtechs = ( + set(tech_settings['DisplayName']) - set(tech_data['DisplayName'].unique()) + ) + tech_data = tech_data.replace({'DisplayName': tech_settings['DisplayName']}) + # verify all expected subtechs were found in the data + if missing_subtechs: + print(f"Warning: the following DisplayName(s) for {tech} were not found in the ATB subset: {sorted(missing_subtechs)}") + else: + print(f"Subsetting to {tech_settings['DisplayName']}") + tech_data = tech_data.loc[tech_data['DisplayName'] == tech_settings['DisplayName']] + if tech_data.empty: + if args.debug: + breakpoint() + else: + raise EmptyDataError(f"\n\nNo data found for {tech} in ATB data. Check file for values for specified subset_rows.") + + # add in columns as needed + if 'addcols' in tech_settings: + print(f"Adding columns {tech_settings['addcols']}") + for col in tech_settings['addcols']: + tech_data[col] = tech_settings['addcols'][col] + + # rename columns as needed + if 'renamecols' in tech_settings: + print(f"Renaming columns {tech_settings['renamecols']}") + tech_data = tech_data.rename(columns=tech_settings['renamecols']) + + # grab columns needed for this particular tech + atb_params = [k for k, v in settings['param_names'].items() if v in tech_settings['cols']] + tech_data_out = tech_data.loc[tech_data.Parameter.isin(atb_params)].copy() + # check if empty + if tech_data_out.empty: + raise EmptyDataError("\n\n" + "Dataframe is empty after subsetting; check 'param_names' in settings.yaml." + f"Current values: {settings['param_names']}" + ) + # check columns and then pivot to wide format + check_columns(tech_data_out, tech_settings['indexcols'], 'indexcols', tech) + tech_data_out = tech_data_out.pivot(index=tech_settings['indexcols'], columns='Parameter', values='value') + tech_data_out = tech_data_out.rename_axis(None, axis=1) + # map parameters to new column names and format + tech_data_out = tech_data_out.rename(columns=settings['param_names']) + # convert numeric columns + tech_data_out = tech_data_out.apply(pd.to_numeric).reset_index() + + # if vom is missing in the input ATB data, assign zero + if tech in zero_vom_techs and 'vom' not in tech_data_out.columns: + print(f"Warning: 'Variable O&M' missing in ATB data for {tech}. Assigning vom = 0.0") + tech_data_out['vom'] = 0.0 + + # if processing continuous battery techs update energy cost data + if 'battery' in tech: + tech_data_out = format_continuous_battery(tech, settings, tech_data_out) + + # Apply technology transformations to the current scraped projections first. + # History is merged afterward so derived historic series are not overwritten. + if 'functions' in tech_settings: + for function_name in tech_settings['functions']: + selected_function = FUNCTION_MAPPING.get(function_name.lower()) + if selected_function: + print(f"Running {function_name}") + tech_data_out = selected_function(tech, settings, tech_data_out) + else: + raise NameError(f"'{function_name}' is not a supported function. " + "Define and add to FUNCTION_MAPPING.") + + # Combine the current projections with the stable local historical baseline. + tech_data_out = merge_historical_atb_data( + tech_data_out, tech, settings, dollaryear, deflator + ) + tech_data_unsmoothed = tech_data_out.copy() + + # This user-facing option is intentionally applied after history is merged, + # so it can bridge the history/current-ATB boundary without changing the + # versioned historical source files. It is disabled by default. + tech_data_out = smooth_cost_curve(tech, settings, tech_data_out) + + smoothing = settings['config']['processing'].get('smooth_cost_curves', {}) + baseline_directory = settings.get('unsmoothed_output_dir') + if smoothing.get('enabled', False) and baseline_directory: + baseline_folder = ( + baseline_directory + if os.path.isabs(baseline_directory) + else os.path.join(ATBDIR, baseline_directory) + ) + write_technology_outputs( + tech_data_unsmoothed, + tech, + settings, + tech_settings, + sensitivity_name, + baseline_folder, + output_label="temporary pre-smoothing", + ) + + write_technology_outputs( + tech_data_out, + tech, + settings, + tech_settings, + sensitivity_name, + outfolder, + filenames=filenames, + ) + +def update_dollaryear(settings, filenames, dollaryear): + """ + function to create dollar year entries for new tech files + + Parameters + ---------- + settings: dict + Parsed settings['dollaryear'] value as the dollar year to assign to the new files + filenames: list[str] + Output filename produced by this run; strips the ".csv" suffix to derive Scenario names + dollaryear: pd.Series + Scenario -> Dollar.Year mapping (index=Scenario, name='Dollar.Year') + """ + # strip .csv from filenames + scennames = [f[:-4] for f in filenames] + # create new dollar year entries + newdollaryear = pd.Series(data=[settings['dollaryear']] * len(scennames), index=scennames) + # combine old and new and format for ReEDS + newdollaryear = pd.concat([dollaryear, newdollaryear]).sort_index() + # drop duplicates + newdollaryear = newdollaryear[~newdollaryear.index.duplicated(keep='first')] + newdollaryear.name = "Dollar.Year" + newdollaryear.index.name = "Scenario" + + return newdollaryear + +def get_financials(atb_data, settings, tech, financial_params, scenarios=["Moderate","*"]): + """ + function to get full financial parameters for a specific technology + + Parameters + ---------- + atb_data: pd.DataFrame + ATB flat file data + settings: dict + Parsed 'decimals' and 'reeds_start_year' values + tech: str + ATB technology name to filter to + financial_params: dict + Mapping of ATB parameter name -> target column name for pivoting + scenarios: list[str], optional + List of scenario names to include (default ["Moderate","*"]) + """ + techfinancial = atb_data.loc[(atb_data.Technology == tech) + & (atb_data.Scenario.isin(scenarios)) + & (atb_data.Parameter.isin(financial_params)) + ].copy() + if techfinancial.empty: + avail_params = atb_data.loc[(atb_data.Technology == tech) + & (atb_data.Scenario.isin(scenarios)), "Parameter" + ].unique() + if args.debug: + breakpoint() + else: + + raise EmptyDataError(f"Looking for the following financial parameters for {tech}:\n{list(financial_params.keys())}.\n" + f"Found the following parameters:\n{list(avail_params)}" + ) + techfinancial = techfinancial.rename(columns={"variable":"t"}) + # reorder columns + techfinancial = techfinancial.pivot(index="t", columns='Parameter', values='value').rename(columns=financial_params) + colorder = [v for k,v in financial_params.items()] + # convert to numeric and round + techfinancial = techfinancial[colorder].apply(pd.to_numeric).round(settings['decimals']) + # reindindex to include years before atb + atb_end_year = techfinancial.index.values.max() + techfinancial = techfinancial.reindex(index=list(range(settings['reeds_start_year'],atb_end_year+1))).bfill().reset_index() + + return techfinancial + +def update_financials(settings, atb_data, outfolder): + """ + function to write updated system and tech financial outputs + + Parameters + ---------- + settings: dict + Expects keys such as 'atbyear', 'reedspath', 'wacc_base_tech', 'decimals', and tech-specific configs under settings['techs']) + atb_data: pd.DataFrame + ATB flat file data used to extract financial parameters and WACC + outfolder: str + Output directory path + """ + print("Processing system financials.") + + financial_params = { + 'Interest Rate Nominal':'interest_rate_nom', + 'Rate of Return on Equity Nominal':'rroe_nom', + 'Debt Fraction':'debt_fraction', + 'Tax Rate (Federal and State)':'tax_rate' + } + + ## system financial file + # uses base tech specified in settings (typically natural gas) + sysfinancial = get_financials(atb_data, settings, settings['wacc_base_tech'], financial_params) + + ## tech financials + # get old tech financials file + print("Processing tech financials.") + prevyear = settings['atbyear']-1 + filename_old = f"financials_tech_ATB{prevyear}.csv" + financials_old = pd.read_csv(os.path.join(settings['reedspath'], 'inputs', 'financials', filename_old)) + + wacc = atb_data.loc[(atb_data['Parameter'] == "WACC Real") & (atb_data['Scenario'] == "Moderate"), ["Technology", "variable", "value"]] + wacc = wacc.rename(columns={"variable": "t", "value":"wacc_real"}) + wacc['wacc_real'] = pd.to_numeric(wacc['wacc_real']) + # use natural gas as baseline + wacc_baseline = wacc.loc[wacc.Technology == settings['wacc_base_tech']] + wacc_baseline = wacc_baseline.rename(columns={"wacc_real":"wacc_real_base"}).drop("Technology", axis=1) + + wacc = wacc.merge(wacc_baseline, on=['t']) + # get financing difference relative to base + wacc['finance_diff_real_update'] = wacc['wacc_real'] - wacc['wacc_real_base'] + + # get WACC values for techs that were processed + financials_update = [] + for tech in settings['techs']: + print(f"Loading financials for {tech}") + + # for batteries we need to calculate the WACC Real value + if tech == "battery": + # batteries are missing WACC, so need to get individual components and recalculate + # since we don't have debt fraction for batteries we take the value from PV for now + battery_params = financial_params.copy() + battery_params.pop('Debt Fraction') + battery_params.update({'Inflation Rate':'inflation_rate'}) + pv_param = {'Debt Fraction': financial_params['Debt Fraction']} + battfinancial = get_financials(atb_data, settings, settings['techs'][tech]['Technology'], battery_params) + pvfinancial = get_financials(atb_data, settings, settings['techs']['upv']['Technology'], pv_param) + # merge and calculate WACC in two steps: + # 1. WACC Nominal = Debt Fraction * Interest Rate Nominal * (1 - Tax Rate) + (1 - Debt Fraction) * Rate of Return + # 2. WACC Real = (1 + WACC Nominal) / (1 + Inflation Rate) - 1 + wacc_tech = battfinancial.merge(pvfinancial) + wacc_tech['wacc_nom'] = (wacc_tech['debt_fraction'] + * wacc_tech['interest_rate_nom'] + * (1 - wacc_tech['tax_rate']) + + (1 - wacc_tech['debt_fraction']) + * wacc_tech['rroe_nom'] + ) + wacc_tech['wacc_real'] = ( + (1 + wacc_tech['wacc_nom']) / (1 + wacc_tech['inflation_rate']) - 1 + ) + wacc_tech = wacc_tech.merge(wacc_baseline, on=['t']) + wacc_tech['finance_diff_real_update'] = wacc_tech['wacc_real'] - wacc_tech['wacc_real_base'] + + else: + wacc_tech = wacc.loc[wacc.Technology == settings['techs'][tech]['Technology']].copy() + + if wacc_tech.empty: + print(f"...no wacc found for {tech}, skipping.") + else: + # overwrite tech with new financial name if specified + if 'financialname' in settings['techs'][tech]: + wacc_tech['i'] = settings['techs'][tech]['financialname'] + wacc_tech = wacc_tech.drop_duplicates() + # otherwise use exist tech name + else: + wacc_tech = wacc_tech.rename(columns={'Technology':'i'}) + # add to list of new financials to update + financials_update.append(wacc_tech[['i', 't', 'finance_diff_real_update']]) + + financials_update = pd.concat(financials_update) + + # merge with old file and rewrite new year values upto the max available ATB year + max_atb_year = int(financials_update['t'].max()) + max_old_year = int(financials_old['t'].max()) + if max_atb_year > max_old_year: + new_years = list(range(max_old_year + 1, max_atb_year + 1)) + # get unique (i, country) combinations from old file + id_combos = financials_old[['i', 'country']].drop_duplicates() + # create new rows for each (i, country) × new year + new_rows = id_combos.merge(pd.DataFrame({'t': new_years}), how='cross') + # merge old file's non-year columns onto new rows (forward-fill from last known year) + last_year_vals = financials_old.loc[financials_old['t'] == max_old_year].drop('t', axis=1) + new_rows = new_rows.merge(last_year_vals, on=['i', 'country'], how='left') + financials_old = pd.concat([financials_old, new_rows], ignore_index=True) + financials_out = financials_old.merge(financials_update, on=['i', 't'], how='left') + # fill forward by country and tech to create values for pre-ATB years + financials_out['finance_diff_real_update'] = financials_out.groupby(['i','country'])['finance_diff_real_update'].bfill() + + # update relevant techs while leaving others in place + financials_out['finance_diff_real'] = np.where(financials_out['finance_diff_real_update'].isna(), + financials_out['finance_diff_real'], + financials_out['finance_diff_real_update'].round(settings['decimals']) + ) + + # write system and tech financial files + sysfile_new = f"financials_sys_ATB{settings['atbyear']}.csv" + sysfinancial.to_csv( + os.path.join(outfolder, sysfile_new), index=False, + lineterminator=CSV_LINE_TERMINATOR, + ) + + techfile_new = f"financials_tech_ATB{settings['atbyear']}.csv" + financials_out = financials_out[financials_old.columns] + financials_out.to_csv( + os.path.join(outfolder, techfile_new), index=False, + lineterminator=CSV_LINE_TERMINATOR, + ) + + return sysfile_new, techfile_new + +#%% =========================================================================== +### --- Main --- +### =========================================================================== +def main(args): + settings = load_processing_settings(args.config) + settings['unsmoothed_output_dir'] = getattr( + args, 'unsmoothed_output_dir', None + ) + processing = settings['config']['processing'] + outfolder = settings['output_dir'] + args.skip_costs = args.skip_costs or not processing.get('update_costs', True) + should_update_financials = processing.get('update_financials', True) + if args.sensitivity_name is None: + args.sensitivity_name = processing.get('sensitivity_name') + + smoothing = processing.get('smooth_cost_curves', {}) + if smoothing.get('enabled', False): + smoothing_techs = smoothing.get('technologies', {}) + if not isinstance(smoothing_techs, dict): + raise TypeError( + "processing.smooth_cost_curves.technologies must map each " + "technology name to its switches." + ) + invalid_entries = [ + tech for tech, config in smoothing_techs.items() + if config is not None and not isinstance(config, dict) + ] + if invalid_entries: + raise TypeError( + "Each processing.smooth_cost_curves.technologies entry " + f"must be a mapping: {invalid_entries}" + ) + unknown_smoothing_techs = [ + tech for tech in smoothing_techs + if tech not in settings['techs'] + ] + if unknown_smoothing_techs: + raise ValueError( + "Unknown technologies in processing.smooth_cost_curves: " + f"{unknown_smoothing_techs}" + ) + baseline_directory = settings.get('unsmoothed_output_dir') + if baseline_directory: + baseline_folder = ( + baseline_directory + if os.path.isabs(baseline_directory) + else os.path.join(ATBDIR, baseline_directory) + ) + os.makedirs(baseline_folder, exist_ok=True) + year_marker = f"_ATB_{settings['atbyear']}_" + for filename in os.listdir(baseline_folder): + if year_marker in filename and filename.endswith('.csv'): + os.remove(os.path.join(baseline_folder, filename)) + + # if output folder does not exist, create it + os.makedirs(outfolder, exist_ok=True) + + # get list of techs to process + configured_techs = processing.get('technologies', 'all') + requested_techs = args.techs if args.techs is not None else configured_techs + if requested_techs == 'all' or requested_techs == ['all']: + techs_to_run = list(settings['techs'].keys()) + else: + techs_to_run = requested_techs if isinstance(requested_techs, list) else [requested_techs] + missing_techs = [t for t in techs_to_run if t not in settings['techs']] + if missing_techs: + raise ValueError(f"The following technologies are not in settings['techs']: {missing_techs}") + + # check if ReEDS repo is correctly specified + if not os.path.isdir(settings['reedspath']): + raise FileNotFoundError(f"Could not find '{settings['reedspath']}'; check config.yaml.") + + # load dollaryear file in ReEDS + dollaryear = pd.read_csv(os.path.join(settings['reedspath'], 'inputs', 'plant_characteristics', 'dollaryear.csv'), + index_col='Scenario').squeeze() + deflator = pd.read_csv(os.path.join(settings['reedspath'], 'inputs', 'financials', 'deflator.csv'), + index_col='*Dollar.Year').squeeze() + if not args.skip_costs: + validate_real_historical_data(settings, techs_to_run, deflator) + # load ATB flat file + atb_data = load_atb_flat_file(settings, args, techs_to_run) + + ## process technology files + filenames = [] + if args.skip_costs: + print("Skipping cost files.") + else: + print("---------------------") + for tech in techs_to_run: + process_tech_file(atb_data, tech, settings, filenames, + dollaryear, deflator, args.sensitivity_name, outfolder, args) + print("---------------------") + # update dollaryear file + newdollaryear = update_dollaryear(settings, filenames, dollaryear) + + ## update financials file + if should_update_financials: + sysfinancialfile, techfinancialfile = update_financials(settings, atb_data, outfolder) + else: + print("Skipping financials (processing.update_financials is false).") + + ## copy new files to ReEDS + if settings['copy_to_reeds']: + + if not args.skip_costs: + # copy tech files + for f in filenames: + shutil.copy(os.path.join(outfolder,f), os.path.join(settings['reedspath'],'inputs','plant_characteristics',f)) + # update dollaryear file for tech files + newdollaryear.to_csv( + os.path.join(settings['reedspath'],'inputs','plant_characteristics','dollaryear.csv'), + index=True, lineterminator=CSV_LINE_TERMINATOR, + ) + + # copy financial files + if should_update_financials: + shutil.copy(os.path.join(outfolder,sysfinancialfile), os.path.join(settings['reedspath'],'inputs','financials',sysfinancialfile)) + shutil.copy(os.path.join(outfolder,techfinancialfile), os.path.join(settings['reedspath'],'inputs','financials',techfinancialfile)) + + +if __name__ == "__main__": + print("Processing ATB files") + parser = argparse.ArgumentParser(description="Generate ATB files.") + parser.add_argument('--config', help='path to config.yaml (default: ../config.yaml)') + parser.add_argument('--techs', '-t', nargs='+', default=None, + help='one or more techs; defaults to processing.technologies in config.yaml') + parser.add_argument('--sensitivity_name', '-s', type=str, + help='suffix to append to file name for sensitivities') + parser.add_argument('--skip_costs', '-c', action="store_true", + help='skip updating cost files for this run') + parser.add_argument('--debug', '-d', action="store_true", + help='option to run in debug mode') + parser.add_argument( + '--unsmoothed-output-dir', + help=( + 'temporary directory for pre-smoothing outputs used by the ' + 'comparison stage' + ), + ) + args = parser.parse_args() + + # list of supported custom functions to call from settings.yml + FUNCTION_MAPPING = { + 'normalize_cf': normalize_cf, + 'apply_offshore_cost_multipliers': apply_offshore_cost_multipliers, + 'apply_coal_projection_overrides': apply_coal_projection_overrides, + 'smooth_hist_cf': smooth_hist_cf, + 'add_csp_techs': add_csp_techs, + 'add_beccs_techs': add_beccs_techs, + } + main(args) diff --git a/atb/scripts/run_pipeline.py b/atb/scripts/run_pipeline.py new file mode 100644 index 0000000..c12fffb --- /dev/null +++ b/atb/scripts/run_pipeline.py @@ -0,0 +1,98 @@ +"""Run the config-selected NLR ATB workflow stages in order.""" + +import argparse +from contextlib import nullcontext +import subprocess +import sys +import tempfile +from pathlib import Path + +from atb_config import DEFAULT_CONFIG_PATH, load_config + + +SCRIPT_DIR = Path(__file__).resolve().parent +STAGES = { + "scrape": ("scrape_raw_data", "scrape_atb_inputs.py"), + "format": ("format_reeds_inputs", "generate_atb_files.py"), + "plot": ("make_plots", "atb_plotting.py"), + "compare": ( + "make_comparison_plots", + "../comparison/compare_atb_outputs.py", + ), +} + + +def run_stage( + name, + script, + config_path, + force_download=False, + unsmoothed_dir=None, +): + command = [sys.executable, str(SCRIPT_DIR / script), "--config", str(config_path)] + if name == "scrape" and force_download: + command.append("--force") + if unsmoothed_dir is not None and name == "format": + command.extend(["--unsmoothed-output-dir", str(unsmoothed_dir)]) + if unsmoothed_dir is not None and name == "compare": + command.extend(["--unsmoothed-dir", str(unsmoothed_dir)]) + print(f"\n=== {name.upper()} ===", flush=True) + subprocess.run(command, check=True) + + +def main(): + parser = argparse.ArgumentParser( + description="Run scrape, ReEDS formatting, and plotting from config.yaml." + ) + parser.add_argument("--config", help="Path to config.yaml (default: ../config.yaml).") + parser.add_argument( + "--only", nargs="+", choices=list(STAGES), + help="Run only the selected stage(s), ignoring workflow switches.", + ) + parser.add_argument( + "--force-download", action="store_true", + help="Replace existing raw files during the scrape stage.", + ) + args = parser.parse_args() + + config_path = Path(args.config).resolve() if args.config else DEFAULT_CONFIG_PATH + config = load_config(config_path) + selected = set(args.only) if args.only else None + + plan = [] + for stage, (switch, script) in STAGES.items(): + enabled = stage in selected if selected is not None else config["workflow"][switch] + plan.append((stage, script, enabled)) + + print(f"Config: {config_path}") + print(f"ATB year: {config['atb']['year']}") + print("Workflow:") + for stage, _, enabled in plan: + print(f" {'RUN ' if enabled else 'SKIP'} {stage}") + + enabled_stages = {stage for stage, _, enabled in plan if enabled} + needs_unsmoothed_data = ( + config['processing'].get('smooth_cost_curves', {}).get('enabled', False) + and {'format', 'compare'}.issubset(enabled_stages) + ) + temporary_output = ( + tempfile.TemporaryDirectory(prefix="atb_unsmoothed_") + if needs_unsmoothed_data + else nullcontext(None) + ) + with temporary_output as unsmoothed_dir: + for stage, script, enabled in plan: + if enabled: + run_stage( + stage, + script, + config_path, + args.force_download, + unsmoothed_dir, + ) + + print("\nPipeline complete.") + + +if __name__ == "__main__": + main() diff --git a/atb/scripts/scrape_atb_inputs.py b/atb/scripts/scrape_atb_inputs.py new file mode 100644 index 0000000..8a2302b --- /dev/null +++ b/atb/scripts/scrape_atb_inputs.py @@ -0,0 +1,138 @@ +"""Download and summarize the raw NLR ATB inputs configured in config.yaml.""" + +import argparse +from pathlib import Path + +import openpyxl +import pandas as pd +import requests + +from atb_config import load_config, raw_file_path + + +def _open_download(url, allow_insecure_ssl_fallback): + """Open a verified download, optionally retrying after a TLS inspection error.""" + request_options = { + "stream": True, + "timeout": 300, + "headers": {"User-Agent": "ReEDS-Input-Processing/1.0"}, + } + try: + return requests.get(url, **request_options) + except requests.exceptions.SSLError as error: + if not allow_insecure_ssl_fallback: + raise RuntimeError( + "TLS certificate verification failed. Install the required CA " + "certificate in this environment or set " + "raw_data.allow_insecure_ssl_fallback: true in config.yaml." + ) from error + + import urllib3 + + urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + print("WARNING: TLS certificate verification failed.") + print(" Retrying this public raw-data download with verify=False.") + return requests.get(url, verify=False, **request_options) + + +def download_file( + url, destination, force=False, allow_insecure_ssl_fallback=False +): + """Download one file atomically, or reuse the existing local copy.""" + destination = Path(destination) + destination.parent.mkdir(parents=True, exist_ok=True) + if destination.exists() and not force: + print(f"Using existing raw file: {destination}") + return destination + + temporary = destination.with_suffix(destination.suffix + ".part") + print(f"Downloading {url}") + print(f" into {destination}") + try: + with _open_download(url, allow_insecure_ssl_fallback) as response: + response.raise_for_status() + with temporary.open("wb") as stream: + for chunk in response.iter_content(chunk_size=1024 * 1024): + if chunk: + stream.write(chunk) + except Exception: + temporary.unlink(missing_ok=True) + raise + temporary.replace(destination) + return destination + + +def summarize_flat_file(path): + """Print a compact, user-visible overview of the downloaded ATBe CSV.""" + path = Path(path) + preview = pd.read_csv(path, nrows=5, low_memory=False) + header = list(preview.columns) + technology_col = "technology" if "technology" in header else "Technology" + year_col = "atb_year" if "atb_year" in header else None + selected = [technology_col] + ([year_col] if year_col else []) + summary = pd.read_csv(path, usecols=selected, low_memory=False) + + print("\nRaw flat file") + print(f" path: {path}") + print(f" size: {path.stat().st_size / 1024**2:,.1f} MiB") + print(f" rows: {len(summary):,}") + if year_col: + years = sorted(summary[year_col].dropna().unique().tolist()) + print(f" ATB years: {years}") + technologies = sorted(summary[technology_col].dropna().astype(str).unique()) + print(f" technologies ({len(technologies)}): {', '.join(technologies)}") + print(" first five rows:") + print(preview.to_string(index=False, max_cols=8)) + + +def summarize_workbook(path): + """Print the path, size, and sheet names for the raw ATB workbook.""" + path = Path(path) + workbook = openpyxl.load_workbook(path, read_only=True, data_only=True) + print("\nRaw workbook") + print(f" path: {path}") + print(f" size: {path.stat().st_size / 1024**2:,.1f} MiB") + print(f" sheets ({len(workbook.sheetnames)}): {', '.join(workbook.sheetnames)}") + workbook.close() + + +def scrape(config, selected="all", force=False): + """Download configured raw inputs and show what is now available.""" + raw = config["raw_data"] + allow_insecure = raw.get("allow_insecure_ssl_fallback", False) + if selected in ("all", "flat"): + flat = download_file( + raw["flat_file"]["url"], + raw_file_path(config, "flat_file"), + force=force, + allow_insecure_ssl_fallback=allow_insecure, + ) + summarize_flat_file(flat) + if selected in ("all", "workbook"): + workbook = download_file( + raw["workbook"]["url"], + raw_file_path(config, "workbook"), + force=force, + allow_insecure_ssl_fallback=allow_insecure, + ) + summarize_workbook(workbook) + + +def main(): + parser = argparse.ArgumentParser( + description="Download and summarize the raw NLR ATB flat file and workbook." + ) + parser.add_argument("--config", help="Path to config.yaml (default: ../config.yaml).") + parser.add_argument( + "--only", choices=["all", "flat", "workbook"], default="all", + help="Download only one raw input (default: all).", + ) + parser.add_argument( + "--force", action="store_true", help="Replace raw files that already exist." + ) + args = parser.parse_args() + scrape(load_config(args.config), selected=args.only, force=args.force) + + +if __name__ == "__main__": + main() diff --git a/atb/scripts/scrape_historical_costs.py b/atb/scripts/scrape_historical_costs.py new file mode 100644 index 0000000..a8d569d --- /dev/null +++ b/atb/scripts/scrape_historical_costs.py @@ -0,0 +1,411 @@ +"""Download and normalize observed generator capital-cost source data.""" + +import argparse +import hashlib +from pathlib import Path + +import openpyxl +import pandas as pd + +from atb_config import load_config, resolve_atb_path +from scrape_atb_inputs import download_file + + +COLUMNS = [ + "technology", + "technology_detail", + "year", + "metric", + "value", + "unit", + "capacity_basis", + "statistic", + "geography", + "dollar_year", + "price_basis", + "sample_count", + "source_id", + "source_file", + "source_sheet", + "source_table", + "source_page_url", + "source_data_url", + "notes", +] + +# Only the three mapped tables give national cost by technology; the rest split +# the same capacity by region, state, size, or panel type. Labels drift between +# editions ("Solar photovoltaic" -> "Solar PV" -> "Solar"), so every spelling stays. +EIA_TECHNOLOGY_MAP = { + "Solar": ("upv", "Utility-scale solar (all reported solar)"), + "Solar PV": ("upv", "Utility-scale solar PV"), + "Solar photovoltaic": ("upv", "Utility-scale solar photovoltaic"), + "Battery storage": ("battery", "Battery storage"), + "Wind": ("wind-ons", "Wind (EIA broad energy-source category)"), + "Natural gas": ("gas", "Natural gas (all reported technologies)"), + "Petroleum liquids": ("petroleum", "Petroleum liquids"), + "Biomass": ("biopower", "Biomass"), + "Geothermal": ("geothermal", "Geothermal"), + "Hydro": ("hydropower", "Hydroelectric"), + "Hydroelectric": ("hydropower", "Hydroelectric"), +} + +# Equipment rather than fuel; the only table with fuel cells. Unmapped on +# purpose: "Steam turbine" (ambiguous fuel), "... (as part of combined cycle)" +# (half a plant, whole in the table below), "Internal combustion engine" (gas +# and oil, no ReEDS counterpart). +EIA_PRIME_MOVER_MAP = { + "Combustion turbine": ("gas", "Natural gas combustion turbine"), + "Onshore wind turbine": ("wind-ons", "Onshore wind turbine"), + "Photovoltaic": ("upv", "Photovoltaic"), + "Energy storage, battery": ("battery", "Battery storage"), + "Battery storage": ("battery", "Battery storage"), + "Fuel cell": ("fuelcell", "Fuel cell"), + "Geothermal turbines": ("geothermal", "Geothermal turbine"), + "Hydroelectric turbine": ("hydropower", "Hydroelectric turbine"), +} + +# Whole plants, so combined cycle arrives as one cost rather than split across +# its turbine halves. Closest observed match to ReEDS Gas-CC and Gas-CT. +EIA_GAS_TECHNOLOGY_MAP = { + "Combined cycle": ("gas", "Natural gas combined cycle"), + "Combustion turbine": ("gas", "Natural gas combustion turbine"), + "Steam turbine": ("gas", "Natural gas steam turbine"), + "Internal combustion engine": ("gas", "Natural gas internal combustion engine"), +} + +# (title fragment, label map, source_table tag), matched against a lowercased +# table title in column A. Natural gas must precede prime mover so its title wins. +EIA_TABLES = ( + ("by major energy source", EIA_TECHNOLOGY_MAP, "major_energy_source"), + ("natural gas generators installed", EIA_GAS_TECHNOLOGY_MAP, "natural_gas_technology"), + ("by prime mover", EIA_PRIME_MOVER_MAP, "prime_mover"), +) + + +def _base_row(source_id, source, filename, sheet): + return { + "metric": "capital_cost", + "source_id": source_id, + "source_file": filename, + "source_sheet": sheet, + "source_page_url": source["page_url"], + "source_data_url": source.get("data_url", ""), + } + + +def _find_header_row(sheet, required_text): + required = required_text.lower() + for row_number, row in enumerate(sheet.iter_rows(values_only=True), 1): + if any(required in str(value).lower() for value in row if value is not None): + return row_number + raise ValueError(f"Could not find {required_text!r} in sheet {sheet.title!r}") + + +def extract_land_based_wind(path, source): + """Extract LBNL's observed annual capacity-weighted installed wind cost.""" + sheet_name = "CapEx Over Time" + workbook = openpyxl.load_workbook(path, read_only=True, data_only=True) + sheet = workbook[sheet_name] + header = _find_header_row(sheet, "Commercial Operation Date") + rows = [] + for year, value, sample_count in sheet.iter_rows( + min_row=header + 1, max_col=3, values_only=True + ): + if isinstance(year, (int, float)) and isinstance(value, (int, float)): + row = _base_row("land_based_wind", source, Path(path).name, sheet_name) + row.update( + technology="wind-ons", + technology_detail="Land-based wind projects", + year=int(year), + value=float(value), + unit="USD/kW", + capacity_basis="nameplate", + statistic="capacity_weighted_mean", + geography="United States", + dollar_year=2024, + price_basis="real", + sample_count=sample_count, + notes="Observed project CapEx; 2024 COD values are preliminary.", + ) + rows.append(row) + workbook.close() + return rows + + +def extract_utility_pv(path, source): + """Extract LBNL's observed PV-only installed costs on AC and DC bases.""" + sheet_name = "CapEx Trend (PV-only)" + workbook = openpyxl.load_workbook(path, read_only=True, data_only=True) + sheet = workbook[sheet_name] + header = _find_header_row(sheet, "Solar COD") + rows = [] + # Each tuple is (year, count, capacity-weighted mean) for one capacity basis. + for basis, year_col, count_col, mean_col in ( + ("AC", 1, 2, 4), + ("DC", 11, 12, 14), + ): + for values in sheet.iter_rows(min_row=header + 1, values_only=True): + year = values[year_col - 1] + value = values[mean_col - 1] + if not isinstance(year, (int, float)) or not isinstance(value, (int, float)): + continue + row = _base_row("utility_pv", source, Path(path).name, sheet_name) + row.update( + technology="upv", + technology_detail="Utility-scale PV-only projects", + year=int(year), + # Source values are $/W; convert to the common $/kW unit. + value=float(value) * 1000, + unit="USD/kW", + capacity_basis=basis, + statistic="capacity_weighted_mean", + geography="United States", + dollar_year=2024, + price_basis="real", + sample_count=values[count_col - 1], + notes="Observed PV-only project CapEx; 2024 COD values are preliminary.", + ) + rows.append(row) + workbook.close() + return rows + + +def extract_offshore_wind(path, source): + """Extract completed-year offshore project CapEx series from NLR Figure 31.""" + sheet_name = "F31, Project CapEx" + workbook = openpyxl.load_workbook(path, read_only=True, data_only=True) + sheet = workbook[sheet_name] + header = _find_header_row(sheet, "Commercial Operation Date") + last_year = int(source["last_historical_year"]) + series = ( + (3, "capacity_weighted_mean", "Global"), + (5, "capacity_weighted_mean", "Europe and United States"), + (7, "capacity_weighted_mean", "Asia"), + ) + rows = [] + for values in sheet.iter_rows(min_row=header + 1, values_only=True): + year = values[1] + if not isinstance(year, (int, float)) or int(year) > last_year: + continue + for column, statistic, geography in series: + value = values[column - 1] + if not isinstance(value, (int, float)) or value <= 0: + continue + row = _base_row("offshore_wind", source, Path(path).name, sheet_name) + row.update( + technology="wind-ofs", + technology_detail="Offshore wind projects", + year=int(year), + value=float(value), + unit="USD/kW", + capacity_basis="nameplate", + statistic=statistic, + geography=geography, + dollar_year=2023, + price_basis="real", + sample_count="", + notes=( + "Figure 31 annual project CapEx. The data file omits units; the " + "report's Figure 31 axis reads USD2023/kW and its section 1.2.2 " + "normalizes all costs to real 2023 USD (FX conversion, then U.S. " + "CPI). Post-2023 pipeline years are excluded." + ), + ) + rows.append(row) + workbook.close() + return rows + + +def _eia_table_for_title(title): + """Return (label_map, table_tag) for a table title, or (None, None). + + The combined-cycle breakdown splits one plant across its turbine halves, so + it is excluded even though its title matches the natural-gas fragment. + """ + lowered = title.lower() + if "at combined-cycle plants" in lowered: + return None, None + for fragment, label_map, tag in EIA_TABLES: + if fragment in lowered: + return label_map, tag + return None, None + + +def extract_eia(path, source, year, data_url): + """Extract every national cost-by-technology table in one EIA workbook.""" + workbook = openpyxl.load_workbook(path, read_only=True, data_only=True) + sheet = workbook[workbook.sheetnames[0]] + rows = [] + label_map = None + table_tag = None + for values in sheet.iter_rows(values_only=True): + label = values[0] + value = values[1] if len(values) > 1 else None + if isinstance(label, str) and "generators installed" in label.lower(): + label_map, table_tag = _eia_table_for_title(label) + continue + if label_map is None or not isinstance(label, str): + continue + entry = label_map.get(label.strip()) + if entry is None or not isinstance(value, (int, float)): + continue + technology, detail = entry + row = _base_row("eia_generator_costs", source, Path(path).name, sheet.title) + row["source_data_url"] = data_url + row.update( + technology=technology, + technology_detail=detail, + year=int(year), + value=float(value), + unit="USD/kW", + capacity_basis="nameplate", + statistic="capacity_weighted_mean", + geography="United States", + dollar_year=int(year), + price_basis="nominal", + sample_count="", + source_table=table_tag, + notes=( + "EIA-860 generators installed in this year. Average construction " + "cost is total cost divided by total capacity. Categories follow " + "EIA definitions and are not one-to-one with ATB technologies." + ), + ) + rows.append(row) + workbook.close() + if not rows: + raise ValueError(f"No EIA cost rows extracted from {path}") + return rows + +def _sha256(path): + digest = hashlib.sha256() + with Path(path).open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _artifact(source_id, source, year=None): + if source_id == "eia_generator_costs": + filename = source["filename"].format(year=year) + if year == int(source["last_year"]): + data_url = source["current_data_url"] + else: + data_url = source["archive_data_url"].format(year=year) + else: + filename = source["filename"] + data_url = source["data_url"] + return filename, data_url + + +def scrape(config, selected="all", force=False, no_download=False): + settings = config["historical_cost_sources"] + output_dir = resolve_atb_path(settings["directory"]) + output_dir.mkdir(parents=True, exist_ok=True) + allow_insecure = settings.get("allow_insecure_ssl_fallback", False) + rows = [] + manifest = [] + source_items = settings["sources"].items() + + for source_id, source in source_items: + if not source.get("enabled", True): + continue + selector = { + "land_based_wind": "wind", + "utility_pv": "solar", + "offshore_wind": "offshore", + "eia_generator_costs": "eia", + }[source_id] + if selected not in ("all", selector): + continue + years = ( + range(int(source["first_year"]), int(source["last_year"]) + 1) + if source_id == "eia_generator_costs" + else [None] + ) + for year in years: + filename, data_url = _artifact(source_id, source, year) + path = output_dir / filename + if not no_download: + download_file( + data_url, + path, + force=force, + allow_insecure_ssl_fallback=allow_insecure, + ) + elif not path.exists(): + raise FileNotFoundError(f"Missing local source file: {path}") + + manifest.append( + { + "source_id": source_id, + "report_year": year or source.get("report_year", ""), + "page_url": source["page_url"], + "data_url": data_url, + "local_file": filename, + "size_bytes": path.stat().st_size, + "sha256": _sha256(path), + } + ) + if source_id == "land_based_wind": + rows.extend(extract_land_based_wind(path, source)) + elif source_id == "utility_pv": + rows.extend(extract_utility_pv(path, source)) + elif source_id == "offshore_wind": + rows.extend(extract_offshore_wind(path, source)) + else: + rows.extend(extract_eia(path, source, year, data_url)) + + normalized = pd.DataFrame(rows, columns=COLUMNS).sort_values( + ["technology", "source_id", "capacity_basis", "geography", "year"] + ) + normalized_path = output_dir / settings["normalized_filename"] + manifest_path = output_dir / settings["manifest_filename"] + normalized.to_csv(normalized_path, index=False) + pd.DataFrame(manifest).to_csv(manifest_path, index=False) + print(f"\nNormalized {len(normalized):,} capital-cost observations:") + print(f" {normalized_path}") + print(f"Recorded {len(manifest):,} source files and checksums:") + print(f" {manifest_path}") + if not normalized.empty: + summary = normalized.groupby(["source_id", "technology"])["year"].agg( + ["min", "max", "count"] + ) + print("\nCoverage") + print(summary.to_string()) + + +def main(): + parser = argparse.ArgumentParser( + description=( + "Download official observed generator-cost workbooks and normalize " + "their capital-cost series." + ) + ) + parser.add_argument("--config", help="Path to config.yaml (default: ../config.yaml).") + parser.add_argument( + "--only", + choices=["all", "wind", "solar", "offshore", "eia"], + default="all", + help="Process one source family (default: all).", + ) + parser.add_argument( + "--force", action="store_true", help="Replace source files already downloaded." + ) + parser.add_argument( + "--no-download", + action="store_true", + help="Rebuild normalized outputs from local source files only.", + ) + args = parser.parse_args() + scrape( + load_config(args.config), + selected=args.only, + force=args.force, + no_download=args.no_download, + ) + + +if __name__ == "__main__": + main() diff --git a/atb/scripts/settings.yaml b/atb/scripts/settings.yaml new file mode 100644 index 0000000..f7e06f5 --- /dev/null +++ b/atb/scripts/settings.yaml @@ -0,0 +1,441 @@ +# Internal formatting rules for converting raw ATB data into ReEDS inputs. +# Edit ../config.yaml for workflow, year, paths, downloads, and output choices. + +# paired entries mapping ATB parameter names into the names used in ReEDS +param_names: + OCC: capcost + Fixed O&M: fom + Variable O&M: vom + CF: cf_improvement + Heat Rate: heatrate + Round-Trip Efficiency: rte +# units for ATB parameters; these are carried over to ReEDS +units: + capcost: $/kW + fom: $/kW/yr + vom: $/MWh + heatrate: MMBtu/MWh +# specifies any columns related to cost (used to determine columns to inflate from historic data) +cost_cols: + - capcost + - fom + - vom +# the ATB technology used as the base tech for computing differences in the weighted average cost of capital +wacc_base_tech: "NaturalGas_FE" + +## Technology format ## + +## see comments below for details on entries +## note the indent when adding new techs + +techs: + # name of the technology in ReEDS; used in the filename and to rename the tech column unless a mapping for 'DisplayName' is present + upv: + # technology group in the ATB data that corresponds to this ReEDS tech + Technology: UtilityPV + # technology sub-class in the ATB data that corresponds to this ReEDS tech; can also be multiple paired entries + DisplayName: Utility PV - Class 4 + # columns in the ATB file to rename; each pairing should be "ATB col name: ReEDS col name" + subset_rows: + Case: Market + CRPYears: 30 + renamecols: + variable: t + # columns to use as index when pivoting the data + indexcols: + - Scenario + - t + # list of columns (in order) to output in the final data for ReEDS + cols: + - t + - capcost + - fom + - vom + - cf_improvement + # option to normalize capacity factor data to a reference point + # specify year (t) and scenario for normalization + cfbase: + t: 2035 + Scenario: Moderate + # name of technology group used in ReEDS for financial parameters + financialname: PV + # any custom functions used to process the data + functions: + - normalize_cf + + wind-ons: + Technology: LandbasedWind + DisplayName: Land-Based Wind - Class 4 - Technology 1 + reeds_name: ons-wind + subset_rows: + Case: Market + CRPYears: 30 + renamecols: + variable: t + "Overnight Cap Cost $/kW": capcost + "Fixed O&M $/(kW-yr)": fom + "Var O&M $/MWh": vom + "Turbine": turbine + "Year": t + "CF_mult": cf_improvement + # adds a column with data to the ReEDS output; paired entry with "col name: value" + addcols: + turbine: 115hh_170rd + indexcols: + - Scenario + - t + - turbine + cols: + - turbine + - t + - capcost + - fom + - vom + - cf_improvement + # ReEDS assigns the onshore wind columns by position rather than by name + # (plantcostprep.py), so the written file must keep this exact order and + # these exact headers. Maps internal column name -> ReEDS header. + output_cols: + turbine: Turbine + t: Year + cf_improvement: CF_mult + capcost: Overnight Cap Cost $/kW + fom: Fixed O&M $/(kW-yr) + vom: Var O&M $/MWh + cfbase: + t: 2035 + Scenario: Moderate + financialname: ONSWIND + decimals: 5 + functions: + - normalize_cf + + wind-ofs: + Technology: OffShoreWind + DisplayName: + Offshore Wind - Class 1: fixed + Offshore Wind - Class 8: floating + reeds_name: ofs-wind + subset_rows: + Case: Market + CRPYears: 30 + addcols: + rsc_mult: 1 + renamecols: + DisplayName: turbine + variable: t + "Overnight Cap Cost $/kW": capcost + "Fixed O&M $/(kW-yr)": fom + "Var O&M $/MWh": vom + "Turbine": turbine + "Wind class": turbine + "Year": t + "CF_mult": cf_improvement + indexcols: + - Scenario + - t + - turbine + - rsc_mult + # column naming the sub-technology, so config.yaml can give each turbine + # class its own historical_data mode + history_class_column: turbine + cols: + - turbine + - t + - capcost + - fom + - vom + - cf_improvement + - rsc_mult + # ReEDS detects the ATB 2024 offshore format by the presence of a 'Turbine' + # column and then assigns the remaining columns by position + # (plantcostprep.py), so both the headers and this order are required. + output_cols: + turbine: Turbine + t: Year + cf_improvement: CF_mult + capcost: Overnight Cap Cost $/kW + fom: Fixed O&M $/(kW-yr) + vom: Var O&M $/MWh + rsc_mult: rsc_mult + cfbase: + t: 2035 + Scenario: Moderate + # preserves additional columns when normalizing capacity factors + keepcols: + - turbine + financialname: OFSWIND + decimals: 5 + functions: + - apply_offshore_cost_multipliers + - normalize_cf + + battery: + Technology: Utility-Scale Battery Storage + DisplayName: + Utility-Scale Battery Storage - 2Hr: battery_li + # note: 2-hour battery is a placeholder for calculating the power and energy costs + # see the 'format_continuous_battery' function + subset_rows: + Case: Market + CRPYears: 30 + renamecols: + DisplayName: i + variable: t + indexcols: + - Scenario + - t + - i + cols: + - i + - t + - capcost + - capcost_energy + - fom + - fom_energy + - vom + - rte + financialname: BATTERY + + csp: + Technology: CSP + DisplayName: + # mapping to csp2 since that corresponds to ATB storage + solar multiple assumption + # 'add_csp_techs' expands this base series to the other CSP configurations + # using the year-specific ratios in manual_input/csp_cost_ratios_.csv. + CSP - Class 2: csp2 + subset_rows: + Case: Market + CRPYears: 30 + renamecols: + variable: t + DisplayName: type + indexcols: + - Scenario + - t + - type + cols: + - type + - t + - capcost + - fom + - vom + financialname: CSP + functions: + - add_csp_techs + + coal: + Technology: Coal_FE + DisplayName: + Coal-new: Coal-new + Coal-IGCC: Coal-IGCC + subset_rows: + Case: Market + CRPYears: 30 + Scenario: Moderate + renamecols: + DisplayName: i + variable: t + indexcols: + - Scenario + - i + - t + cols: + - i + - t + - capcost + - fom + - vom + - heatrate + functions: + - apply_coal_projection_overrides + + coal-ccs: + Technology: Coal_FE + DisplayName: + Coal-95%-CCS: coal-CCS_mod + subset_rows: + Case: Market + CRPYears: 30 + renamecols: + DisplayName: i + variable: t + indexcols: + - Scenario + - i + - t + cols: + - i + - t + - capcost + - fom + - vom + - heatrate + functions: + - apply_coal_projection_overrides + + gas: + Technology: NaturalGas_FE + DisplayName: + NG 2-on-1 Combined Cycle (F-Frame): Gas-CC + NG Combustion Turbine (F-Frame): Gas-CT + NG 1-on-1 Combined Cycle (H-Frame): Gas-CC_H_1x1 + NG 2-on-1 Combined Cycle (H-Frame): Gas-CC_H_2x1 + subset_rows: + Case: Market + CRPYears: 30 + Scenario: Moderate + renamecols: + DisplayName: i + variable: t + indexcols: + - Scenario + - i + - t + # column naming the sub-technology, so config.yaml can give each plant + # configuration its own historical_data mode + history_class_column: i + cols: + - i + - t + - capcost + - fom + - vom + - heatrate + + gas-ccs: + Technology: NaturalGas_FE + DisplayName: + NG 2-on-1 Combined Cycle (F-Frame) 95% CCS: Gas-CC-CCS_mod + NG 1-on-1 Combined Cycle (H-Frame) 95% CCS: Gas-CC_H_1x1-CCS_mod + NG 2-on-1 Combined Cycle (H-Frame) 95% CCS: Gas-CC_H_2x1-CCS_mod + subset_rows: + Case: Market + CRPYears: 30 + renamecols: + DisplayName: i + variable: t + indexcols: + - Scenario + - i + - t + cols: + - i + - t + - capcost + - fom + - vom + - heatrate + + fuelcell: + Technology: NaturalGas_FE + DisplayName: + NG Fuel Cell: ng-fuel-cell + subset_rows: + Case: Market + CRPYears: 30 + renamecols: + DisplayName: i + variable: t + indexcols: + - Scenario + - i + - t + cols: + - i + - t + - capcost + - fom + - vom + - heatrate + + nuclear: + Technology: Nuclear + DisplayName: + Nuclear - Large: Nuclear + subset_rows: + Case: Market + CRPYears: 30 + renamecols: + DisplayName: i + variable: t + indexcols: + - Scenario + - i + - t + cols: + - i + - t + - capcost + - fom + - vom + - heatrate + + nuclear-smr: + Technology: Nuclear + DisplayName: + Nuclear - Small: Nuclear-SMR + subset_rows: + Case: Market + CRPYears: 30 + renamecols: + DisplayName: i + variable: t + indexcols: + - Scenario + - i + - t + cols: + - i + - t + - capcost + - fom + - vom + - heatrate + + biopower: + Technology: Biopower + DisplayName: + Biopower - Dedicated: biopower + subset_rows: + Case: Market + CRPYears: 30 + renamecols: + DisplayName: i + variable: t + indexcols: + - Scenario + - i + - t + cols: + - i + - t + - capcost + - fom + - vom + - heatrate + + # NOTE: beccs is not available in ATB 2024 (only 'Biopower - Dedicated' exists, + # and there is no 'R&D + TC' case). Disabled for 2024 runs; re-enable for ATB 2025. + # beccs: + # Technology: Biopower + # DisplayName: + # Biopower - 95% CCS: beccs_mod + # subset_rows: + # Case: R&D + TC + # CRPYears: 30 + # renamecols: + # DisplayName: i + # variable: t + # indexcols: + # - Scenario + # - i + # - t + # cols: + # - i + # - t + # - capcost + # - fom + # - vom + # - heatrate + # filename_historical: "beccs_reference.csv"