diff --git a/.github/workflows/refresh-snapshots.yml b/.github/workflows/refresh-snapshots.yml index 7920602..bfcfbc7 100644 --- a/.github/workflows/refresh-snapshots.yml +++ b/.github/workflows/refresh-snapshots.yml @@ -20,7 +20,8 @@ # Who gets told, per AGENTS.md "Refresh, break, or schema change": a break is # an issue HERE; a merged refresh reaches consumers per the manifest's # `on_refresh` (the fan-out is not wired yet — the PR body lists what it would -# do, and today no snapshot has a consumer). +# do; the first consumer, lecture-wasm's business_cycle, reads raw/main in +# the reader's browser and needs no dispatch, PLAN Phase 5). # # Token: the PR is opened with QUANTECON_SERVICES_PAT when the org secret is # available to this repo, falling back to the workflow token. The fallback diff --git a/CATALOG.md b/CATALOG.md index f450a4f..db7e9d4 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -6,7 +6,7 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest, which may be before its consuming lectures are repointed — an empty **Used by** column means the file is here and documented but no lecture reads it from this repo yet. Files still to migrate are tracked in [PLAN.md](PLAN.md). -**44 datasets** · 40 read by lectures today, 4 awaiting repoint · 113.2 MB total · 38 permitted / 6 restricted redistribution +**44 datasets** · 44 read by lectures today · 113.2 MB total · 38 permitted / 6 restricted redistribution | Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by | | --- | --- | --- | --- | --- | --- | --- | --- | --- | @@ -32,7 +32,7 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/* | [**forbes-global2000.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-global2000.csv)
Forbes Global 2000 — firm size measures | constructed | [Forbes Global 2000](https://www.forbes.com/lists/global2000/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 115.6 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | | [**fp.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fp.dta)
Treisman (2016) Russia's Billionaires — country-year panel of billionaire counts and covariates | verbatim | [Replication package for Treisman (2016), "Russia's Billionaires" (AER Papers & Proceedings)](https://www.aeaweb.org/articles?id=10.1257/aer.p20161068) | | ✅ permitted | ✅ verified | n/a (verbatim) | 1000.1 KB | [lecture-python.myst · mle.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/mle.md)
[lecture-python.zh-cn · mle.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/mle.md)
[lecture-stats · mle.md](https://github.com/QuantEcon/lecture-stats/blob/main/lectures/mle.md) | | [**fred_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fred_data.csv)
US Treasury yields and NBER recessions — GS1/GS5/GS10, DFII5/DFII10 and USREC, monthly 1953-04 to 2024-12 | constructed | [FRED (Federal Reserve Bank of St. Louis) — series GS1, GS5, GS10, DFII5, DFII10, USREC](https://fred.stlouisfed.org) | | ✅ permitted | ✅ verified | ✅ committed | 27.3 KB | [lecture-python-advanced.myst · risk_aversion_or_mistaken_beliefs.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/risk_aversion_or_mistaken_beliefs.md) | -| [**gdp_growth_annual.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/gdp_growth_annual.csv)
World Bank GDP growth (annual %) — nine economies, 1960 onward | dynamic-snapshot | [World Bank, World Development Indicators (national accounts data, and OECD National Accounts data files)](https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG) | CC BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 10.4 KB | — | +| [**gdp_growth_annual.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/gdp_growth_annual.csv)
World Bank GDP growth (annual %) — nine economies, 1960 onward | dynamic-snapshot | [World Bank, World Development Indicators (national accounts data, and OECD National Accounts data files)](https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG) | CC BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 10.4 KB | [lecture-wasm · business_cycle.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/business_cycle.md) | | [**hansen_jagannathan_1991_data.json**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_jagannathan_1991_data.json)
Hansen-Jagannathan (1991) replication — US asset returns 1891-1986, with annual consumption (three-table bundle) | constructed | [Robert J. Shiller's public data workbooks (chapt26.xlsx, ie_data.xls) and FRED](http://www.econ.yale.edu/~shiller/data.htm) | | ✅ permitted | ✅ verified | ⚠️ unrecovered | 62.0 KB | [lecture-python-advanced.myst · hansen_jagannathan_1991.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/hansen_jagannathan_1991.md) | | [**hansen_singleton_1982_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_singleton_1982_data.csv)
Hansen-Singleton (1982) replication — monthly US gross real market return and consumption growth, 1959-1978 | constructed | [FRED (BEA and BLS monthly series) and the Ken French data library (F-F_Research_Data_Factors)](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) | | ✅ permitted | ✅ verified | ✅ committed | 11.4 KB | [lecture-python.myst · hansen_singleton_1982.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/hansen_singleton_1982.md)
[lecture-python.zh-cn · hansen_singleton_1982.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/hansen_singleton_1982.md) | | [**hansen_singleton_1983_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_singleton_1983_data.csv)
Hansen-Singleton (1983) replication — monthly US returns, consumption and inflation, 1959-1978 | constructed | [FRED (BEA and BLS monthly series) and the Ken French data library (F-F_Research_Data_Factors)](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) | | ✅ permitted | ✅ verified | ✅ committed | 25.5 KB | [lecture-python.myst · hansen_singleton_1983.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/hansen_singleton_1983.md)
[lecture-python.zh-cn · hansen_singleton_1983.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/hansen_singleton_1983.md) | @@ -47,12 +47,12 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/* | [**maketable4.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/maketable4.dta)
Acemoglu-Johnson-Robinson (2001) colonial origins — table 4 replication data | verbatim | [Daron Acemoglu data archive — "The Colonial Origins of Comparative Development", table 4 (program and data files)](https://economics.mit.edu/people/faculty/daron-acemoglu/data-archive) | | ✅ permitted | ✅ verified | n/a (verbatim) | 11.7 KB | [lecture-python.myst · ols.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/ols.md)
[lecture-python.zh-cn · ols.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/ols.md) | | [**mpd2020.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/mpd2020.xlsx)
Maddison Project Database 2020 — GDP per capita and population, 1 CE to 2018 | constructed | [Maddison Project Database, version 2020](https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2020) | CC BY 4.0 | ✅ permitted | ⇄ diverged | ⚠️ unrecovered | 1.7 MB | [lecture-python-intro · long_run_growth.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/long_run_growth.md)
[lecture-wasm · long_run_growth.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/long_run_growth.md)
[lecture-intro.zh-cn · long_run_growth.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/long_run_growth.md)
[test-actions-lecture-intro · long_run_growth.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/long_run_growth.md) | | [**nom_balances.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/nom_balances.npy)
French Revolution — monthly nominal assignat balances, 1789-1796 | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.4 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)
[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md)
[lecture-intro.zh-cn · french_rev.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/french_rev.md)
[test-actions-lecture-intro · french_rev.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/french_rev.md)
⚠️ Reads a local `datasets/` copy, not this file
[tom-econ370-2025 · french_rev.md](https://github.com/QuantEcon/tom-econ370-2025/blob/main/lectures/french_rev.md)
⚠️ Course fork with a live Pages site; reads its own blob-identical `datasets/` copy (french_rev.md:715-716), not this file, and its `base_url` still points at lecture-python-intro
[python-lecture-sandpit.myst · french_rev.md](https://github.com/QuantEcon/python-lecture-sandpit.myst/blob/main/lectures/french_rev.md)
⚠️ Public sandpit holding `lectures/_static/` copies | -| [**private_credit_to_gdp.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/private_credit_to_gdp.csv)
World Bank domestic credit to the private sector (% of GDP) — United Kingdom, 1960 onward | dynamic-snapshot | [World Bank, World Development Indicators (International Monetary Fund, International Financial Statistics, and World Bank and OECD GDP estimates)](https://data.worldbank.org/indicator/FS.AST.PRVT.GD.ZS) | CC BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 1.6 KB | — | +| [**private_credit_to_gdp.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/private_credit_to_gdp.csv)
World Bank domestic credit to the private sector (% of GDP) — United Kingdom, 1960 onward | dynamic-snapshot | [World Bank, World Development Indicators (International Monetary Fund, International Financial Statistics, and World Bank and OECD GDP estimates)](https://data.worldbank.org/indicator/FS.AST.PRVT.GD.ZS) | CC BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 1.6 KB | [lecture-wasm · business_cycle.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/business_cycle.md) | | [**realwage.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/realwage.csv)
OECD real minimum wages — 32 countries, 2006–2016 | constructed | [OECD — Real minimum wages (RMW)](https://stats.oecd.org/Index.aspx?DataSetCode=RMW) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 118.7 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md)
[lecture-python.zh-cn · pandas_panel.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/pandas_panel.md) | | [**test_pwt.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/test_pwt.csv)
Penn World Table teaching extract — eight countries, year 2000 | constructed | [Penn World Table (Heston, Summers and Aten; PWT 6.x/7.x era) — exact vintage unestablished](https://www.rug.nl/ggdc/productivity/pwt/pwt-releases/pwt-7.0) | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 793 B | [lecture-python-programming · pandas.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas.md)
[lecture-python-programming · polars.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/polars.md)
[lecture-python-programming.zh-cn · pandas.md](https://github.com/QuantEcon/lecture-python-programming.zh-cn/blob/main/lectures/pandas.md)
[lecture-python-programming.zh-cn · polars.md](https://github.com/QuantEcon/lecture-python-programming.zh-cn/blob/main/lectures/polars.md)
[lecture-python-programming.fr · pandas.md](https://github.com/QuantEcon/lecture-python-programming.fr/blob/main/lectures/pandas.md)
[lecture-python-programming.fr · polars.md](https://github.com/QuantEcon/lecture-python-programming.fr/blob/main/lectures/polars.md)
[lecture-python-programming.fa · pandas.md](https://github.com/QuantEcon/lecture-python-programming.fa/blob/main/lectures/pandas.md)
[lecture-python-programming.fa · polars.md](https://github.com/QuantEcon/lecture-python-programming.fa/blob/main/lectures/polars.md) | -| [**unemployment_rate_annual.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/unemployment_rate_annual.csv)
World Bank unemployment rate (annual, % of labour force) — USA, FRA, GBR, JPN, 1960 onward | dynamic-snapshot | [World Bank, World Development Indicators (International Labour Organization, ILOSTAT database, national estimates)](https://data.worldbank.org/indicator/SL.UEM.TOTL.NE.ZS) | CC BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 1.8 KB | — | +| [**unemployment_rate_annual.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/unemployment_rate_annual.csv)
World Bank unemployment rate (annual, % of labour force) — USA, FRA, GBR, JPN, 1960 onward | dynamic-snapshot | [World Bank, World Development Indicators (International Labour Organization, ILOSTAT database, national estimates)](https://data.worldbank.org/indicator/SL.UEM.TOTL.NE.ZS) | CC BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 1.8 KB | [lecture-wasm · business_cycle.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/business_cycle.md) | | [**us_adult_heights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/us_adult_heights.csv)
United States — adult standing height by sex, NHANES 2015-2018 | constructed | [National Health and Nutrition Examination Survey (NHANES), US Centers for Disease Control and Prevention, National Center for Health Statistics](https://www.cdc.gov/nchs/nhanes/index.htm) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 123.1 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md)
[lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md)
[lecture-intro.zh-cn · fitting_distributions.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/fitting_distributions.md)
[lecture-intro.zh-cn · observed_distributions.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/observed_distributions.md)
[lecture-intro.zh-cn · prob_dist.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/prob_dist.md)
⚠️ Reads a local `_static/lecture_specific/prob_dist/` copy, not this file, under a comment that still says to switch once the datasets repo exists | -| [**us_business_cycle_monthly.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/us_business_cycle_monthly.csv)
US business-cycle indicators, monthly — unemployment, NBER recessions, consumer sentiment, core CPI, industrial production, 1919 onward | dynamic-snapshot | [FRED (Federal Reserve Bank of St. Louis) — series UNRATE, USREC, UMCSENT, CPILFESL, INDPRO, M0892AUSM156SNBR](https://fred.stlouisfed.org) | | ✅ permitted | ✅ verified | ✅ committed | 41.7 KB | — | +| [**us_business_cycle_monthly.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/us_business_cycle_monthly.csv)
US business-cycle indicators, monthly — unemployment, NBER recessions, consumer sentiment, core CPI, industrial production, 1919 onward | dynamic-snapshot | [FRED (Federal Reserve Bank of St. Louis) — series UNRATE, USREC, UMCSENT, CPILFESL, INDPRO, M0892AUSM156SNBR](https://fred.stlouisfed.org) | | ✅ permitted | ✅ verified | ✅ committed | 41.7 KB | [lecture-wasm · business_cycle.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/business_cycle.md) | | [**usa-gini-nwealth-tincome-lincome.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/usa-gini-nwealth-tincome-lincome.csv)
US Gini coefficients — net wealth, total income and labour income, 1950-2016 | constructed | [Derived from SCF_plus_mini.csv (this repo), an extract of the SCF+ panel](https://github.com/QuantEcon/data-lectures/blob/main/lectures/SCF_plus_mini.csv) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 1.2 KB | [lecture-python-intro · inequality.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/inequality.md)
[lecture-wasm · inequality.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/inequality.md)
[lecture-intro.zh-cn · inequality.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/inequality.md)
[test-actions-lecture-intro · inequality.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/inequality.md) | --- diff --git a/PLAN.md b/PLAN.md index 2a0a190..80861fa 100644 --- a/PLAN.md +++ b/PLAN.md @@ -300,7 +300,7 @@ Full automation: - [x] Retrofit `builders/business_cycle.py` to the four-stage builder contract — **done 2026-09-01**, with the two provenance dumps moved out of the published tree to `provenance/` ([#13](https://github.com/QuantEcon/data-lectures/issues/13)). Its `validate()` is the first to face a *revised* upstream: it bounds the overlap window (5 pp) and prints the revision summary rather than asserting equality, which is the review surface the refresh-as-PR workflow below will use. It previously had fetch/transform/write but no validate stage. Builder architecture and a copy-able template: [#14](https://github.com/QuantEcon/data-lectures/issues/14) - [x] Scheduled refresh workflow for dynamic datasets — **landed 2026-09-01** as `.github/workflows/refresh-snapshots.yml`, manifest-driven rather than cron-per-class: a weekly run asks `scripts/snapshots.py due` which `dynamic-snapshot` datasets have their cadence elapsed (or are `diverged`, or were never refreshed), runs each builder in place, stamps the manifest (`retrieved`, `sha256`, `integrity.upstream: verified`, `date_range.end`), regenerates the catalog, and opens a PR on `refresh/` whose body is the builder's overlap summary. Nothing auto-merges; the first consumer is `business_cycle_data.csv`, not UNRATE — the pilot's order inverted once the World Bank file turned out to be the one already here - [x] Weekly sources-alive canary: fetch + validate, no commit, opens an issue on failure — **landed 2026-09-01** as the `canary` job of the same workflow: every dynamic snapshot's builder runs with `--out-dir`, and a failure opens or updates one `upstream-break` issue classified by exit code (2 = the data broke the contract, a human; anything else = the fetch, a retry). Covers the live APIs only as their snapshot twins land here — the 23 live-API lectures without a twin are still guarded by nothing but their own CI -- [ ] Consumer fan-out: a merged refresh or in-place correction dispatches rebuilds of the repos in the dataset's machine-readable `consumers` list — **policy recorded 2026-09-01** (AGENTS.md "Refresh, break, or schema change": per-consumer `on_refresh: rebuild | review`), and the refresh PR body already lists what the fan-out would do; the dispatch itself waits for the first snapshot with a consumer +- [ ] Consumer fan-out: a merged refresh or in-place correction dispatches rebuilds of the repos in the dataset's machine-readable `consumers` list — **policy recorded 2026-09-01** (AGENTS.md "Refresh, break, or schema change": per-consumer `on_refresh: rebuild | review`), and the refresh PR body already lists what the fan-out would do; the dispatch itself waits for the first consumer that executes at build time. **The first snapshot consumer needs no dispatch** (P4, [#118](https://github.com/QuantEcon/data-lectures/issues/118), "Not in scope"): `lecture-wasm` runs `business_cycle` in the reader's browser and reads `raw/main` when the page runs, so a merged refresh reaches its readers with no rebuild. Its CI never executes code and has no dispatch path that builds, so a dispatched rebuild would change nothing a reader sees - [ ] Package the refresh job as a reusable workflow (`quantecon/actions`) once it stabilizes ### Phase 6 — Metadata backfill for existing holdings @@ -340,7 +340,13 @@ The first end-to-end deployment: one dataset per hosting pattern, each the harde **The flip was the acceptance test, and it measured as one.** Dry-run locally in both directions before pushing: `landed` → exit 1 with 6 warnings; `repointed` → exit 0. So the red window was real, opened when the last consuming PR merged, and closed with the flip. **Do this both-directions dry-run on every future wave** — it converts "same-day, trust me" into a measurement. Five things P3 proved that were not on its test list. A `constructed` dataset's builder must land in the **same** PR as the data, because `check_consumed_files.py` asserts the `builder:` path resolves. `builders/README.md`'s coverage table is a real coverage report and goes stale silently. The plain-git decision costs ~10 MB of packed history for 110 MB of working tree, since CSV compresses 5-22×. The **C0 → C1 → C2 ordering worked and proved less than it looks like** — the sync PR it was designed to defuse (QuantEcon/lecture-intro.zh-cn#293) touched zero data-read lines, zero `# i18n` markers and zero protected localisations, but nothing ever asked the model to rewrite those cells, so the markers remain unexercised, prompt-level protection and **the hand-diff is what protects a localisation**. And the translation sync is **`.md`-only**, so no hand-localised `_static` asset can be created, updated or repaired by it — every `data.ipynb` copy had to be repointed by hand in all four repos, filed upstream as QuantEcon/action-translation#271 -- [ ] **P4 — dynamic snapshot twin**: originally `UNRATE` alone; **reframed 2026-09-01** as the `business_cycle` set, because the lecture that needs a twin is excluded from `lecture-wasm` for want of one and a partial twin buys it nothing. Done so far: `business_cycle_data.csv` (renamed `gdp_growth_annual.csv` 2026-09-07 under the naming policy, [#113](https://github.com/QuantEcon/data-lectures/issues/113)) manifested and its builder retrofitted ([#109](https://github.com/QuantEcon/data-lectures/pull/109)); the refresh-as-PR and canary workflow ([#110](https://github.com/QuantEcon/data-lectures/pull/110)); the first real refresh ([#112](https://github.com/QuantEcon/data-lectures/pull/112)); the World Bank set extended to three tables and the FRED half landed as one composite monthly file on a shared `builders/_fred.py` library ([#114](https://github.com/QuantEcon/data-lectures/pull/114)). Remaining: the `lecture-wasm` adoption ([QuantEcon/lecture-wasm#70](https://github.com/QuantEcon/lecture-wasm/issues/70) — intro keeps its live calls as the lesson), the flip with `on_refresh: rebuild`, and a canary run catching an induced failure +- [x] **P4 — dynamic snapshot twin**: originally `UNRATE` alone; **reframed 2026-09-01** as the `business_cycle` set, because the lecture that needed a twin was excluded from `lecture-wasm` for want of one and a partial twin would have bought it nothing. **Complete B_MERGE_DATE**, run from 2026-09-29 as the `wasm-dynamic-snapshots` project ([#118](https://github.com/QuantEcon/data-lectures/issues/118)). + + **Data half** — `business_cycle_data.csv` (renamed `gdp_growth_annual.csv` 2026-09-07 under the naming policy, [#113](https://github.com/QuantEcon/data-lectures/issues/113)) manifested and its builder retrofitted ([#109](https://github.com/QuantEcon/data-lectures/pull/109)); the refresh-as-PR and canary workflow ([#110](https://github.com/QuantEcon/data-lectures/pull/110)); the first real refresh ([#112](https://github.com/QuantEcon/data-lectures/pull/112)); the World Bank set extended to three tables and the FRED half landed as one composite monthly file on a shared `builders/_fred.py` library ([#114](https://github.com/QuantEcon/data-lectures/pull/114)). + + **Consumer half and the flip, B_MERGE_DATE** — `lecture-wasm` re-enabled `business_cycle`, reading all four files from `raw/main` ([QuantEcon/lecture-wasm#85](https://github.com/QuantEcon/lecture-wasm/pull/85), for [QuantEcon/lecture-wasm#70](https://github.com/QuantEcon/lecture-wasm/issues/70)); intro keeps its live calls as the lesson. The same day the flip ([#146](https://github.com/QuantEcon/data-lectures/issues/146)) recorded `lecture-wasm` as the consumer of all four, with `on_refresh: rebuild`, and moved their `migration.yml` records to `repointed`. Dry-run in both directions first, as P3 asks: with `lecture-wasm` at the adoption branch, `landed` → exit 1 with 4 warnings, one per record, and `repointed` → exit 0; with `lecture-wasm` at its `main`, `repointed` → exit 1. + + **Canary proof complete 2026-09-30** ([#147](https://github.com/QuantEcon/data-lectures/issues/147)): on a throwaway branch that narrowed `UNRATE`'s band in `builders/business_cycle_fred.py`, two dispatched runs of `refresh-snapshots` exited 2 in the canary and opened, then updated, the `upstream-break` issue. The second, [36654357802](https://github.com/QuantEcon/data-lectures/actions/runs/36654357802), forced the file due, so it also shows a failing canary keeping the refresh job from running; the first is [36654189187](https://github.com/QuantEcon/data-lectures/actions/runs/36654189187). Nothing reached `main` - [ ] Verify each migrated URL with a pyodide/JupyterLite fetch (CORS, meta#143) - [ ] Fold every validated decision into the draft `styleguide/datasets.md` (manual#108) as it is proven diff --git a/lectures/gdp_growth_annual.csv.yml b/lectures/gdp_growth_annual.csv.yml index 57cf5fa..e82fe81 100644 --- a/lectures/gdp_growth_annual.csv.yml +++ b/lectures/gdp_growth_annual.csv.yml @@ -40,10 +40,9 @@ source: url: https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG doi: null version: > - Unknown for the committed bytes — WDI carries a database-level - `lastupdated` stamp but the export recorded none (see `retrieved`). The - live database at the 2026-09-01 check reported lastupdated 2026-07-13 and - had moved on from these bytes (see integrity.upstream). + Live WDI at each refresh; the vintage is pinned by `retrieved` and + integrity.upstream, not by a source-side edition (WDI carries only a + database-level lastupdated stamp). citation: > World Bank, World Development Indicators, series NY.GDP.MKTP.KD.ZG (GDP growth, annual %), from World Bank national accounts data and OECD @@ -70,36 +69,30 @@ license: is the model case for this repo's cache-with-attribution policy (AGENTS.md, "Licensing and attribution"). -# `retrieved` is stamped by scripts/snapshots.py on every merged refresh; null -# until the first one. The inherited bytes were produced by the committed -# builder in QuantEcon/data commit b857c5c of 2025-02-16 — but that is when -# QuantEcon ran the script and committed the output, and AGENTS.md says not to -# reconstruct a retrieval date from git history. The vintage is pinned by -# content instead: YR2023 is the last column and is populated for all five -# economies, which places the export after the WDI release that first carried -# 2023 growth, and integrity.upstream records how far the live series has -# moved since. +# Stamped by scripts/snapshots.py on every merged refresh, so it dates the +# bytes committed now. It was null until the first refresh (#112) replaced the +# inherited bytes (QuantEcon/data b857c5c, 2025-02-16), which carried no +# retrieval date; AGENTS.md says not to reconstruct one from git history. retrieved: 2026-09-01 maintainer: QuantEcon # --------------------------------------------------------------------------- # Integrity (PLAN Phase 7) # --------------------------------------------------------------------------- -# No migration check applies: no lecture has ever read this file (it was -# "never adopted" — PLAN "The audit (2026-07-15)"), so there is no prior copy -# to byte-match and no figure a change here could alter today. +# No migration check applied: no lecture read this file before lecture-wasm +# adopted it in P4 (it was "never adopted" — PLAN "The audit (2026-07-15)"), +# so there was no prior copy to byte-match. integrity: sha256: 41df23233ea238e4166c5c21cc383791015d4c9af6868b840901a8ce083c9da8 upstream: - # `diverged`, not `failing`: the builder was re-run in full against live - # WDI and its output differs from these bytes in the way a revised - # aggregate is expected to. For a dynamic snapshot this is the normal - # state between refreshes, not a defect — the refresh is the resolution, - # and it is a deliberate PR, not something this manifest asks for. # status / date / against / note are stamped by scripts/snapshots.py on - # every merged refresh (the delta block below is dropped at the same time, - # since a fresh snapshot is `verified` by construction). + # every merged refresh: a fresh snapshot is `verified` by construction, + # since these bytes are the builder's validated output from the live + # source that day. The World Bank goes on revising the series, so a + # re-run between refreshes is expected to differ from these bytes. For a + # dynamic snapshot that is the normal state, not a defect: the next + # refresh is the resolution, and it is a deliberate PR. status: verified date: 2026-09-01 against: builders/business_cycle.py @@ -143,13 +136,26 @@ schema: # --------------------------------------------------------------------------- # Consumers — how a correction knows what to rebuild # --------------------------------------------------------------------------- -# None yet. Decided 2026-09-01: lecture-python-intro KEEPS its live wbgapi -# calls (the API is the lesson there), and lecture-wasm — where business_cycle -# is currently excluded from the build because pyodide cannot reach the API — -# will read this file and its two siblings plus us_business_cycle_monthly.csv -# (QuantEcon/lecture-wasm#70). When that repoint lands, record it here with -# `on_refresh: rebuild`, and `cadence` below becomes load-bearing. -consumers: [] +# lecture-wasm's business_cycle reads this file, its two World Bank siblings +# and us_business_cycle_monthly.csv from raw/main (QuantEcon/lecture-wasm#70). +# The snapshots are what let lecture-wasm re-enable the lecture, which it had +# excluded because its live reads could not run in the browser: neither +# wbgapi nor pandas_datareader ships with Pyodide, and FRED's fredgraph.csv +# sends no CORS header. Decided 2026-09-01: +# lecture-python-intro KEEPS its live wbgapi calls (the API is the lesson +# there), so it is not a consumer. With a reader, `cadence` below is +# load-bearing: the page shows whatever vintage is committed. +# +# `on_refresh: rebuild`, because no prose in the lecture quotes a value a +# refresh can move: its numbers are the years of historical events, and its +# ranges run "to the present". The value dispatches nothing today. The page +# runs in the reader's browser and reads raw/main when it runs, so a merged +# refresh reaches its readers with no rebuild (PLAN Phase 5, consumer +# fan-out). +consumers: + - repo: QuantEcon/lecture-wasm + file: lectures/business_cycle.md + on_refresh: rebuild builder: builders/business_cycle.py builder_status: committed # four-stage since 2026-09-01 (PLAN Phase 5); diff --git a/lectures/private_credit_to_gdp.csv.yml b/lectures/private_credit_to_gdp.csv.yml index 7fbae8d..10cf080 100644 --- a/lectures/private_credit_to_gdp.csv.yml +++ b/lectures/private_credit_to_gdp.csv.yml @@ -73,8 +73,13 @@ schema: known_nulls: {} nulls: {along: columns, leading: true, recent: 2, ended: [], inner: {}} -# None yet — see gdp_growth_annual.csv.yml (QuantEcon/lecture-wasm#70). -consumers: [] +# lecture-wasm's business_cycle reads this file (QuantEcon/lecture-wasm#70); +# lecture-python-intro keeps its live call. See gdp_growth_annual.csv.yml for +# the decision and for why `on_refresh` is `rebuild`. +consumers: + - repo: QuantEcon/lecture-wasm + file: lectures/business_cycle.md + on_refresh: rebuild builder: builders/business_cycle.py builder_status: committed diff --git a/lectures/unemployment_rate_annual.csv.yml b/lectures/unemployment_rate_annual.csv.yml index 4012d93..d202521 100644 --- a/lectures/unemployment_rate_annual.csv.yml +++ b/lectures/unemployment_rate_annual.csv.yml @@ -82,10 +82,13 @@ schema: known_nulls: {} nulls: {along: columns, leading: true, recent: 2, ended: [], inner: {}} -# None yet — see gdp_growth_annual.csv.yml: lecture-wasm will read this -# file once business_cycle is re-enabled there (QuantEcon/lecture-wasm#70); -# lecture-python-intro keeps its live call. -consumers: [] +# lecture-wasm's business_cycle reads this file (QuantEcon/lecture-wasm#70); +# lecture-python-intro keeps its live call. See gdp_growth_annual.csv.yml for +# the decision and for why `on_refresh` is `rebuild`. +consumers: + - repo: QuantEcon/lecture-wasm + file: lectures/business_cycle.md + on_refresh: rebuild builder: builders/business_cycle.py builder_status: committed diff --git a/lectures/us_business_cycle_monthly.csv.yml b/lectures/us_business_cycle_monthly.csv.yml index 3c7aa8b..8b4b1e6 100644 --- a/lectures/us_business_cycle_monthly.csv.yml +++ b/lectures/us_business_cycle_monthly.csv.yml @@ -139,13 +139,16 @@ schema: CPILFESL: [2025-10-01] UMCSENT: {sparse_until: 1978-01-01} -# None yet. Decided 2026-09-01: lecture-python-intro keeps its live -# pandas_datareader calls (the API is the lesson there); lecture-wasm will -# read this file once business_cycle is re-enabled in its build -# (QuantEcon/lecture-wasm#70) — record it here with `on_refresh: rebuild`. -# The python.myst unemployment_linear and unemployment_shocks lectures read -# UNRATE live over the same window and are candidates for a later adoption. -consumers: [] +# lecture-wasm's business_cycle reads this file (QuantEcon/lecture-wasm#70). +# Decided 2026-09-01: lecture-python-intro keeps its live pandas_datareader +# calls (the API is the lesson there). See gdp_growth_annual.csv.yml for why +# `on_refresh` is `rebuild`. The python.myst unemployment_linear and +# unemployment_shocks lectures read UNRATE live over the same window and are +# candidates for a later adoption. +consumers: + - repo: QuantEcon/lecture-wasm + file: lectures/business_cycle.md + on_refresh: rebuild builder: builders/business_cycle_fred.py builder_status: committed diff --git a/migration.yml b/migration.yml index f14f35a..f3d0ff5 100644 --- a/migration.yml +++ b/migration.yml @@ -833,67 +833,83 @@ datasets: # The dynamic-snapshot prototype. The BYTES have been in the published tree # since the 2026-07-16 flatten (#10; originally QuantEcon/data b857c5c, # 2025-02-16), but `landed` means file + manifest (see the lifecycle above), - # and the manifest arrived with #109 — so that is the landing PR. No - # consumer: intro's business_cycle still makes the live wbgapi call the - # builder reproduces; whether it adopts the snapshot is the P4 decision. - # Renamed from business_cycle_data.csv on 2026-09-07 under the naming policy - # (#113), while `consumers` was still empty; no URL was ever read. + # and the manifest arrived with #109 — so that is the landing PR. Renamed + # from business_cycle_data.csv on 2026-09-07 under the naming policy (#113), + # while `consumers` was still empty; no URL was ever read under the old name. + # Its one consumer is lecture-wasm's business_cycle, which reads all four P4 + # files (QuantEcon/lecture-wasm#70). Intro's business_cycle keeps the live + # wbgapi call the builder reproduces, because there the API is the lesson + # (decided 2026-09-01). gdp_growth_annual.csv: pilot: P4 - status: landed + status: repointed prior_pattern: null # never read from anywhere — the lecture - # fetches live (api pattern), so there is no + # fetched live (api pattern), so there was no # prior file read to migrate landed: pr: QuantEcon/data-lectures#109 date: 2026-09-01 - repoints: [] + repoints: + - repo: QuantEcon/lecture-wasm + pr: QuantEcon/lecture-wasm#85 + date: B_MERGE_DATE cutover: null # The rest of the business_cycle set (#114): two more World Bank tables from # the same builder, and the FRED half as one composite monthly file. Names - # settled 2026-09-07 (#113). Consumer-to-be: lecture-wasm (lecture-wasm#70). + # settled 2026-09-07 (#113). One lecture-wasm PR repointed all four, and the + # records flipped the same day (#146), after P3's acceptance test: the strict + # audit dry-run in both directions (PLAN Phase 8, P4). unemployment_rate_annual.csv: pilot: P4 - status: landed + status: repointed prior_pattern: null landed: pr: QuantEcon/data-lectures#114 date: 2026-09-01 - repoints: [] + repoints: + - repo: QuantEcon/lecture-wasm + pr: QuantEcon/lecture-wasm#85 + date: B_MERGE_DATE cutover: null private_credit_to_gdp.csv: pilot: P4 - status: landed + status: repointed prior_pattern: null landed: pr: QuantEcon/data-lectures#114 date: 2026-09-01 - repoints: [] + repoints: + - repo: QuantEcon/lecture-wasm + pr: QuantEcon/lecture-wasm#85 + date: B_MERGE_DATE cutover: null us_business_cycle_monthly.csv: pilot: P4 - status: landed + status: repointed prior_pattern: null landed: pr: QuantEcon/data-lectures#114 date: 2026-09-01 - repoints: [] + repoints: + - repo: QuantEcon/lecture-wasm + pr: QuantEcon/lecture-wasm#85 + date: B_MERGE_DATE cutover: null # Planned waves that have not landed anything here yet. `datasets` names the # files as the audit sees them today, so the dashboard can join the two views. # `title` is the reader-facing milestone name (the dashboard is read by people -# who are not doing the migration); `scope` carries the insider detail. -pending: - - pilot: P4 - title: Maintained snapshots for lectures that fetch live API data - scope: the dynamic-snapshot case — a maintained UNRATE twin for the incidental FRED consumers - datasets: [] # produces a NEW file here; no existing static file moves - tracking: - - QuantEcon/meta#338 - -# Interim → final URL cutover (status: repointed → final) is tracked as one -# sweep, not per dataset: QuantEcon/data-lectures#15, gated on Phase 4 DNS. +# who are not doing the migration); `scope` carries the insider detail. None is +# planned: the last entry, P4 ("a maintained UNRATE twin"), became the four +# records above and was removed when they were repointed. +pending: [] + +# repointed → final is qeld adoption (the lifecycle above; D11, 2026-08-12): +# every consumer code read resolves through qeld.url(). That step is the +# proposed qeld-package project's (PLAN-QELD-PACKAGE.md; #66, #88) and is +# tracked there, not per dataset here. The data.quantecon.org URL cutover this +# comment used to point at (#15, gated on Phase 4 DNS) was retired with the +# custom-domain plan (#37). diff --git a/scripts/audit_annotations.yml b/scripts/audit_annotations.yml index f3bc2b7..55ec421 100644 --- a/scripts/audit_annotations.yml +++ b/scripts/audit_annotations.yml @@ -45,7 +45,11 @@ api: lecture-python-intro:business_cycle:pandas_datareader: series: UNRATE, USREC, M0892AUSM156SNBR, UMCSENT, CPILFESL, INDPRO pedagogy: mixed - note: snapshot pipeline already exists (data-lectures gdp_growth_annual.csv, business_cycle_data.csv until 2026-09-07) but is unadopted + note: > + intro keeps this live call by decision (2026-09-01), because the API is + part of the lesson here; its snapshot twin, data-lectures + us_business_cycle_monthly.csv, is what lecture-wasm's business_cycle + reads (QuantEcon/lecture-wasm#70) lecture-python-intro:commod_price:yfinance: series: CT=F pedagogy: incidental @@ -133,14 +137,6 @@ api: # lecture-wasm mirrors intro's sources; live APIs are the repo's core problem # (pyodide cannot reach them — meta#143), so every entry here is a migration # target, not an endorsement. - lecture-wasm:business_cycle:wbgapi: - series: NY.GDP.MKTP.KD.ZG, SL.UEM.TOTL.NE.ZS, FS.AST.PRVT.GD.ZS - pedagogy: mixed - note: mirror of intro's business_cycle; cannot run under pyodide - lecture-wasm:business_cycle:pandas_datareader: - series: UNRATE, USREC, M0892AUSM156SNBR, UMCSENT, CPILFESL, INDPRO - pedagogy: mixed - note: mirror of intro's business_cycle; cannot run under pyodide lecture-wasm:commod_price:yfinance: series: CT=F pedagogy: incidental diff --git a/scripts/snapshots.py b/scripts/snapshots.py index fd76e0a..92a752f 100644 --- a/scripts/snapshots.py +++ b/scripts/snapshots.py @@ -331,7 +331,7 @@ def cmd_pr_body(args) -> int: action = c.get("on_refresh", "unset") lines.append(f"- `{c.get('repo')}` — `{c.get('file')}` — `on_refresh: {action}`" + (" → open an issue there with this summary" if action == "review" - else " → dispatch a rebuild" if action == "rebuild" + else " → dispatch a rebuild (not wired yet: PLAN Phase 5)" if action == "rebuild" else " → decide, then record `on_refresh` in the manifest")) else: lines.append("- none recorded — nothing to rebuild or notify.")