From 4df591dca5e6ed46327e6624aff53a3b734ca005 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Wed, 30 Sep 2026 14:32:16 +1000 Subject: [PATCH] licence: rest UMCSENT on Michigan's FAQ and record FRED's caching clause Records two licensing decisions taken on 2026-09-30 (#35, #118), and two corrections the check turned up. - us_business_cycle_monthly.csv keeps UMCSENT, `permitted` on the University of Michigan's FAQ ("Any data, tables, or charts available to the public on our main website can be used without permission"): the headline index is main-website data, and every committed value equals Michigan's own tbmics.csv. bbh_michigan_monthly.csv stays `restricted`, because its columns come from portal-only tables that the portal's usage agreement governs. Both manifests record the line. - AGENTS.md records FRED's known yes with Terms of Use clause (p), on storing or caching FRED content, in view. The summarized terms and each series' copyright label are the grounds, neither stands apart from (p), and asking the Bank for consent waits for the promotion gate. The seven FRED-derived manifests point to that paragraph. - Corrections: FRED labels USREC and M0892AUSM156SNBR "Copyrighted: Citation Required" as well as UMCSENT (the composite and fred_data.csv said otherwise), and M0892AUSM156SNBR is seasonally adjusted, not NSA. No bytes, filenames or `redistribution` values change, and CATALOG.md regenerates identically. validate_datasets.py (with and without --builders) and check_consumed_files.py pass. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 + lectures/bbh_macro_quarterly.csv.yml | 5 +- lectures/bbh_michigan_monthly.csv.yml | 20 +++++- lectures/fred_data.csv.yml | 17 +++-- .../hansen_jagannathan_1991_data.json.yml | 5 +- lectures/hansen_singleton_1982_data.csv.yml | 4 +- lectures/hansen_singleton_1983_data.csv.yml | 4 +- lectures/us_business_cycle_monthly.csv.yml | 62 ++++++++++++------- 8 files changed, 87 insertions(+), 32 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 16b3b56..d693bc1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -150,6 +150,8 @@ Because this repo is a **stability cache, not a content-distribution host** (see For the public data sources most snapshots come from (World Bank, FRED, Eurostat, …) the answer is a **known yes, recorded once per source** — permissive terms plus attribution. Record what the source states and move on; don't re-litigate it per snapshot. Treat the manifest's `redistribution` field as a **cheap binary gate** (`permitted` / `restricted`): a fast `permitted` for public statistics agencies, `restricted` blocking only the genuinely restricted source before it goes public — e.g. FRED re-serves third-party series that may not be redistributed, and anything under non-commercial or no-redistribution terms must not be cached here, since attribution alone does not cure those. Capture licence detail richly when the source provides it; where it is genuinely unavailable, record the gap rather than blocking the file. +**FRED's known yes is recorded with its caching clause in view** (decided 2026-09-30, [#35](https://github.com/QuantEcon/data-lectures/issues/35)). FRED's full Terms of Use say "You may not, without the Bank's prior written consent: … (p) Store, cache, or archive any portion of the FRED® Services or FRED® Content; provide any stored, cached, or archived portion … to any third party; or incorporate any FRED® Content in any database, compilation, archive, cache, or other medium." Read literally, that describes this repo. The summarized terms FRED puts in front of its users encourage educational use and list no caching prohibition, and each series page states the series' own copyright label. Neither stands apart from clause (p): the summary grants use "subject to the full terms and conditions of the service", and both of FRED's citation labels grant use only to a user who engages in no prohibited use. The repo still reads FRED as a known yes on those two grounds, names clause (p) in every FRED-derived manifest's `license.note` rather than leaving it unsaid, and leaves asking the Bank for consent to the promotion gate below, which for FRED covers new snapshots such as `us_business_cycle_monthly.csv` as well as inherited files. A third-party series is judged on its owner's own terms, not on FRED's label alone: Michigan's consumer sentiment index is `permitted` on Michigan's FAQ for data on its main website (`us_business_cycle_monthly.csv`), while tables Michigan publishes only on its data portal stay `restricted` (`bbh_michigan_monthly.csv`). + **Licensing does not gate migration** (settled 2026-08-06, [#35](https://github.com/QuantEcon/data-lectures/issues/35)). A file the lectures have **already served publicly** migrates here with its licence recorded *as found* — including `redistribution: restricted` and a null licence name where that is the honest answer — and is logged in the inventory ([#35](https://github.com/QuantEcon/data-lectures/issues/35), feeding [workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)) with a `note`. Resolve it (permission, an open replacement, or removal) before this repo is ever promoted as a branded public open-data host: **that promotion is the gate, not the file's move** — and with the custom domain deferred indefinitely (2026-08-12, D11), no promotion is scheduled, so the gate looms over nothing while the [#35](https://github.com/QuantEcon/data-lectures/issues/35) inventory stays tracked. Rehosting the same bytes with better provenance and an explicit licence field improves on the status quo, so a licence question is never a reason to stall a migration. This covers **inherited** data only. A genuinely new dataset — one with no prior life in a lecture repo — still has its licence established *before* it lands, as the P5 additions all did. diff --git a/lectures/bbh_macro_quarterly.csv.yml b/lectures/bbh_macro_quarterly.csv.yml index 4e544d5..a6c633b 100644 --- a/lectures/bbh_macro_quarterly.csv.yml +++ b/lectures/bbh_macro_quarterly.csv.yml @@ -120,7 +120,10 @@ license: 2026-08-17 against a positive control (FRED's SP500 page, which does carry S&P Dow Jones Indices' "Reproduction … is prohibited" notice), so the zero is a real zero and not a broken check. Nothing to register on - QuantEcon/data-lectures#35. + QuantEcon/data-lectures#35. These bytes came from the Zenodo deposit, not + from FRED directly. FRED's Terms of Use clause (p), on storing or caching + FRED content, is recorded once, for all FRED-derived files, in AGENTS.md, + "Licensing and attribution" (2026-09-30). retrieved: null # inherited-undated: the bytes arrived in # lecture-python-advanced.myst with no recorded diff --git a/lectures/bbh_michigan_monthly.csv.yml b/lectures/bbh_michigan_monthly.csv.yml index 23b1663..895e267 100644 --- a/lectures/bbh_michigan_monthly.csv.yml +++ b/lectures/bbh_michigan_monthly.csv.yml @@ -106,7 +106,7 @@ license: # (AGENTS.md, "Licensing and attribution"). The gate answer is `restricted` # because the layer beneath the deposit says so — see the note. redistribution: restricted - verified: 2026-08-17 + verified: 2026-09-30 note: > Split answer, and the halves disagree. The Zenodo record declares "cc-by-4.0" (Zenodo API, metadata.license.id, read 2026-08-17), and the @@ -131,9 +131,25 @@ license: have to re-derive them: what is published here is four aggregate columns for the all-households cell only — not the demographic breakdowns the deposit ships, and not the microdata — and these are the same published aggregates - Michigan puts on its own tables pages. Michigan's terms bind someone who + Michigan puts on its data portal's tables pages + (data.sca.isr.umich.edu/tables.php). Michigan's terms bind someone who obtains the data from its website; QuantEcon obtained them from a CC BY 4.0 deposit. That argument is not obviously wrong and is not obviously enough. + Re-examined 2026-09-30 alongside us_business_cycle_monthly.csv (#35). + Michigan also publishes a FAQ (data.sca.isr.umich.edu/faq.php) that + answers the reprint question with "Any data, tables, or charts available + to the public on our main website can be used without permission". It does + not reach this file. The main site's tables page + (www.sca.isr.umich.edu/tables.html) offers the sentiment index, its + components and the median inflation expectations, while this file's + px1_mean and three expectation shares come from the portal's tables 32 and + 30 only. So the main site's public tables are `permitted` and the + portal-only tables `restricted`: this file stays `restricted`, and the + composite's UMCSENT column sits on the permitted side. The UNRATE column is + FRED data, though these bytes came from the Zenodo deposit, not from FRED + directly. FRED's Terms of Use clause (p), on storing or caching FRED + content, is recorded once, for all FRED-derived files, in AGENTS.md, + "Licensing and attribution" (2026-09-30). Inherited exposure: served publicly from lecture-python-advanced.myst since 2026-06-12, and reachable today at https://python-advanced.quantecon.org/_static/lecture_specific/subjective_beliefs_business_cycles/bbh_michigan_monthly.csv diff --git a/lectures/fred_data.csv.yml b/lectures/fred_data.csv.yml index 87120df..85539f6 100644 --- a/lectures/fred_data.csv.yml +++ b/lectures/fred_data.csv.yml @@ -58,7 +58,8 @@ source: (H.15)", retrieved from FRED, Federal Reserve Bank of St. Louis (series GS1, GS5, GS10, DFII5, DFII10); Federal Reserve Bank of St. Louis, "NBER based Recession Indicators for the United States" (USREC), from the - National Bureau of Economic Research's business cycle chronology. + National Bureau of Economic Research's business cycle chronology, retrieved + from FRED. note: > The lecture's prose names exactly these six series and FRED as the source, and for once the prose survives measurement: the committed file reproduces @@ -72,7 +73,7 @@ license: name: null url: https://fred.stlouisfed.org/series/USREC redistribution: permitted - verified: 2026-08-18 + verified: 2026-09-30 note: > Five of the six series are the Board of Governors' H.15 release delivered through FRED — US Government works. The sixth, USREC, is constructed by the @@ -83,8 +84,16 @@ license: federal series page; the same check could not be re-run for this manifest on 2026-08-18 because fred.stlouisfed.org returned 403 to both the target pages and the control from the authoring environment, so `permitted` rests - on the series' federal provenance plus that dated precedent. Nothing to - register on QuantEcon/data-lectures#35. + on the series' federal provenance plus that dated precedent. That precedent + covered other series, not USREC. Checked 2026-09-30, the five H.15 series + are labelled "Public Domain: Citation Requested" and USREC "Copyrighted: + Citation Required", usable, "provided you have not engaged in any prohibited + uses", "with proper attribution of the source and acknowledgment that you + obtained the data from FRED", which `source.citation` gives, so all six + answers now stand on their labels. Nothing to + register on QuantEcon/data-lectures#35. FRED's Terms of Use clause (p), on + storing or caching FRED content, is recorded once, for all FRED-derived + files, in AGENTS.md, "Licensing and attribution" (2026-09-30). retrieved: null # inherited-undated: the bytes arrived in # lecture-python-advanced.myst with no recorded diff --git a/lectures/hansen_jagannathan_1991_data.json.yml b/lectures/hansen_jagannathan_1991_data.json.yml index 1512dda..4b30e8a 100644 --- a/lectures/hansen_jagannathan_1991_data.json.yml +++ b/lectures/hansen_jagannathan_1991_data.json.yml @@ -126,7 +126,10 @@ license: `annual.consumption` is his `C` column times a constant. An inherited exposure, served publicly from lecture-python-advanced.myst since 2026-04-10. No permission has been sought from Yale or from Professor Shiller. - Registered for licence review on QuantEcon/data-lectures#35. + Registered for licence review on QuantEcon/data-lectures#35. FRED's Terms of + Use clause (p), on storing or caching FRED content, is recorded once, for + all FRED-derived files, in AGENTS.md, "Licensing and attribution" + (2026-09-30). retrieved: null # no upstream-retrieval date was recorded, and # none is reconstructed from git history diff --git a/lectures/hansen_singleton_1982_data.csv.yml b/lectures/hansen_singleton_1982_data.csv.yml index 61a607f..8fd555d 100644 --- a/lectures/hansen_singleton_1982_data.csv.yml +++ b/lectures/hansen_singleton_1982_data.csv.yml @@ -98,7 +98,9 @@ license: values cannot be recovered without the deflator. An inherited exposure, served publicly from lecture-python.myst since 2026-06-26. No permission has been sought from either provider. Registered for licence review on - QuantEcon/data-lectures#35. + QuantEcon/data-lectures#35. FRED's Terms of Use clause (p), on storing or + caching FRED content, is recorded once, for all FRED-derived files, in + AGENTS.md, "Licensing and attribution" (2026-09-30). retrieved: null # the builder fetches live and no retrieval date # was recorded upstream; not reconstructed from diff --git a/lectures/hansen_singleton_1983_data.csv.yml b/lectures/hansen_singleton_1983_data.csv.yml index 84e99d7..d5cb0b6 100644 --- a/lectures/hansen_singleton_1983_data.csv.yml +++ b/lectures/hansen_singleton_1983_data.csv.yml @@ -104,7 +104,9 @@ license: and the reason it is written down rather than waved through. An inherited exposure, served publicly from lecture-python.myst since 2026-06-26. No permission has been sought from either provider. Registered for licence - review on QuantEcon/data-lectures#35. + review on QuantEcon/data-lectures#35. FRED's Terms of Use clause (p), on + storing or caching FRED content, is recorded once, for all FRED-derived + files, in AGENTS.md, "Licensing and attribution" (2026-09-30). retrieved: null # the builder fetches live and no retrieval date # was recorded upstream; not reconstructed from diff --git a/lectures/us_business_cycle_monthly.csv.yml b/lectures/us_business_cycle_monthly.csv.yml index 3c7aa8b..42adc58 100644 --- a/lectures/us_business_cycle_monthly.csv.yml +++ b/lectures/us_business_cycle_monthly.csv.yml @@ -41,16 +41,16 @@ source: all items less food and energy, index 1982-84=100, SA, from 1957-01 (BLS). INDPRO — industrial production: total index, 2017=100, SA, from 1919-01 (Board of Governors). M0892AUSM156SNBR — unemployment rate for - the United States, %, NSA, 1929-04 to 1942-06 (NBER Macrohistory, + the United States, %, SA, 1929-04 to 1942-06 (NBER Macrohistory, chapter 8). citation: > U.S. Bureau of Labor Statistics (UNRATE, CPILFESL); Federal Reserve Bank of St. Louis, NBER based Recession Indicators (USREC); University of - Michigan, Surveys of Consumers (UMCSENT); Board of Governors of the - Federal Reserve System, Industrial Production and Capacity Utilization - (INDPRO); National Bureau of Economic Research, Macrohistory Database - (M0892AUSM156SNBR). All retrieved from FRED, Federal Reserve Bank of St. - Louis. + Michigan, Survey Research Center, Surveys of Consumers (UMCSENT); Board of + Governors of the Federal Reserve System, Industrial Production and Capacity + Utilization (INDPRO); National Bureau of Economic Research, Macrohistory + Database (M0892AUSM156SNBR). All retrieved from FRED, Federal Reserve Bank + of St. Louis. note: > fredgraph.csv titles its date column `observation_date`; the library renames it to `DATE`, the header every other FRED file here carries. @@ -63,22 +63,40 @@ license: name: null url: https://fred.stlouisfed.org/legal/ redistribution: permitted - verified: 2026-09-01 + verified: 2026-09-30 note: > - Split provenance, one answer. FRED classifies each series on its page - (checked 2026-09-01): UNRATE is "public domain: citation requested", and - CPILFESL, INDPRO and USREC are US federal statistics or built from the - NBER's published dates (the fred_data.csv manifest's 2026-08-17 check of - federal series pages against FRED's S&P 500 page as a positive control is - the precedent). UMCSENT is the one third-party series: FRED classifies it - "copyrighted: citation required" (fred.stlouisfed.org/legal/#copyright- - citation-required) — the University of Michigan's terms permit use with - citation, which is exactly what this repo's cache-with-attribution policy - provides; the manifest's `source.citation` names the Surveys of - Consumers. M0892AUSM156SNBR is NBER Macrohistory data the NBER publishes - openly. So `permitted`, with the Michigan attribution as the condition - that makes it so; if Michigan's terms are ever read as forbidding a - cached CSV, the alternative is to drop UMCSENT from the composite. + Split provenance, one answer, re-examined 2026-09-30 (#35). FRED labels each + series on its page (checked 2026-09-30): UNRATE, CPILFESL and INDPRO "Public + Domain: Citation Requested"; USREC, M0892AUSM156SNBR and UMCSENT + "Copyrighted: Citation Required", which FRED's legal page + (fred.stlouisfed.org/legal/#copyright-citation-required) defines as usable, + "provided you have not engaged in any prohibited uses", "with proper + attribution of the source and acknowledgment that you obtained the data from + FRED". USREC is built from the NBER's published chronology, and + M0892AUSM156SNBR is NBER Macrohistory data the NBER publishes openly. + UMCSENT is the one series whose FRED notes carry its owner's copyright + notice, "Copyright, 2016, Surveys of Consumers, University of Michigan. + Reprinted with permission.", and FRED sends any use of such a series beyond + your own personal use to the owner. Michigan publishes two texts that point + different ways. Its data portal's usage agreement + (data.sca.isr.umich.edu/agreement.php) forbids redistributing "the data and + materials from the Surveys of Consumers website" without written consent. + Its FAQ (data.sca.isr.umich.edu/faq.php) answers the reprint question with + "Any data, tables, or charts available to the public on our main website can + be used without permission", and asks for the citation "University of + Michigan, Survey Research Center, Surveys of Consumers". The headline index + is main-website data: www.sca.isr.umich.edu/tables.html serves it as + tbmics.csv, and on 2026-09-30 every UMCSENT value then committed here + equalled that table. So `permitted` on the FAQ, with Michigan's citation in + `source.citation` as the condition. The line this draws, the main site's + public tables permitted and the portal-only tables restricted, is the one + bbh_michigan_monthly.csv records as well, and it is why that file stays + `restricted`. If Michigan ever reads its terms against a cached CSV, UMCSENT + leaves the composite: in place while `consumers` is empty; once it is not, a + UMCSENT-free file lands under a new filename, the consumers move to it, and + this file is then withdrawn rather than kept serving. FRED's Terms of Use + clause (p), on storing or caching FRED content, is recorded once, for all + FRED-derived files, in AGENTS.md, "Licensing and attribution" (2026-09-30). # Stamped by scripts/snapshots.py on every merged refresh. retrieved: 2026-09-01 @@ -103,7 +121,7 @@ schema: - {name: UMCSENT, dtype: float64, description: "Michigan consumer sentiment index, 1966Q1=100; empty before 1952-11, sparse (quarterly then irregular) until 1978-01, monthly since; validate() bounds it in [20, 150]"} - {name: CPILFESL, dtype: float64, description: "CPI less food and energy, index 1982-84=100, SA; empty before 1957-01 and for 2025-10; validate() bounds it in [20, 1000]"} - {name: INDPRO, dtype: float64, description: "industrial production index, 2017=100, SA; complete from 1919-01; validate() bounds it in [1, 300]"} - - {name: M0892AUSM156SNBR, dtype: float64, description: "unemployment rate, %, NSA, from the NBER Macrohistory database; populated 1929-04 to 1942-06 only; validate() bounds it in [0, 40]"} + - {name: M0892AUSM156SNBR, dtype: float64, description: "unemployment rate, %, SA, from the NBER Macrohistory database; populated 1929-04 to 1942-06 only; validate() bounds it in [0, 40]"} row_count_floor: 1291 # 1919-01 .. 2026-07 in the first vintage; # grows by one row per month. validate() # asserts the grid and recency (newest month