Repository navigation
Schema decision: semantics of the column pattern entry for wide files #120
Description
A schema.columns entry carries either name (one exact column) or pattern (a regex matching a run of columns). pattern was proposed in #14 and first used on 2026-09-01; four manifests use it today (business_cycle_data.csv.yml, unemployment_rate_annual.csv.yml, private_credit_to_gdp.csv.yml, chapter_3.xlsx.yml), all for wide files whose column set grows. manifest-schema.yml documents the field but not its semantics, and no validator reads it yet.
To decide before the manifest-driven validate() enforces it:
- Regex dialect and anchoring: full-match or search; Python
reas the reference. - Ordering and contiguity: must the matched run be contiguous, and in the position it appears in the
columnslist, or is it a set membership test? business_cycle additionally asserts an unbroken year grid, which is stronger than any regex says. - Minimum run length: does a pattern that matches zero columns pass (schema silent) or fail (file lost its wide part)?
- Exhaustiveness: must every column in the file be claimed by exactly one
nameorpatternentry, so an unexpected column fails validation? (business_cycle's own check says yes.) - Whether
date_rangefor a wide file is derived from the pattern's matched columns, asscripts/snapshots.pyalready does when stampingdate_range.end.
Record the outcome in manifest-schema.yml's comment on pattern, and touch the four manifests only if the decision changes what they mean.
Sub-issue of #14.
Activity
mmcky commentedon Sep 6, 2026
Decision, 2026-09-07: ordered, contiguous, exhaustive, at least one match, and a capture group carries the date
Settled the five questions against what the repo does today rather than in the abstract. Three CSV manifests carry the same entry (
YR\d{4}, float64) for the three World Bank tables, each with 68 header columns on an unbroken grid from YR1960 to YR2025;chapter_3.xlsx.yml'ssheets_patternis a sibling key for sheet names and is not covered here.builders/business_cycle.py's hand-writtenvalidate()is already stricter than any regex: it requires the year columns to be every column afterCountry, to start at 1960, to have no gap, and to be float, and it derivesdate_range.endfrom the last year column, whichscripts/snapshots.py stampthen writes into the manifest. So the de facto semantics were already ordered, contiguous, exhaustive and date-bearing; this records them so the sharedvalidate()(#119) can enforce them.re,fullmatch—YR\d{4}cannot matchYR19601columnsin order; apatternentry consumes the maximal run of consecutive matching columns, in the position it appears. Column order is part of the contract because positional reads exist in the lecturesmin_countfield until a file needs a floor above onenameorpatternentry; an unclaimed column failsdate_rangefor wide filesYR(\d{4}). The grid check (no gaps) stays builder-specific, layered on topWhat the validator reads. The manifest describes the CSV header as written, so the validator reads the raw file with pandas defaults, honouring the manifest's existing
delimiterandread_askeys, and noindex_col— even thoughbusiness_cycle.pyreads withindex_col=0and seesCountryfirst. The manifest stays a description of the bytes on disk; the read-options half of this is recorded on #122.Edits this implies.
YR\d{4}→YR(\d{4})inbusiness_cycle_data.csv.yml,unemployment_rate_annual.csv.ymlandprivate_credit_to_gdp.csv.yml(a two-character change that alters nothing about what they mean), and thepatterncomment inmanifest-schema.ymlrewritten to say the above. Both ride with #119 or a small precursor PR; this issue closes whenmanifest-schema.ymlrecords it.Record the four schema and naming decisions: rename, pattern capture …
Manifest-driven validate(): the schema block is the spec, shared by t…