Repository navigation
Schema decision: is known_nulls an exact count or a ceiling? #121
Description
Activity
Decision, 2026-09-07: the integer stays exact; dynamic snapshots declare null placement, not a count
Option 4 from the body, chosen after measuring what the manifests declare and what the two dynamic builders already assert.
What is in the tree today. 12 of 41 manifests declare non-empty
known_nulls, all as plain integers; ten are static files where exact is right. Two manifests (unemployment_rate_annual.csv.yml,private_credit_to_gdp.csv.yml) use a third form,known_nulls_total: N, described in their own comments as a ceiling — and that key is declared nowhere: not inmanifest-schema.yml, not in any script. So three shapes are in the wild and one is documented.Which counts actually move on a refresh. The FRED composite's nulls split by position (first vintage, 1919-01 to 2026-07):
column total leading inner trailing UNRATE 349 348 1 0 UMCSENT 616 406 210 0 CPILFESL 457 456 1 0 M0892AUSM156SNBR 1132 123 0 1009 Only M0892's count grows, by one per month, because the series ended in 1942-06 and every new row adds a trailing null; the other three are stable unless the source backfills. The inner nulls are the 2025-10 shutdown hole and UMCSENT's sparse pre-1978 years.
business_cycle_data's{YR1960: 9}is stable by construction; the two World Bank siblings have leading nulls for GBR and FRA and one trailing null for an unpublished newest year. Neither dynamic builder asserts a count — both assert placement (before first observation, in known inner holes, after last observation), and the manifest comments say the counts are "recorded for the reader, not asserted".The rules.
- A plain integer under
known_nullsis exact:n != knownfails. Required for any nulled column in averbatimorconstructedfile. - A
dynamic-snapshotdeclares placement per column with a small closed vocabulary —leading,trailing, andinner: [...]listing the known holes (dates for a time index, column names for a wide file). The validator checks that every null sits where declared and nowhere else. A dynamic file may still write an integer for a column whose count is stable (YR1960: 9) to get the stronger check. - The default is by
class, which every manifest already carries — no new per-file judgement field. known_nulls_totalis retired: a file-level total cannot say where the nulls are. Its two users move toleadingon the affected columns andtrailingon the newest year.- No
{max: N}ceiling form. A ceiling silently accepts a column emptying out below the bound, which is the failure the check exists to catch.
Edits this implies. Placement blocks in the four dynamic manifests, the
known_nullscomment and anullsplacement entry inmanifest-schema.yml, and the template's check replaced by the shared one from #119. Closes with the PR that records the vocabulary inmanifest-schema.yml.- A plain integer under
Correction to the decision above, found while writing it into the tree:
known_nulls_totalis used by four manifests, not two. Besides the two World Bank siblings (where it was a ceiling and is now replaced by anulls:placement block), it appears nine times inside thesheets:entries ofassignat.xlsx.ymlanddette.xlsx.yml. Those are positional reads withheader: null, so the range has no column names to count against, and a file-level exact total is the only form a count can take. There it is not a ceiling and not drift — it is an exact count for a static file, which is exactly what the integer rule wants.So the rule as recorded in
manifest-schema.yml:known_nulls_total: Nis legal only inside aheader: nullsheet entry, as an exact total; it is retired for named columns and never a ceiling. The rest of the decision stands unchanged.- added a commit that references this issue
on Sep 6, 2026 - added a commit that references this issue
on Sep 7, 2026
schema.known_nullsmaps a column to a null count, and #14's template treats it as exact:n != known.get(col, 0)fails. Two dynamic snapshots now carry counts that a refresh will change:business_cycle_data.csv.yml:{YR1960: 9}— stable by construction, growth is undefined in the first year for every economy.{UNRATE: 349, UMCSENT: 616, CPILFESL: 457, M0892AUSM156SNBR: 1132}— leading nulls before each series starts, on a monthly panel whose length grows with every refresh, so at least some of these counts move as soon as the panel is extended.Frozen extracts want exact counts (a changed null is a changed file). Tracking snapshots want a shape constraint, not a number.
Options.
retrievedanddate_range.end, so the manifest diff shows them. Simple, but it turns a data invariant into a field the pipeline rewrites.n <= known. Never fails on a refresh that fills in history, but it silently accepts a column going empty below the ceiling.{UNRATE: {max: 400}}or a{col: "<=400"}string. Two forms to document and validate.validate()already implements by hand), with the count kept as documentation.Decide, then update
manifest-schema.yml'sknown_nullscomment and the template's check in step with the manifest-drivenvalidate().Sub-issue of #14.