Track and migrate policyengine-us-data construction dependencies

[anth-volk]

Context

microplex-us currently uses policyengine-us-data in two materially different ways:

1. Construction dependencies: PE-US-data code, files, or storage artifacts help build source frames, target constraints, calibration/selection weights, or other inputs that can affect the final Microplex H5.
2. Validation/comparison dependencies: PE-US-data is used after a candidate H5 exists, usually as the incumbent comparator or as the legacy PE-native scoring surface.

If the goal is to fully migrate/supersede PE-US-data with microplex-us, I recommend duplicating or re-homing the construction-related functionality inside microplex-us or a Microplex-owned data/artifact package. The validation/comparison uses should not necessarily be removed, because PE-US-data remains useful as an incumbent benchmark.

Construction-related PE-US-data dependencies to migrate

These are the current places where PE-US-data can affect the final product, not just evaluate it afterward.

Donor survey loading

microplex-us can load ACS and SCF donor frames through PE-US-data dataset classes, then converts those outputs into Microplex household/person source frames.

Relevant paths:

• src/microplex_us/manifests/pe_source_impute_blocks.json
  • policyengine_us_data.datasets.acs.acs.ACS_2022
  • policyengine_us_data.datasets.scf.scf.SCF_2022
• src/microplex_us/data_sources/donor_surveys.py
  • builds a subprocess loader for those PE-US-data dataset classes
  • parses the resulting arrays into ObservationFrame tables

Recommended migration:

• Move/port the ACS and SCF loading contracts into microplex-us source providers.
• Keep the current variable mapping manifests, but make their loaders Microplex-owned rather than PE-US-data imports.
• Add parity tests showing the Microplex-owned loaders reproduce the existing PE-US-data-backed source frames for the canonical years.

SIPP donor source files

SIPP donor blocks currently download raw files from the Hugging Face repo PolicyEngine/policyengine-us-data, e.g. pu2023_slim.csv and pu2023.csv, then parse them into donor source frames.

Relevant paths:

• src/microplex_us/manifests/pe_source_impute_blocks.json
• src/microplex_us/data_sources/donor_surveys.py
  • _download_policyengine_us_data_file(...)

Recommended migration:

• Re-home the required SIPP files in a Microplex-owned dataset/artifact location, or define a source contract that fetches them from their original public source.
• Keep a manifest with checksums, year, schema, and provenance.
• Update donor survey loaders to reference the Microplex-owned source contract.

PUF raw files and uprating inputs

When a PE-US-data checkout is supplied, the PUF provider can use repo-local PE-US-data files and tables during source construction:

• raw PUF CSVs and demographics files
• soi.csv
• uprating_factors.csv

These affect the PUF source before final export and can therefore affect the final H5.

Relevant paths:

• src/microplex_us/data_sources/puf.py
  • _resolve_policyengine_repo_local_puf_paths(...)
  • _resolve_pe_soi_path(...)
  • _resolve_pe_uprating_factors_path(...)
  • uprate_raw_puf_pe_style(...)
  • uprate_mapped_puf_with_pe_factors(...)
• src/microplex_us/pipelines/pe_us_data_rebuild.py
  • canonical rebuild provider bundle sets PE-style PUF uprating via PUF_UPRATING_MODE_PE_SOI

Recommended migration:

• Move the required PE-style SOI/uprating tables into a Microplex-owned artifact or source manifest.
• Make PUF raw artifact resolution independent of a local PE-US-data checkout.
• Preserve PE-style parity behavior as a named Microplex-owned mode, with tests against the current PE-US-data-backed results.

PUF pre-tax contribution imputation support

The optional PE-style pre-tax contribution imputation path can use PE-US-data artifacts/code for training data:

• extended_cps_2024.h5 or extended_cps_<year>.h5 under PE-US-data storage
• fallback subprocess path using policyengine_us_data.datasets.cps.CPS_2021

Relevant path:

• src/microplex_us/data_sources/puf.py
  • _load_pe_extended_cps_pre_tax_training_frame(...)
  • _default_pe_style_puf_pre_tax_contribution_model(...)
  • PEStyleSubprocessImputationPredictor

Recommended migration:

• Decide whether this PE-style imputation path is part of the canonical build.
• If yes, port the training frame and model construction into microplex-us with Microplex-owned fixtures/artifacts.
• If no, mark it explicitly as a PE-compatibility optional path and exclude it from the canonical no-PE-US-data build contract.

Calibration target DB ownership

The active PE-US target DB is often supplied from PE-US-data storage and is used to build calibration constraints, not only to evaluate final H5s. This means it affects final weights.

Relevant paths:

• src/microplex_us/policyengine/us.py
  • PolicyEngineUSDBTargetProvider
• src/microplex_us/pipelines/us.py
  • calibrate_policyengine_tables(...)
  • _load_policyengine_target_set(...)

Recommended migration:

• Treat the target DB as a first-class Microplex/PolicyEngine target artifact, not as an incidental file inside a PE-US-data checkout.
• Keep the DB schema reader if it is the canonical target schema, but document and package the target DB as an explicit input to Microplex builds.
• Add clear provenance/version metadata to saved artifacts.

Optional solver and PE-native selection paths

Some optional construction paths directly reuse PE-US-data code:

• calibration_backend="pe_l0" calls policyengine_us_data.calibration.unified_calibration.fit_l0_weights.
• policyengine_selection_backend="pe_native_loss" and PE-native optimization extract policyengine_us_data.utils.loss.build_loss_matrix(...) and can affect household selection/weights.

Relevant paths:

• src/microplex_us/pipelines/pe_l0.py
• src/microplex_us/pipelines/pe_native_optimization.py
• src/microplex_us/pipelines/us.py

Recommended migration:

• Port any solver/loss-matrix functionality intended for canonical builds into microplex-us or microplex core.
• If these remain experimental/benchmark-only paths, gate and document them as optional PE-US-data compatibility features.

Validation/comparison PE-US-data uses that should not necessarily be removed

These usages are different from construction dependencies. They are useful for benchmarking Microplex against the incumbent PE-US-data dataset and for preserving continuity with existing PE-US evaluation conventions.

Examples:

• src/microplex_us/pipelines/pe_native_scores.py
  • runs PE-US-data's native enhanced-CPS broad-loss scorer over candidate and baseline H5s
  • useful for incumbent comparison and historical continuity
• src/microplex_us/pipelines/pe_us_data_rebuild_checkpoint.py
  • can attach policyengine_native_scores.json and PE-native audit evidence after the candidate H5 is saved
• src/microplex_us/pipelines/pe_us_data_rebuild_audit.py
• src/microplex_us/pipelines/backfill_pe_native_scores.py
• src/microplex_us/pipelines/backfill_pe_native_audit.py
• src/microplex_us/pipelines/pe_focus_targets.py
• PE baseline H5s such as enhanced_cps_2024.h5
  • useful as incumbent comparator datasets

Recommendation: keep these validation/comparison paths available, but document them as benchmark/comparator dependencies rather than construction dependencies.

Suggested acceptance criteria

• The canonical microplex-us build path can run without a local PE-US-data checkout, except when the user explicitly opts into PE-US-data validation/comparison.
• Construction-time PE-US-data imports are removed from the canonical build path or replaced by Microplex-owned loaders/artifacts.
• Source manifests identify Microplex-owned provenance, checksums, and schemas for ACS, SCF, SIPP, PUF, SOI, and uprating inputs.
• Optional PE-US-data compatibility paths are clearly gated and documented.
• Docs distinguish:
  • construction dependencies that affect final outputs
  • validation/comparison dependencies that benchmark final outputs
• Tests cover the migrated construction paths against current PE-US-data-backed behavior where parity is expected.

[MaxGhenis]

Progress update: merged #35 to keep CBO projection facts reference-only in Microplex's Arch adapter. This pairs with PolicyEngine/ledger#35: CBO income-source projection facts can now live in Arch with R2-backed provenance, but Microplex will not treat them as current-year calibration targets unless we add an explicit projection-to-target adapter. The guard also extends the broad-business blocklist for CBO net business income so it cannot become plain self_employment_income through a future path.

[MaxGhenis]

Progress update: merged #36 after Arch #36. Microplex now maps ACS local Arch consumer facts into calibration targets: census_acs.person_count maps to person_count, census_acs.household_count maps to household_count, 119th Congress ACS GEOIDs (5001900US....) normalize to Microplex congressional_district_geoid, and ACS S2201 snap_receipt_status rows become snap > 0 / snap == 0 filters. This gives the local-target path source-backed congressional-district age and SNAP household controls rather than relying on ad hoc side inputs.

[MaxGhenis]

Progress update: merged Arch source-data fix PolicyEngine/ledger#47 to add BEA regional SAINC5N line 36 (contributions for government social insurance) and line 42 (residence adjustment) for all US/state rows.

Microplex impact verified from the target adapter side:

• Built package-only Arch bundle at /tmp/arch-bea-state-wage-components-20260529 during validation.
• Reran pe_native_broad 2024 coverage using existing Arch main 2023/2024 bundles plus the new BEA package bundle.
• Coverage improved from 170/220 to 171/220 cells.
• employment_income_before_lsr improved from 1/2 covered cells to 2/2; the state cell is now BEA-backed.
• Durable coverage output: /Users/maxghenis/CosilicoAI/microplex-us/artifacts/arch_bea_state_wage_components_coverage_20260529/pe_native_broad_2024_coverage.json.

No Microplex code patch was needed for this one; the existing Arch adapter already knew how to derive state employment income before LSR once Arch exposed all four BEA regional components.

[MaxGhenis]

Follow-up merged in Microplex #51 after Arch #47: state employment_income_before_lsr is now included in pe_native_broad_source_backed.

Validation:

• target-profile tests pass with 171 source-backed cells
• saved post-Arch coverage artifact confirms 171/171 current source-backed cells are covered
• no adapter changes were required; this only removed the stale pending-source exclusion

PR: #51

[MaxGhenis]

Microplex adapter status: PR #52 is merged (da2cf84). Microplex now maps Arch SSA SSI recipient-count and payment-amount consumer facts, including SSI aged/blind/disabled category constraints and state geography, into canonical calibration targets. Next blocker is adding the exact SSA SSI detail facts to Arch so this path has real source-package records.

[MaxGhenis]

Arch SSI update: PolicyEngine/ledger#48 is merged. It adds SSA Supplement 2025 Table 7.B1 as 416 source-package consumer facts for 2024 SSI recipient counts and payments across All areas, states/DC, and total/aged/blind/disabled categories. Microplex #52 smoke-loaded those facts into 416 SSI calibration targets; e.g. CA disabled recipient count maps to is_ssi_disabled == 1, state_fips == 06, ssi > 0, and U.S. total payments maps to person-level ssi.

[MaxGhenis]

Microplex SSI update: #53 is merged. With Arch SSA SSI Table 7.B1 facts available through PolicyEngine/ledger#48 and mapped by #52, the PE-native broad source-backed profile now treats the SSI recipient/payment cells as source-backed instead of pending.

Profile impact: source-backed broad cells increased from 171 to 183, and documented exclusions fell from 49 to 37. The added cells include state SSI recipient detail such as person_count | state | ssi,is_ssi_disabled, state ssi payment totals, and national aged/blind/disabled SSI payment totals.

Validation on #53:

• uv run --python 3.13 --extra dev ruff check src/microplex_us/policyengine/target_profiles.py tests/policyengine/test_target_profiles.py
• uv run --python 3.13 --extra dev pytest -q tests/policyengine/test_target_profiles.py
• CI: implementation-tests and site-snapshot passed; artifact-gates skipped as expected.

[MaxGhenis]

Post-#53 coverage verification is clean against current Arch main bundles. I rebuilt Arch bundles from PolicyEngine/arch-data@5e851c5 for 2023 and 2024, then refreshed Microplex coverage on origin/main@52d9397:

• pe_native_broad_source_backed: 183 / 183 covered, 0 uncovered, coverage rate 100%.
• pe_native_broad: 183 / 220 covered, 37 uncovered; the remaining uncovered cells are the documented exclusions.
• Snapshot artifacts: /Users/maxghenis/CosilicoAI/microplex-us/artifacts/arch_current_main_after_ssi_coverage_20260529/pe_native_broad_source_backed_2024_coverage.json and pe_native_broad_2024_coverage.json.

This means the SSI source-backed promotion in #53 is backed by actual Arch consumer facts, not only by target-profile bookkeeping.

[MaxGhenis]

Update: Microplex PR #62 is merged. The source-backed Arch coverage evidence is now wired into the MP artifact gates and packaged gate inputs.

The gate requires the pe_native_broad_source_backed coverage payload for the same period to report complete coverage (covered_cell_count == target_cell_count, uncovered_cell_count == 0, coverage_rate >= 1.0). This turns the current 183/183 source-backed coverage check into a release requirement for MP replacement candidates.

[anth-volk]

Follow-up from the PE-SOI rebuild work in #155: the current PUF_UPRATING_MODE_PE_SOI path still relies on the PolicyEngine/policyengine-us-data repo artifact for the historical PE-style SOI target table, not the policyengine-us-data Python package.

Specifically, #155 now resolves policyengine_us_data/storage/calibration_targets/soi_targets.csv from a pinned policyengine-us-data commit into a versioned Microplex cache when --soi-path is not supplied. This removes the need for a local PE-US-data checkout to provide policyengine_us_data/storage/soi.csv, but it is still a construction-time dependency on PE-US-data-owned SOI data.

Longer term, this should migrate to Arch: Arch should own the source-backed IRS SOI facts/provenance across the needed historical years, and microplex-us should own the adapter/export that turns those facts into the PE-style SOI target surface needed for PUF uprating.