Run the backfill simulation on Modal; the workflow becomes trigger/commit only - #112
Open
DTrim99 wants to merge 1 commit into
Open
Run the backfill simulation on Modal; the workflow becomes trigger/commit only#112DTrim99 wants to merge 1 commit into
DTrim99 wants to merge 1 commit into
Conversation
…mmit only Post-PolicyEngine#107 runs still die on ubuntu-latest: even a single 16-spec levels chunk gets the runner shutdown signal ~5.5 minutes in, before its first simulation completes - one Microsimulation plus the populace/torch import stack no longer fits 7GB, so no chunk size saves the hosted runner. New tools/reform_validation/modal_backfill_app.py runs the IDENTICAL producer path (backfill.py mounted verbatim) in a 64GB Modal container: clones populace at a requested ref, installs the release-exact policyengine-us/-core from the release manifest at runtime, drives the batches with a Volume-backed workdir (partials survive interruptions and resume), and publishes reform_validation_<release_id>.json to the cd-reform-validation Volume. The workflow keeps its schedule and override-exists guard but never simulates: a tick either spawns the Modal run (fire-and-forget, with a started-marker double-spawn guard) or harvests a finished artifact into the usual auto-merging override PR. Ticks take seconds; timeout drops 330 -> 30 minutes. Requires MODAL_TOKEN_ID / MODAL_TOKEN_SECRET repository secrets (fails with an explicit error until they are set). The Volume is pre-seeded with the Build O artifact, so the first live tick can harvest it directly if PolicyEngine#109 has not merged by then. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@DTrim99 is attempting to deploy a commit to the PolicyEngine Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the reform-validation backfill for good. #107's chunking was necessary but not sufficient: every scheduled run through 7/27 still dies — the logs show even a single 16-spec levels chunk receiving the runner shutdown signal ~5.5 minutes in, before its first simulation completes. One
Microsimulationplus the populace/torch import stack no longer fits ubuntu-latest's 7GB, so no chunk size saves the hosted runner.Design
The simulation moves to Modal; GitHub Actions keeps the schedule, the override-exists guard, and the commit/auto-merge PR role. Every tick now finishes in seconds:
modal deploy+ spawn the run (fire-and-forget, pinned to populace main's current sha for provenance) and exit. Astartedmarker in the Modal workdir prevents double-spawns.gen-reform-overrides.mjs, lint, auto-merging override PR (unchanged from today's flow).tools/reform_validation/modal_backfill_app.pyruns the identical producer path —backfill.pymounted verbatim into a 64GB container — cloning populace at the requested ref and installing the release-exactpolicyengine-us/-corefrom the release manifest at runtime (engine pins vary per release, so they don't belong in the image). The workdir lives on thecd-reform-validationVolume, so batch partials survive any interruption and a re-spawn resumes rather than restarts.This pattern has now produced three correct artifacts (Build O twice — including the post-populace#513 236-row payload on #109 — with the chunked-levels path exercised end-to-end).
What admins need to do
Add repository secrets
MODAL_TOKEN_ID/MODAL_TOKEN_SECRET(PolicyEngine Modal workspace). Until they exist the workflow fails with an explicit error naming them. Compute cost is ~$1–2 per release; ticks with nothing to do never touch Modal compute.Deployed + pre-seeded
The app is already deployed (
cd-reform-validation-backfill) and the Volume is pre-seeded with the Build O artifact — so the first live tick after secrets land will harvest Build O directly if #109 hasn't merged by then (making #109 closable either way).🤖 Generated with Claude Code