HistoryPubnetParallelCatchupV2 Mission Rewrite - #423
Open
Jonathan-Eid wants to merge 115 commits into
Open
Conversation
Replaces the redis queue + long-lived worker pods with one Kubernetes Job per ledger range, driven by job_monitor, with a log_collector sidecar that streams each worker's log to a durable volume while the pod is still alive. Checkpoint commit on a work-in-progress branch, not a finished change. Monitor - Job per range; the monitor owns retries (backoffLimit 0) so disruption, OOM, disk eviction and timeout are classified separately and get their own attempt budgets. - Finished Jobs are deleted once their record is durable; TTL is only a backstop. 459 dead Jobs accumulated in 28 minutes at 2048 parallelism and every one inflated the reconcile LIST. - PVCs are released as each range completes; 2032 bound PVCs and 79 TiB had accumulated a third of the way through a 3982-range run. Profiling - Peaks come from kubelet /stats/summary, not Prometheus: same payload already fetched for ephemeral storage, ~10s vs a 30s scrape, and no dependency on Prometheus being up or still retaining the window. - Size memory from anon (rssBytes), never working set: page cache grows to fill whatever limit it is given, so memory.peak is always ~= the limit. Measured 862 MiB of anon reporting a 12704 MiB peak under a 24000 MiB limit. - Peaks are maxed across a contiguous chain of resumed attempts. A pod killed after replay starts resumes at LCL+1 and skips the download and bucket apply, where peak memory actually happens, so profiling only the winning attempt under-reports. A fresh retry ran new-db and supersedes everything before it. - cpu is no longer profiled; the request is fixed. Fixes - tx_apply regex missed scientific notation, silently dropping the metric for 91-99% of ranges above ledger 35M. - COLLECTOR_MAX_STREAMS is derived from worker.replicas. It caps the aiohttp connection pool with no semaphore above it, so a fixed 1200 against 2048 workers starved the excess indefinitely rather than queueing, and retries -- created last -- never got a slot. - A pod deleted while Running never became terminal, so its stream retried every 30s for the life of the run. Vanished pods are now reaped. - Two exit paths returned without finalizing, dropping tx_apply and peaks for any pod that outlived its object or whose last read threw. Known gaps - txApply and seconds are still tail-only for a resumed range; both need summing across the chain, and per-attempt duration is not yet persisted. - EPH_EVICT_JOB_CONDITION and EPH_EVICT_MESSAGE in the tests are reconstructions, not verbatim captures. Re-pin from a real eviction. 117 unit tests, each mutation-checked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Peaks already aggregated across a resumed chain; the two timings did not, so a range interrupted mid-replay reported only its final leg. - txApply: medida's total is per-process, so a pod resuming at LCL+1 reports only the transactions it replayed. Summed across the chain. - seconds: a failed attempt's duration was never persisted anywhere -- reconcile computes it solely on the success path. record_outcome now stores attemptSeconds while it still holds the pod, and the range total sums the chain. - _resumed_chain is now shared by all three aggregations. txApply slightly over-counts: replay restarts at the checkpoint boundary containing LCL, so up to 64 ledgers can be applied twice. Against a 16320-ledger range that is <=0.4%, but it is a fixed ledger cost rather than a percentage and grows as ranges shrink. seconds is compute, not elapsed -- gaps between attempts (scheduling, image pull, node startup) are not in it. wallSeconds still covers those. 123 unit tests, each mutation-checked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_collector_fn and its header went stale when the collector stopped querying Prometheus: nothing calls it, and the comment described a sampling trade-off that no longer exists. Adds two tests for behaviour that was only implied. An unclassifiable Job failure (BackoffLimitExceeded carries no rule index and no exit code, so classify returns nothing) must land in ENVIRONMENTAL_OUTCOMES and get the disruption budget, alongside admission rejections -- both mean the cluster did this, not the ledger range. A genuine non-zero catchup exit stays the only outcome with no retry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Audit of every test against the current code.
Dead code removed
- _sized_cpu was defined but never called once cpu stopped being profiled,
and PROFILE_CPU_MARGIN only fed it. Chart value and env removed with it.
- overrides.pop('cpu') could no longer hit: _profile_overrides stopped
emitting cpu.
- peakCpuCores stays in _PROFILE_ONLY_FIELDS deliberately. That is a strip
list, not a produce list -- a progress record resumed from an older run
still carries the field, and letting it through is what pushes the
ConfigMap mirror toward the 1 MiB cap. A test caught this when it was
removed.
Coverage gaps
A sweep of 14 semantically meaningful mutations found 8 that broke no test
at all. Every one guarded a decision the design depends on:
backoffLimit 0 the monitor, not the Job controller, owns retries
restartPolicy Never in-place restart would loop at the limit that
OOMed and never advance the attempt counter
ttlSecondsAfterFinished backstop for Jobs reconcile did not reach
MEM_BUMP_FACTOR 1.0 would retry an OOM at the identical limit
attempt budgets ordering encodes whose fault a failure was
atomic writes a half-written .outcome downgrades a classified
failure to "unknown"
sinceTime resume without it every reconnect re-reads the whole log
overlap dedup the resume overlap is deliberate and must be
removed per line
All 8 now fail under mutation. 136 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found by the live 2096-worker spot run. record_outcome writes attemptSeconds from the pod's terminated timestamps, but only when the monitor still has the pod -- and a spot eviction reaps the node first. Measured on ssc-test: 212 of 212 disruptions were classified from the Job condition with no pod, so no .outcome and no duration. Peaks were unaffected (the collector writes .metrics regardless), but the resumed-chain time total silently dropped every evicted leg -- the exact path the chain sum was added for. The collector watched the container run, so it is the only observer left. It now stamps the stream lifetime into .metrics, and seconds_for_range prefers .outcome and falls back to it. The collector figure is an approximation: the stream opens up to COLLECTOR_POLL_SECONDS after the container does. 139 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found live on the 2096-worker spot run. The AWS CLI draws its transfer meter with carriage returns and no newline, so a 628 MiB bucket download reaches the log endpoint as one multi-megabyte "line". The collector read the stream line-wise and aiohttp raises above 512 KiB, so every large download killed its own stream; the reconnect resumed from sinceTime and hit the same wall. The resulting spin consumed the collector, and no retry pod ever got a stream -- 289 a2 pods, zero a2 archives or metrics, which is exactly the resumed-attempt data this run existed to capture. Two fixes: - aws s3 cp gains --no-progress. The meter is noise in an archive and was the bulk of every large range's log. - The collector reads 64 KiB chunks and splits on \r as well as \n, with MAX_LINE_CHARS bounding any single unterminated blob. A stream must not be destroyable by whatever a worker happens to print. 143 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on the live 2096-worker run: the collector sits at 1444 MiB of a 2048 MiB limit with memory.events max=2617 and 1.00 of 2 cpu, holding 1797 established connections. follow=true charges a gzip deflate buffer and aiohttp read buffers per stream for the pod's whole life, so that cost scales with parallelism. Linear extrapolation to 4096 streams is ~2.7 GiB and ~2.0 cpu -- it runs out of both at once. - MAX_LINE_CHARS 256 KiB -> 64 KiB. It is charged per live stream, so the old value was another 512 MiB at 2096 and 1 GiB at 4096, enough to OOM the sidecar by itself. stellar-core lines are well under a kilobyte. - The retry-path job deletion is now gated on the collector having finalized that attempt, matching the success path. delete_job reaps the pod and backstop_save_pod_log stands down for any range the collector claimed, so nothing else would ever read that log. Benign under follow=true, required before any move to polling. 145 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on the live 2096-worker run: the collector sat at 1444 MiB of a 2048 MiB limit with memory.events max=2617, 1.00 of 2 cpu, and 1797 held connections. follow=true charges a connection, a gzip deflate buffer and aiohttp read buffers per pod for the pod's whole life, so the cost scales with parallelism -- 4096 streams extrapolates past both limits at once. Polling makes concurrency a tuning parameter instead of a function of pod count. A single poll measured ~0.22s, so 2096 pods on a 10s interval need ~46 in-flight polls against 2096 permanently-held connections. The starvation class of bug goes with it: no pool size can be "too small" when slots are time-shared rather than held for hours. - _poll_once does one short read under a semaphore and opens the archive per poll, so nothing is retained between polls. - poll_pod samples terminal BEFORE each poll, so a pod that exits mid-poll still has its final output read; checking after would race it. - A terminal pod whose polls keep failing finalizes after TERMINAL_POLL_ATTEMPTS rather than spinning on a dead pod forever. follow=true finalized there because it already held the bytes; the suite caught this when polling did not. - COLLECTOR_MAX_STREAMS is gone: nothing holds a stream, and deriving it from worker.replicas was what starved retries twice. Not yet exercised against a cluster. 152 tests, each new behaviour mutation-checked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An allowlist of Running/Succeeded/Failed, not "skip Pending". Measured after the polling switch: 60 of 88 poll failures were 400 "container is waiting to start". Unknown is excluded for the same reason -- the node has stopped reporting, so the poll cannot succeed. Terminal phases stay in: that is where a pod's final output lives. 153 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 2096-worker spot run delivered a profile with txApply on 356 of 356 completed ranges and peaks on 0, while 1936 .metrics files on the same volume carried peakAnonBytes. Two causes, both on the success path: - The reap was gated on tx_apply, which is the wrong signal. tx_apply_for_range falls back to the archive and then the pod, so it is available almost immediately; peaks_for_range has no fallback at all. Gating on the always-available field let delete_job reap the pod before the collector had finalized, and .metrics is the only place peaks live. _reap_if_complete now requires both. - The peak read was one-shot, inside `if end not in completed`. A range recorded before the collector finalized never got a second chance, and the reap removed the Job so reconcile never revisited it. Later passes now backfill peaks while the Job is still present. JOB_TTL_SECONDS remains the backstop, so a range whose collector never finalizes costs a late Job rather than a stuck one. 155 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces inferring "the collector has finished" from peaks being present. That inference was wrong in both directions: tx_apply falls back to the archive so it lands long before the collector finishes, and an attempt can legitimately finalize with no peaks at all, which left its Job waiting out the TTL for no reason. finalize now writes range-<end>-a<n>.done on the shared volume, last, after .metrics -- written any earlier it would authorise exactly the reap it exists to prevent. The monitor stats it and only then deletes the Job, which is what reaps the pod, the one place peaks can still be read from. Atomic (.tmp + os.replace) and best-effort: a failed marker costs a Job that waits out JOB_TTL_SECONDS, never correctness. 158 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
poll_pod slept LOG_POLL_SECONDS between polls and only noticed a terminal pod on its next tick, so up to 10s separated the container exiting from the final read. That window belongs to whatever deletes the pod next -- a spot reclaim, a node reap -- and takes the last lines with it. The main loop now sets a per-pod Event when it first observes the pod terminal, and when the pod leaves the pod list entirely. Gone is terminal too: without that its poller sleeps out the interval before taking the 404, delaying finalize and the .done the monitor waits for. Polling faster instead would not help: sinceTime has second granularity, so anything under ~1s re-reads the same second. One behaviour change worth naming: an Event stays set, so after a pod is terminal the backoff no longer applies and a failing poll retries at once. Bounded by TERMINAL_POLL_ATTEMPTS, and on an evicted pod retrying fast is the point. Also makes the retry-path reap wait for the same .done marker as the success path. Peaks were a proxy: an attempt with none would never be reaped, and one whose peaks landed early could be reaped mid-read. 162 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fresh-start rule dropped it, which broke the self-correcting loop the profile depends on. A range that OOMs at L really did allocate L and want more, so L is a lower bound on demand and the next run must size above it: L * PROFILE_MARGIN + PROFILE_CACHE_HEADROOM clears it. Discard that and the range is sized from whichever attempt happened to survive, and OOMs again. It only bit some of the time, which is worse. Measured on ssc-test 2026-07-30 with tip-first ordering: an OOM during replay resumes, so the attempt stays in the chain (224 of 252), but an OOM during download does not (25 of 252). A run at higher cpu is download-bound -- the on-demand run at 750m had every OOM in download -- so the loop would go quiet exactly where it is needed most. Peaks only. tx_apply and seconds are summed across the chain, and a fresh start redoes the work the dropped attempt already did, so counting it there would double-count. 165 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The collector reads LABEL_ATTEMPT off the pod to decide which range-<end>-a<n>.* files an attempt owns, and defaults to "1". The label was set only on the Job, so every attempt claimed attempt 1's files. Measured on the live 2096-worker run: 2246 metrics files, all a1, while 475 a2 pods were running. Each retry merged its peak over the first attempt's instead of being maxed against it, so a range that OOMed at a1 and succeeded at a2 recorded only a2's smaller peak -- destroying exactly the ceiling-hit evidence the chain aggregation was built to preserve. peaks_for_range(end, 2) also found nothing, and those Jobs were never reaped because .done was written under the wrong name. 167 tests. Nothing in the suite asserted the pod carried the label. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both found on the live 2096-worker spot run.
The cpu limit was only removed inside `if overrides:`, which silently
excluded two populations, because _profile_overrides returns {} for an
unmeasured range AND for an escalated attempt. Measured: 214 a1 and 256 a2
pods capped at cpu 2 while their peers ran uncapped. For the retries that
meant more memory and less cpu at the same time, immediately after an OOM
-- and less cpu means less download concurrency means a lower peak, so the
retry records a figure an unthrottled run cannot reproduce. The cap is now
gone for every worker unless PROFILE_CPU_LIMIT is set explicitly; packing
is driven by the request, which is unchanged.
Memory escalation keyed on the attempt index, so evictions climbed the
ladder. On spot they dominate -- 288 disruption retries against 7 OOM
retries -- and a range disrupted three times then OOMing once jumped to
base * 1.5^4, a 5x request for a single OOM. Escalation now counts OOM
outcomes among prior attempts.
169 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Root cause of the 2096-worker run aborting at 61% complete. Range 16752063: attempt 1 replayed to the target ledger and was evicted before it could exit 0, so the Job never reported success. Attempt 2 read LCL == TARGET, and the resume guard accepted it (`-le "$TARGET"`), so it skipped new-db and ran catchup against a database with nothing left to apply. stellar-core applied 0 transactions and exited 2 -- identically on every retry. The range burned its whole budget, the monitor reported one failed job, and the mission aborts the run on any failure. The work had actually been done. We discarded ~1500 ranges of profile data because we could not recognise a completed range. The script now exits 0 when LCL >= TARGET, and the resume branch is narrowed to a strict `-lt`. Also adds an executable test of all three resume decisions -- already-complete, partial, never-started -- driving the real script against a stubbed stellar-core. Separately: RESUME_SCRIPT is %-formatted at dispatch, so a bare % anywhere in it, including a comment, raises and kills every job dispatch. A comment reading "61%-complete" nearly shipped exactly that; there is now a test that formats the script and rejects stray percent signs. 173 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The probe used `grep -A8 '"ledger":'`, but offline-info puts ~40 lines of bucketlist hashes between that key and "num", so the window never reached it. LCL came back empty every time and the probe fell through to the log grep it was meant to replace -- with no error, so the change read as working while doing nothing. Verified against 27.1.1 on ssc-test 2026-07-30: exactly one "num" key in the document and it is the ledger's. A plain sed over the whole output reads it. Confirmed end to end with the exact line that ships: empty on a volume with no DB, "1" after new-db. Adds a test rejecting any line-windowed grep in the probe. 174 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`started` measures how long this poller has been watching, not how long the container ran. A pod already terminal when poll_pod first executes -- it finished while the collector was down, or between pod-list polls -- finalizes on the first pass and records ~0 seconds beside a real memory peak. Measured on the recovered artifacts from the 2096-worker run, across two collector restarts: of 3237 metrics files, 140 recorded under 1s and 150 recorded under 5s alongside an anon peak above 500 MiB, which no real attempt produces. Now reports nothing in that case. The monitor's own figure, taken from the pod's terminated timestamps, is authoritative and seconds_for_range already prefers it; the collector value exists only as a fallback for earlier legs of a resumed chain, where a fabricated near-zero silently shrinks the sum. The profile's `seconds` was not affected -- the winning attempt always used the pod timestamps (verified: min 128s across 2437 completed ranges). 175 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
write_metrics merged with `{**on_disk, **new}`, so the newer value won every
collision. That is wrong for a monotonic quantity: after a collector restart
the fresh poller's high-water starts at zero, and its first in-flight flush
replaced the higher pre-restart peak with a lower one. The flush exists to
survive a restart, and the merge then undid it.
Lowering a peak undersizes the range on the next run, which is the single
direction that costs an OOM.
Peaks now take the max on merge; every other field still takes the newest
value. Found while asking why 1141 of 4213 recovered metrics files carried
an anon peak but no working set -- only anon is flushed mid-flight, so it is
the only one exposed to this.
176 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
attemptSeconds came from `asyncio.get_event_loop().time()` sampled when poll_pod started, which measures how long THIS POLLER existed, not how long the container ran. For a pod already terminal on the first pass -- finished while the collector was down -- that is ~0. Measured across two collector restarts: 150 of 3237 metrics files recorded under 5s beside an anon peak above 500 MiB. The previous commit suppressed the bad value. That was the lazy fix: a terminal pod still carries startTime and terminated.finishedAt until it is deleted, which is the same source the monitor uses, and the collector's main loop already has the whole pod object and was discarding everything but `phase`. It now records the pod's own duration while the pod still exists, and finalize prefers that. The poller's elapsed time remains only as a fallback for a pod that vanished before its timestamps could be read. 178 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MAX_ATTEMPTS is effectively the OOM budget: `failed` is the only other outcome that reaches it, and that one sets no retry reason. At 5 it condemned a range before the escalation ladder had run its course, and a condemned range aborts the entire run -- which is what ended a 61%-complete 2096-worker run tonight. With escalation now counting OOMs rather than attempts, rung N means the range genuinely wanted more N times, so spending rungs is evidence rather than churn. MEM_ESCALATION_CAP (48Gi) is the real ceiling: a 1.6GiB base reaches it on OOM 10, a 4.2GiB base on OOM 8. So the practical effect is that a hungry range escalates until the cap instead of dying at 5. Adds a test tying the budget to the ladder: the two must stay consistent, or the budget silently becomes the binding limit again. 179 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reverts the raise to 10. 5 rungs is 1.5^4 = 5x the profile figure; a range needing more than that is broken rather than mis-sized, and chasing it to MEM_ESCALATION_CAP parks a whole r8a.2xlarge on one range for hours. The price is that such a range is condemned, and today a condemned range aborts the entire run. That coupling -- not this number -- is the thing to fix. Raising the budget only pushes the same failure further out. Test now pins the budget to a sane band and asserts the ladder can at least treble the request before giving up, rather than requiring it to reach the cap. 179 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…broken"
Found by a sandbox edge-case suite run against the monitor while the main
run was in flight. Three of eight scenarios failed on one root cause.
stellar-core catches SIGTERM, drains, and exits 3 in ~7s. A corrupt archive
also exits 3. Nothing in the exit code separates them -- only a
DisruptionTarget condition does, and that is gone the moment the pod is.
classify() mapped exit 3 to outcome `failed`, which is the ZERO-retry
outcome, so every graceful kill condemned its range, and one condemned
range aborts the mission.
Measured in sandbox: a pod deleted mid-replay, a pod deleted mid-download,
and an attempt-deadline kill were all condemned at attempt 1. The resume
logic verified earlier tonight was unreachable through every one of them.
Four fixes:
- exit 3 now retries on the ordinary range budget. A genuinely corrupt
range still exhausts MAX_ATTEMPTS and fails with evidence; an interrupted
one succeeds, usually by resuming at LCL+1. It deliberately does NOT join
ENVIRONMENTAL_OUTCOMES, which would give it 20 attempts.
- A DeadlineExceeded Job condition now outranks the pod-derived verdict.
The deadline kills with SIGTERM, so the pod says exit 3 / `failed`, and
whichever of the two won the race decided retry vs condemn.
- Condemnation is logged loudly. The zero-retry path emitted nothing at
all: the range appeared under failed{} and the mission aborted with no
line explaining why.
- txApply is backfilled like the peaks. progress.json carried txApply=null
while range-4000-a1.metrics durably held txApplySeconds -- the same
one-shot read race the peaks had, with the same fix.
184 tests, each fix mutation-checked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…fact rangeProfileFields predates peakAnonBytes, so the mission's writeRangeProfile silently stripped the one field the sizing consumer prefers. Measured on the 2026-07-30 full run: the artifact carried 0% peakAnonBytes and 0% wallSeconds while the monitor's own progress.json carried both at 99-100%. The real profile had to be recovered from the worker-logs tar, twice in one night. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The status loop threw at the first failed range, which abandons every range still in flight. The ranges alive at that moment are the expensive tip ones this mission exists to measure. Measured 2026-07-30: one condemned range at 97% completion discarded 123 ranges of completed and in-flight work and left a hole in the profile. Failures are now recorded once (deduped -- the same range reappears on every poll), logged loudly as they happen, and reported after the run drains. The mission still fails; it just finishes the work it can first. dumpLogs is guarded because after a full drain the failed pod is almost certainly reaped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dispatch was gated on `not state['halted'] and not failed`, so the first condemned range stopped the monitor sending any further work. Combined with the driver now draining before it reports, that deadlocks the run outright: the mission waits for `remaining == 0 and in_progress == []`, and a frozen dispatch pins `remaining` at however many ranges were never sent, forever. Gate on `halted` alone. `halted` still means what it did -- the durable record went backwards, so nothing can be trusted. A condemned range is an ordinary failure: the mission fails, but only after the work already paid for finishes. Found by an adversarial audit of the reconcile loop, not by a test run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 185-test suite could not have caught any of these. job_monitor imported
a FileHandler under /data and called load_incluster_config() at module scope,
so the module was unimportable off-cluster and every test operated on source
text -- 151 of 185 regex-extract a function body and exec it, or assert a
substring exists. Nothing drove reconcile(). An adversarial audit found eight
races in the loop those tests never entered.
Import is now side-effect-free (log dir falls back to a temp dir; the
in-cluster load is gated on KUBERNETES_SERVICE_HOST, which is the variable
that call keys on anyway, so in a pod it is unconditional as before).
fake_k8s.py is an in-memory cluster returning real client models and real
ApiExceptions with 404/409 semantics; conftest.py drives real reconcile()
passes against it. 45 new behavioral tests assert on observed state -- objects
created and deleted, progress.json contents, reconcile's return -- never on
source text. Each was observed red against the bug and green against the fix.
Races fixed:
1 a completed range was re-dispatched; the reap is now range-scoped and the
failed branch consults `completed`
2 a backfilled txApply never reached the histogram; `replayed` keys on
(end, field) so a late field is still counted once
3 a torn gzip member raised EOFError and aborted the whole reconcile pass;
the member is built in a local buffer and appended in one write, and the
reader catches EOFError/zlib.error so a bad file costs one range
4 a sticky _wake Event made the terminal-poll backoff dead code and spent
the retry budget in milliseconds
5 retry budgets were spent from one shared attempt counter, so spot churn
drained the OOM budget and the first real OOM condemned the range
6 activeDeadlineSeconds sat on the JobSpec and charged Pending time; a
Job-level timeout also masked the pod's own oom/disrupted verdict
7 regression test for the dispatch freeze fixed in b09e5f6
8 the ConfigMap fallback produced a measurement-free profile that passed the
entry.Count > 0 guard, because count was attached before the guard
Two pre-existing tests went red as collateral -- both pinned literals the
fixes replaced while the invariant they named still held. Rewritten to assert
the invariant instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds 82 tests that restart the monitor at every point a real process death can
land -- a seeded fuzz across 30 seeds x 24 passes (soaked to 300x40), a crash
injected at each side-effect boundary in reconcile(), hostile durable state,
and independent collector restarts. All assert on observed state: the durable
record, the objects in the fake cluster, and reconcile's own return.
Six defects, each observed failing before the fix and passing after:
- a crash between save_progress() and the end of the success arm left the
range recorded but its PVC never released and its Job never reaped, and
permanently: the first-sight branch cannot run again and the backfill
branch is skipped once the record is whole. Both calls are hoisted to a
shared per-sighting tail; both were already idempotent.
- a 409 on the dispatch create did not spend a slot, so losing the create
race ran PARALLELISM+1 workers and reported the running range as remaining.
- `remaining` was a subtraction over three independently maintained lengths,
so a progress record carrying ends from a run with a different
ledgersPerJob drove it negative -- and to 0 on the first pass, which reads
as a finished run that never dispatched anything. It is now a count over
this run's own ranges.
- a progress.json that parsed but had the wrong shape crashed reconcile()
after dispatch, leaving the run with no status and no remaining, forever.
- peakEphemeralBytes was never flushed mid-flight, and ephemeral use is not
monotonic, so a collector OOM lost a high-water no successor could re-derive.
- attemptSeconds was newest-wins, so a second finalize could overwrite a real
duration with a near-zero one.
Two further defects are recorded as strict xfail rather than fixed, because
neither fix is obviously correct: the backwards-progress guard keeps its
high-water only in memory, so a restart disarms the guard that exists for
exactly that event; and load_progress()'s ConfigMap fallback returns a record
with every measurement stripped, which the next save_progress() then persists
over the real one.
Also removes dead weight found by audit: MAX_LINE_CHARS (referenced by no code,
yet plumbed through the chart and asserted by two tests), STATE_FLUSH_SECONDS,
was_disrupted, _cpu_millis, an `if True:`, two F# calls that read a ConfigMap
and discarded it, and the worker ping loop -- PARALLELISM HTTP GETs every 10s
for a liveness signal the driver never reads. Net -102 lines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The directory had 13 test files beside the two modules they test, named after the bug that prompted them (test_race_4.py) rather than the behaviour they pin. Groups them by subject -- reconcile/, collector/, resilience/ -- and gives each file a name that says what breaks if it fails. Adds pytest.ini because the suite no longer sits next to its imports: pytest was only resolving `import job_monitor` by inserting each test file's own directory. Source and chart lookups inside the suite were built by string- replacing the test's own filename, which broke the moment the file moved; they resolve from the module directory now. No test changed behaviour. 346 passed, 2 xfailed before and after. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The guard halted dispatch when `completed` shrank, on the theory that the durable record had been tampered with and redoing hours of work silently was worse than stopping. Its high-water mark lived in state['max_completed'], which a fresh process resets to zero -- so a monitor restart disarmed the guard for precisely the event it existed to survive, and it could only ever fire for a fault it had already failed to protect against. A reconciler must not gate a decision on state a restart erases. The alternative was persisting the high-water to the volume, which trades an automatic failure for a manual one: a legitimate reset would wedge the run until someone knew which file to delete. Dropping it is the cheaper mistake. Re-running a range is idempotent -- the PVC still holds /data, so the attempt resumes from its last closed ledger and the measurements are re-recorded rather than lost. Nothing gates dispatch now. `failed` stopped gating it because it deadlocked the driver; `halted` stops gating it for the reason above. The five tests that pinned the halt now pin the resumption, including the two that had encoded "the guard latches, it does not flap". Audited the rest of the reconcile state against the same rule: `replayed`, `counted` and `last_counts` also reset on restart, but they only feed metrics, and the Prometheus registry resets with the process anyway, so re-observing is correct rather than a bug. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Jenkinsfile options it describes belong with whoever owns the Jenkinsfile, not checked into this repo where they would drift from it silently.
Unlike the eight cut before it, this one was not redundant: it was the sole killer of two mutants, and both now survive. Dropping peakEphemeralBytes or peakWorkingSetBytes from rangeProfileFields no longer fails anything. What that costs, so it is not rediscovered the hard way: the artifact would still be written, still carry the right number of ranges, and still look complete, while the next run sizes every range from defaults on the missing axis. That is the RACE stellar#8 failure shape -- nothing downstream can tell a measurement-free profile from a good one -- narrowed to one field. drop-seconds and volume-peak-profiled are still covered, by "a measured run still produces a complete profile" and "range profile does not carry a pvc volume peak" respectively.
It asserted "peakVolumeBytes" is absent from rangeProfileFields. Nothing in the repo produces that field: no collector writes it, no sizer reads it, and the name appears nowhere outside the test. It guarded against a change nobody was making, and anyone deliberately profiling volume size would have deleted the assertion in the same edit. It scored as irreplaceable in the mutation run only because the mutant that killed it -- adding peakVolumeBytes to the field list -- was one I wrote to give it something to kill. A test that dies only to a mutant designed around it measures the mutant, not the test.
The mission sent 2Gi/4Gi for pvc runs, a figure invented to be small rather than
measured. In pvc mode /data is on the volume and the node disk holds only logs
and tmp, so any request there reserves disk nothing uses and makes disk rather
than cpu the binding dimension for packing.
The monitor was already written for this. _resources reads an empty
REQ_EPHEMERAL as "leave both axes off the pod" and drops any profile-derived
disk override with it, and eph_for_attempt returns None when no limit is
configured, so escalation is a no-op rather than a wrong number. Those branches
were unreachable while the mission always sent a value. Verified: a pvc pod now
comes out with requests {cpu, memory} and no limits at all, an ephemeral one is
unchanged at 35Gi/40Gi.
Consequence worth naming: a pvc worker now has no ephemeral-storage limit, so
it cannot be evicted for exceeding its own. Node disk pressure can still evict
it. That is the accepted trade -- the limit only ever guarded a dimension this
mode does not write to.
The test that covered this goes too. It asserted on the text of the mission
source, matching "2Gi", "4Gi" -- a string that no longer exists.
They asserted wallSeconds and txApply do not reach the artifact. Both are still recorded per range -- verified against a real ssc-test run, whose progress.json carried txApply=0.00025 and wallSeconds=42.0 while the artifact correctly held only peakAnonBytes, peakWorkingSetBytes, seconds and count. But leaking them would break nothing. load_profile_doc copies each record wholesale and sizing reads named keys with .get(), so an unknown field is ignored. The assertions guarded artifact weight, not behaviour, and weight is already the stated reason those fields are excluded. The three mutants this test uniquely kills -- storageMode and ledgersPerRange missing from the doc, and the count-before-guard defect -- still die. Also fixes unmeasured's doc comment, which described the record "as it survives the ConfigMap mirror". That mirror was deleted when status moved to /status; the helper now says what it actually builds.
It was "a run whose ranges measured nothing produces no profile artifact",
which describes the least interesting case it covers. The empty-artifact
scenario is caught by a human looking at the profile they pass in.
What is not caught by looking is a MIX. profile_for resolves a range to the
nearest measured end ABOVE it, so one measurement-free entry captures every
range beneath it and hides the real measurement further up. Demonstrated
against the live sizing code: with a junk entry at 1200 alongside a real one at
1600, range 1100 resolves to {'count': 1600} and routes to protostar; with the
guard doing its job it resolves to the 5 GiB record and routes to supergiant.
It is silent the whole way. The junk entry is a truthy record, so
_profile_overrides proceeds, finds no peakAnonBytes, _tier_for_bytes returns
None, and pool_for treats the range as past the profile. Nothing logs a
difference, and the run just costs more.
Test body unchanged -- it already asserted the right thing. Only the name and
the comment were wrong about why.
The two existing tests feed all-measured or all-unmeasured records, and a guard that decides per RUN rather than per RECORD passes both. Demonstrated with a sticky guard -- one that latches on the first measurement and lets every later record through: all-unmeasured never latches so the document is still refused, all-measured latches immediately so everything is still written, and both tests go green while every junk record following a real one now reaches the artifact. That is the case that costs something, and it is also the realistic one: a run misses a few ranges, not zero and not all. profile_for resolves a range to the nearest measured end ABOVE it, so one junk entry captures every range beneath it and hides the real measurement further up -- range 1100 routes to protostar with a junk entry at 1200 present, and to supergiant without it. The new test is the sole killer of the sticky-guard mutant; the other two do not notice it.
Same facts, a third of the lines. The shadowing explanation appeared in two test comments and the module header; it now lives once, in the header, since it is the reason the whole group exists rather than a property of any one test.
Six tests do not need a file of their own. They go under a section banner, with Tests.fs regaining the two opens it lost when they moved out. Verified as a move: same 22 tests, and the count-before-guard defect still fails the same two.
helm reads <chart>/values.yaml as its base already, so --values with that same path re-applied the file on top of itself. Verified byte-identical renders with and without it. It was only ever there to carry the second file: the on-demand overlay rode as a second --values, and the base had to be named explicitly so layering worked. With the overlay gone the base names itself, and valuesFilePath, valuesArgs and the argv entry all go with it. Precedence is unchanged -- chart defaults, then --set -- because a -f of the chart's own defaults changed nothing to begin with.
Unset, nothing is passed and helm uses the chart's own values.yaml as its base -- which it does regardless, so naming that path explicitly only re-applied the file to itself. Set, the named file is layered on top. Layered rather than substituted, so an override carries only the keys it changes: verified with a two-line file that moves poolPrefix while poolTiers keeps the chart's ladder. SUPERCLUSTER_CHART_PATH already repoints the whole chart, values included. This is the narrower case -- the baked chart run against experimental numbers, without a working copy.
Only https was recognised as a URL, so an http spec fell to the else branch and was read as a FILENAME. It failed as "could not load range profile" without ever naming the scheme, and the run proceeded unprofiled -- every range sized from defaults, visible only as a run that cost more. The codebase already speaks plain http to the monitor, so there was no https-only posture to keep, and a profile moves resource REQUESTS only: a tampered one costs node size, not code execution, and it is parsed and range-counted before use. Four tests, one per path, each the sole killer of its own mutant: fetched over http (against a loopback listener), read from a local path, refused when it carries no ranges, and never fatal when the spec is empty, missing or unreachable. The listener helper swallows exceptions inside its serving async deliberately. Without that, a test where no request arrives faults the pending GetContextAsync when Stop() runs, on a background thread, and takes the test host down -- the run then reports "26 passed" as "8 passed" and the failure it was supposed to surface disappears. Caught while checking the http test could fail at all.
Four tests and a loopback HTTP listener for paths whose failure is a warning and a run sized from defaults, not a fatal outcome. The http path is better verified by running it.
Every run shipped two log files: http_server_<stamp>.log holding the whole monitor's output, and job_monitor_<stamp>.log holding nothing. Seen in run stellar#181 at 13.27 KiB and 0 B, and reproduced locally. build_logger calls logging.basicConfig, which is a no-op once the root logger has handlers. job_monitor imports http_server before configuring itself, so http_server's call ran first and won; job_monitor's was discarded. The FileHandler it had already constructed still created and opened its file, which is why an empty one appeared. Both modules take the ROOT logger, so everything landed in the file named after whichever module imported first. The entrypoint should configure logging and nothing else should. http_server now takes a plain named logger, so job_monitor's build_logger is the first and only call: one file, job_monitor_<stamp>.log, carrying both modules' records. Verified -- one file, 276 bytes, records from both. Image rebuilt as 2026-08-11a and pinned.
The bundle was flat. A 10-range run puts 43 files beside the five that summarise it; a 4000-range run puts ~16000. The three per-range artifacts now land in range-logs/, metrics/ and state/, with mission_started joining the other state markers, leaving the top level as the monitor log, the driver log, progress.json, run.json and the profile. Sorted on the way out rather than on the volume. The monitor's paths are an implementation detail, and keeping the volume flat leaves /logs/<name> taking one path element with no separator -- which is what stops the route, reachable from outside the cluster once its HTTPRoute is attached, being walked out of LOG_DIR. Verified on ssc-test: top level 6 files, range-logs/ 3, metrics/ 3, state/ 6. DumpPodInfo now logs a count per phase instead of a line per pod. It fires every 5 minutes for the whole mission, so at 1024 workers it wrote ~1026 lines a time -- roughly 57000 over a 4.7h catchup -- and the only thing worth reading in it, anything not Running, was buried. Shorter than the code it replaces, and a 5-pod mission now reads "Pods: 5 total Running=5".
The status line went out every minute -- ~280 lines on a 4.7h run, saying
nothing new most of the time. It now prints one poll in ten.
Printed less, not polled less: jobMonitorStatusCheckTimeOutSecs is spent in
units of the interval, so 600 over a 60s poll tolerates ten consecutive
failures. Slowing the poll to 600s instead would leave a single transient blip
failing the run outright. Detection is unchanged; only the logging thins. A
range failure still logs the moment it is seen, on its own line.
The pod summary timer goes from 5 minutes to 10. That callback lists every pod
in the namespace, so on a 1024-worker run it is a large response fetched to
print one line.
Also drops the ephemeral request and limit on pooled runs that were given a
profile. Such a range has its node to itself -- the tier's memory cut is sized
to exclude a second pod -- so the limit guards no neighbour and only turns
spare disk into an eviction: measured, a dwarf range capped at 5211Mi alone on
a 20Gi root, dying once it exceeded its profiled peak by more than the margin.
The request goes too, since its only remaining job is scheduling and the tier
label already decides placement. Unprofiled pooled runs keep both, nothing
having measured them, so their disk escalation path is intact.
Verified across the four combinations: pooled+profiled ephemeral drops to
requests {cpu, memory} with no limits; pooled with no profile keeps 35Gi/40Gi;
unpooled+profiled keeps its profile-derived figure; pvc is unchanged.
Image rebuilt as 2026-08-11b and pinned.
collectLogs resumed a partial file with a Range request and appended the
answer, and skipped a file whose length already matched. Both assume the file
only ever grew, which is true of a worker log and of nothing else it collects.
progress.json is rewritten whole on every reconcile and grows as ranges
complete, so the second pass asked for bytes past the length of the FIRST
document and appended that tail to it. Reproduced on ssc-test with a 32-range
run: pass one left 2675 bytes at 18 completed ranges, pass two saw 4708 at 32,
and splicing them the way the old code would gives "Extra data: line 1 column
2676" -- the first document's closing "failed":{}} followed by a fragment of
the second.
Silent, too: every prod run collects on a 10-minute timer over ~4.7h, so this
landed a truncated progress.json in the bundle while nothing complained. The
profile artifact is unaffected, being built from the copy read off the pod.
Equal length is no safer than a short read: a rewrite can change a value
without changing the length -- seconds 100.0 to 200.0 -- so both the skip and
the resume now apply only to .log.gz. Refetching the rest costs little against
what the optimisation is actually for: progress.json is ~1 MB at 4000 ranges,
the worker logs are ~1.4 GB.
Verified on the same run with the fix: 4708 bytes, 32 ranges, valid.
Runs name this image by a mutable tag -- the mission passes --job-monitor-image-pc-v2 stellajuna/ssc-jm:latest -- and IfNotPresent pinned each node to whatever it had cached the first time it saw that tag. Worker nodes churn under Karpenter and would repull regardless, but the monitor carries no tolerations and lands on the long-lived default pool, where the layer persists indefinitely. Run stellar#183 shipped a 0-byte job_monitor.log and a populated http_server log from an image whose fix for precisely that had been pushed hours before. The registry tag was correct throughout; verified by pulling latest fresh, which carries the logging fix, REQUIRE_NODE_LABELS, the pooled+profiled ephemeral change and the /start pool-map guard. Kubernetes defaults :latest to Always for this reason and the explicit IfNotPresent was overriding it. One extra pull per run for one pod, and a registry outage now fails the run at start rather than quietly running something stale.
monitor.capacityType and the python that read it both went when capacity became an ordinary entry in REQUIRE_NODE_LABELS, but the env block survived. It rendered from a value that no longer exists, so every run shipped a CAPACITY_TYPE with no value that nothing reads -- visible in run stellar#184's pod spec. Harmless, and invisible to the contract tests: they walk the config module and pair each attribute with its chart env, so an env var with no counterpart in the code is never looked at. Not worth a test of its own -- the failure is a dead line, not a wrong run.
The monitor asked for nothing -- no nodeSelector, no tolerations beyond the default two -- so it could only land on an untainted node. Every catchup pool is tainted, which left the shared EKS managed node group: on ssc-eks that is a node up since June, hosting traefik, argocd, kyverno and another engineer's stellar-core pods. It was there by omission, not by choice, and it is the pod whose collector streams logs from up to 1024 workers. --job-monitor-node-labels and --job-monitor-tolerate-taints take the same `key:value` form as their worker counterparts and are empty by default, so an installation without tiered pools sets nothing and sees exactly today's behaviour. Both are needed together to move it -- labels alone leave it unschedulable on a tainted pool. The chart gains only monitor.nodeSelector, rendered with toYaml. tolerateNodeTaints already took native toleration objects, and a plain equality map is enough here: the monitor's node is chosen once for the run, unlike a worker's, which is resolved per range and per attempt. Verified on ssc-test with purpose:catchup-giant catchup-capacity:od and the catchup toleration: the monitor landed on a karpenter catchup-giant-od-w100 node, on-demand, m8a.large, and the run completed. Worth knowing before pinning it to a small tier: the monitor holds the logs PVC for the whole run, so its node cannot be consolidated away while it is there, and it reserves that node from the ranges.
Five knobs carried the same reasoning in both places, and config.py had the fuller account each time -- 29 lines against 14 for the attempt deadline, 30 against 7 for the blocked rungs. The chart now says what the knob is and which way it fails, and leaves the derivation beside the code that reads it, where it cannot drift from the logic. Two of the removed blocks were also wrong rather than merely long: the attempt deadline explained itself in terms of "the progress ConfigMap", deleted when status moved to /status, and a sentence about cpu margin sat above a memory ceiling it did not describe. Deliberately kept: prestopSleepSeconds, createRbac, imagePullPolicy, storageSize, poolPrefix, the worker cpu block and watchTimeoutSeconds. Those have no counterpart in config.py, so the chart is their only record -- the file reads comment-heavy because most of it is the sole account of a decision, not because it repeats one. 301 -> 276 lines.
The formatting check fails the build outright (exit 99) and three files had drifted: the mission, Tests.fs and Program.fs. Formatting only, no behaviour change -- same 22 F# tests, same build. values.yaml rides along rather than as its own commit, which was an accident of staging, but the two do not conflict. Nine comment blocks ran past three sentences; the long ones now lead with what the knob is and which way it fails, and keep only the measurement that pins the number -- the 90 OOMKills behind profileCacheHeadroom, the 1.03x margin behind the tier cuts, the 60s hook that still lost txApply behind prestopSleepSeconds.
Contributor
There was a problem hiding this comment.
Pull request overview
Rewrites Parallel Catchup V2 around dynamically sized Kubernetes Jobs, durable monitoring, retry-aware state, and profile-driven scheduling.
Changes:
- Replaces Redis/StatefulSet workers with per-range Jobs and PVC-backed state.
- Adds profiling, tiered placement, retry classification, log collection, and HTTP control APIs.
- Extends mission options, Helm deployment, artifacts, and tests.
Reviewed changes
Copilot reviewed 34 out of 36 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
.gitignore |
Ignores Python test artifacts. |
src/App/Program.fs |
Adds V2 mission CLI options. |
src/FSLibrary/MissionHistoryPubnetParallelCatchupV2.fs |
Rewrites mission orchestration and artifact handling. |
src/FSLibrary/StellarKubeSpecs.fs |
Clarifies V1/V2 resource ownership. |
src/FSLibrary/StellarMissionContext.fs |
Adds V2 configuration fields. |
src/FSLibrary/StellarSupercluster.fs |
Summarizes pod phases in logs. |
src/FSLibrary.Tests/FSLibrary.Tests.fsproj |
Reformats the test project. |
src/FSLibrary.Tests/Tests.fs |
Adds profile and Helm-option tests. |
src/MissionParallelCatchup/.dockerignore |
Excludes Python bytecode. |
src/MissionParallelCatchup/Dockerfile.jobmonitor |
Builds the monitor and collector image. |
src/MissionParallelCatchup/apps/job_monitor.py |
Implements Job dispatch, retries, and state. |
src/MissionParallelCatchup/apps/log_collector.py |
Collects logs and resource measurements. |
src/MissionParallelCatchup/job_monitor.py |
Removes the Redis-based monitor. |
src/MissionParallelCatchup/lib/attempts.py |
Aggregates attempt results and metrics. |
src/MissionParallelCatchup/lib/config.py |
Defines monitor configuration and policies. |
src/MissionParallelCatchup/lib/http_server.py |
Exposes control, status, and artifact APIs. |
src/MissionParallelCatchup/lib/kube.py |
Creates shared Kubernetes clients. |
src/MissionParallelCatchup/lib/logger.py |
Centralizes logging setup. |
src/MissionParallelCatchup/lib/medida.py |
Parses stellar-core timing metrics. |
src/MissionParallelCatchup/lib/metrics.py |
Declares Prometheus metrics. |
src/MissionParallelCatchup/lib/profiles.py |
Loads and resolves range profiles. |
src/MissionParallelCatchup/lib/ranges.py |
Generates and orders ledger ranges. |
src/MissionParallelCatchup/lib/records.py |
Manages durable attempt files. |
src/MissionParallelCatchup/lib/sizing.py |
Calculates resources and pool tiers. |
src/MissionParallelCatchup/lib/units.py |
Converts Kubernetes quantities. |
src/MissionParallelCatchup/lib/worker_liveness.py |
Probes worker health concurrently. |
src/MissionParallelCatchup/parallel_catchup_helm/files/logarithmic_range_generator.sh |
Removes Redis range generation. |
src/MissionParallelCatchup/parallel_catchup_helm/files/uniform_range_generator.sh |
Removes Redis range generation. |
src/MissionParallelCatchup/parallel_catchup_helm/files/worker.sh |
Removes Redis worker loop. |
src/MissionParallelCatchup/parallel_catchup_helm/templates/_helpers.tpl |
Adds label rendering helpers. |
src/MissionParallelCatchup/parallel_catchup_helm/templates/catchup_workers.yaml |
Removes the worker StatefulSet. |
src/MissionParallelCatchup/parallel_catchup_helm/templates/core_config.yaml |
Extracts stellar-core configuration. |
src/MissionParallelCatchup/parallel_catchup_helm/templates/job_monitor.yaml |
Deploys monitor, collector, storage, and RBAC. |
src/MissionParallelCatchup/parallel_catchup_helm/templates/job_preload_redis.yaml |
Removes Redis queue preloading. |
src/MissionParallelCatchup/parallel_catchup_helm/templates/redis_queue.yaml |
Removes Redis deployment. |
src/MissionParallelCatchup/parallel_catchup_helm/values.yaml |
Defines sizing, retry, storage, and monitoring defaults. |
Suppressed comments (1)
src/FSLibrary/MissionHistoryPubnetParallelCatchupV2.fs:893
- Signal cleanup is documented as having roughly five seconds before SIGKILL and as needing to uninstall resources first, but this unconditional profile read performs a pod lookup plus remote exec before entering that branch and has no short timeout. If the API server or monitor is slow, SIGKILL occurs before
helm uninstall, leaking the full mission fleet. Keep the signal path bounded and prioritize uninstall; use an already-fetched local progress artifact if a partial profile is needed there.
try
writeRangeProfile context
with ex -> LogWarn "Failed to write range profile: %s" ex.Message
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # TEMPORARY dev pin: stellar/ssc-job-monitor:latest predates the apps/+lib/ | ||
| # split and ATTEMPT_BUDGETS, so this chart would ship env vars it cannot read. | ||
| # Revert to the stellar/ repo once there is a push path for it. | ||
| image: "stellajuna/ssc-jm:2026-08-11b" |
Comment on lines
+76
to
+77
| - kind: ServiceAccount | ||
| name: {{ .Release.Name }}-job-monitor |
Comment on lines
+811
to
+813
| return [client.V1OwnerReference(api_version='v1', kind='ConfigMap', | ||
| name=cm.metadata.name, uid=cm.metadata.uid, | ||
| block_owner_deletion=True)] |
| // "did not tolerate taint (taint=catchup:NoSchedule)". | ||
| setOptions.Add(sprintf "worker.requireNodeLabels[0]=purpose:%s" context.pubnetParallelCatchupPoolPrefix) | ||
|
|
||
| setOptions.Add(sprintf "worker.tolerateNodeTaints[0]=%s" context.pubnetParallelCatchupPoolPrefix) |
Comment on lines
+1077
to
+1080
| # On the JobSpec, not the pod: a pod-level deadline is immutable once | ||
| # the pod exists, so a mis-set value could not be corrected on a live | ||
| # run. | ||
| active_deadline_seconds=config.ATTEMPT_DEADLINE_SECONDS or None, |
Comment on lines
+50
to
+51
| range: | ||
| overlapLedgers: 320 |
Comment on lines
+536
to
+540
| [<Option("pubnet-parallel-catchup-storage-mode", | ||
| HelpText = "worker /data backing: 'pvc' keeps it across pods so an evicted range resumes at L+1 (needed for spot); 'ephemeral' puts it on the node disk where retries cannot resume (only supported for V2)", | ||
| Required = false, | ||
| Default = "pvc")>] | ||
| member self.PubnetParallelCatchupStorageMode : string = pubnetParallelCatchupStorageMode |
Comment on lines
+548
to
+551
| [<Option("pubnet-parallel-catchup-range-order", | ||
| HelpText = "dispatch order: 'tip-first' (default) or 'oldest-first'. Generators emit tip-first, which front-loads the most expensive ranges; oldest-first profiles the cheap early ones first (only supported for V2)", | ||
| Required = false, | ||
| Default = "tip-first")>] |
Comment on lines
+97
to
+101
| profile = profiles.load_profile_doc(doc.get('profile') or {}) | ||
| # The whole config, judged at the first moment it is complete. Anything | ||
| # wrong rejects the POST with the reason rather than dispatching a run that | ||
| # is already misconfigured. | ||
| validate_config() |
Comment on lines
+202
to
+206
| if r.IsSuccessStatusCode then | ||
| LogInfo "Mission started: profile POSTed to %s/start" (monitorEndpoint context) | ||
| started <- true | ||
| else | ||
| Thread.Sleep(5000) |
The merge appended them after this branch's module-level V2 tests, so five `member __.` definitions ended up outside the type they belong to. F# read the first `[<Fact>]` at that indent as a syntax error, which is why fantomas could not parse the file rather than merely wanting to reformat it -- the build was failing too. Moved verbatim to the end of the class body, ahead of the V2 section. The suite goes 22 -> 34: main's twelve were not being run at all, since a member outside its type is not a member of anything.
- lib/ splits: shared (config, logger, records), lib/monitor/, lib/collector/ - config.py 555 -> 53 lines; 60 monitor-only names to monitor_config - collector settings to collector_config; log_collector has no os.getenv left - new lib/collector: kube_http, verdicts, tx_scan - log_collector opens on main(); all runtime state above it - delete load_profile()/PROFILE_PATH: no production caller, /start carries the profile; its cross-mode strip was inert (pvc sets no ephemeral limit) - delete base()/done_path(): duplicated records.py, which both processes import - Dockerfile flattens each lib dir explicitly -- a directory COPY would leave subdirectories in /app and break bare-name imports Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…he pod list The collector kept a task per pod, and that one decision forced a registry, wake events, cancellation, grace cycles and a peak cache -- roughly half its state existed to coordinate tasks rather than to collect anything. One cycle now: list pods, then read every log and sample every kubelet off that list. Two conditions carry the work -- the first read of an attempt records its resume decision, the read that finds the pod terminal records the verdict, txApply, duration and .done. - module state is 5 names (3 semaphores, follows, last_ts), from 17 - peaks go straight to .metrics, which already max-merges: no in-memory copy to lose on restart, and no flush ratio to approximate it - keyed by (end, attempt) like every file, so a vanished attempt is identifiable once its pod is gone - .done on disk replaces the finalized set, so a restart stops re-finalizing - no persistent scanner: the terminal read reaches back TERMINAL_REREAD_SECONDS so a straddled medida block is whole in one read, and _ingest dedups the overlap - records.py is the cross-process contract only; retry accounting moved to lib/monitor/attempt_files, resume points to lib/collector/state_files - medida folded into tx_scan: the second reader it existed for is gone - delete tx_scan.scan_archive and 5 chart knobs the rewrite orphaned log_collector.py: 1350 -> 524 lines. Fixed on the way: _poll_once acquired the same semaphore the gather held, which deadlocks every holder at full occupancy; hydration dropped the timestamp validation that keeps a poisoned .state from making every request 400. Testing: 3-range and 4-range missions on ssc-test, the latter with maxConcurrentPolls pinned to 2 against 4 pods so the bound was exercised. 4/4 ranges completed with txApply, peaks and exact durations. 140 contract tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HistoryPubnetParallelCatchupV2 Mission Rewrite
Purpose
HistoryPubnetParallelCatchupV2 is the longest and most expensive mission in the Supercluster mission suite.
This is due to:
This PR addresses the above 4 by:
Components
Karpenter NodePool Tiers
Ledger range jobs are sized into tiers. The smallest ledger range jobs will run on the
dwarftiers, the largest ledger range jobs will run on thehypergianttier.This implementation supports running this mission on either On-Demand nodes using ephemeral storage or Spot Instance nodes on PVC storage. Due to the capacity availability differences between On-Demand and Spot, the pool tiers are setup a bit differently for each.
Ondemand nodes use the fastest AMD chips, sized to allow 1 node to fit exactly 1 job. This is the fastest configuration that allows each job to closes ledgers at a rate of at least 2 ledgers per second. Allowing the mission to complete in less than 3 hours.
Spot instances primarily use 8th gen Intel Chips, sized for 1 node to fit exactly 2 jobs because each tier uses double the cpu and ram of the equivalent Ondemand tiers. Intel machines are weaker than AMD, and appear to reach AMD performance when the size is doubled. Ledger ranges close at a rate of atleast ~1.5 ledgers per second, causing the mission to finish in less than 4 hours.
Missions without a profile input run on the
nebulatier, 2 pods per r8.xl, 3 pods per m8.2xl, 4 pods per r8.2xlUnprofiled ranges in a profile inputted mission run on the
protostartier.Job Monitor
Emits K8s Jobs at a maximum of the parallelism. 4000 ranges, parallelism 1000, max 1000 jobs running at a time.
Singleton writer of state, as opposed to the previous system where each worker wrote to the redis state.
Retries job based on reason:
Emits metrics
log-collectorwrites job as.donein the PVCtxApply,seconds,peakAnonBytes,peakWorkingSetBytesfrom all job attempts and emits them to prometheusRuns http server
{"num_remain":0,"queue_remain_count":0,"queue_succeeded_count":0,"queue_failed_count":0,"queue_in_progress_count":10,"jobs_failed":[],"workers_refresh_duration":3.0994415283203125e-06,"mission_duration":30.198680877685547,"started":true}Produces a profile to use for next mission
peakAnonBytesis used to set nodepool tier on tiered mission runs. This is the max non-cache memory observed needed for the pod to run.peakWorkingSetBytes- cached used by the pod, a large cache causes the job monitor to escalate the job range to the next node tierseconds- seconds is how long a job ran, used to emit the longest jobs first.peakEphemeralBytescaptures max amount of ephemeral storage the job pod used. Unused for tiered runs, used to set ephemeralStorage req/limits for untiered mission runsLog Collector
Mission Options and Outcomes
Full Mission Run Length on Spot + PVC: