You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
template-manager: reverse the next release's node_dirty_ref_revisions migration so its rollback stays image-only - #1277
Template-manager's node-local SQLite database can be migrated by a surge pod while the previous pod still serves. The rule in crates/libs/sqlite/src/template_manager.rs is that a rollback must need nothing beyond the image. A runner refuses a database that records a migration it doesn't bundle (unbundled migration). So rolling the image back past #1235's 20261031050126_node_dirty_ref_revisions would crash-loop template-manager on every rolled-back node.
This PR ships one release before #1235, using the two-release pattern from #1070, so the image a rollback returns to can reverse that migration.
A list instead of one hard-coded migration. The runner's single reversal for 20261026220935 becomes a list (UNBUNDLED). Each entry has a version, description, original up SQL (for the recorded checksum), down SQL, and a check that refuses when reversing would drop data worth keeping.
Check everything, then reverse in one transaction. The runner first checks every recorded, unbundled entry: success and checksum, then the data-loss check. Any failed check stops before a write. It then applies their down SQL newest first (sorted by version, whatever the list order) and deletes their history rows, all in one transaction, so a failure partway leaves the database as it was. Other recorded versions it doesn't bundle are still refused by history validation, which runs after the reversal commits.
20261026220935 behaves as before. Same checks, same refusal messages, same log line, and its tests pass unchanged. Only the context strings on SQLite I/O failures during down SQL, history delete and commit are now generic: "failed to reverse retired template-manager migration 20261026220935" instead of "failed to reverse retired template cache partitions".
Docs.
The module doc keeps its first paragraph and the image-only rule.
db/README.md gains an "Unbundled template-manager migrations" section: the list, the check-then-reverse transaction, the rule that a new migration joins the list one release before it is bundled and must then ship unchanged, and the in-flight behaviour of 20261031050126. The existing "Retired template-manager migration" section is unchanged. The SQLite history paragraph now points to the new section.
reverse_retired_partitions is renamed reverse_unbundled; the only caller is in crates/libs/sqlite/src/lib.rs.
Validation
Everything below ran at 1aff5639. The current head differs from it only in doc comments in template_manager.rs and prose in db/README.md.
Unit tests. With space-nextest-postgres-setup built first, cargo nextest run -p pplx-space-sqlite -p pplx-space-template-mgr -p pplx-space-db-migrate -p pplx-space-db-migrator ran 378 tests: 378 passed, 0 skipped. cargo clippy on the same packages with --all-targets -- -D warnings is clean, cargo fmt --check is clean, and cargo arch reports 0 required findings.
a_database_with_dirty_ref_revisions_reverses_and_reapplies: next catalog → next catalog (untouched) → this runner (reversed) → next catalog (re-applied) → this runner. Each step checks rows, outbox rows, history, and that the schema equals a fresh database exactly.
a_failed_or_modified_dirty_ref_revisions_refuses_reversal: a modified checksum or failed row is refused and the schema is left alone.
every_unbundled_migration_reverses_together_or_not_at_all: with both migrations recorded, they reverse together; a non-legacy partition row refuses in the checks, before any down SQL.
a_failure_after_the_first_reversal_rolls_back_every_reversal: with both migrations recorded and every check passing, a trigger aborts the older migration's history delete, after the newer migration's down SQL and history delete and the older down SQL have run. Migration fails with the injected error, and rows, outbox, schema and history are unchanged.
dirty_ref_revisions_is_the_migration_the_next_release_records: pins the checksum.
"A database without it is untouched" is already covered by a_bundled_schema_is_left_untouched.
Red first: before the change, the three original behaviour tests failed with _sqlx_migrations has failed, modified, or unbundled migration 20261031050126.
Mutation check (not committed): I kept the shared checks but made reverse_unbundled commit after each migration's reversal. a_failure_after_the_first_reversal_rolls_back_every_reversal then fails: 20261031050126 was reversed and removed from history while 20261026220935 stayed. The other 11 template-manager tests, including every_unbundled_migration_reverses_together_or_not_at_all, still pass. Without the mutation all 12 pass.
Real-binary rollback. I built space-db-migrator from #1235 (1265e6eb), from main (d397b69d) and from this branch, each in its own worktree and its own CARGO_TARGET_DIR. A target dir shared across worktrees can produce a stale binary. strings <bin> | grep -c 'reversed newer node dirty ref revision migration' gives 0, 0 and 1. I then ran them in order on one database file:
main's migrator up on a copy fails: unbundled migration 20261031050126.
This branch's migrator up logs reversed newer node dirty ref revision migration and exits 0. History is baseline only, the outbox rows are kept, and the schema matches a fresh main database exactly.
Rollback on Kind with a real pod. I brought up a fresh cluster from this branch with SPACE_KIND_TELEMETRY_MODE=otel bash infra/space/kind/scripts/up.sh (Kubernetes substrate; about 5.5 minutes); the pods report this head's git_sha. I built #1235's migrator in the repo's bookworm builder image, in its own target dir, to match the node's glibc. I ran it as space-tmplmgr on worker space-local-worker's live database, beside the serving pod, and it recorded 20261031050126. I then deleted the pod. This branch's template-manager reversed the migration at startup (WARN log) and was Ready 6s after the delete, with 0 restarts. Running #1235's migrator again re-applied it, and another restart reversed it again (Ready in 6s).
Kind smoke, on the same cluster.
Template scenarios (rg -l template space_smoke/scenarios, 23 IDs). smoke-kind-typed.sh only takes categories, so I ran each ID with space_smoke run <id> after the script's own token and org setup. 21 passed in 16m42s.
native.aws_secret_network_credential_injection was inconclusive because it needs SPACE_SMOKE_SECRET_ARN.
node.windows_sandbox_lifecycle refuses Kind; it needs a deployed backend.
all: SPACE_SMOKE_KIND_COLD_RECEIPTS=<dir> bash scripts/tests/smoke-kind-typed.sh all took 61m56s, with the run exiting 0.
ECW pass: 89 passed, 0 failed, 0 inconclusive, including ecw.cold_create_environment and ecw.warm_pool_environment, which need the receipts directory.
Native pass: 35 passed, 0 failed.
Severity and risk
Severity: high once #1235 ships. Without this, an automatic canary rollback across it crash-loops template-manager on every rolled-back node.
Risk: the change is runner-only plus history SQL; this release has no schema change. Before #1235 ships, no database records 20261031050126, so the new entry is never used. To roll back, revert this commit. That only matters after #1235 ships: without this commit, a rollback past #1235 crash-loops again.
Rollout
During a rollback from #1235's release, the serving #1235 pod still reads revision after this release's migrator reverses it. Its outbox reads and acknowledgements fail until it exits or a migrator re-applies the migration, and its rows stay queued. A send it started before the reversal and acknowledges after a re-apply can match a restarted revision and delete a re-enqueue made during that send. This is reasoned from #1235's code; Kind has no surge migrator to observe it. In the forward direction, this release writes the outbox without naming revision, so it keeps serving while #1235's migrator applies it.
Don't change 20261031050126's up SQL, comments included. This release would refuse a database that recorded a different checksum. A changed migration needs a new version.
Adapt the tests built on REVISIONED and PARTITIONED_REVISIONED. template-manager: ack node dirty refs by revision so a re-enqueue during a send survives #1235's BUNDLED already contains 20261031050126, so both catalogs would list it twice, and BUNDLED is no longer the release that reverses it. Give those tests a catalog without 20261031050126 as the rolling-back release; with BUNDLED in that role, a_failure_after_the_first_reversal_rolls_back_every_reversal still passes but reverses only one migration. The runner itself already skips any version the catalog bundles.
Its module-doc change (rollback past 20261031050126 "needs the newer release's space-db-migrator template-manager down") is no longer true and should be dropped.
native.aws_secret_network_credential_injection: needs a real AWS secret.
node.windows_sandbox_lifecycle: needs a deployed backend.
AI assistance: Claude Code (an AI agent) wrote the code, tests, docs and this description, and ran the validation.
Summary by CodeRabbit
New Features
Dirty-reference records now receive revision numbers when created or updated, with a node-wide counter tracking changes.
Bug Fixes
Template-manager startup can now reverse certain recorded migrations that are no longer included in the service. Reversal checks migration history and data safety first, and applies eligible changes in a single transaction.
Reversing dirty-reference revision tracking removes the associated revision data.
Automatic approval is disabled by the configured approval policy for this pull request. Review comments and requests for changes continue normally.
Commit: 2c64bdf210cc.
No actionable comments were generated in the recent review.
Walkthrough
The template-manager store now reverses recorded migrations that are not bundled by the current service. It checks migration history and data loss before applying down SQL in one transaction. The change adds dirty-ref revision migration SQL, rollback tests, and database migration guidance.
The dirty-ref revisions migration adds a revision counter and triggers. The unbundled migration descriptor lists its SQL and reversal check.
Validate and reverse unbundled migrations crates/l10_foundation/sqlite/src/lib.rs, crates/l10_foundation/sqlite/src/template_manager.rs
Template-manager migration startup selects recorded unbundled migrations, checks history and losslessness, then applies down SQL and removes history rows in one transaction.
Test rollback behavior and document migration rules crates/l10_foundation/sqlite/src/template_manager/tests.rs, db/README.md
Tests cover checksums, repeated migration, rejected reversals, and transaction rollback. The database guide describes release requirements and the effects of reversing dirty-ref revision tracking.
sequenceDiagram
participant TemplateManagerStartup
participant reverse_unbundled
participant SQLiteDatabase
TemplateManagerStartup->>reverse_unbundled: request migration reversal
reverse_unbundled->>SQLiteDatabase: inspect migration history
SQLiteDatabase-->>reverse_unbundled: return recorded candidates
reverse_unbundled->>SQLiteDatabase: validate checksums and losslessness
reverse_unbundled->>SQLiteDatabase: apply down SQL and delete history rows
reverse_unbundled->>SQLiteDatabase: commit transaction
Loading
Suggested labels:kind-smoke
Suggested reviewers:the-gigi-pplx
Merge Risk: Minimal · previous review 3beb5
No actionable issue is established for this change. The documented mixed-version rollback behavior remains an operational consideration for the release that bundles the migration.
Security architecture risk: Moderate · up to 3beb5
The rollback implementation has strong transaction and validation safeguards. However, reversing the revision schema while a newer pod remains active creates a conditional reconciliation risk that transaction atomicity alone does not address. Compatibility with that newer pod and deployment-level coordination remain unverified.
Retained concerns
Medium · reliability · inferred: Automatic reversal removes the revision contract without coordinating with an active revision-aware consumer in the inspected standalone migration path. Under the documented overlapping-pod rollout, this can interrupt outbox processing; reversal followed by reapplication also reuses revision identities and may let an old acknowledgement delete a later re-enqueue. The daemon's ownership mutex and SQLite transaction protect different boundaries and do not establish standalone-migrator coordination. This is a conditional rollback-safety concern: the current consumer remains schema-compatible, while the future consumer and deployment fencing were unavailable for verification.
Security review details
Security Blast Radius
inferred — Each reversal targets one writable template-manager database and can affect all its outbox obligations. A fleet rollback repeats that exposure per node. The inspected trigger is migration or startup authority, not an ordinary request selecting arbitrary rollback SQL.
Trust Boundaries and Controls
observed — Database history selects only compiled, allowlisted reversal descriptors. Failed or checksum-mismatched candidates are refused, the retired-partition data check remains, and general history validation still rejects unsupported versions. These are migration-integrity controls, not substitutes for filesystem access control or active-consumer coordination.
Resilience and Maintainability Implications
observed — The current worker's key-only acknowledgement can lose a same-key re-enqueue during an in-flight send, but its database and worker implementations are unchanged between the PR base and head. That existing behavior is counterevidence against attributing a new current-image acknowledgement vulnerability to this PR; the distinct unresolved risk concerns newer consumers during schema reversal.
Hardening Proposals
proposed — Before deploying the revision-aware release, establish a rollback barrier that stops or fences its outbox worker before reversal and prevents acknowledgements from surviving revision-counter reset. Validate that barrier using overlapping consumers and interrupted rollback/reapplication, rather than schema-only round trips.
Pre-merge checks: 4 · 1
Failed checks: 1 warning
Check name
Status
Explanation
Resolution
Docstring Coverage
Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…
Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed checks: 4 passed
Check name
Status
Explanation
Title check
The title names the change: the next release's node_dirty_ref_revisions migration gets a reversal entry, so its rollback needs only the image. The changeset matches this: it adds a generalized unbundl…
Description check
The description follows the template. It has Summary, Validation, Severity and risk, Rollout, Stack, Not covered, and AI assistance sections. The Summary explains the effect of merging, the rollback r…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title names the change: the next release's node_dirty_ref_revisions migration gets a reversal entry, so its rollback needs only the image. The changeset matches this: it adds a generalized unbundled-migration reversal list with the 20261031050126 entry, renames reverse_retired_partitions to reverse_unbundled, and adds the matching tests and docs. The title is long (112 characters) and uses some internal jargon, but it is specific and not misleading.
Full details: Description check
Explanation
The description follows the template. It has Summary, Validation, Severity and risk, Rollout, Stack, Not covered, and AI assistance sections. The Summary explains the effect of merging, the rollback rule, and the two-release pattern. Validation lists test runs, a real-binary rollback, a Kind rollback, and smoke results, and it names what was not validated. Severity and risk states the impact and the rollback path. The optional Rollout, Stack, and Not covered sections are filled in with specifics. The description is long for a change of this size, but the template asks for risk-sized detail and the content is relevant.
Full details: Docstring Coverage
Explanation
Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 7 functions across 3 files. (3 skipped: 3 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
Finishing Touches
Generate docstrings
Commit to this branch
Create a new PR
Generate unit tests (beta)
Commit to this branch
Create a new PR
Recent review info
Automatic approval policy
Commit: 2c64bdf210cc.
Setting
Effective value
Source
Automatic approvals
Disabled
Override 4
Merge risk
Low
Override 2
Security risk
Not set
Global configuration
Matching overrides (numbered as in the resolved configuration):
Any matching disabled override withholds approval, regardless of risk. Otherwise, the strictest explicitly set limit wins for each score. Limits omitted by all matching overrides inherit the global configuration.
Not set means no limit was configured after inheritance. Use @coderabbitai configuration to inspect the resolved settings and their sources.
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Enterprise
Run ID: a9782fbf-3706-4ee7-89e6-a6b22f33bab0
Commits
Reviewing files that changed from the base of the PR and between 259d227 and 3beb564.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
… migration so its rollback stays image-only
The next release adds template-manager migration
20261031050126_node_dirty_ref_revisions (a revision column on
node_dirty_refs, a singleton counter and two triggers). An image rollback
past it would leave a database that records a migration this runner
doesn't bundle, which it refuses ("unbundled migration"), so
template-manager would crash-loop on every rolled-back node.
Generalize the runner's single retired-migration reversal into a list of
unbundled migrations, each with its version, description, original up SQL
(for the recorded checksum), down SQL and a check that refuses when
reversing would drop data worth keeping. The runner checks every recorded
one (checksum, then that check) before any write, then applies their down
SQL newest first, whatever their order in the list, and removes their
history rows in one transaction, so a failure partway leaves the database
as it was. Other recorded versions it doesn't bundle are still refused by
history validation, which runs after the reversal commits.
20261026220935 keeps its checks, refusal messages and log line; only the
context strings on SQLite failures during the down SQL, history delete
and commit are now generic.
Add 20261031050126 to the list, with its up and down SQL copied
byte-for-byte into db/template-manager/history/ so the checksum matches
what the next release's runner records. Reversing it keeps every outbox
row, so its check never refuses. It gives up the revision itself: a
release without the migration acknowledges rows by key, so a re-enqueue
made while a send is in flight can be lost, as before the migration, and
re-applying restarts revisions at 0. A catalog that bundles the version
applies it normally.
A release that lists a migration compares the recorded checksum only with
its own copy, so the release that bundles the migration must ship the
same up SQL byte for byte. A new test fails once a catalog bundles a
listed version with different up SQL, and db/README.md states the rule.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Template-manager's node-local SQLite database can be migrated by a surge pod while the previous pod still serves. The rule in
crates/libs/sqlite/src/template_manager.rsis that a rollback must need nothing beyond the image. A runner refuses a database that records a migration it doesn't bundle (unbundled migration). So rolling the image back past #1235's20261031050126_node_dirty_ref_revisionswould crash-loop template-manager on every rolled-back node.This PR ships one release before #1235, using the two-release pattern from #1070, so the image a rollback returns to can reverse that migration.
20261026220935becomes a list (UNBUNDLED). Each entry has a version, description, original up SQL (for the recorded checksum), down SQL, and a check that refuses when reversing would drop data worth keeping.20261031050126is added to the list. Its up and down SQL are copied byte-for-byte from template-manager: ack node dirty refs by revision so a re-enqueue during a send survives #1235 at1265e6ebintodb/template-manager/history/. Its checksumc92d5187…matches what template-manager: ack node dirty refs by revision so a re-enqueue during a send survives #1235's migrator records. Reversing it keeps every outbox row, so its check never refuses: it drops only therevisioncolumn, thenode_dirty_ref_revisioncounter and the two triggers. It does give up the revision guarantee: a release without the migration acknowledges rows by key, as before template-manager: ack node dirty refs by revision so a re-enqueue during a send survives #1235, and re-applying restarts revisions at 0.db/accepted-checksums.toml. So if template-manager: ack node dirty refs by revision so a re-enqueue during a send survives #1235 changed its up SQL, even a comment, this release would refuse template-manager: ack node dirty refs by revision so a re-enqueue during a send survives #1235's databases on rollback. A new test,every_listed_migration_matches_the_catalog_that_bundles_it, fails once a catalog bundles a listed version with different up SQL.db/README.mdstates the rule.20261026220935behaves as before. Same checks, same refusal messages, same log line, and its tests pass unchanged. Only the context strings on SQLite I/O failures during down SQL, history delete and commit are now generic: "failed to reverse retired template-manager migration 20261026220935" instead of "failed to reverse retired template cache partitions".db/README.mdgains an "Unbundled template-manager migrations" section: the list, the check-then-reverse transaction, the rule that a new migration joins the list one release before it is bundled and must then ship unchanged, and the in-flight behaviour of20261031050126. The existing "Retired template-manager migration" section is unchanged. The SQLite history paragraph now points to the new section.reverse_retired_partitionsis renamedreverse_unbundled; the only caller is incrates/libs/sqlite/src/lib.rs.Validation
Everything below ran at
1aff5639. The current head differs from it only in doc comments intemplate_manager.rsand prose indb/README.md.Unit tests. With
space-nextest-postgres-setupbuilt first,cargo nextest run -p pplx-space-sqlite -p pplx-space-template-mgr -p pplx-space-db-migrate -p pplx-space-db-migratorran 378 tests: 378 passed, 0 skipped.cargo clippyon the same packages with--all-targets -- -D warningsis clean,cargo fmt --checkis clean, andcargo archreports 0 required findings.template_manager/tests.rsare unchanged and pass. That file only has additions.a_database_with_dirty_ref_revisions_reverses_and_reapplies: next catalog → next catalog (untouched) → this runner (reversed) → next catalog (re-applied) → this runner. Each step checks rows, outbox rows, history, and that the schema equals a fresh database exactly.a_failed_or_modified_dirty_ref_revisions_refuses_reversal: a modified checksum or failed row is refused and the schema is left alone.every_unbundled_migration_reverses_together_or_not_at_all: with both migrations recorded, they reverse together; a non-legacy partition row refuses in the checks, before any down SQL.a_failure_after_the_first_reversal_rolls_back_every_reversal: with both migrations recorded and every check passing, a trigger aborts the older migration's history delete, after the newer migration's down SQL and history delete and the older down SQL have run. Migration fails with the injected error, and rows, outbox, schema and history are unchanged.dirty_ref_revisions_is_the_migration_the_next_release_records: pins the checksum.every_listed_migration_matches_the_catalog_that_bundles_it: has nothing to check here, since this catalog doesn't bundle20261031050126. It fires in template-manager: ack node dirty refs by revision so a re-enqueue during a send survives #1235's CI once template-manager: ack node dirty refs by revision so a re-enqueue during a send survives #1235 rebases. Checked locally by adding the version todb/template-manager/(not committed): with one comment line added to the up SQL it fails on20261031050126, and with a byte-identical copy it passes.a_bundled_schema_is_left_untouched._sqlx_migrations has failed, modified, or unbundled migration 20261031050126.reverse_unbundledcommit after each migration's reversal.a_failure_after_the_first_reversal_rolls_back_every_reversalthen fails:20261031050126was reversed and removed from history while20261026220935stayed. The other 11 template-manager tests, includingevery_unbundled_migration_reverses_together_or_not_at_all, still pass. Without the mutation all 12 pass.Real-binary rollback. I built
space-db-migratorfrom #1235 (1265e6eb), frommain(d397b69d) and from this branch, each in its own worktree and its ownCARGO_TARGET_DIR. A target dir shared across worktrees can produce a stale binary.strings <bin> | grep -c 'reversed newer node dirty ref revision migration'gives 0, 0 and 1. I then ran them in order on one database file:upon a fresh file, then two outbox rows inserted and one re-enqueued. This recorded checksumC92D5187….main's migratorupon a copy fails:unbundled migration 20261031050126.uplogsreversed newer node dirty ref revision migrationand exits 0. History is baseline only, the outbox rows are kept, and the schema matches a freshmaindatabase exactly.upagain changes nothing.upre-applies cleanly, with the same checksum and the same schema as in step 1. Existing rows get revision 0, and a new insert gets revision 1.Rollback on Kind with a real pod. I brought up a fresh cluster from this branch with
SPACE_KIND_TELEMETRY_MODE=otel bash infra/space/kind/scripts/up.sh(Kubernetes substrate; about 5.5 minutes); the pods report this head'sgit_sha. I built #1235's migrator in the repo's bookworm builder image, in its own target dir, to match the node's glibc. I ran it asspace-tmplmgron workerspace-local-worker's live database, beside the serving pod, and it recorded20261031050126. I then deleted the pod. This branch's template-manager reversed the migration at startup (WARN log) and was Ready 6s after the delete, with 0 restarts. Running #1235's migrator again re-applied it, and another restart reversed it again (Ready in 6s).Kind smoke, on the same cluster.
rg -l template space_smoke/scenarios, 23 IDs).smoke-kind-typed.shonly takes categories, so I ran each ID withspace_smoke run <id>after the script's own token and org setup. 21 passed in 16m42s.native.aws_secret_network_credential_injectionwas inconclusive because it needsSPACE_SMOKE_SECRET_ARN.node.windows_sandbox_lifecyclerefuses Kind; it needs a deployed backend.all:SPACE_SMOKE_KIND_COLD_RECEIPTS=<dir> bash scripts/tests/smoke-kind-typed.sh alltook 61m56s, with the run exiting 0.ecw.cold_create_environmentandecw.warm_pool_environment, which need the receipts directory.Severity and risk
Severity: high once #1235 ships. Without this, an automatic canary rollback across it crash-loops template-manager on every rolled-back node.
Risk: the change is runner-only plus history SQL; this release has no schema change. Before #1235 ships, no database records
20261031050126, so the new entry is never used. To roll back, revert this commit. That only matters after #1235 ships: without this commit, a rollback past #1235 crash-loops again.Rollout
During a rollback from #1235's release, the serving #1235 pod still reads
revisionafter this release's migrator reverses it. Its outbox reads and acknowledgements fail until it exits or a migrator re-applies the migration, and its rows stay queued. A send it started before the reversal and acknowledges after a re-apply can match a restarted revision and delete a re-enqueue made during that send. This is reasoned from #1235's code; Kind has no surge migrator to observe it. In the forward direction, this release writes the outbox without namingrevision, so it keeps serving while #1235's migrator applies it.Stack
DIRTY_REF_REVISIONS,dirty_ref_revisions_is_the_migration_the_next_release_recordsandevery_listed_migration_matches_the_catalog_that_bundles_it. Once template-manager: ack node dirty refs by revision so a re-enqueue during a send survives #1235's catalog bundles the version, the entry is never used at runtime, but those tests are what keep template-manager: ack node dirty refs by revision so a re-enqueue during a send survives #1235's file equal to the copy this release compares against.20261031050126's up SQL, comments included. This release would refuse a database that recorded a different checksum. A changed migration needs a new version.REVISIONEDandPARTITIONED_REVISIONED. template-manager: ack node dirty refs by revision so a re-enqueue during a send survives #1235'sBUNDLEDalready contains20261031050126, so both catalogs would list it twice, andBUNDLEDis no longer the release that reverses it. Give those tests a catalog without20261031050126as the rolling-back release; withBUNDLEDin that role,a_failure_after_the_first_reversal_rolls_back_every_reversalstill passes but reverses only one migration. The runner itself already skips any version the catalog bundles.20261031050126"needs the newer release'sspace-db-migrator template-manager down") is no longer true and should be dropped.20261026220935and deletes this module and itslib.rscall. Whichever lands second adapts:RETIRED_PARTITIONSleaves the list; the module, the call andDIRTY_REF_REVISIONSstay.Not covered
native.aws_secret_network_credential_injection: needs a real AWS secret.node.windows_sandbox_lifecycle: needs a deployed backend.AI assistance: Claude Code (an AI agent) wrote the code, tests, docs and this description, and ran the validation.
Summary by CodeRabbit