0.3.75: adopt ciris-server v0.5.206 — the unbrick - #87
Merged
Merged
Conversation
…that proves our CI blind
Substrate: persist v44.0.0 -> v44.1.1 / edge v23.0.0 -> v23.1.1; verify
v15.1.0 and leviculum v0.26.0+ciris.1 unchanged.
0.5.205 — which this repo tagged yesterday as 0.3.74 — could not open a
database that had history. CIRISPersist#840: a one-word edit inside a
SQL COMMENT in migration V070 changed its refinery checksum, so any node
that applied V070 before 2026-09-09 failed validation inside
`Engine::with_*` BEFORE ANYTHING BOUND. persist v43.0.0 through v44.1.0
are all in that range. It took the canonical down ~7 minutes and
crash-looped it 16 times. v44.1.1 reverts V070 to its v42.1.0 bytes
(byte-identical upstream, both dialects), normalises the history row for
nodes created under the broken releases, and pins every migration
checksum so the class cannot recur.
THIS NODE WOULD HAVE BRICKED, AND THE CHECKSUM SAYS SO RATHER THAN AN
ARGUMENT. Read from prod:
sqlite3 'file:/data/cirisstatus/data/ciris_engine.db?mode=ro' \
"select version,name,checksum from refinery_schema_history where version=70;"
70|ceg_018_at_rest_blob_key_grants|7163486563775091993
That is the SAME checksum upstream cites for the canonical, so their
both-directions proof (0.5.205 -> RuntimeError building the Engine;
0.5.206 -> opened, refinery at 143, nothing to apply) covers this home
verbatim. 0.3.74 survived contact with prod only because it was never
deployed — prod is on 0.3.70 / persist v42.1.0, below the broken range.
The deploy recommended the night before this tag would have taken the
status page down.
AND OUR CI IS BLIND TO THE WHOLE CLASS, not just to this instance. A
fresh database applies the current migration and records the current
checksum, so there is nothing to mismatch; the failure cannot be
observed by a node that starts from an empty data dir. Upstream's 122
integration binaries and both ladders were green on the broken pin, and
so was our `image-runtime-smoke` — it boots the release image on a clean
volume, which is exactly the shape of test that cannot see this. Every
upgrade this node performs is an upgrade onto existing state and every
test we run is a test of a node with none. Filed as CIRISStatus#86.
edge v23.1.1 (CIRISEdge#586/#581): `chat::Body` gains `Pointer`; no chat
surface here, and `BlobChunkVerifier` has no call sites in this crate.
Verified: cargo build --release --locked clean; 131 tests green; clippy
--all-targets -D warnings clean; fmt clean. v0.3.74 has been retitled
DO NOT DEPLOY with the reason, since `latest` pointed at it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HhKY2YZNH2LzNTQVwgjnBh
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adopts ciris-server v0.5.206. Substrate: persist v44.0.0 → v44.1.1, edge v23.0.0 → v23.1.1; verify v15.1.0 and leviculum v0.26.0+ciris.1 unchanged.
What 0.5.205 did
It could not open a database that had history. CIRISPersist#840: a one-word edit inside a SQL comment in migration V070 changed its refinery checksum, so any node that applied V070 before 2026-09-09 failed validation inside
Engine::with_*before anything bound. persist v43.0.0 through v44.1.0 are all in that range — four releases. Upstream: ~7 minutes down, 16 crash-loops on the canonical.v44.1.1 reverts V070 to its v42.1.0 bytes (verified byte-identical upstream, both dialects), normalises the history row for nodes created under the broken releases, and pins every migration checksum so the class cannot recur.
This node would have bricked — the checksum says so, not an argument
Read from prod:
That is the same checksum upstream cites for the canonical, so their both-directions proof applies to this home verbatim — 0.5.205 →
RuntimeError: build shared persist Engine, 0.5.206 → engine opened, refinery at current version, nothing to apply.0.3.74 (tagged yesterday, pinning 0.5.205) survived contact with prod only because it was never deployed: prod is still on 0.3.70 / persist v42.1.0, below the broken range. The deploy I recommended the night before this tag would have taken the status page down.
v0.3.74has been retitled DO NOT DEPLOY with the reason, sincelatestpointed at it. This PR moveslatestto a build that opens the corpus.Our CI is blind to the class, not just the instance
A fresh database applies the current migration and records the current checksum, so there is nothing to mismatch — the failure is structurally invisible to any node starting from an empty data dir. Upstream's 122 integration binaries and both ladders were green on the broken pin. So was our
image-runtime-smoke: it boots the release image on a clean volume, which is precisely the test shape that cannot observe this.Every upgrade this node performs is an upgrade onto existing state; every test we run is a test of a node with none. Filed as #86 (one extra container start: boot clean, stop, boot the new image on the same volume, assert
/health).Other changes
edge v23.1.1 (CIRISEdge#586/#581):
chat::BodygainsPointer— group content moves to the room's encrypted blob store and the row carries a pointer. No chat surface here;BlobChunkVerifierhas no call sites in this crate.Verification
cargo build --release --locked— cleancargo test --locked— 131 passed, 0 failedcargo clippy --all-targets --locked -- -D warnings— cleancargo fmt --check— clean🤖 Generated with Claude Code
https://claude.ai/code/session_01HhKY2YZNH2LzNTQVwgjnBh