[ac-pplx]

Summary

Narrow behaviour change. Node-publication ingest read PublishedSandboxState.status two ways. The Rust paths (transition-receipt check, quarantined report, state_changed_at check, the projection prefilters) trimmed it and compared it ignoring ASCII case. Ten SQL statements applied LOWER(TRIM(..)) under the database collation. Under a libc UTF-8 collation such as the test fixture's en_US.utf8, LOWER('FAİLED') = 'failed'. So a node publishing "FAİLED" moved a k8s row to failed and opened a failure episode, while the Rust paths treated the same text as unknown. A Turkish-collated database would lower-case plain FAILED differently again.

The status is now parsed once per published state into PublishedStatus: Known(StoredSandboxLifecycle) after trimming and ASCII case folding, or Unrecognized(String). Every Rust decision reads it. The ten ingest statements that normalized the status text now compare the canonical published_lifecycle that the publish path serializes (NULL when absent or unrecognized). Ingest no longer lets the database collation decide what a node reported.

Changed outcome: a non-ASCII status such as "FAİLED" is unrecognized and handled like any other unrecognized status. Under a collation that lower-cases İ to i, main read it as failed. Now it no longer moves a row to failed or opens a failure episode, and it closes a failure episode that is already open, so the public API no longer reports failure_code: runtime_failure for that sandbox. Current nodes publish only enum labels, so this needs a non-SPACE or faulty publisher. ASCII spellings reach the same decisions as before.

Unchanged:

• the trimmed original spelling written to sandbox_observed_states.status;
• the stored-column LOWER(TRIM(s.lifecycle)) checks, which wait on a census of stored values;
• the three non-canonical stored-lifecycle rows from test: one stored-lifecycle matrix for the sandbox row writers #695.

Adds SandboxTransitionTargetState::lifecycle() for the receipt check.

sql! text changes (recordset column status → published_lifecycle, and LOWER(TRIM(input.status)) → input.published_lifecycle):

• published_state.rs: failure-episode insert, failure-episode clear, local-suspend handoff (adds the column; WHERE input.status IS NOT NULL is kept)
• ingest.rs: TRANSITION_RECEIPT_RECORD_COLUMNS, mark_published_failed, converge_published_restoring_to_started, converge_published_stopped, release_completed_activations
• sandbox_quarantine_store.rs: select_activation_rollback_candidates, recover_pause_failed_from_inventory (SELECT list and WHERE), select_suspended_projection_candidates

Validation

Test expectations:

• Changed: two new tests pin the changed outcome. Both fail with main's production code and pass here.
  • published_status::non_ascii_failed_spelling_is_unrecognized: on main, left: "failed", right: "started".
  • published_status::non_ascii_failed_spelling_closes_an_open_failure_episode: an ASCII failed opens the episode, then "FAİLED" closes it. On main the episode stays open.
• New, equivalence: one Postgres sweep per changed statement (10) over ASCII spellings: canonical, upper, padded with spaces/tabs/CR/LF/VT/FF, and near misses. All 10 pass with main's production code and here. A case-sensitive parse fails all 10.
• New, unit: PublishedStatus parse tests for canonical values, padding, case, blank and unknown strings.
• Fixture key only: published_state_lifecycle_race_tests.rs renames its hand-built JSON key status → published_lifecycle. Expectation unchanged.
• lifecycle_writers.rs and status_tests.rs are unchanged.

Commands:

• cargo fmt --all -- --check, scripts/check-rust-error-policy.sh, cargo clippy --workspace --locked --all-targets --all-features -- -D warnings: pass.
• cargo arch: 0 required findings, no sandbox_writes growth. A local rebase onto current main has no findings.
• SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run -p pplx-space-internal-api-server --all-features: 453 of 453 pass, Postgres required, none skipped.
• The wider runs below used this head's production code (only doc comments differ) and every test here except non_ascii_failed_spelling_closes_an_open_failure_episode:
  • SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --all-features for common-core, internal-api-server, sandbox-store, placement and every dependent (internal-api-svc, api, template-mgr, build-mgr, cluster-controller-server, snapshot-retention): 4897 of 4899 pass, Postgres required. The other 2 (template-mgr NoCoW tests) need the btrfs fixture and pass under ci/scripts/with-nocow-test-filesystem.sh. 1 #[ignore] benchmark was not run.
  • Fresh Kind, SPACE_KIND_TELEMETRY_MODE=otel, CI's admin-token and cold-receipt directories: up.sh 8m24s. smoke-kind-typed.sh all 65m55s: ECW 91/91 and native 42/42 passed, 0 failed or inconclusive, no reruns. Includes ecw.cold_create_environment and local.telemetry_reaches_local_backends. 6 console scenarios were skipped (no Playwright); 6 declared-ignored.

Not validated: an end-to-end Kind reproduction. Real nodes cannot publish a non-ASCII status, so only the Postgres test shows it.

Severity and risk

Severity: low. Fixes a collation-dependent disagreement on inputs no SPACE node produces.

Risk: a publisher sending a non-ASCII status no longer gets fail/episode handling where the collation folded it to failed. ASCII behaviour is pinned per statement. Rollback: revert; no migration or schema change.

Rollout

Nothing persisted or on the wire changes. published_lifecycle is built and read inside one internal-api process; nodes and protos are untouched. During a rolling deploy, old and new replicas are each self-consistent and differ only on non-ASCII statuses.

Not covered

Readers of the stored spelling outside ingest are unchanged. The public API's has_current_failure still lower-cases sandbox_observed_states.status, but it also requires the failure episode, which ingest now opens only for a status that parses as failed.

Overlaps open #1399 (published_state.rs, sandbox_quarantine_store.rs, ingest.rs; no status-comparison changes), #1406 (ingest.rs), #1204 (status.rs, unchanged here) and #319 (common/core/src/types/sandbox.rs). These conflicts are textual at most, with one exception. #1399 also adds a_revert_leaving_its_execution_active_takes_the_activation_its_node_completed (in crates/libs/api/src/controllers/substrate/kubernetes/store/k8s_store_postgres_tests/restore_revert_tests.rs). That test calls release_completed_activations with a hand-built recordset keyed status. After this PR the key must be published_lifecycle: a missing key reads as NULL, the receipt matches nothing, and the test's retirement-off case fails. Whichever of the two PRs lands second renames the key.

AI assistance: Claude Code wrote the code, the tests and this description; a human chose the behaviour change.

[coderabbitai]

Automatic approval withheld

Assessment	Current risk	Maximum for approval
Merge risk	High	Low

Why: Unrecognized statuses can now clear failure episodes, and lifecycle write guards have changed. The rollout carries high risk.

Commit: 864947b86bfa.

No actionable comments were generated in the recent review.

Walkthrough

Published sandbox statuses are parsed into recognized lifecycle values or unrecognized text. Lifecycle-dependent receipt validation, failure handling, quarantine, projections, and convergence use parsed or canonical lifecycle values. Published lifecycle recordsets use exact values, while stored-lifecycle checks retain their existing behavior. Added tests cover status spelling variants, unrecognized input, receipts, recovery, and suspend flows.

Possibly related PRs

• perplexityai/SPACE#1399: Updates activation-completion receipt selection and lifecycle projection in the same receipt-related data flow.
• perplexityai/SPACE#1420: Uses the published_lifecycle contract for activation rollback and pause-failed recovery decisions.

Unblocks: 1 PR

Merge Risk: High · up to 86494

Unrecognized statuses can now clear failure episodes, and lifecycle write guards have changed. The rollout carries high risk.

Security architecture risk: Low · up to 86494

Status interpretation becomes consistent across publication and storage without weakening the reviewed node-ownership or transition-receipt controls. The main behavioral risk is the intentional change in how unrecognized spellings affect lifecycle transitions and existing failure indicators.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — A faulty or compromised authenticated publisher can influence lifecycle and failure observations for multiple sandboxes within its existing node-assignment authority. The inspected changes do not expand that authority to other nodes: failed projection, activation completion, quarantine recovery, and failure-episode updates retain node-ownership predicates.

Trust Boundaries and Controls

• observed — A matching canonical label alone cannot release an activation attempt. SQL additionally requires node ownership, an active execution, matching execution generation and activation attempt, matching target state, and a route activation. Terminal and quarantine lifecycle exclusions remain in the final sandbox update. Pause-failed recovery likewise retains reason/source restrictions and locks both sandbox and quarantine rows.

Pre-merge checks: 1 · 4

Failed checks: 4 inconclusive

Check name	Status	Explanation	Resolution
Description Follows The Pr Template		Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or task execution failures.
Bug Fixes Name A Regression Test		Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or task execution failures.
Weakened Safeguards Are Justified		Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or task execution failures.
Public-Safe Content		Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or task execution failures.

Passed checks: 1 passed

Check name	Status	Explanation
Title check		The title starts with the lowercase area "internal-api:" and a lowercase summary. It accurately describes the main change and has no trailing period, ticket ID, or internal link.

Recent review info

Automatic approval policy

Commit: 864947b86bfa.

Setting	Effective value	Source
Automatic approvals	Enabled	Global configuration
Merge risk	Low	Override 2
Security risk	Not set	Global configuration

Matching overrides (numbered as in the resolved configuration):

• Override 2 — any author; paths: **, !ui/**; merge risk: Low.

Any matching disabled override withholds approval, regardless of risk. Otherwise, the strictest explicitly set limit wins for each score. Limits omitted by all matching overrides inherit the global configuration.

Not set means no limit was configured after inheritance. Use @coderabbitai configuration to inspect the resolved settings and their sources.

Run configuration

• Source mode: Auto
• Repository: perplexityai/SPACE
• Branch: ac/lifecycle-ingest-status-parse
• File: .coderabbit.yaml
• Commit: 864947b
• Layers: Repository YAML; Enforced overrides: Organization
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 556889f4-b9fa-44dd-a2f2-ebcaa3229130

Commits

Reviewing files that changed from the base of the PR and between 4449fcd and 864947b.

Files selected for processing: 10

• crates/l30_storage/sandbox/store/src/ingest.rs
• crates/l40_api/internal/api/server/src/published_state.rs
• crates/l40_api/internal/api/server/src/published_state/input.rs
• crates/l40_api/internal/api/server/src/published_state/published_status.rs
• crates/l40_api/internal/api/server/src/published_state/published_status_tests.rs
• crates/l40_api/internal/api/server/src/published_state_lifecycle_race_tests.rs
• crates/l40_api/internal/api/server/src/published_state_tests.rs
• crates/l40_api/internal/api/server/src/published_state_tests/published_status.rs
• crates/l40_api/internal/api/server/src/sandbox_quarantine_store.rs
• crates/libs/common/core/src/types/sandbox.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

[github-actions]

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

[ac-pplx]

@coderabbitai full review

[coderabbitai]

Action performed

Full review finished.

[ac-pplx]

On the review summary's "unrecognized statuses can now clear failure episodes": that already happens on main, and this PR leaves it unchanged except for one collation-dependent case.

• Clear rule on main: the failure-episode clear runs when LOWER(TRIM(input.status)) IS DISTINCT FROM 'failed'. Any published status that doesn't lower-case to failed closes an open episode, an unknown word or a blank status included. "A publication that doesn't say failed ends the episode" is the existing rule.
• Clear rule here: the same rule over the parsed value, input.published_lifecycle IS DISTINCT FROM 'failed'. published_lifecycle is NULL for an unrecognized or absent status, so those still close the episode, exactly as on main. The insert side is symmetric. Only a status that parses as failed opens or keeps an episode.
• What does change: text outside ASCII that the database collation lower-cases to a lifecycle, for example FAİLED. On main it read as failed and opened or kept an episode. Here it is unrecognized, so it doesn't. That is the narrow change the description calls out, and non_ascii_failed_spelling_is_unrecognized and non_ascii_failed_spelling_closes_an_open_failure_episode pin it. Nodes publish lifecycle names from a fixed ASCII vocabulary, so a real node doesn't send such text.
• Lifecycle write guards: the ten ingest statements now compare the canonical parsed lifecycle instead of LOWER(TRIM(status)). Whitespace was already trimmed in Rust before the JSON on main. One Postgres-backed sweep per statement, over canonical, upper-case, padded and near-miss ASCII spellings, passes on both main and this head.

So nothing that a node actually publishes gets a new state change from this PR; only collation-folded non-ASCII text is read differently.