Skip to content

db-test-fixtures: run the Postgres suites against execution-record compatibility views - #1309

Merged
ac-pplx merged 1 commit into
mainfrom
ac/schema-compat-view-fixture
Nov 5, 2026
Merged

ac-pplx merged 1 commit into
mainfrom
ac/schema-compat-view-fixture

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A planned change renames sandbox_k8s_executions to sandbox_execution_records and node_k8s to node_identities in place, leaving compatibility views under the old names. This change adds the test infrastructure to check, before those migrations are written, that today's code works unchanged through the views. It changes no migration and no production code.

  • Opt-in schema variant. pplx-space-db-test-fixtures applies a candidate DDL file to the migrated test template after the migrations when SPACE_TEST_POSTGRES_SCHEMA_VARIANT=execution-record-views is set. The file is crates/test_support/db/test/fixtures/schema-variants/execution_record_views.sql, marked as a test candidate and not a migration. Unset or empty, the template is unchanged. An unknown name, or a value that is not valid UTF-8, fails template creation before the migrations run, so a typo cannot pass as a variant run. The candidate renames the tables and pod_materialized_at, k8s_node_name and k8s_node_uid, adds the new columns, makes pod_name and pod_namespace nullable, and creates the views. The views keep the old column lists in order, including the three runtime_upgrade_* execution columns from [db] record a stay-stopped runtime upgrade decision on its Kubernetes execution #1304, and use WITH CASCADED CHECK OPTION. It also grants space_credmgr the base column reads the invoker views need. The service logins' privileges on the views come from space_migrator's default privileges and bootstrap-roles, as for any new relation.

  • Test-side fixes. Two api test helpers create row triggers on sandbox_k8s_executions, which Postgres refuses on a view (42809). They now name the table through pplx_space_db_test_fixtures::physical_table(...), which returns the renamed table in variant mode and the old name otherwise.

  • Pins. crates/l30_storage/db/migrate/src/bootstrap_roles/tests/execution_record_view_tests.rs runs statement shapes from today's code and tests as the real bootstrap-roles logins. The server is migrated with control-plane up and the candidate is applied as space_migrator. Shapes covered:

    • targetless ON CONFLICT DO NOTHING … RETURNING, and the retired-generation INSERT … SELECT … WHERE NOT EXISTS
    • the claims' generation allocation MAX(generation) + 1 through the view
    • the node inventory's aliased ON CONFLICT (node_id) DO UPDATE … WHERE node_k8s.k8s_node_name IS DISTINCT FROM EXCLUDED…
    • the check option refusing a non-k8s row
    • the base default filling an omitted managed_by
    • FOR UPDATE OF … NOWAIT and SKIP LOCKED on view joins
    • LOCK TABLE through a view, as the api's create-readiness contention test takes it (it locks view and base)
    • UPDATE … FROM
    • statements prepared before the rename (re-resolved to the views, kept prepared with the same prepare_time, no 0A000)
    • the views' privileges from default privileges alone, for space_user, space_creds and space_ro
    • invoker-view privilege checks for each view separately, including the space_credmgr column grants

    Under nextest they share one bare Postgres server that the bare-postgres setup script (ci/scripts/setup-nextest-postgres.sh --bare; its default mode is unchanged) starts for the run. The first pin bootstraps and migrates a template database on it under an advisory lock, and each pin clones a database of its own from it. They still share cluster-global state: the roles bootstrap provisions and a probe role the template build creates. A pin must not change it; the invoker pin's second bootstrap is safe only because it converges the shared roles to the values the template build gave them. Under plain cargo test each pin starts its own container. They need Docker, so they stay in the nextest privileged group. The LOCK TABLE pin's blocked writes use a 20 ms lock_timeout (was 200 ms): lock_timeout bounds only lock waits, so an unblocked write succeeds whatever the bound.

  • Docs. The fixtures crate README ("Schema variants") and a pointer in docs/rust/space-conventions.md explain how to run the variant. There is no CI job for it: it would repeat the whole Rust test job.

Constraints for the real migrations. The Postgres behaviour in 1 and 2 is pinned on PG16 and PG17; what it means for the production callers comes from reading their code, and 3 comes from reading the candidate:

  1. Invoker views check every base column the view reads. Through a security_invoker view, Postgres checks the caller's base-table SELECT on every column the view's query reads, not only the ones the caller selects. The service logins are unaffected: bootstrap-roles grants space_user and space_creds SELECT, INSERT, UPDATE, DELETE on both base tables, and the credential manager connects as space_creds. A session that holds only the space_credmgr capability role is affected, such as the credential-injection store's resolution test (SET LOCAL ROLE space_credmgr). With its old column grants plus managed_by, its reads through either view fail with 42501. The candidate therefore grants space_credmgr base SELECT on every column the views read, so that role can now read pod_uid and the other base columns directly. The view's column grants still refuse pod_uid through the view. A view that checks base privileges as its owner, the default without security_invoker, would keep that role's old base fence. Every column later added to either table must also be added to the view and to that grant.

  2. Base unique indexes still arbitrate rows the views hide. A targetless ON CONFLICT DO NOTHING arbitrates every unique index on the base table, and the views hide other substrates' rows:

    • with a non-k8s active execution, the execution claim through the view returns no row, even though the view shows no active execution;
    • claims allocate their generation as MAX(generation) + 1 through the view, while (sandbox_id, generation) is unique across substrates, so a hidden row of any state at that generation also makes the claim return no row, and every retry allocates the same generation;
    • the node inventory upsert ON CONFLICT (node_id) DO UPDATE on a hidden node_identities row fails with 44000 (new row violates check option for view "node_k8s").

    In both claim cases, the claims' follow-up read through the view finds no active execution: the api claim fails with failed to create active Kubernetes execution row, and the placement claim's read finds no row. None of this happens today, because no non-k8s rows exist yet.

  3. Nullable pod_name/pod_namespace. Making them nullable means an insert through the view no longer rejects a NULL pod_name. The views' WHERE managed_by = 'k8s' does not restore that. Today's code never writes one.

Validation

At bac756cd (rebased on main past #1407, #1408 and #1426), on a Linux host with Docker, Rust 1.99.1 and nextest 0.9.133, on PG16:

  • cargo nextest run -p pplx-space-db-migrate -p pplx-space-db-test-fixtures -p pplx-space-api --locked --all-features --run-ignored all --ignore-default-filter --retries 1 --no-fail-fast with SPACE_CI_REQUIRE_TEST_DEPS=1, once unset and once with SPACE_TEST_POSTGRES_SCHEMA_VARIANT=execution-record-views: 2602/2602 passed in each mode, no skips, no dependency-skip marker, identical per-test results. python3 ci/scripts/check-test-durations.py target/nextest/default/junit.xml ci/test-duration-budget.toml passes after both. A bogus variant value fails template setup.
  • The pplx-space-db-migrate privileged group, run as root from the build as CI does, 12 threads, three times, then the duration check: the five pins took 0.49-0.66 s each, and the check passed. With a container per pin they took 2.7-4.6 s, and the check failed on all three runs. PG17 (postgres:17-alpine): 16/16. Plain cargo test, a container per pin: 5/5.
  • Each pin's key assertion fails under a mutation of what it pins: execution view column order; node_k8s check option; the LOCK TABLE released before the blocked writes; the execution view's managed_by filter; node_k8s without security_invoker.
  • cargo fmt --all -- --check, cargo check --workspace --all-targets --all-features --locked, and cargo clippy -p pplx-space-db-migrate --all-targets --all-features -- -D warnings: clean. bash ci/scripts/run-archcheck.sh "$(git rev-parse origin/main)" "": no findings.

At this head (ef1f9305, which adds review fixes on top: comments on the shared roles, and the test helper's bare-server handling in pg_test_support.rs, which now errors on a non-UTF-8 server URL and fails, rather than skips, when a published server is unreachable): the migrate privileged group passed 16/16 with the duration check (pins 0.51-0.61 s); plain cargo test 5/5; fmt and clippy clean. CI on GitHub at bac756cd ran the bare setup in the privileged container and the pins in 0.53-0.72 s.

From the rebase across the crate move: when main gained #1304's runtime_upgrade_* execution columns, two pins failed as they should (views_keep_the_old_row_types_and_writes_reach_the_base_tables on the row type, statements_prepared_before_the_rename_run_through_the_views with 0A000 cached plan must not change result type) until the candidate's view and grant carried them.

From an earlier head, before the rebase across the crate move, and not rerun here: the PG17 runs, the full rdeps(pplx-space-db-test-fixtures) comparison over 46 packages on PG16 and PG17 (no per-test difference between modes or versions), clippy, and the checks that each pin's key assertion fails without the property it pins. Before the trigger fix, a PG16 variant run of the api suite had five variant-only failures, all 42809 "sandbox_k8s_executions" is a view, in controllers::sandbox::lifecycle::characterization_tests::kubernetes and controllers::substrate::kubernetes::tests::kubernetes_tests.

Severity and risk

Severity: does not apply (test infrastructure). Risk: with the variable unset the fixtures behave as before, as the default-mode run above shows. The pins add five tests to the privileged CI half, about 0.5-0.7 s each, on one Postgres server the bare-postgres setup script starts for the run; the script's default mode is unchanged. Roll back by reverting the commit.

Rollout

None: no migration and no production code. Mixed-version: does not apply. The candidate DDL runs only in test databases: the shared fixtures apply it when the variable is set, and the five pins always apply it, each in its own database on their shared server (its own container under cargo test).

Not covered

  • The fixture suites run as the superuser, so a variant run checks privileges only where a test switches role (the credential-injection resolution test, as space_credmgr). The pins cover the bootstrap-roles logins.
  • No CI job runs the variant.
  • Once the real rename lands, the default physical_table result names a view and the pins' candidate no longer applies. A crates change must make them work on either schema before that migration, because the migration-isolation guard rejects changes to physical_table, the api test helpers and the pins in a migration PR.
  • The constraints above need decisions before the real migrations; they are tracked separately.

AI assistance: An AI agent (Claude Code) wrote the code, tests, candidate DDL, documentation, and this description, rebased the change across the crate move and onto main past #1407, #1408 and #1426, and restructured the pins to share one server under the test-time gate.

@github-actions github-actions Bot added the triage:pending Waiting for a maintainer to triage label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

@ac-pplx
ac-pplx force-pushed the ac/schema-compat-view-fixture branch 2 times, most recently from 2d4e83d to fd5178f Compare November 4, 2026 05:46
@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Explain Change →

Automatic approval withheld

Automatic approval is disabled by the configured approval policy for this pull request. Review comments and requests for changes continue normally.

Commit: ab3b3333e378.

No actionable comments were generated in the recent review.

Walkthrough

The change adds an opt-in Postgres test schema with renamed base relations and compatibility views. Test fixtures select and apply the schema variant, and tests can resolve renamed physical relations. New migrator tests exercise view behavior and role privileges. The test setup supports a shared bare Postgres server for these tests. Documentation describes variant selection and test commands.

Possibly related PRs

  • perplexityai/SPACE#1310: Changes bootstrap grant convergence for excluded relations, which intersects with this PR’s compatibility-view grant and privilege tests.
Merge Risk: High · up to ab3b3

The compatibility schema changes how the credential-manager role reads the test views. Its privileged tests should be validated before merge. Production database permissions are unchanged.

Security architecture risk: Low · up to ab3b3

The effective exposure is limited to disposable test databases and a separate CI Postgres instance. The candidate schema deliberately broadens credential-manager reads on underlying tables; this does not change deployed permissions or establish that the same grants would be appropriate in production.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The introduced schema and grant changes affect opted-in fixture templates and explicitly transformed migrator-test clones. The full-base comparison and inspected activation paths do not establish a change to deployed database privileges, tenant data access, or production role provisioning.

Trust Boundaries and Controls

  • observed — The candidate uses caller privileges rather than view-owner authority. Its tests assert failures when required base-column grants are revoked, deny a read-only login's DELETE, and check credential-manager reads after bootstrap convergence. These assertions document the intended privilege contract; they are not execution results from this review.

Resilience and Maintainability Implications

  • observed — The shared compatibility-test clones have no per-database deletion path, including when fixture acquisition fails after CREATE DATABASE. Cleanup ownership instead rests with the run-level container: setup errors remove it, and a detached watcher removes it when nextest exits. This permits run-local retention but bounds ordinary cleanup to the disposable test-server lifecycle.
Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title "db-test-fixtures: run the Postgres suites against execution-record compatibility views" starts with a lowercase area and colon, followed by a lowercase summary word, with no trailing period…
Description Follows The Pr Template Passed The description includes all required items: a substantive “## Summary” section, a “## Validation” section with commands and results, a “## Severity and risk” section with severity and risk, and an “A…
Bug Fixes Name A Regression Test Passed PASS: This pull request is test infrastructure, not an application bug fix. The only changes under crates/l40_api are test helpers that resolve the physical table before creating test triggers. The …
Weakened Safeguards Are Justified Passed No changed safeguard is weakened. The diff adds execution-record view tests, assigns them to a separate bare-Postgres setup, and adds them to the existing privileged group. The removed executable line…
Public-Safe Content Passed The title, description, and reviewed diff contain no real credentials, tokens, private keys, internal hostnames or URLs, private tracker/chat/document links, cloud account identifiers, personal workst…
Full details: Title check

Explanation

The title "db-test-fixtures: run the Postgres suites against execution-record compatibility views" starts with a lowercase area and colon, followed by a lowercase summary word, with no trailing period or ticket ID. It matches the configured format. It also describes the main change: the new opt-in execution-record-views schema variant that the Postgres fixtures apply, and the test suites run against it. The raw summary confirms the fixtures crate, the variant SQL, and the compatibility views.

Full details: Description Follows The Pr Template

Explanation

The description includes all required items: a substantive “## Summary” section, a “## Validation” section with commands and results, a “## Severity and risk” section with severity and risk, and an “AI assistance:” line that identifies the tool and drafted or changed parts.

Full details: Bug Fixes Name A Regression Test

Explanation

PASS: This pull request is test infrastructure, not an application bug fix. The only changes under crates/l40_api are test helpers that resolve the physical table before creating test triggers. The new behavior tests are under crates/l30_storage/db/migrate and fixtures, and the description states that no production code or migrations change. Therefore the regression-test requirement does not apply.

Full details: Weakened Safeguards Are Justified

Explanation

No changed safeguard is weakened. The diff adds execution-record view tests, assigns them to a separate bare-Postgres setup, and adds them to the existing privileged group. The removed executable lines only refactor test support and trigger setup; bootstrap behavior and trigger execution remain. Default Postgres setup behavior is unchanged. No test, assertion, lint, CI check, repository guard, allowlist, or skip mechanism is removed or loosened.

Full details: Public-Safe Content

Explanation

The title, description, and reviewed diff contain no real credentials, tokens, private keys, internal hostnames or URLs, private tracker/chat/document links, cloud account identifiers, personal workstation paths, production logs or measurements, incident details, or undisclosed security vulnerability. The PostgreSQL connection strings use test-only templates and environment-provided addresses without embedded passwords. The only IP literal added is localhost test data.

Recent review info

Automatic approval policy

Commit: ab3b3333e378.

Setting Effective value Source
Automatic approvals Disabled Override 4
Merge risk Low Override 2
Security risk Not set Global configuration

Matching overrides (numbered as in the resolved configuration):

  • Override 2 — any author; paths: **, !ui/**; merge risk: Low.
  • Override 4 — any author; paths: .coderabbit.yaml, .coderabbit.yml, coderabbit.yaml, coderabbit.yml, .coderabbit.config.ts, **/AGENTS.md, **/CLAUDE.md, .agents/**, .claude/**, .claude-plugin/**, CONTRIBUTING.md, docs/versioning.md, docs/rust/space-conventions.md, docs/rust/space-review-lessons.md, .github/**, ci/**, scripts/source_guards/**, scripts/check*, clippy.toml, deny.toml, .gitleaks.toml, .pre-commit-config.yaml, lefthook.yaml, rust-toolchain.toml, .cargo/**, supply-chain/**, pyproject.toml, uv.lock, **/package.json, **/package-lock.json, **/go.mod, **/go.sum, **/requirements*.txt, **/requirements*.in, proto/**, db/**, crates/l40_api/api/src/auth.rs, crates/l40_api/api/src/auth/**, crates/l40_api/api/src/middleware/auth*, crates/l40_api/api/src/middleware/auth/**, crates/l40_api/api/src/routes/authz.rs, crates/l10_foundation/bearer/**, crates/l10_foundation/rustls/**, crates/l10_foundation/guest/**, crates/l10_foundation/pii/**, shim/**, docker/**, templates/**, infra/space/aws/**, infra/space/cluster/**, infra/space/identity/**, infra/space/observability/**, infra/space/packer/**, infra/space/kind/charts/**, LICENSE, SECURITY.md; automatic approvals: disabled.

Any matching disabled override withholds approval, regardless of risk. Otherwise, the strictest explicitly set limit wins for each score. Limits omitted by all matching overrides inherit the global configuration.

Not set means no limit was configured after inheritance. Use @coderabbitai configuration to inspect the resolved settings and their sources.

Run configuration

Commits

Reviewing files that changed from the base of the PR and between a466def and ab3b333.

Files ignored due to path filters: 1

  • Cargo.lock is excluded by !**/*.lock, !**/Cargo.lock
Files selected for processing: 14

  • .config/nextest.toml
  • ci/scripts/setup-nextest-postgres.sh
  • crates/l30_storage/db/migrate/Cargo.toml
  • crates/l30_storage/db/migrate/src/bootstrap_roles/tests.rs
  • crates/l30_storage/db/migrate/src/bootstrap_roles/tests/execution_record_view_tests.rs
  • crates/l30_storage/db/migrate/src/bootstrap_roles/tests/postgres_bootstrap_tests.rs
  • crates/l30_storage/db/migrate/src/pg_test_support.rs
  • crates/l40_api/api/src/controllers/sandbox/lifecycle/characterization_tests/kubernetes.rs
  • crates/l40_api/api/src/controllers/substrate/kubernetes/tests.rs
  • crates/test_support/db/test/fixtures/README.md
  • crates/test_support/db/test/fixtures/schema-variants/execution_record_views.sql
  • crates/test_support/db/test/fixtures/src/lib.rs
  • crates/test_support/db/test/fixtures/src/tests.rs
  • docs/rust/space-conventions.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 1 remain after this review.

@ac-pplx
ac-pplx force-pushed the ac/schema-compat-view-fixture branch from fd5178f to 48e1869 Compare November 4, 2026 17:01
@ac-pplx
ac-pplx marked this pull request as ready for review November 4, 2026 17:52
@ac-pplx

ac-pplx commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Action performed

Full review finished.

@ac-pplx

ac-pplx commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Action performed

Full review finished.

A planned change renames sandbox_k8s_executions and node_k8s in place
and leaves compatibility views under the old names. Before writing
those migrations we want evidence that today's code works unchanged
through the views.

Add an opt-in schema variant to the shared Postgres fixtures:
SPACE_TEST_POSTGRES_SCHEMA_VARIANT=execution-record-views applies a test
candidate DDL file kept with the fixtures (schema-variants/, not a
migration) to the migrated template. Unset, the template is unchanged;
an unknown or non-UTF-8 value fails template creation before the
migrations run. physical_table() names the renamed table for test DDL
that only a table accepts. The README runs the variant over every
package that depends on the fixtures,
rdeps(pplx-space-db-test-fixtures).

Against the variant, those suites fail only where two api test helpers
create row triggers on sandbox_k8s_executions, which a view refuses
(42809). Those helpers now target the physical table. With that, no
test result differs between the default and variant runs on PG16 and
PG17, so no production statement these suites run fails through the
views.

Pin the statement shapes the code and its tests depend on under the
real application roles in the migrator's bootstrap tests (PG16 and PG17
via TESTCONTAINERS_POSTGRES_IMAGE): targetless ON CONFLICT DO NOTHING,
generation allocation through the view, the node inventory's aliased
upsert, the check option, base defaults, RETURNING, FOR UPDATE NOWAIT /
SKIP LOCKED on view joins, LOCK TABLE through a view, UPDATE ... FROM,
statements prepared before the rename, the views' privileges from
space_migrator's default privileges, and each invoker view's
base-privilege check.

The five pins share one bare Postgres server per nextest run, which a
--bare mode of setup-nextest-postgres.sh starts. The first pin
bootstraps and migrates a template database on it under an advisory
lock, and each pin clones a database of its own from that. They still
share the cluster-global roles: those bootstrap provisions and the
credmgr_probe login the template build creates. The invoker pin's second
bootstrap rewrites the shared roles while other pins run, which is safe
only because it converges them to the values the template build gave
them. Under a plain cargo test each pin still starts its own container.
A container per pin put them over the 4 s test-time failure bound;
sharing the server keeps each well under the 2 s limit. The LOCK TABLE
pin's blocked writes now wait 20 ms rather than 200 ms each for
lock_timeout: any bound proves the write waited, since an unblocked one
succeeds.

The pins record constraints for the real migrations. An invoker view
checks the caller's base privileges on every column the view reads. The
service logins hold full privileges on both tables, but a session
holding only the space_credmgr capability role needs base SELECT on the
whole view column list, which also lets it read those base columns
directly. And the base unique indexes still arbitrate rows the views
hide: a claim through a view skips on a hidden active row, or on a
hidden row of any state at the generation it allocates through the
view, and the node upsert's DO UPDATE fails the check option on a
hidden row.
@ac-pplx
ac-pplx force-pushed the ac/schema-compat-view-fixture branch from ef1f930 to ab3b333 Compare November 5, 2026 02:15
@ac-pplx

ac-pplx commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Action performed

Full review finished.

@johnlouie-pplx johnlouie-pplx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers from me. The five compatibility tests passed in CI, and the application queries still go through the views rather than bypassing them.

I haven't run Postgres locally. The actual migration and rollback still need their own validation.

@ac-pplx
ac-pplx added this pull request to the merge queue Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage:pending Waiting for a maintainer to triage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants