Skip to content

[db] add node_k8s.instance_id and the node_k8s_superseded_instances table - #1686

Closed
ac-pplx wants to merge 1 commit into
ac/node-k8s-instance-ownerfrom
ac/node-k8s-instance-migration
Closed

ac-pplx wants to merge 1 commit into
ac/node-k8s-instance-ownerfrom
ac/node-k8s-instance-migration

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds node_k8s.instance_id (nullable TEXT) and the node_k8s_superseded_instances table, in db/postgres/20261106232823_node_k8s_instance_id.{up,down}.sql. Nothing writes or reads them until #1676.

The dead-node check finds a node's instance in EC2 by its private DNS name. EC2 forgets a terminated instance about an hour after it terminates, so an old dead node returns no match, as a wrong region, account or name does, and a same-IP replacement reuses the node id and answers for the new instance. Nothing records which instance a node ran on, or which one it ran on before.

node_k8s_superseded_instances keeps each instance a node_k8s row named before a different one replaced it, with the Kubernetes Node UID the row held then (id, node_id, instance_id, k8s_node_uid, superseded_at). Its rows go with their node_k8s row (ON DELETE CASCADE), indexed on (node_id, superseded_at). The node registration store owns it, declared in the previous pull request.

The test fixtures' execution-record-views schema variant carries the new column through its node_k8s view. That view runs with its caller's privileges, so the credential manager's base grant on node_identities gains the column too.

Validation

At this head (CI's test profile, Postgres fixtures required):

  • Migration guards against the previous pull request's head: check-migration-versions.sh, check-postgres-migrations-isolated.sh, check-postgres-migrations-squawk.sh (0 issues) and check-migration-isolation-per-commit.sh: pass.
  • pplx-space-db-migrate privileged group as root: 16 passed, including the 5 execution_record_view_tests.
  • pplx-space-db-migrate, pplx-space-db-test-fixtures, pplx-space-node-registration-store unprivileged: 56 passed, 0 failed.
  • The crates that read node_k8s (pplx-space-api, pplx-space-cluster-controller-server, pplx-space-internal-api-server, pplx-space-node-registration-store), --all-features: 4297 passed, 0 failed, 0 skipped. No code reads node_k8s with SELECT *.

Severity and risk

Severity: does not apply; new evidence storage, not a fix.

Risk: the migration runs with lock_timeout = '1s' and statement_timeout = '5s'. Adding a nullable column without a default changes the catalog only and rewrites no row; the new table starts empty, and its foreign key briefly locks node_k8s. If a lock is not granted within 1 s, the migration fails and nothing is applied. Code deployed today never names the new column. Roll back with the down migration, after rolling back #1676's binaries: it drops the table and the column, and the instances recorded in them.

Rollout

Deploy this migration before #1676's binaries, which write the column and the table.

Stack

On #1685 (the table's owner and the guard change that lets the fixture travel with this migration), which is on #1673. #1676 builds on this one. Merge in that order.

AI assistance: Claude Code made the change, ran the validation and drafted this description. A human reviews before merge.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Explain Change →

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

No actionable comments were generated in the recent review.

Walkthrough

The migration adds nullable instance_id to node_k8s and creates an indexed table for superseded instances. The rollback removes the table and column. The schema fixture adds instance_id to the node_k8s compatibility view and the space_credmgr grant.

Possibly related PRs

Merge Risk: High · previous review 14e8c

The migration adds instance tracking, while the compatibility fixture expands the credential manager’s read access in test databases. Confirm that the grant has the intended scope before merging.

Security architecture risk: Low · up to 14e8c

The immediate exposure is limited: this change prepares storage without adding instance-based control decisions, and the additional read privilege is confined to an opt-in test schema. Rollback permanently removes recorded history. Future identity-update behavior remains outside this change, and incomplete security coverage limits assurance.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — Within the opt-in test variant, space_credmgr can directly select instance_id from node_identities without going through the compatibility view’s Kubernetes-only predicate. The direct-grant pattern predates this PR; instance_id is the newly exposed column. This does not establish production exposure: the variant is test-only, and the production baseline’s node-column grants are unchanged.

Trust Boundaries and Controls

  • observed — The compatibility view continues to execute with caller privileges, selects only managed_by = 'k8s' rows, and uses WITH CASCADED CHECK OPTION for writes through the view. These view controls do not impose the same row predicate on separately granted direct base-table reads.

Resilience and Maintainability Implications

  • inferred — The up migration is transactional rather than marked no-transaction. Under the existing migration runner, a failed transactional file rolls back together with its migration-history entry, and eligible lock-timeout failures can be retried without adopting partially applied schema. This contains interruption during the multi-statement schema addition; it does not verify future registration-write atomicity.
Pre-merge checks: Passed 6

Check name Status Explanation
Title check Passed The title "[db] add node_k8s.instance_id and the node_k8s_superseded_instances table" uses the bracketed area prefix the requirements allow, then a summary of what merging does. The summary starts wit…
Description Follows The Pr Template Passed The description contains all four required items: Summary, Validation, Severity and risk, and an AI assistance line with the tool and its contributions.
Bug Fixes Name A Regression Test Passed The PR adds database storage and updates a test fixture. It does not fix application behavior under crates/, shim/, or ui/, so the check does not apply.
Weakened Safeguards Are Justified Passed The diff does not remove or weaken a test, assertion, lint, CI check, or repository guard. It does not change review guidance.
Agent Tooling Runs No New Commands Passed The diff changes only SQL migrations and a database fixture. It adds no agent tooling commands, server entries, or skill frontmatter.
Public-Safe Content Passed The diff, title, and description contain no content prohibited by this check.
Full details: Title check

Explanation

The title "[db] add node_k8s.instance_id and the node_k8s_superseded_instances table" uses the bracketed area prefix the requirements allow, then a summary of what merging does. The summary starts with the lowercase word "add" and keeps the code identifiers in their original case. It has no trailing period, ticket ID, or internal link. It matches the main change: a nullable instance_id column on node_k8s and a new node_k8s_superseded_instances table. The title meets the stated format and describes the main change.

Recent review info

Automatic approval policy

Commit: 14e8c52c4af0.

Setting Effective value Source
Automatic approvals Disabled Override 3
Merge risk Moderate Global configuration
Security risk Not set Global configuration

Matching overrides (numbered as in the resolved configuration):

  • Override 3 — any author; paths: .coderabbit.yaml, .coderabbit.yml, coderabbit.yaml, coderabbit.yml, .coderabbit.config.ts, .claude/settings*.json, **/.mcp.json, **/hooks.json, .agents/plugins/**/hooks/**, .agents/plugins/**/scripts/**, .agents/plugins/**/commands/**, .agents/plugins/**/agents/**, **/skills/**/hooks/**, **/skills/**/scripts/**, **/skills/**/commands/**, **/skills/**/agents/**, .claude/commands/**, .claude/agents/**, **/marketplace.json, .github/**, ci/**, !ci/scripts/setup-nextest-*.sh, scripts/source_guards/**, scripts/check*, clippy.toml, deny.toml, .gitleaks.toml, .pre-commit-config.yaml, lefthook.yaml, rust-toolchain.toml, .cargo/**, supply-chain/**, pyproject.toml, uv.lock, **/package.json, **/package-lock.json, **/go.mod, **/go.sum, **/requirements*.txt, **/requirements*.in, proto/**, db/**, docs/versioning.md, crates/l40_api/api/src/auth.rs, crates/l40_api/api/src/auth/**, crates/l40_api/api/src/middleware/auth*, crates/l40_api/api/src/middleware/auth/**, crates/l40_api/api/src/routes/authz.rs, crates/l10_foundation/bearer/**, crates/l10_foundation/rustls/**, crates/l10_foundation/guest/**, crates/l10_foundation/pii/**, shim/**, docker/**, !docker/Dockerfile.rust-build-kind, templates/**, infra/space/aws/**, infra/space/cluster/**, infra/space/identity/**, infra/space/observability/**, infra/space/packer/**, infra/space/kind/charts/**, LICENSE, SECURITY.md; automatic approvals: disabled.

Any matching disabled override withholds approval, regardless of risk. Otherwise, the strictest explicitly set limit wins for each score. Limits omitted by all matching overrides inherit the global configuration.

Not set means no limit was configured after inheritance. Use @coderabbitai configuration to inspect the resolved settings and their sources.

Run configuration

Commits

Reviewing files that changed from the base of the PR and between ea9c0c4 and 14e8c52.

Files selected for processing: 3

  • crates/test_support/db/test/fixtures/schema-variants/execution_record_views.sql
  • db/postgres/20261106232823_node_k8s_instance_id.down.sql
  • db/postgres/20261106232823_node_k8s_instance_id.up.sql

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.

@github-actions github-actions Bot added the triage:pending Waiting for a maintainer to triage label Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

@ac-pplx
ac-pplx force-pushed the ac/node-k8s-instance-migration branch from 89d7963 to 14e8c52 Compare November 7, 2026 03:18
@ac-pplx
ac-pplx marked this pull request as ready for review November 7, 2026 03:58
@ac-pplx
ac-pplx added this pull request to stack #1689 November 7, 2026 04:19
@ac-pplx
ac-pplx marked this pull request as draft November 7, 2026 04:30
…able

The dead-node check finds a node's instance in EC2 by its private DNS
name. EC2 forgets a terminated instance about an hour after it
terminates, so an old dead node returns no match, as a wrong region,
account or name does, and a same-IP replacement reuses the node id and
answers for the new instance. Nothing records which instance a node ran
on, or which one it ran on before.

node_k8s gains a nullable instance_id, and node_k8s_superseded_instances
keeps each instance a node_k8s row named before a different one replaced
it, with the Kubernetes Node UID the row held then. Its rows go with
their node_k8s row (ON DELETE CASCADE). The node registration store,
declared as the table's owner in the previous change, writes both in the
next one; until then nothing writes or reads them.

The nullable column has no default, so adding it changes the catalog and
rewrites no row, and the new table starts empty. The migration runs with
a 1 s lock timeout and a 5 s statement timeout. The down migration drops
the table and the column, and the instances recorded in them.

The test fixtures' execution-record-views schema variant carries the new
column through its node_k8s view. That view runs with its caller's
privileges, so the credential manager's base grant on node_identities
gains the column too.
@ac-pplx
ac-pplx force-pushed the ac/node-k8s-instance-migration branch from 14e8c52 to 086d6e7 Compare November 7, 2026 05:06
@ac-pplx

ac-pplx commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Closing for now: this piece is being rebuilt as part of one combined drain v2 build that drops several of the structures it adds. It will come back as part of a smaller set of PRs once that build passes end to end.

@ac-pplx ac-pplx closed this Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage:pending Waiting for a maintainer to triage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant