Repository navigation
fix(server): scope the SSH identity lock per sandbox and bound data-pool lock holders - #4696
Conversation
|
This pull request has had no activity for 14 days and is now marked stale. |
2863a12 to
af95972
Compare
|
/ok to test af95972 |
|
Label |
|
/ok to test af95972 |
The SSH identity and provider-refresh locks each hold their PostgreSQL advisory lock on a data-pool connection, and their holders then query the same pool. At a small database_max_connections, a few holders could take every data connection and wait for one more until sqlx's 30-second acquire timeout, stalling every database request on the replica. Each PostgresStore now has max(1, max_connections / 2) data-pool lock slots. A holder takes a slot before its connection and keeps it until that connection has closed, so every holder can still get a second connection for its queries. A slot that stays busy for the 10-second mutation lock timeout fails with a lock timeout and a warning. The floor rationale in the config docs, man page and startup error now names both holders, and mutation_lock.rs documents the slot-sharing cycle the timeouts break at small pools. Part of NVIDIA#4307. Signed-off-by: Emilien Macchi <emacchi@redhat.com>
SSH host identity provisioning and cleanup took one fleet-wide advisory key and a process-wide mutex, so create, start, restart and delete ran that step for one sandbox at a time across the fleet, with each waiting create holding a lock-pool connection. The SSH identity guard now takes the legacy key in shared mode and then an exclusive key derived from the sandbox id, on one data-pool connection and one data-pool lock slot. The local mutex is keyed by sandbox id too. Steps for different sandboxes run at once, steps for the same sandbox still exclude each other on every replica, and gateways from earlier releases, which hold the legacy key exclusively, keep excluding new ones during a rolling upgrade. One 10-second deadline now covers a data-pool lock holder's slot, its connection and every lock statement, as it already did for mutation locks, so the second lock statement doesn't extend the wait under create's mutation guard. A lock that times out returns a lock timeout. Part of NVIDIA#4307. Closes NVIDIA#4686 Signed-off-by: Emilien Macchi <emacchi@redhat.com>
af95972 to
6e31bef
Compare
Sandbox creation held its PostgreSQL mutation guard while it waited for the SSH identity keys, so each replica admitted at most one in-flight create per lock-pool connection. The per-sandbox SSH identity lock already fences a concurrent delete, so release the guard before that step. Data-pool lock connections were closed on every release, which opened a new PostgreSQL session per create. Unlock and reset an idle session and return it to the pool; close it only when a lock statement may still be waiting. Signed-off-by: divesh <dgude@nvidia.com>
|
/ok to test 9b4dbd5 |
|
Since the SSH identity step now fails with UNAVAILABLE (MUTATION_LOCK_TIMEOUT, 1 s retry hint) after 10 s instead of queuing, should creates retry it automatically? Neither the SDK nor the CLI retries create_sandbox today, so every caller has to add its own retry loop. Do you want that handled in this PR, or as a follow-up on the client side? |
Looked at 9b4dbd5 LGTM. The early guard drop is fenced right (per-sandbox key, plus the reload that aborts if the sandbox went away), and scrubbing the session only when no statement is in flight is the conservative call I wanted.
follow-up, client side. Server-side auto-retry would just re-queue creates under saturation and hide the signal the 10s timeout is there for. That needs its own design IMO. |
Summary
SSH host identity setup took one fleet-wide lock, so create, start, restart and delete ran it for one sandbox at a time across the fleet. The SSH identity and provider-refresh locks also hold a data-pool connection while their holders query the same pool, so a few of them could exhaust a small pool and stall every request on the replica. This PR scopes the SSH identity lock per sandbox and caps those lock holders at half the data pool.
Related Issue
Closes #4686 (needs
state:acceptedfor the issue acceptance check)Part of #4307
Changes
max(1, database_max_connections / 2)slots per replica, so each holder can still get a connection for its queries. A holder keeps its slot until its connection has closed. One 10-second deadline covers the slot, the connection and the locks, and a slot timeout logs a warning.UNAVAILABLEafter 10 seconds when every slot stays busy, like other lock waits. At small pools, refresh executors and creates in the same workspace can briefly wait on each other until those timeouts break it;mutation_lock.rsdocuments it.database_max_connectionsfloor rationale (configuration page, CLI help, man page, startup error), the HA guide, thedebug-openshell-clusterskill and the lock rules inmutation_lock.rs.Testing
mise run rust:lint,rust:format:check,markdown:lint,docs:build:strict,pre-commit,cargo test -p openshell-server --features test-supportandmise run test:rust:postgrespass.Checklist