Repository navigation
fix(runtime): graceful shutdown, fail fast on a lost worker, refuse public listeners (#31, #148, #30) - #154
Merged
Merged
Conversation
Verified on this branch (Ballista 55.0.0, static planner per #134), real binaries on 127.0.0.1: - `kill -TERM` on oxide-scheduler: status 143, every running query lost. Ballista's scheduler installs no handler (axum::serve, no shutdown). - `kill -TERM` on oxide-worker: exit 0. Ballista's executor already fences itself, tells the scheduler (which re-runs its tasks elsewhere) and waits for its tasks, but with no bound: a task that does not finish keeps the worker up until it is killed. The issue's "no signal handler" is true of the scheduler only. - No flag reached task_max_failures, stage_max_failures, the executor timeout or the heartbeat interval. Changes: - `shutdown` module: ShutdownSignal (SIGTERM/SIGINT, listening from startup so an early signal is held), Stopped { Drained, Abandoned }. - `cluster::run_scheduler_with_shutdown`: on the signal, new queries fail in the session builder ("this scheduler is shutting down ..."; it is called for every query and nothing a running one needs), the running ones are waited for up to --drain-timeout by polling the job state, plus a one-second linger so their clients hear they finished; then exit 0. - `cluster::run_executor_with_shutdown`: Ballista's own graceful stop, bounded by --drain-timeout; past it the worker exits 0 at once (process::exit: dropping the runtime would wait for the tasks). A second signal stops either at once. - Scheduler flags --task-max-failures, --stage-max-failures (0 refused, 1 = no retry), --worker-timeout (cluster::with_worker_timeout also sets Ballista's dead-worker check to a third of it, at most 15 s, and the no-worker grace to it, as Ballista's binary does), --drain-timeout (20). Worker flags --heartbeat-interval and --drain-timeout (20). Defaults are Ballista's (unit-tested), so a deployment that sets none is unchanged. Worker loss, measured (2 cpu workers x 2 tasks, 600k-row fixture, self-join on k, `kill -9` as the victim starts its first join task): - With Ballista's defaults the query came back WRONG in 9 of 12 runs (7 of 8 from a shell loop, 2 of 4 of the committed test): the 100 groups plus 13 of them again (two hash partitions twice), exit 0; the same query without a kill is byte-equal to embedded. - A rollback leaks the survivors' task slots: a cancelled task's status arrives once its stage is UnResolved, sum_vcores_for_statuses refunds 0 for it, and with no slot left the query waits for a new worker (one run: 670 s, ended when I started one). SIGTERM on a worker mid-query takes the same path: 2 of 2 runs hung. - With --stage-max-failures 1 the first failed fetch (Ballista retries it 3 times, 3 s apart) fails the query before any rollback, naming the worker's address; the next query runs on the worker left. Both bugs are in Ballista's scheduler, outside our hooks, so this documents them (README Cluster mode, STATUS decision, CHANGELOG known issue) and recommends --stage-max-failures 1 and stopping workers between queries. Tests (tests/cli.rs, real processes and signals, all on 127.0.0.1): - sigterm_drains_a_scheduler_and_exits_zero: a new query is refused naming the shutdown, the running join finishes equal to embedded, exit 0. - sigterm_stops_a_worker_within_its_drain_timeout_with_exit_zero: idle worker exits 0 at once and stops counting; a busy one with --drain-timeout 1 exits 0 after 1..10 s, logging the timeout. - a_query_that_loses_a_worker_fails_naming_it_when_stage_retries_are_off. - a_silent_worker_stops_counting_after_the_worker_timeout (a 5 s timeout over a 1 s heartbeat: Ballista keeps heartbeat times in whole seconds). - a_query_that_loses_a_worker_comes_back_with_the_right_rows: what the issue asked for; #[ignore]d as the reproduction of the Ballista bug (fails in about half its runs, not every one). - the_resilience_flags_appear_in_help; unit tests for the drain, the refusal, and the flags reaching SchedulerConfig. Broken to watch each go red: no drain bound (busy worker: no timeout line), draining never set (the new query ran), no drain wait (scheduler gone before the refusal), --stage-max-failures not applied (the query "recovered"), --heartbeat-interval not applied (live worker expired too: not ready 0/1), --worker-timeout not applied (dead worker still counted). Refs #148 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Verified on this branch: `oxide-scheduler --bind-host 0.0.0.0` started and listened on every interface without a word, although its port, like the workers', has no authentication or encryption (SECURITY.md: whoever connects can run queries that read any file a worker can read). The same held for --metrics-host. oxide-worker had no --bind-host at all: it listened where Ballista's ExecutorProcessConfig default puts it, 127.0.0.1, so a scheduler on another machine could not reach it. The first half of the issue, the enforced private-network posture; TLS and authentication are not built (see below), so #30 stays open for them. - `exposure` module: an address is Loopback, Private (RFC 1918, IPv6 unique-local, link-local; IPv4-mapped IPv6 judged as its IPv4) or Public (anything else, 0.0.0.0 and :: included, and the carrier-shared 100.64/10). check_listen_address refuses Public unless --allow-insecure-bind, judges a hostname by the widest address it resolves to, and refuses one that does not resolve. A bracketed IPv6 host ([::1]), the way the scheduler's and Ballista's `host:port` parsing needs one written, is judged without its brackets (broken once: [::1] was refused as a name that does not resolve). - oxide-scheduler and oxide-worker check --bind-host, and --metrics-host when --metrics-port is set, before anything listens, exit 1 naming the risk and the two ways out, and warn at startup whenever they listen beyond loopback. oxide-worker gains --bind-host (default 127.0.0.1, as before) under the same rule. - README (flag tables and a "Production deployments" paragraph) and SECURITY.md (trust model, the same paragraph word for word), CHANGELOG (Security), STATUS (decision). After: `oxide-scheduler --bind-host 0.0.0.0` exits 1 at once with "--bind-host 0.0.0.0 would listen where any network that routes to it can connect, and OxideLake's cluster ports have no authentication or encryption ... pass --allow-insecure-bind if a firewall or an authenticating proxy already decides who can connect (SECURITY.md)". Tests: a_public_or_wildcard_listener_is_refused_without_the_flag (assert_cmd: scheduler 0.0.0.0, ::, 203.0.113.7, public --metrics-host; worker 0.0.0.0 and public --metrics-host; both --help name the flag). Every case passes --port 1, which an unprivileged process cannot bind, so a regression fails on the bind instead of listening anywhere. Broken once (the check never refuses): red, the scheduler reached the bind and failed with "Permission denied". Unit tests classify the range edges (172.15/172.32, 192.169, 100.64, mapped addresses) and check the flag. No test listens beyond 127.0.0.1. Not done, proposed for the rest of #30: Ballista 55 can make its clients speak TLS (ballista.client.use_tls, plus an endpoint override for certificates) but serves the scheduler's port and the executors' gRPC port in plain text with no hook; only the executors' Flight service can be replaced. TLS therefore needs either an upstream server-side TLS option (tonic's ServerTlsConfig on create_grpc_server and the scheduler's listener) or a terminating proxy per port, with every OxideLake process told to dial TLS (use_tls on clients, the scheduler's use_tls for its proxy, the endpoint override for a private CA), tested against a local mTLS-terminating proxy. Refs #30 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Ballista 55's recovery from a lost worker rolls back every stage that read the worker's shuffle output and runs it again, and in 55.0.0 that recovery returns duplicated rows with exit 0 (fixed upstream by apache/datafusion-ballista#2677, merged 2026-10-28, unreleased) or never finishes (the tasks the rollback cancels never return their slots; see apache/datafusion-ballista#2539 and #2698). Decision for 0.3.0: correctness over availability. Reproduction (this branch before this commit; debug build, deps at -O2): - fixture: `oxide gen-data --rows 600000 --row-group-rows 4096 --out D` (seed 42, zstd: D/t.parquet, 12,301,854 bytes, 16 scan partitions). - cluster on 127.0.0.1: `OXIDE_CLUSTER_BACKEND=cpu oxide-scheduler --port P` and two `oxide-worker --scheduler-host 127.0.0.1 --scheduler-port P --port F --grpc-port G --concurrent-tasks 2 --work-dir W`, RUST_LOG=info. - query: `oxide sql --cluster df://127.0.0.1:P --table t=D -q "SELECT a.k, COUNT(*) AS n, SUM(a.v + b.v) AS sv FROM t a JOIN t b ON a.k = b.k WHERE a.id < 200000 GROUP BY a.k ORDER BY a.k"` (four stages: 1 and 2 scan, 3 joins, 4 aggregates; embedded prints 100 groups, 104 lines). - the kill: `kill -9` on the second worker as soon as its log shows its first join task, `Execute task : [TID <job>/3.0/<partition>.0]`, polled every 20 ms. - with Ballista's stage_max_failures 4: 12 of 15 runs (7 of 8 from a shell loop, 5 of 7 of the test) printed the 100 sorted groups plus 13 or 14 of them again (whole hash partitions twice), exit 0. The same query with no kill is byte-equal to embedded. - with --worker-timeout 10 and --heartbeat-interval 1 one run waited 670 s, until I started a third worker: the cancelled join task's status was "Ignored ... as the Stage ... is in UnResolved status", so sum_vcores_for_statuses refunded its slot to nobody. - `kill -TERM` instead of -9 hung in 2 of 2 runs, and still hung (150 s) with --stage-max-failures 1. Change: --stage-max-failures defaults to 1 (cluster::DEFAULT_STAGE_MAX_ FAILURES, also in cluster::scheduler_config). The first failed fetch from a lost worker, what usually happens first when a worker is killed, then fails the query, naming the worker's address. Ballista's own default is 4; an operator can raise it, and --help, README, SECURITY.md, STATUS and the CHANGELOG (Changed, with a migration line) say that brings the bug back. What this does NOT cover, said in the same places: only a failed fetch counts toward the limit. A worker stopped with SIGTERM, or silent past --worker-timeout, is rolled back through Ballista's ExecutorLost with count_to_failures = false, whatever the limit, so a query running then can still return duplicated rows or hang; the docs say to stop workers between queries. The next commit closes those paths. Tests: a_query_that_loses_a_worker_fails_naming_it now runs at the defaults (kill -9 as above): exit non-zero, empty stdout, the worker's Flight address in stderr ("Stage 3 has failed 1 times ... Error connecting to ... http://127.0.0.1:<port>", about 11 s after the kill: Ballista retries the fetch 3 times, 3 s apart), and the next query equal to embedded. The #[ignore]d a_query_that_loses_a_worker_comes_back_with_the_ right_rows runs at --stage-max-failures 4 and points to #148 and apache/datafusion-ballista#2677; it failed 3 of 3 runs, as intended. The scheduler's unit test pins the default against Ballista's; set back to 4 it went red (left 4, right 1), and with the limit at 4 the kill test's query "recovered" instead of failing. Refs #148 Refs #31 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
The previous commit's --stage-max-failures 1 covers a failed shuffle fetch, what usually happens first when a worker is killed. It does not cover a worker that is stopped (SIGTERM: Ballista's executor_stopped) or that goes silent past --worker-timeout: those reach Ballista's ExecutorLost, which rolls the query back with count_to_failures = false, whatever the limit. Measured on the previous commit, same setup as there (fixture, two workers, the self-join, the signal as the worker's first join task starts): `kill -TERM` hung, killed by a 150 s timeout, no rows. Change, inside OxideLake's hooks only (no Ballista code copied): - oxide-scheduler builds Ballista's scheduler itself (create_scheduler + start_grpc_service, what start_server does) and keeps a handle to it. - worker_loss::WorkerLoss: every removal of a worker passes through our cluster state's remove_executor, and Ballista posts ExecutorLost only after it returns. There, after removing the worker, it cancel_job()s each running query with a finished task on that worker, so the cancellation is queued ahead of the rollback and Ballista's own abort refunds the task slots. A query whose tasks on the worker had finished nothing is left to Ballista: no stage is rolled back, the tasks run again elsewhere. - worker_loss::ReportingJobState: Ballista's in-memory job state, saving such a query's failure as "worker <id> (<host:port>) was lost while this query ran, and it held shuffle output the query still needed ... (#148). Run the query again" where Ballista says "Cancelled". - --help, README, SECURITY.md, STATUS and the CHANGELOG now say every loss path fails the query; the in-process start_standalone cluster keeps Ballista's defaults (its builder hard-codes them, its workers live and die with the process). After, every case exits 1, prints no rows, and names the worker: - kill -TERM mid-join: at once; the stopped worker exits 0. - silent (kill -9, --worker-timeout 5, --heartbeat-interval 1): about 5 s after the kill, 2 runs of 2. - the #31 drain on a busy worker (the only worker, --drain-timeout 1): the query fails at once instead of waiting on Ballista's no-worker grace. In each case the next query on the workers left is byte-equal to embedded. Tests (tests/cli.rs, real processes on 127.0.0.1): a_query_whose_worker_stops_fails_naming_it, a_query_whose_worker_goes_silent_fails_naming_it, and the drain test sigterm_stops_a_worker_within_its_drain_timeout_with_exit_zero now also requires its query to fail naming the worker. Broken once (the hook not called): the stop and silent tests, "`query` was still running after 120s"; the drain test, "after 60s". With the reason not rewritten the client saw "Job ... failed: Cancelled". Refs #148 Refs #31 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
`make test` on 61f455f failed a_query_that_loses_a_worker_fails_naming_it in wait_ready(2): w1 registered, then "Could not start built-in arrow flight server ... Address already in use (os error 98)" on the port free_port() had picked, and it stopped. An earlier full run failed another cluster test the same way. free_port() bound 127.0.0.1:0 and let the port go; that range (ip_local_port_range, 32768-60999 here) is where the kernel also takes source ports for outgoing connections, so on a busy machine a connection reused the port before the worker bound it. It now hands out ports from 20000-31999, which no one but this function hands out: below the ephemeral ranges of Linux (32768) and macOS (49152). Each call in the binary gets a different port, starting from an offset in the process id, and each is checked free by binding it. Two full runs of tests/cli.rs after: 24 passed, 3 ignored, then 23 passed and 1 failed for a different reason (the next commit). Refs #31 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
A run of tests/cli.rs on a579a5a failed a_query_whose_worker_goes_silent_fails_naming_it although the query did what it should: it failed at once, printed no rows, and nothing hung. The error named no worker: "Job ... failed: Job failed due to stage 3 failed: Task failed due to runtime execution error: ... IpcError(\"... h2 protocol error: error reading a body from connection\" ... broken pipe)". The worker was killed while it was sending a shuffle block to a join task on the other worker; the broken stream is an execution error, not a failed fetch, so Ballista fails the stage at once whatever --stage-max-failures says, with no rollback, and its message carries no address. 1 run in about 8 of that test; the kill -9 test can meet it the same way. The tests' helper now takes how the worker was lost: a stopped worker (SIGTERM) keeps serving its shuffle data while it drains, and its query's error must name it; a killed one may also show as that broken stream. README, the CHANGELOG, STATUS and docs/verification.md said the error always names the worker; they now say when it does not. After: three full runs of tests/cli.rs, 24 passed, 3 ignored each. Refs #148 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
A run of tests/cli.rs failed sigterm_stops_a_worker_within_its_drain_timeout_with_exit_zero with "an idle worker ignored SIGTERM". Its log: the scheduler had counted the worker (/readyz said 2) and the test sent SIGTERM at 09:53:12.823, but the worker's own registration finished at 12.849, and Ballista's executor starts listening for signals only after that. OxideLake's listener, which is installed at startup so an early signal is not fatal, saw it alone; Ballista never ran its graceful stop, and the worker exited 0 when its 20 s drain timeout passed, just after the test's 20 s wait. Passing the signal on to Ballista would mean raising it again through FFI, which this crate does not do outside allocator code, so the edge is documented on run_executor_with_shutdown instead: a signal in a worker's first fraction of a second stops it at the drain timeout, with exit 0, and the scheduler learns from the missed heartbeats. The test now waits for the worker's Flight server line, Ballista's last step before it listens, and 300 ms more. After: three full runs of tests/cli.rs, 24 passed, 3 ignored each. Refs #31 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
test-metal-device runs every #[ignore]d test in its graph, and a_query_that_loses_a_worker_comes_back_with_the_right_rows is the #[ignore]d reproduction of Ballista 55's rollback bug (#148): it fails until a Ballista release carries apache/datafusion-ballista#2677, which is why it is ignored. The macOS job ran it and went red (the query came back wrong, as the reproduction expects). Skipped by name, as mixed_build is for the same reason. Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com> # Conflicts: # crates/oxidelake-runtime/tests/cli.rs
This was referenced Nov 18, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cluster fixes from the 0.3.0 review, part 2, one commit each, with the evidence in each commit body.
/healthz,/readyzon the scheduler).SIGTERMkilledoxide-scheduler(status 143) and every running query with it. OnSIGTERMorSIGINTit now refuses new queries, saying it is shutting down, gives the running ones up to--drain-timeout(20 s) to finish, and exits 0.oxide-workeralready stopped through Ballista's graceful path, but waited for its tasks with no bound.--drain-timeoutnow bounds that wait.--task-max-failures,--stage-max-failuresand--worker-timeouton the scheduler, and--heartbeat-intervalon the worker. Without the last two, a dead worker went unnoticed for 180 s.--stage-max-failuresnow defaults to 1, so a failed fetch from a lost worker fails the query. That limit does not cover a worker that stops or goes silent: those take Ballista'sExecutorLostrollback whatever the limit.oxide-schedulernow builds Ballista's scheduler itself:config.validate(), thencreate_scheduler, thenstart_grpc_service, which is allstart_serverdoes.remove_executorruns before Ballista postsExecutorLost. There it cancels every running query that holds finished output on the removed worker, and a job-state wrapper reports the reason, naming the worker, in place of "Cancelled".kill -9,SIGTERMand a silent worker mid-join each fail the query at once, naming the worker, with no rows printed, and the next query is byte-equal to embedded.--stage-max-failuresbrings the bug back;--help, README, SECURITY.md, STATUS and the CHANGELOG say so. The wrong-rows reproduction is committed#[ignore]d at--stage-max-failures 4.--bind-hostor--metrics-hostunless--allow-insecure-bindis given. They log a warning whenever they listen beyond loopback.oxide-worker --bind-hostis new. Before, a worker could listen only on 127.0.0.1, so no worker on another machine could join.Verified:
make -k fmt-check lint lint-cuda lint-predict test test-predict check-cuda check-predict-no-second-cuda doc coherencepassed on the fix branch: 38 test binaries, 209 passed, 0 failed, 8 ignored. Real processes on 127.0.0.1 cover the scheduler drain, the worker drain (idle and busy), worker loss bykill -9,SIGTERMand silence, and the refused listeners. Each guard was broken once and its test went red.A worker lost before it finished any task is left to Ballista, which runs that task again elsewhere. That path was checked with 4
kill -9and 2SIGTERMruns at the first scan task: every run exited 0 with stdout byte-equal to embedded, with no rollback in the scheduler log and only the one running task reset.Three test-only commits fix what the gate runs found:
SIGTERMin a worker's first ~30 ms, before Ballista's executor listens, exits 0 only at the drain timeout; this is documented.The last commit makes the macOS on-device pass (
make test-metal-device, which runs every#[ignore]d test) skip the lost-worker reproduction by name, as it already skipsmixed_build. That test fails by design until a Ballista release carries the fix.Locally on this branch:
cargo test -p oxidelake-runtime --lib --bins --test cli(lib 36, bins 3, cli 26 passed and 3 ignored), runtime clippy, fmt,crate-metadata,ci-scripts,release-scriptsand zizmor all pass.Closes #31
Refs #30
Refs #148