Skip to content

fix(runtime): graceful shutdown, fail fast on a lost worker, refuse public listeners (#31, #148, #30) - #154

Merged
vyncint merged 9 commits into
mainfrom
pr/030-cluster2
Nov 18, 2026
Merged

vyncint merged 9 commits into
mainfrom
pr/030-cluster2

Conversation

@vyncint

@vyncint vyncint commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Cluster fixes from the 0.3.0 review, part 2, one commit each, with the evidence in each commit body.

  • Test worker loss, expose Ballista retry settings and handle SIGTERM in scheduler and worker #31 Graceful shutdown, Ballista's retry limits, and documented health endpoints (/healthz, /readyz on the scheduler).
    • Before, SIGTERM killed oxide-scheduler (status 143) and every running query with it. On SIGTERM or SIGINT it now refuses new queries, saying it is shutting down, gives the running ones up to --drain-timeout (20 s) to finish, and exits 0.
    • oxide-worker already stopped through Ballista's graceful path, but waited for its tasks with no bound. --drain-timeout now bounds that wait.
    • A second signal stops either process at once.
    • New flags: --task-max-failures, --stage-max-failures and --worker-timeout on the scheduler, and --heartbeat-interval on the worker. Without the last two, a dead worker went unnoticed for 180 s.
  • A worker lost mid-query can make a cluster return duplicated rows with exit 0, or hang (upstream Ballista 55 rollback) #148 A query that loses a worker fails, naming the worker, instead of coming back wrong.
    • Ballista 55's recovery from a lost worker rolls back the stages that read its shuffle output. In 55.0.0 that recovery returned duplicated rows with exit 0 in 12 of 15 runs. It can also leak the survivors' task slots, so a query hangs until a new worker joins (one run took 670 s).
    • The upstream fix, fix(scheduler): discard a rolled-back stage's partial outputs before it re-runs apache/datafusion-ballista#2677, is merged but not released. Decision for 0.3.0: correctness over availability.
    • --stage-max-failures now defaults to 1, so a failed fetch from a lost worker fails the query. That limit does not cover a worker that stops or goes silent: those take Ballista's ExecutorLost rollback whatever the limit.
    • So oxide-scheduler now builds Ballista's scheduler itself: config.validate(), then create_scheduler, then start_grpc_service, which is all start_server does.
    • Our cluster state's remove_executor runs before Ballista posts ExecutorLost. There it cancels every running query that holds finished output on the removed worker, and a job-state wrapper reports the reason, naming the worker, in place of "Cancelled".
    • After the change, kill -9, SIGTERM and a silent worker mid-join each fail the query at once, naming the worker, with no rows printed, and the next query is byte-equal to embedded.
    • Raising --stage-max-failures brings the bug back; --help, README, SECURITY.md, STATUS and the CHANGELOG say so. The wrong-rows reproduction is committed #[ignore]d at --stage-max-failures 4.
  • Production cluster mode needs TLS/auth or an enforced private-network posture #30, first half Both binaries now refuse a public or wildcard --bind-host or --metrics-host unless --allow-insecure-bind is given. They log a warning whenever they listen beyond loopback.

Verified: make -k fmt-check lint lint-cuda lint-predict test test-predict check-cuda check-predict-no-second-cuda doc coherence passed on the fix branch: 38 test binaries, 209 passed, 0 failed, 8 ignored. Real processes on 127.0.0.1 cover the scheduler drain, the worker drain (idle and busy), worker loss by kill -9, SIGTERM and silence, and the refused listeners. Each guard was broken once and its test went red.

A worker lost before it finished any task is left to Ballista, which runs that task again elsewhere. That path was checked with 4 kill -9 and 2 SIGTERM runs at the first scan task: every run exited 0 with stdout byte-equal to embedded, with no rollback in the scheduler log and only the one running task reset.

Three test-only commits fix what the gate runs found:

  • Ports are now taken below the kernel's ephemeral range. A worker's Flight port had been grabbed first ("Address already in use").
  • A worker killed while sending a shuffle block surfaces as Ballista's broken-stream error. That still fails the query at once with no rollback, but does not name the worker (about 1 run in 8). The tests accept it and the docs say so.
  • An idle worker is signalled only once it listens. A SIGTERM in a worker's first ~30 ms, before Ballista's executor listens, exits 0 only at the drain timeout; this is documented.

The last commit makes the macOS on-device pass (make test-metal-device, which runs every #[ignore]d test) skip the lost-worker reproduction by name, as it already skips mixed_build. That test fails by design until a Ballista release carries the fix.

Locally on this branch: cargo test -p oxidelake-runtime --lib --bins --test cli (lib 36, bins 3, cli 26 passed and 3 ignored), runtime clippy, fmt, crate-metadata, ci-scripts, release-scripts and zizmor all pass.

Closes #31
Refs #30
Refs #148

Verified on this branch (Ballista 55.0.0, static planner per #134), real
binaries on 127.0.0.1:
- `kill -TERM` on oxide-scheduler: status 143, every running query lost.
  Ballista's scheduler installs no handler (axum::serve, no shutdown).
- `kill -TERM` on oxide-worker: exit 0. Ballista's executor already fences
  itself, tells the scheduler (which re-runs its tasks elsewhere) and waits
  for its tasks, but with no bound: a task that does not finish keeps the
  worker up until it is killed. The issue's "no signal handler" is true of
  the scheduler only.
- No flag reached task_max_failures, stage_max_failures, the executor
  timeout or the heartbeat interval.

Changes:
- `shutdown` module: ShutdownSignal (SIGTERM/SIGINT, listening from
  startup so an early signal is held), Stopped { Drained, Abandoned }.
- `cluster::run_scheduler_with_shutdown`: on the signal, new queries fail
  in the session builder ("this scheduler is shutting down ..."; it is
  called for every query and nothing a running one needs), the running
  ones are waited for up to --drain-timeout by polling the job state, plus
  a one-second linger so their clients hear they finished; then exit 0.
- `cluster::run_executor_with_shutdown`: Ballista's own graceful stop,
  bounded by --drain-timeout; past it the worker exits 0 at once
  (process::exit: dropping the runtime would wait for the tasks).
  A second signal stops either at once.
- Scheduler flags --task-max-failures, --stage-max-failures (0 refused,
  1 = no retry), --worker-timeout (cluster::with_worker_timeout also sets
  Ballista's dead-worker check to a third of it, at most 15 s, and the
  no-worker grace to it, as Ballista's binary does), --drain-timeout (20).
  Worker flags --heartbeat-interval and --drain-timeout (20). Defaults are
  Ballista's (unit-tested), so a deployment that sets none is unchanged.

Worker loss, measured (2 cpu workers x 2 tasks, 600k-row fixture,
self-join on k, `kill -9` as the victim starts its first join task):
- With Ballista's defaults the query came back WRONG in 9 of 12 runs (7
  of 8 from a shell loop, 2 of 4 of the committed test): the 100 groups
  plus 13 of them again (two hash partitions twice), exit 0; the same
  query without a kill is byte-equal to embedded.
- A rollback leaks the survivors' task slots: a cancelled task's status
  arrives once its stage is UnResolved, sum_vcores_for_statuses refunds 0
  for it, and with no slot left the query waits for a new worker (one
  run: 670 s, ended when I started one). SIGTERM on a worker mid-query
  takes the same path: 2 of 2 runs hung.
- With --stage-max-failures 1 the first failed fetch (Ballista retries it
  3 times, 3 s apart) fails the query before any rollback, naming the
  worker's address; the next query runs on the worker left.
Both bugs are in Ballista's scheduler, outside our hooks, so this
documents them (README Cluster mode, STATUS decision, CHANGELOG known
issue) and recommends --stage-max-failures 1 and stopping workers between
queries.

Tests (tests/cli.rs, real processes and signals, all on 127.0.0.1):
- sigterm_drains_a_scheduler_and_exits_zero: a new query is refused
  naming the shutdown, the running join finishes equal to embedded, exit 0.
- sigterm_stops_a_worker_within_its_drain_timeout_with_exit_zero: idle
  worker exits 0 at once and stops counting; a busy one with
  --drain-timeout 1 exits 0 after 1..10 s, logging the timeout.
- a_query_that_loses_a_worker_fails_naming_it_when_stage_retries_are_off.
- a_silent_worker_stops_counting_after_the_worker_timeout (a 5 s timeout
  over a 1 s heartbeat: Ballista keeps heartbeat times in whole seconds).
- a_query_that_loses_a_worker_comes_back_with_the_right_rows: what the
  issue asked for; #[ignore]d as the reproduction of the Ballista bug
  (fails in about half its runs, not every one).
- the_resilience_flags_appear_in_help; unit tests for the drain, the
  refusal, and the flags reaching SchedulerConfig.
Broken to watch each go red: no drain bound (busy worker: no timeout
line), draining never set (the new query ran), no drain wait (scheduler
gone before the refusal), --stage-max-failures not applied (the query
"recovered"), --heartbeat-interval not applied (live worker expired too:
not ready 0/1), --worker-timeout not applied (dead worker still counted).

Refs #148

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Verified on this branch: `oxide-scheduler --bind-host 0.0.0.0` started
and listened on every interface without a word, although its port, like
the workers', has no authentication or encryption (SECURITY.md: whoever
connects can run queries that read any file a worker can read). The same
held for --metrics-host. oxide-worker had no --bind-host at all: it
listened where Ballista's ExecutorProcessConfig default puts it,
127.0.0.1, so a scheduler on another machine could not reach it.

The first half of the issue, the enforced private-network posture; TLS
and authentication are not built (see below), so #30 stays open for them.

- `exposure` module: an address is Loopback, Private (RFC 1918, IPv6
  unique-local, link-local; IPv4-mapped IPv6 judged as its IPv4) or
  Public (anything else, 0.0.0.0 and :: included, and the carrier-shared
  100.64/10). check_listen_address refuses Public unless
  --allow-insecure-bind, judges a hostname by the widest address it
  resolves to, and refuses one that does not resolve. A bracketed IPv6
  host ([::1]), the way the scheduler's and Ballista's `host:port` parsing
  needs one written, is judged without its brackets (broken once: [::1]
  was refused as a name that does not resolve).
- oxide-scheduler and oxide-worker check --bind-host, and --metrics-host
  when --metrics-port is set, before anything listens, exit 1 naming the
  risk and the two ways out, and warn at startup whenever they listen
  beyond loopback. oxide-worker gains --bind-host (default 127.0.0.1, as
  before) under the same rule.
- README (flag tables and a "Production deployments" paragraph) and
  SECURITY.md (trust model, the same paragraph word for word), CHANGELOG
  (Security), STATUS (decision).

After: `oxide-scheduler --bind-host 0.0.0.0` exits 1 at once with
"--bind-host 0.0.0.0 would listen where any network that routes to it
can connect, and OxideLake's cluster ports have no authentication or
encryption ... pass --allow-insecure-bind if a firewall or an
authenticating proxy already decides who can connect (SECURITY.md)".

Tests: a_public_or_wildcard_listener_is_refused_without_the_flag
(assert_cmd: scheduler 0.0.0.0, ::, 203.0.113.7, public --metrics-host;
worker 0.0.0.0 and public --metrics-host; both --help name the flag).
Every case passes --port 1, which an unprivileged process cannot bind,
so a regression fails on the bind instead of listening anywhere. Broken
once (the check never refuses): red, the scheduler reached the bind and
failed with "Permission denied". Unit tests classify the range edges
(172.15/172.32, 192.169, 100.64, mapped addresses) and check the flag.
No test listens beyond 127.0.0.1.

Not done, proposed for the rest of #30: Ballista 55 can make its clients
speak TLS (ballista.client.use_tls, plus an endpoint override for
certificates) but serves the scheduler's port and the executors' gRPC
port in plain text with no hook; only the executors' Flight service can
be replaced. TLS therefore needs either an upstream server-side TLS
option (tonic's ServerTlsConfig on create_grpc_server and the
scheduler's listener) or a terminating proxy per port, with every
OxideLake process told to dial TLS (use_tls on clients, the scheduler's
use_tls for its proxy, the endpoint override for a private CA), tested
against a local mTLS-terminating proxy.

Refs #30

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Ballista 55's recovery from a lost worker rolls back every stage that read
the worker's shuffle output and runs it again, and in 55.0.0 that recovery
returns duplicated rows with exit 0 (fixed upstream by
apache/datafusion-ballista#2677, merged 2026-10-28, unreleased) or never
finishes (the tasks the rollback cancels never return their slots; see
apache/datafusion-ballista#2539 and #2698). Decision for 0.3.0:
correctness over availability.

Reproduction (this branch before this commit; debug build, deps at -O2):
- fixture: `oxide gen-data --rows 600000 --row-group-rows 4096 --out D`
  (seed 42, zstd: D/t.parquet, 12,301,854 bytes, 16 scan partitions).
- cluster on 127.0.0.1: `OXIDE_CLUSTER_BACKEND=cpu oxide-scheduler --port
  P` and two `oxide-worker --scheduler-host 127.0.0.1 --scheduler-port P
  --port F --grpc-port G --concurrent-tasks 2 --work-dir W`, RUST_LOG=info.
- query: `oxide sql --cluster df://127.0.0.1:P --table t=D -q "SELECT a.k,
  COUNT(*) AS n, SUM(a.v + b.v) AS sv FROM t a JOIN t b ON a.k = b.k WHERE
  a.id < 200000 GROUP BY a.k ORDER BY a.k"` (four stages: 1 and 2 scan,
  3 joins, 4 aggregates; embedded prints 100 groups, 104 lines).
- the kill: `kill -9` on the second worker as soon as its log shows its
  first join task, `Execute task  : [TID <job>/3.0/<partition>.0]`, polled
  every 20 ms.
- with Ballista's stage_max_failures 4: 12 of 15 runs (7 of 8 from a
  shell loop, 5 of 7 of the test) printed the 100 sorted groups plus 13 or
  14 of them again (whole hash partitions twice), exit 0. The same query
  with no kill is byte-equal to embedded.
- with --worker-timeout 10 and --heartbeat-interval 1 one run waited
  670 s, until I started a third worker: the cancelled join task's status
  was "Ignored ... as the Stage ... is in UnResolved status", so
  sum_vcores_for_statuses refunded its slot to nobody.
- `kill -TERM` instead of -9 hung in 2 of 2 runs, and still hung (150 s)
  with --stage-max-failures 1.

Change: --stage-max-failures defaults to 1 (cluster::DEFAULT_STAGE_MAX_
FAILURES, also in cluster::scheduler_config). The first failed fetch from
a lost worker, what usually happens first when a worker is killed, then
fails the query, naming the worker's address. Ballista's own default is 4;
an operator can raise it, and --help, README, SECURITY.md, STATUS and the
CHANGELOG (Changed, with a migration line) say that brings the bug back.

What this does NOT cover, said in the same places: only a failed fetch
counts toward the limit. A worker stopped with SIGTERM, or silent past
--worker-timeout, is rolled back through Ballista's ExecutorLost with
count_to_failures = false, whatever the limit, so a query running then can
still return duplicated rows or hang; the docs say to stop workers between
queries. The next commit closes those paths.

Tests: a_query_that_loses_a_worker_fails_naming_it now runs at the
defaults (kill -9 as above): exit non-zero, empty stdout, the worker's
Flight address in stderr ("Stage 3 has failed 1 times ... Error connecting
to ... http://127.0.0.1:<port>", about 11 s after the kill: Ballista
retries the fetch 3 times, 3 s apart), and the next query equal to
embedded. The #[ignore]d a_query_that_loses_a_worker_comes_back_with_the_
right_rows runs at --stage-max-failures 4 and points to #148 and
apache/datafusion-ballista#2677; it failed 3 of 3 runs, as intended. The
scheduler's unit test pins the default against Ballista's; set back to 4
it went red (left 4, right 1), and with the limit at 4 the kill test's
query "recovered" instead of failing.

Refs #148
Refs #31

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
The previous commit's --stage-max-failures 1 covers a failed shuffle
fetch, what usually happens first when a worker is killed. It does not
cover a worker that is stopped (SIGTERM: Ballista's executor_stopped) or
that goes silent past --worker-timeout: those reach Ballista's
ExecutorLost, which rolls the query back with count_to_failures = false,
whatever the limit. Measured on the previous commit, same setup as there
(fixture, two workers, the self-join, the signal as the worker's first
join task starts): `kill -TERM` hung, killed by a 150 s timeout, no rows.

Change, inside OxideLake's hooks only (no Ballista code copied):
- oxide-scheduler builds Ballista's scheduler itself (create_scheduler +
  start_grpc_service, what start_server does) and keeps a handle to it.
- worker_loss::WorkerLoss: every removal of a worker passes through our
  cluster state's remove_executor, and Ballista posts ExecutorLost only
  after it returns. There, after removing the worker, it cancel_job()s
  each running query with a finished task on that worker, so the
  cancellation is queued ahead of the rollback and Ballista's own abort
  refunds the task slots. A query whose tasks on the worker had finished
  nothing is left to Ballista: no stage is rolled back, the tasks run
  again elsewhere.
- worker_loss::ReportingJobState: Ballista's in-memory job state, saving
  such a query's failure as "worker <id> (<host:port>) was lost while this
  query ran, and it held shuffle output the query still needed ... (#148).
  Run the query again" where Ballista says "Cancelled".
- --help, README, SECURITY.md, STATUS and the CHANGELOG now say every
  loss path fails the query; the in-process start_standalone cluster keeps
  Ballista's defaults (its builder hard-codes them, its workers live and
  die with the process).

After, every case exits 1, prints no rows, and names the worker:
- kill -TERM mid-join: at once; the stopped worker exits 0.
- silent (kill -9, --worker-timeout 5, --heartbeat-interval 1): about 5 s
  after the kill, 2 runs of 2.
- the #31 drain on a busy worker (the only worker, --drain-timeout 1):
  the query fails at once instead of waiting on Ballista's no-worker grace.
In each case the next query on the workers left is byte-equal to embedded.

Tests (tests/cli.rs, real processes on 127.0.0.1):
a_query_whose_worker_stops_fails_naming_it,
a_query_whose_worker_goes_silent_fails_naming_it, and the drain test
sigterm_stops_a_worker_within_its_drain_timeout_with_exit_zero now also
requires its query to fail naming the worker. Broken once (the hook not
called): the stop and silent tests, "`query` was still running after
120s"; the drain test, "after 60s". With the reason not rewritten the
client saw "Job ... failed: Cancelled".

Refs #148
Refs #31

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
`make test` on 61f455f failed a_query_that_loses_a_worker_fails_naming_it
in wait_ready(2): w1 registered, then "Could not start built-in arrow
flight server ... Address already in use (os error 98)" on the port
free_port() had picked, and it stopped. An earlier full run failed
another cluster test the same way. free_port() bound 127.0.0.1:0 and let
the port go; that range (ip_local_port_range, 32768-60999 here) is where
the kernel also takes source ports for outgoing connections, so on a
busy machine a connection reused the port before the worker bound it.

It now hands out ports from 20000-31999, which no one but this function
hands out: below the ephemeral ranges of Linux (32768) and macOS (49152).
Each call in the binary gets a different port, starting from an offset
in the process id, and each is checked free by binding it. Two full runs
of tests/cli.rs after: 24 passed, 3 ignored, then 23 passed and 1 failed
for a different reason (the next commit).

Refs #31

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
A run of tests/cli.rs on a579a5a failed
a_query_whose_worker_goes_silent_fails_naming_it although the query did
what it should: it failed at once, printed no rows, and nothing hung. The
error named no worker: "Job ... failed: Job failed due to stage 3 failed:
Task failed due to runtime execution error: ... IpcError(\"... h2 protocol
error: error reading a body from connection\" ... broken pipe)". The
worker was killed while it was sending a shuffle block to a join task on
the other worker; the broken stream is an execution error, not a failed
fetch, so Ballista fails the stage at once whatever --stage-max-failures
says, with no rollback, and its message carries no address. 1 run in
about 8 of that test; the kill -9 test can meet it the same way.

The tests' helper now takes how the worker was lost: a stopped worker
(SIGTERM) keeps serving its shuffle data while it drains, and its query's
error must name it; a killed one may also show as that broken stream.
README, the CHANGELOG, STATUS and docs/verification.md said the error
always names the worker; they now say when it does not.

After: three full runs of tests/cli.rs, 24 passed, 3 ignored each.

Refs #148

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
A run of tests/cli.rs failed
sigterm_stops_a_worker_within_its_drain_timeout_with_exit_zero with "an
idle worker ignored SIGTERM". Its log: the scheduler had counted the
worker (/readyz said 2) and the test sent SIGTERM at 09:53:12.823, but
the worker's own registration finished at 12.849, and Ballista's executor
starts listening for signals only after that. OxideLake's listener, which
is installed at startup so an early signal is not fatal, saw it alone;
Ballista never ran its graceful stop, and the worker exited 0 when its
20 s drain timeout passed, just after the test's 20 s wait.

Passing the signal on to Ballista would mean raising it again through
FFI, which this crate does not do outside allocator code, so the edge is
documented on run_executor_with_shutdown instead: a signal in a worker's
first fraction of a second stops it at the drain timeout, with exit 0, and
the scheduler learns from the missed heartbeats. The test now waits for
the worker's Flight server line, Ballista's last step before it listens,
and 300 ms more.

After: three full runs of tests/cli.rs, 24 passed, 3 ignored each.

Refs #31

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
@vyncint vyncint added this to the v0.3.0 milestone Nov 18, 2026
test-metal-device runs every #[ignore]d test in its graph, and
a_query_that_loses_a_worker_comes_back_with_the_right_rows is the
#[ignore]d reproduction of Ballista 55's rollback bug (#148): it fails
until a Ballista release carries apache/datafusion-ballista#2677, which
is why it is ignored. The macOS job ran it and went red (the query came
back wrong, as the reproduction expects). Skipped by name, as
mixed_build is for the same reason.

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>

# Conflicts:
#	crates/oxidelake-runtime/tests/cli.rs
@vyncint
vyncint merged commit ccd80d7 into main Nov 18, 2026
19 checks passed
@vyncint
vyncint deleted the pr/030-cluster2 branch November 18, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test worker loss, expose Ballista retry settings and handle SIGTERM in scheduler and worker

1 participant