A GPU-accelerated, Arrow-native distributed analytical query engine, written in Rust. It reads and writes Parquet on object storage; a catalog, a metastore and transactions are out of scope, so it is a query engine over a lake rather than a lakehouse in the full sense.
Status: internal phases 0–7 complete; published 0.3.0, the correctness release. It moves to Ballista 55, DataFusion 55.2 and arrow 59, and fixes everything a review of the whole engine found. Among those findings: places where GPU placement could change a query's result, files that could panic the engine, and a cluster that now fails a query on a lost worker rather than risk Ballista 55's rollback. Three Phase 8 lines remain open: double-buffered transfers, TLS for the cluster ports, and a runner for the on-demand CUDA job. The default build is pure CPU, and the whole gate is green on Linux x86_64 and macOS arm64. Both GPU backends have executed on real hardware:
- Metal on Apple silicon, conformance-tested against stock DataFusion on an M4 Pro.
- CUDA on an NVIDIA T4:
- on 2026-09-06, where every kernel was NVRTC-compiled and launched, and a correctness bug in the aggregation kernel was found;
- again on 2026-11-18, for every CUDA change in 0.3.0. STATUS.md holds the verification matrix, component by component, and is the ledger ADR-0012 makes it. The specification the engine was built to is docs/SPEC.md.
brew install vyncint/tap/oxidelake # macOS and Linux: static binaries, no Rust toolchain
cargo install oxidelake-runtime # or build from crates.ioBoth give you the oxide, oxide-scheduler and oxide-worker binaries.
The Homebrew formula is generated by the release itself from the archives'
checksums, so it never describes a build that did not happen.
Or download a static archive for Linux (x86_64, aarch64, musl) or macOS
(Apple silicon, Intel) from the latest release,
verify it against the .sha256 beside it, and put the three binaries on your
PATH. Using the engine as a library is cargo add oxidelake-api.
OxideLake runs the same plans in two modes over the Apache Arrow columnar memory model:
- Embedded mode (DuckDB-like) — in-process DataFusion; operators exchange
Arc<RecordBatch>over bounded async channels with no serialization. - Cluster mode (Spark-like) — Apache DataFusion Ballista schedules stages across executors and streams shuffle partitions over Arrow Flight; OxideLake supplies the GPU operators, the placement rule and the plan codec that ships them to executors.
Acceleration backends: CPU (always available, the correctness reference), NVIDIA CUDA (opt-in feature, NVRTC-JIT at runtime), Apple Metal on unified memory (opt-in, macOS only, MSL compiled at runtime).
The differentiator is the operator layer: GpuFilterExec (fused filter + projection), GpuHashJoinExec, GpuAggregateExec and GpuVectorDistanceExec are DataFusion ExecutionPlans with their own CUDA and Metal kernels behind one object-safe hardware abstraction. A placement rule rewrites eligible plan nodes for the target backend — EXPLAIN shows GpuFilterExec[metal] — and every operator still selects the real local backend at execution time and falls back per batch to the CPU reference, which is what keeps a heterogeneous cluster correct. Storage is Parquet pruned by DataFusion's statistics, page index and Bloom filters, with Arrow IPC for zero-decode spill and caches.
Everything below runs as written (Rust 1.98 via rust-toolchain.toml; the workspace builds on 1.94.1 and up — or just make quickstart):
cargo build --release -p oxidelake-runtime # or `cargo install oxidelake-runtime`
# 1M-row demo table: Parquet with Bloom filters on id and k, page statistics on
./target/release/oxide gen-data --rows 1000000 --out data/
# embedded SQL over it
./target/release/oxide sql -q "SELECT k, SUM(v) FROM t GROUP BY k ORDER BY k" --table t=data/
# vector search: l2_distance / cosine_distance are built-in SQL UDFs
./target/release/oxide sql -q "SELECT id, l2_distance(emb, [1.0,0.0,2.0,0.5,-1.0,0.25,3.0,0.0]) AS d \
FROM t ORDER BY d LIMIT 5" --table t=data/
# what the plan looks like for a GPU cluster: placement tags without needing the GPU
./target/release/oxide explain -q "SELECT k, SUM(v) FROM t WHERE k >= 2 AND v < 4.0 GROUP BY k" \
--table t=data/ --target cuda
# the terminal dashboard over a real query: plan DAG, per-operator telemetry, column profiles
./target/release/oxide tui -q "SELECT k, SUM(v) FROM t WHERE k >= 2 GROUP BY k" --table t=data/ --target cudaScore a row against a trained model without leaving SQL. Off by default; it pulls in a tensor library.
cargo build --release -p oxidelake-runtime --features predict
# an oxmera Sequential/Linear model saved with safetensors, whose header
# declares {"__metadata__": {"oxidelake.activation": "relu"}}
./target/release/oxide sql --table t=data/ \
-q "SELECT id, predict('scorer.safetensors', emb) AS logits FROM t LIMIT 5"
# a checkpoint whose header says nothing: state the activation in the query
./target/release/oxide sql --table t=data/ \
-q "SELECT id, predict('scorer.safetensors', emb, 'gelu') AS logits FROM t LIMIT 5"+----+--------------------------+
| id | logits |
+----+--------------------------+
| 0 | [-31.006792, 3.9050333] |
| 1 | [7.2003703, -4.237343] |
+----+--------------------------+
It composes with everything else — including the vector search that was already here, which is the point: OxideLake could find the nearest rows and could not score them.
./target/release/oxide sql --table t=data/ -q \
"SELECT id, l2_distance(emb, [1.0,0.0,2.0,0.5,-1.0,0.25,3.0,0.0]) AS dist,
predict('scorer.safetensors', emb)[2] AS score
FROM t ORDER BY dist LIMIT 5"The model file is read for its architecture, not just its weights: predict
rebuilds an oxmera::nn::Sequential of Linear layers from the 0.weight,
1.weight, … naming, and takes the activation between them from the
safetensors __metadata__ key oxidelake.activation — relu, gelu,
sigmoid, tanh or none, applied between layers and never after the last.
A file that declares nothing is refused, because the alternative is
scoring it with an activation it was not trained with and returning a
confident wrong number; the third argument predict(path, features, 'relu')
is how you say it yourself, and it must agree with the file when the file has
an opinion. A NULL feature vector scores NULL, and so does one with a NULL
element: no number is made up to stand in for the missing value. The
CPU-only execution, and why this is a UDF rather than an operator, are
recorded in
ADR-0015.
--target cpu|cuda|metal picks the placement target; execution always uses the hardware that is present (operators planned for an absent GPU take the per-batch CPU path — exactly what cluster executors do with a scheduler's plan).
# OXIDE_CLUSTER_BACKEND declares the cluster's placement capability (default: cpu, no rewrites)
OXIDE_CLUSTER_BACKEND=cuda ./target/release/oxide-scheduler --port 50050 &
./target/release/oxide-worker --scheduler-port 50050 --port 50051 --grpc-port 50052 &
./target/release/oxide sql --cluster df://127.0.0.1:50050 \
-q "SELECT k, SUM(v) FROM t GROUP BY k ORDER BY k" --table t=data/The end-to-end test in crates/oxidelake-runtime/tests/cli.rs spawns exactly this topology on ephemeral ports and asserts the cluster output is byte-identical to embedded mode.
A scheduler admits only workers of its own build: the same OxideLake release, built with predict if and only if the scheduler is. cuda and metal do not count, so a default-build scheduler with --features cuda workers is a supported cluster. A worker of another build is refused when it registers and exits non-zero naming both builds, and a query sent to a cluster whose workers were all refused fails at once with the same reason. The check guards against a mismatched deployment, not against an attacker: a worker declares its own build (SECURITY.md).
SIGTERM or SIGINT stops either binary gracefully, and a second signal stops it at once. A scheduler refuses new queries, saying it is shutting down, gives the running ones up to --drain-timeout to finish, and exits 0. A worker takes no new tasks and exits 0 once its running ones finish or --drain-timeout passes, and a query that held its output fails, naming it (below); keep the timeout under your supervisor's own limit (Kubernetes sends SIGKILL 30 s after SIGTERM by default). A worker that dies without a word is presumed lost once the scheduler's --worker-timeout passes with no heartbeat from it (the worker's --heartbeat-interval, which must be well under the timeout). Until then a query holding tasks on it waits, unless a shuffle fetch from it fails first: Ballista retries a fetch three times, three seconds apart, and then the query fails. The scheduler's port serves Ballista's /healthz (always ok) and /readyz (ready: N executors registered, or 503 before the first); a worker serves nothing over HTTP except --metrics-port.
Production deployments. The cluster's ports (the scheduler's one port for gRPC, REST and probes, each worker's Flight and gRPC ports, and any --metrics-port) have no authentication and no encryption: whoever can connect can run queries, and a query can read any file a worker can read. Keep them on a network that only the cluster's own hosts and trusted clients can reach, such as a private subnet or a VPN, behind a firewall. oxide-scheduler and oxide-worker refuse to listen on a public address, or on every address (0.0.0.0, ::), unless --allow-insecure-bind says that something else, such as that firewall, already decides who can connect, and they log a warning whenever they listen beyond loopback. TLS and authentication in OxideLake itself are tracked in #30.
A query that loses a worker fails, naming it (#148). Ballista 55 would recover such a query by rolling back every stage that read the lost worker's output and running it again, and that recovery is not safe: the rolled-back stage keeps its partial outputs, so in our runs most such queries came back with some groups twice, with exit 0 (fixed upstream by apache/datafusion-ballista#2677, merged after 55.0.0 and not yet released), and the tasks it cancels never return their slots, so a query can wait for capacity that does not come back (see apache/datafusion-ballista#2539 and #2698). So oxide-scheduler fails any query that held output on a worker it loses, killed, stopped with SIGTERM, or silent past --worker-timeout, and --stage-max-failures defaults to 1, so a failed fetch from a lost worker ends the query too. The error names the worker, unless the worker was killed in the middle of sending shuffle data, which Ballista reports as a broken stream that fails the query just as cleanly; either way, run the query again. A query whose tasks on the lost worker had not finished anything is not failed: those tasks run again elsewhere. Raising --stage-max-failures brings Ballista's stage retries back, with the bug.
use oxidelake_api::prelude::*;
let session = OxideSession::local()?;
session.register_parquet("t", "data/").await?;
let nearest = session
.table("t").await?
.filter(col("k").gt_eq(lit(2)))?
.vector_distance("emb", &[0.0; 8], DistanceMetric::L2, "d")?
.sort(vec![col("d").sort(true, false)])?
.limit(10)?
.collect().await?;The fluent verbs build ordinary DataFusion plans, so the placement rule lowers them the same way it lowers SQL — .vector_distance(…) plans GpuVectorDistanceExec[metal] on a Metal target.
# CUDA: compiles and lints with no CUDA installed (cudarc dynamic loading + NVRTC)
cargo check -p oxidelake-runtime --features cuda
# Metal (macOS): the conformance suite executes on the device
cargo test -p oxidelake-compute --features metal -- --ignored
# On a CUDA machine (run on a T4 on 2026-09-06 and 2026-11-18; see STATUS.md):
OXIDE_BACKEND=cuda cargo test -p oxidelake-compute --features cuda -- --ignoredRunning the CUDA backend needs the NVIDIA driver and NVRTC, which ships
with the CUDA toolkit rather than the driver (the cuda-nvrtc package). A
machine with the driver alone is detected as a CPU machine — RUST_LOG=warn
says why — and OXIDE_BACKEND=cuda there fails at startup naming libnvrtc.
The GPU must have compute capability 7.0 (Volta) or newer, and the NVRTC must
be able to compile for it (CUDA 13 can no longer target Volta); otherwise CUDA
is passed over, or refused, the same way.
Every knob OxideLake reads, in one table. Flags win over environment variables; an explicit choice that the machine cannot honour is a startup error, never a silent fallback.
| Knob | Where | Default | What it does |
|---|---|---|---|
--query, -q |
sql, explain, tui |
— | The SQL statement (optional for tui, which then shows the demo model). |
--table, -t |
sql, explain, tui |
— | Registers NAME=PATH (a Parquet file or directory) before planning. Repeatable. |
--cluster |
sql |
— | Runs on a Ballista scheduler (df://host:port) instead of in-process. |
--target |
sql, explain, tui |
detected | Plans for cpu, cuda or metal. Placement only: execution uses the hardware that is present. Embedded mode only — on a cluster the scheduler decides. |
--batch-size |
sql, explain, tui |
8192, or 65536 on a GPU target | Rows per record batch. Batch boundaries never change results; larger batches amortise the host↔device round trip. 0 is refused. With --cluster the GPU default is the scheduler's call (its target), and a size given here is kept. |
--output |
sql |
table |
table, json (an array of objects, one per row, each carrying every column) or csv (RFC 4180 with a header). In json a NULL is null, and NaN, infinity and negative infinity, which JSON has no number for, are the strings "NaN", "Infinity" and "-Infinity", which Python's float() and JavaScript's Number() read back. An empty result still prints the CSV header and [], so a script can tell "no rows" from "the query failed". |
--rows |
gen-data |
1000000 |
Rows in the demo table. |
--out |
gen-data |
— | Output directory; writes <out>/t.parquet. |
--seed |
gen-data |
42 |
PRNG seed. The same seed always writes the same bytes. |
--row-group-rows |
gen-data |
65536 |
Rows per Parquet row group. |
--compression |
gen-data |
zstd |
none, lz4 or zstd. |
--special-floats |
gen-data |
off | Puts IEEE special values into v on every 16th row — NaN, -0.0, +0.0, +inf, -inf and a negative NaN, in turn — to compare backends where float comparison is subtle. Every other value, and the default table, stay the same. |
| Knob | Default | What it does |
|---|---|---|
--bind-host |
127.0.0.1 |
Address to bind: loopback or private (10/8, 172.16/12, 192.168/16, fc00::/7). A public address or every address (0.0.0.0, ::) is refused without --allow-insecure-bind, because the port is unauthenticated. |
--allow-insecure-bind |
off | Accepts a public or wildcard --bind-host/--metrics-host. Only when a firewall or an authenticating proxy already decides who can connect (Production deployments). |
--port |
50050 |
gRPC port. |
--cluster-backend |
$OXIDE_CLUSTER_BACKEND, then cpu |
The capability the cluster declares, which is what placement rewrites against. |
--metrics-port / --metrics-host |
off / 127.0.0.1 |
As for the worker. A scheduler plans but does not execute, so its counters are empty; the endpoint exists so every process is scraped the same way. |
--task-max-failures |
4 |
Times one task may fail before its query fails; 1 means no retry. A task lost with its worker runs again elsewhere without counting. Ballista's task_max_failures. |
--stage-max-failures |
1 |
Times a stage may fail because a shuffle fetch failed, its input lost with a worker, before its query fails. The default ends the query at the first such failure, naming the worker; a worker that is stopped or goes silent fails its queries whatever this says (#148, Cluster mode). Raising it turns on Ballista 55's re-run of the stage, which can return duplicated rows with exit 0 or never finish. Ballista's stage_max_failures, whose own default is 4. |
--worker-timeout |
180 |
Seconds without a heartbeat before a worker is presumed lost and its tasks run elsewhere. A query holding tasks on a dead worker waits this long. |
--drain-timeout |
20 |
On SIGTERM/SIGINT, new queries are refused and the running ones get this many seconds to finish; then the scheduler exits 0. |
| Knob | Default | What it does |
|---|---|---|
--scheduler-host / --scheduler-port |
localhost / 50050 |
The scheduler to join. |
--bind-host |
127.0.0.1 |
Address the Flight and gRPC ports bind, under the scheduler's rule: loopback or private, anything else only with --allow-insecure-bind. The scheduler connects back to the address it sees the worker connect from, so bind the interface that faces it. |
--allow-insecure-bind |
off | As for the scheduler. |
--port |
50051 |
Arrow Flight port for shuffle data. |
--grpc-port |
50052 |
gRPC control port. |
--concurrent-tasks |
available parallelism | Tasks run at once. |
--work-dir |
a temporary directory | Where shuffle files go. |
--metrics-port / --metrics-host |
off / 127.0.0.1 |
Serve Prometheus metrics at /metrics (needs --features metrics). Unauthenticated, so --metrics-host follows the --bind-host rule. Without the feature the flag is refused, never ignored. |
--backend |
$OXIDE_BACKEND, then detected |
The backend this worker executes on. Selected before the first task, so it reaches the operators; a backend the machine cannot provide is a startup error. |
--heartbeat-interval |
60 |
Seconds between heartbeats to the scheduler; keep it well under the scheduler's --worker-timeout. |
--drain-timeout |
20 |
On SIGTERM/SIGINT, no new tasks are taken (the scheduler runs this worker's tasks elsewhere) and the running ones get this many seconds to finish; then the worker exits 0. |
| Variable | Read by | What it does |
|---|---|---|
OXIDE_BACKEND |
every process that executes operators | Forces cpu, cuda or metal instead of the detected backend. oxide-worker --backend overrides it. |
OXIDE_CLUSTER_BACKEND |
oxide-scheduler |
The cluster's declared placement capability (default cpu: no GPU rewrites). --cluster-backend overrides it. |
RUST_LOG |
every binary | tracing filter; logs go to stderr, so they never mix into --output json, and are uncoloured when stderr is not a terminal. RUST_LOG=oxidelake_runtime=info gives one line per query with the mode, rows, elapsed time and CPU fallbacks. |
| Path | Purpose |
|---|---|
crates/oxidelake-core |
EngineError, BackendKind, operator parameter types, TelemetryHub |
crates/oxidelake-memory |
64/128-byte-aligned buffers, pinned/UMA allocators, 3-tier SpillManager (a library: no query path registers batches with it in 0.x — see architecture), Arrow IPC codec |
crates/oxidelake-device |
object-safe GpuBackend; CPU reference, CUDA (NVRTC) and Metal (MSL) backends; HardwareDetector |
crates/oxidelake-compute |
the four Gpu*Exec operators, per-batch CPU fallback, l2_distance/cosine_distance UDFs, conformance suite |
crates/oxidelake-storage |
Parquet writer/pruning config with proofs, Arrow IPC spill files, io_uring ObjectStore (Linux — implemented and conformance-tested, not yet used by sessions), gen-data generator |
crates/oxidelake-planner |
HardwarePlacementRule (+ vector-distance lowering before projection pushdown), OxidePhysicalCodec |
crates/oxidelake-runtime |
OxideSession (embedded + Ballista cluster), scheduler/worker wrappers, the oxide CLI, dashboard builder |
crates/oxidelake-tui |
ratatui dashboard, TestBackend snapshots and termlens PTY tests |
crates/oxidelake-api |
OxideFrame fluent DataFrame API and prelude |
crates/oxidelake-device/kernels/ |
CUDA C (.cu, NVRTC at runtime) and MSL (.metal, newLibraryWithSource) sources — inside the device crate so the published crate is self-contained (include_str! cannot reach outside a crate's directory at publish time; ADR-0016) |
docs/ |
roadmap · architecture · dependencies · verification · ADRs |
docs/SPEC.md |
The executable spec the build agent executed, phase by phase |
STATUS.md |
What is done, how it was verified, what was deferred — the honesty ledger |
Makefile · deny.toml |
make gate and friends; supply-chain policy for cargo deny |
Every change lands behind required-green and commit-policy. make gate
runs formatting, feature-specific Clippy, the test suites, all-feature rustdoc,
dependency-coherence and supply-chain checks, plus release and CI script tests.
CI
separates the Linux feature tests and checks the MSRV; macOS runs Metal and
PTY tests and executes conformance when a Metal device exists. Dependency
caching reuses compilation while tests still run; release CI retains a clean
full gate. Known documentation-only changes may skip Rust jobs under the
tested verification policy.
The honesty rule applies throughout: nothing is claimed as working unless it
ran; GPU paths that only compiled are recorded as exactly that in
STATUS.md. The
2026-09-02 security and performance audit, with every finding's status, is in
docs/audit-2026-09-02.md.
See CONTRIBUTING.md for the workflow (make help lists every target), SECURITY.md for the trust model and how to report vulnerabilities, and CHANGELOG.md for what changed.
Apache-2.0 — see LICENSE.