diff --git a/CHANGELOG.md b/CHANGELOG.md index 23bc0c1f..cd94eacc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,24 @@ reconstructed from Git history, release commits, and the shipped docs. - Validate row-to-Arrow conversion with borrowed identity membership instead of constructing discarded adjacency, and copy string properties directly into Arrow buffers without temporary owned String clones. +- Add automatic Cypher routing through `grust_datafusion::cypher::RoutedGraph`: + one graph captured as a typed index and an Arrow snapshot, one bounded-read + admission per query, and DataFusion execution for single-node scan plans + whose every physical operator charges candidate work and intermediate bytes + against the caller's `ReadQueryPolicy`. Relationship joins, unsupported + shapes, small graphs and plans over a limit stay on the reference executor; + explain reports the chosen route and why DataFusion was declined. +- Add `run_prepared_read_query_indexed`, so a declined route keeps the original + admission and deadline instead of re-preparing the query. +- Split large single-batch Arrow tables into contiguous zero-copy partitions. + Upstream round-robin repartitioning charged every queued slice the full size + of its shared parent buffers, which intermittently exhausted a 256 MiB pool on + a one-million-node scan with nothing copied. +- Add `LadybugConfig::max_db_bytes`. Ladybug reserves its maximum database + size (8 TiB by default) as virtual address space per open database, so one + x86-64 Linux process can hold only about fifteen at once. `grust-ladybug` + unit tests now cap each database at 16 GiB, as Ladybug's own tests do; + parallel test threads previously failed intermittently with `Mmap ... failed`. ## 0.20.0 — Brine — 2026-09-14 @@ -21,6 +39,19 @@ reconstructed from Git history, release commits, and the shipped docs. - Admit snapshot relationship-ordinal payload and construction work before allocation, retaining reservations with emitted buffers. Combined input-policy capture preserves the prepared request deadline and exact serialized limits. +- grust-turso no longer creates `_edges_from_idx`, and bootstrap + drops it from existing stores. The edge primary key + `(from_id, label, to_id, identity_key)` leads with `from_id`, and Turso's + planner already answered every source lookup and traversal hop from that + key, so the index only cost a write per edge. Query plans and results are + unchanged. Other SQL dialects keep the index (`GraphSqlDialect:: + edge_source_index`, default `true`). With the bulk_load example on a + 4-core host, 1M edges: MVCC 12.7k -> 15.9k edges/s, WAL 43.6k -> 51.5k. +- MVCC loads end in a `TRUNCATE` checkpoint inside `put_graph`, as WAL loads + already did, so a load returns durable in the database file with the MVCC + in-memory store drained. +- Optional `mimalloc` feature on grust-turso: Turso's own default allocator, + installed as the binary's global allocator (MVCC loads about 9% faster). ## 0.19.0 — Mantis — 2026-09-14 diff --git a/Cargo.lock b/Cargo.lock index 0696c6ae..9bbc72db 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -6309,6 +6309,15 @@ version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" +[[package]] +name = "libmimalloc-sys" +version = "0.1.49" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a45a52f43e1c16f667ccfe4dd8c85b7f7c204fd5e3bf46c5b0db9a5c3c0b8e9" +dependencies = [ + "cc", +] + [[package]] name = "libredox" version = "0.1.17" @@ -6629,6 +6638,15 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "mimalloc" +version = "0.1.52" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d4139bb28d14ad1facf21d5eb8825051b326e172d216b39f6d31df53cc97862" +dependencies = [ + "libmimalloc-sys", +] + [[package]] name = "mime" version = "0.3.17" @@ -10212,6 +10230,7 @@ dependencies = [ "hyper", "hyper-rustls", "hyper-util", + "mimalloc", "thiserror", "tokio", "tracing", diff --git a/Cargo.toml b/Cargo.toml index 0d6c3a99..870f37b9 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -39,7 +39,7 @@ categories = ["database", "data-structures"] [workspace.dependencies] async-trait = "0.1.89" helix-db = "=3.0.0" -lbug = "0.20.2" +lbug = "0.20.4" lancedb = "0.30.0" redis = "1.6.0" r2d2 = "0.8.10" diff --git a/benchmarks/arrow-pipelines/README.md b/benchmarks/arrow-pipelines/README.md index 85ce738e..7178a4d0 100644 --- a/benchmarks/arrow-pipelines/README.md +++ b/benchmarks/arrow-pipelines/README.md @@ -103,3 +103,16 @@ diagnostic for physical plans and batch-versus-slice memory reporting. It executes the captured physical plan, retains each error/mismatch and checks the arithmetic count oracle. It does not report performance timings. See [evidence](evidence/cypher-plan-memory-c658abc) for the captured partition plans. + +### Warm routed-read profile + +`cypher_route ...` captures one `RoutedGraph` per size and then +times the complete bounded-read entrypoint on each forced route, alternating +route order: admission, route planning, execution and portable output. Capture +(index plus Arrow snapshot) is reported once per size and excluded from trials, +so this measures reuse of warm representations, unlike `cypher_end_to_end`. +Both routes run under the same `ReadQueryPolicy`; DataFusion additionally uses a +1 GiB tracked pool, no spill and four target partitions. Each query's first +result is the oracle for every later trial on either route; mismatches, errors +and all timings are retained. Its measurements set +`DEFAULT_MIN_DATAFUSION_NODES` for this scan workload only. diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/README.md b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/README.md new file mode 100644 index 00000000..4e1fb7ed --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/README.md @@ -0,0 +1,57 @@ +# Automatic routing: partition fix and warm route profile + +Capitola (aarch64 macOS, 10 cores, 64 GiB), stable Rust 1.97.1, run +2026-09-14 23:19–23:22 UTC. The host was an interactive workstation during the +run (Chrome, WindowServer, Spotlight, Docker Desktop); load averages were about +27 at start and end, recorded in `host-start.txt` and `host-end.txt`. Exact +binary hashes are in `hashes.txt`: + +- `before_cypher_end_to_end` is the retained `6544dc4` binary that produced the + [original failure](../cypher-conversion-6544dc4) (same SHA-256). +- `after_cypher_end_to_end` and `cypher_route` are built from branch + `claude/auto-datafusion-route` (base `a134e05`) with the zero-copy partition + split and the router. `cypher_route` forces each route, so the threshold + constant later set from these results does not affect them. + +## Partition fix, one million nodes + +Ten process pairs, alternating order, each running three trials per route with +the unchanged 256 MiB pool, spill disabled and four target partitions: + +| Binary | DataFusion trials | Memory exhaustion | Median DataFusion total (s) | +| --- | ---: | ---: | ---: | +| before | 30 | 2 | 0.798 | +| after | 30 | 0 | 0.797 | + +Both before-binary failures are the retained `Memory Exhausted while SpillPool +(DiskManager is disabled)` error; all 60 indexed trials passed. Two of thirty +against none of thirty is not statistically decisive by itself (two-sided +Fisher p ≈ 0.49). The structural evidence is stronger: the fixed physical plan +contains no `RoundRobinBatch` repartition (regression +`single_large_batches_are_split_without_round_robin`), and DataFusion 55.1 +`repartition/mod.rs` is where each queued slice was charged its shared parent +buffers. The fix did not change measured DataFusion time. + +## Warm route profile + +`cypher_route 5 1000 3000 10000 30000 100000 300000 1000000`: one `RoutedGraph` +per size, then the complete bounded-read entrypoint on each forced route, five +alternating trials. All trials passed with identical results on both routes. +Median seconds per query (q0 filtered count, q1 top-7 ids by ORDER BY, q2 +group-by with ORDER BY): + +| Nodes | q0 reference | q0 DataFusion | q1 reference | q1 DataFusion | q2 reference | q2 DataFusion | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 1,000 | 0.00087 | 0.00098 | 0.00088 | 0.00055 | 0.00172 | 0.00090 | +| 3,000 | 0.00223 | 0.00058 | 0.00227 | 0.00051 | 0.00449 | 0.00090 | +| 10,000 | 0.00809 | 0.00072 | 0.00828 | 0.00096 | 0.01746 | 0.00106 | +| 30,000 | 0.02698 | 0.00086 | 0.02802 | 0.00099 | 0.05764 | 0.00109 | +| 100,000 | 0.09058 | 0.00096 | 0.09583 | 0.00109 | 0.19741 | 0.00119 | +| 300,000 | 0.28274 | 0.00141 | 0.30139 | 0.00143 | 0.60173 | 0.00144 | +| 1,000,000 | 0.97369 | 0.00224 | 1.07067 | 0.00304 | 1.99825 | 0.00228 | + +One-time capture (index plus Arrow snapshot) took 0.0012 s at 1,000 nodes, +0.057 s at 100,000 and 1.055 s at 1,000,000, and is excluded from trials. The +reference route was faster only for q0 at 1,000 nodes, so +`DEFAULT_MIN_DATAFUSION_NODES` is 3,000. These are single-node scans on one +loaded host; they do not qualify joins, other shapes, other hosts or backends. diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/hashes.txt b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/hashes.txt new file mode 100644 index 00000000..10f44d83 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/hashes.txt @@ -0,0 +1,3 @@ +3f9578586e43a07c8f7c2e970ba4489ab0c8f3ece5e7542a2384d89469f54443 before_cypher_end_to_end +d4f4f68ad822f96c8fa54792f13004d4b49d681775b59782ae4aa1aa3341fd6c after_cypher_end_to_end +5e262f2d5a0914099da9991efd82cd3698e5e738184ee6d5a89c5393675520aa cypher_route diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/host-end.txt b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/host-end.txt new file mode 100644 index 00000000..797f41f9 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/host-end.txt @@ -0,0 +1,2 @@ +Mon Sep 14 23:22:45 UTC 2026 + 1:22 up 3 days, 7:24, 6 users, load averages: 26.96 25.72 24.88 diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/host-start.txt b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/host-start.txt new file mode 100644 index 00000000..513200ff --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/host-start.txt @@ -0,0 +1,12 @@ +Mon Sep 14 23:19:22 UTC 2026 + 1:19 up 3 days, 7:20, 6 users, load averages: 27.14 26.65 24.98 +10 +68719476736 + %CPU COMMAND + 43.8 /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/152.0.7977.84/Helpe + 24.1 /System/Library/PrivateFrameworks/SkyLight.framework/Resources/WindowServer -daemon + 24.1 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Support/mds + 23.6 /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/152.0.7977.84/Helpe + 15.3 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Support/mds_ + 12.0 /System/Library/Frameworks/WebKit.framework/Versions/A/XPCServices/com.apple.WebKit.WebContent.xpc/Contents/MacOS/ + 10.2 /Applications/Docker.app/Contents/MacOS/Docker Desktop.app/Contents/Frameworks/Docker Desktop Helper (Renderer).ap diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-0-after.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-0-after.jsonl new file mode 100644 index 00000000..e538c5f8 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-0-after.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"c658abce7ff26101f1ea75474098fac002e21de9-dirty","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.1175565,"query_seconds":1.641187583,"total_seconds":1.758744083} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.787802083,"query_seconds":0.011109917,"total_seconds":0.798912} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.7819865,"query_seconds":0.002232125,"total_seconds":0.7842186250000001} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.106516167,"query_seconds":1.526532,"total_seconds":1.633048167} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.117787666,"query_seconds":1.321058083,"total_seconds":1.438845749} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.791805042,"query_seconds":0.00229525,"total_seconds":0.794100292} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-0-before.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-0-before.jsonl new file mode 100644 index 00000000..faf27db0 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-0-before.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"6544dc410cf57addd06e2ba2057762ba713e953c","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.113686375,"query_seconds":1.9037511249999999,"total_seconds":2.0174374999999998} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.8163245,"query_seconds":0.012801583,"total_seconds":0.829126083} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.809607208,"query_seconds":0.002892708,"total_seconds":0.8124999159999999} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.116661792,"query_seconds":1.557994375,"total_seconds":1.674656167} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.106825584,"query_seconds":1.303186875,"total_seconds":1.410012459} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.79118775,"query_seconds":0.002813458,"total_seconds":0.794001208} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-1-after.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-1-after.jsonl new file mode 100644 index 00000000..28f4772d --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-1-after.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"c658abce7ff26101f1ea75474098fac002e21de9-dirty","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.119819209,"query_seconds":1.682823875,"total_seconds":1.802643084} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.8161745,"query_seconds":0.003446916,"total_seconds":0.819621416} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.795003042,"query_seconds":0.0020855,"total_seconds":0.797088542} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.108852917,"query_seconds":1.5090406669999998,"total_seconds":1.617893584} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.119839833,"query_seconds":1.2847146249999999,"total_seconds":1.4045544579999998} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.786687583,"query_seconds":0.002204083,"total_seconds":0.7888916659999999} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-1-before.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-1-before.jsonl new file mode 100644 index 00000000..b2995b3b --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-1-before.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"6544dc410cf57addd06e2ba2057762ba713e953c","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.108929542,"query_seconds":1.602852959,"total_seconds":1.711782501} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.790513042,"query_seconds":0.004116166,"total_seconds":0.794629208} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.792127334,"query_seconds":0.0027765,"total_seconds":0.794903834} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.106862708,"query_seconds":1.534455042,"total_seconds":1.64131775} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.111730125,"query_seconds":1.302880542,"total_seconds":1.414610667} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.791784167,"query_seconds":0.002815917,"total_seconds":0.7946000839999999} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-2-after.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-2-after.jsonl new file mode 100644 index 00000000..53c04ce4 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-2-after.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"c658abce7ff26101f1ea75474098fac002e21de9-dirty","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.124976875,"query_seconds":1.613661167,"total_seconds":1.738638042} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.799491125,"query_seconds":0.003307584,"total_seconds":0.8027987089999999} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.805642917,"query_seconds":0.002171541,"total_seconds":0.8078144580000001} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.112691958,"query_seconds":1.543033416,"total_seconds":1.6557253740000002} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.108725583,"query_seconds":1.302997875,"total_seconds":1.411723458} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.795059167,"query_seconds":0.002525375,"total_seconds":0.7975845420000001} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-2-before.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-2-before.jsonl new file mode 100644 index 00000000..d7a17f78 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-2-before.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"6544dc410cf57addd06e2ba2057762ba713e953c","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.1063815,"query_seconds":1.6543884580000001,"total_seconds":1.760769958} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.81134625,"query_seconds":0.003722416,"total_seconds":0.815068666} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.788453,"query_seconds":0.002969083,"total_seconds":0.7914220829999999} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.105954125,"query_seconds":1.509353459,"total_seconds":1.615307584} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.110220083,"query_seconds":1.326196458,"total_seconds":1.436416541} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.785775125,"query_seconds":0.003000083,"total_seconds":0.788775208} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-3-after.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-3-after.jsonl new file mode 100644 index 00000000..ec0396b4 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-3-after.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"c658abce7ff26101f1ea75474098fac002e21de9-dirty","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.125100791,"query_seconds":1.670597292,"total_seconds":1.795698083} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.8032505,"query_seconds":0.002983833,"total_seconds":0.806234333} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.799016333,"query_seconds":0.00198075,"total_seconds":0.8009970829999999} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.116069292,"query_seconds":1.560836291,"total_seconds":1.676905583} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.11724025,"query_seconds":1.329798084,"total_seconds":1.4470383340000001} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.79966675,"query_seconds":0.002332167,"total_seconds":0.801998917} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-3-before.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-3-before.jsonl new file mode 100644 index 00000000..795d8515 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-3-before.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"6544dc410cf57addd06e2ba2057762ba713e953c","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.125746584,"query_seconds":1.647837459,"total_seconds":1.773584043} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.796415666,"query_seconds":0.004196584,"total_seconds":0.80061225} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.783534042,"query_seconds":0.00267925,"total_seconds":0.786213292} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.101808666,"query_seconds":1.495656375,"total_seconds":1.597465041} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.104616209,"query_seconds":1.264162292,"total_seconds":1.368778501} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.776293,"query_seconds":0.002791125,"total_seconds":0.779084125} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-4-after.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-4-after.jsonl new file mode 100644 index 00000000..1371754c --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-4-after.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"c658abce7ff26101f1ea75474098fac002e21de9-dirty","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.127027459,"query_seconds":1.638463834,"total_seconds":1.765491293} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.803479917,"query_seconds":0.00287425,"total_seconds":0.806354167} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.790684834,"query_seconds":0.002081417,"total_seconds":0.792766251} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.107107708,"query_seconds":1.53724425,"total_seconds":1.6443519580000001} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.110954792,"query_seconds":1.3035311250000001,"total_seconds":1.4144859170000001} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.801743167,"query_seconds":0.002517833,"total_seconds":0.804261} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-4-before.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-4-before.jsonl new file mode 100644 index 00000000..c61ffc0c --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-4-before.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"6544dc410cf57addd06e2ba2057762ba713e953c","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.1200725,"query_seconds":1.692859,"total_seconds":1.8129315} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.798808208,"query_seconds":0.003666375,"total_seconds":0.802474583} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.796136834,"query_seconds":0.003045708,"total_seconds":0.799182542} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.105534125,"query_seconds":1.52270125,"total_seconds":1.6282353749999998} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.109070917,"query_seconds":1.32120875,"total_seconds":1.430279667} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.790427458,"query_seconds":0.002860167,"total_seconds":0.793287625} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-5-after.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-5-after.jsonl new file mode 100644 index 00000000..6e492dd2 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-5-after.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"c658abce7ff26101f1ea75474098fac002e21de9-dirty","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.126865583,"query_seconds":1.661257416,"total_seconds":1.788122999} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.788443959,"query_seconds":0.003469709,"total_seconds":0.791913668} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.791794,"query_seconds":0.002092834,"total_seconds":0.793886834} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.113126167,"query_seconds":1.541162875,"total_seconds":1.6542890419999998} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.110109125,"query_seconds":1.3153685419999999,"total_seconds":1.4254776669999998} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.788705209,"query_seconds":0.002345042,"total_seconds":0.7910502509999999} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-5-before.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-5-before.jsonl new file mode 100644 index 00000000..a588081a --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-5-before.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"6544dc410cf57addd06e2ba2057762ba713e953c","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.111283917,"query_seconds":1.6658263340000001,"total_seconds":1.777110251} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.81283025,"query_seconds":0.066450625,"total_seconds":0.879280875} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.791115875,"query_seconds":0.002763292,"total_seconds":0.793879167} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.10062625,"query_seconds":1.518578542,"total_seconds":1.619204792} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.102051708,"query_seconds":1.271767708,"total_seconds":1.3738194160000001} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.785395,"query_seconds":0.0033135,"total_seconds":0.7887084999999999} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-6-after.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-6-after.jsonl new file mode 100644 index 00000000..524bd60c --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-6-after.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"c658abce7ff26101f1ea75474098fac002e21de9-dirty","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.123249791,"query_seconds":1.600961458,"total_seconds":1.7242112490000001} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.801432375,"query_seconds":0.003352917,"total_seconds":0.8047852919999999} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.792244125,"query_seconds":0.002138833,"total_seconds":0.7943829579999999} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.108243666,"query_seconds":1.5383877080000001,"total_seconds":1.646631374} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.114342333,"query_seconds":1.284766083,"total_seconds":1.399108416} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.803029542,"query_seconds":0.003023625,"total_seconds":0.806053167} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-6-before.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-6-before.jsonl new file mode 100644 index 00000000..189dfbaa --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-6-before.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"6544dc410cf57addd06e2ba2057762ba713e953c","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.110853042,"query_seconds":1.595240709,"total_seconds":1.706093751} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.795525417,"query_seconds":0.004429,"total_seconds":0.799954417} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.804042333,"query_seconds":0.002685333,"total_seconds":0.806727666} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.100918,"query_seconds":1.5191712499999999,"total_seconds":1.62008925} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.111093791,"query_seconds":1.318318292,"total_seconds":1.429412083} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.799847625,"query_seconds":0.003403625,"total_seconds":0.80325125} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-7-after.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-7-after.jsonl new file mode 100644 index 00000000..fbfa02a0 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-7-after.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"c658abce7ff26101f1ea75474098fac002e21de9-dirty","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.12699525,"query_seconds":1.621926875,"total_seconds":1.748922125} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.792199916,"query_seconds":0.003162583,"total_seconds":0.795362499} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.7853145,"query_seconds":0.002110625,"total_seconds":0.7874251250000001} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.11249275,"query_seconds":1.5381845410000001,"total_seconds":1.650677291} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.12624,"query_seconds":1.324577917,"total_seconds":1.450817917} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.793801208,"query_seconds":0.002450833,"total_seconds":0.796252041} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-7-before.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-7-before.jsonl new file mode 100644 index 00000000..698713f1 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-7-before.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"6544dc410cf57addd06e2ba2057762ba713e953c","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.116719375,"query_seconds":1.655886041,"total_seconds":1.772605416} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.797733209,"query_seconds":0.008380083,"total_seconds":0.806113292} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.79434625,"query_seconds":0.002400916,"total_seconds":0.7967471660000001} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.104994917,"query_seconds":1.537308541,"total_seconds":1.642303458} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.105070708,"query_seconds":1.316973333,"total_seconds":1.422044041} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.799885875,"query_seconds":0.003036625,"total_seconds":0.8029225} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-8-after.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-8-after.jsonl new file mode 100644 index 00000000..1553fc53 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-8-after.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"c658abce7ff26101f1ea75474098fac002e21de9-dirty","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.12661925,"query_seconds":1.6195422499999999,"total_seconds":1.7461615} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.792280958,"query_seconds":0.00306225,"total_seconds":0.795343208} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.790088375,"query_seconds":0.0018995,"total_seconds":0.791987875} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.11561875,"query_seconds":1.531565875,"total_seconds":1.6471846250000002} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.110925625,"query_seconds":1.289372584,"total_seconds":1.400298209} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.802696417,"query_seconds":0.002168208,"total_seconds":0.804864625} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-8-before.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-8-before.jsonl new file mode 100644 index 00000000..d7d1d239 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-8-before.jsonl @@ -0,0 +1,8 @@ +{"event":"configuration","source":"6544dc410cf57addd06e2ba2057762ba713e953c","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.118263,"query_seconds":1.703276458,"total_seconds":1.821539458} +{"event":"trial","route":"datafusion","trial":0,"status":"error","error":"Resources exhausted: Memory Exhausted while SpillPool (DiskManager is disabled)"} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.804689875,"query_seconds":0.003078166,"total_seconds":0.807768041} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.10673325,"query_seconds":1.527333959,"total_seconds":1.634067209} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.099357958,"query_seconds":1.307706458,"total_seconds":1.4070644159999999} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.785979,"query_seconds":0.002519833,"total_seconds":0.788498833} +{"event":"error","error":"one or more trials did not pass; all outcomes retained"} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-9-after.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-9-after.jsonl new file mode 100644 index 00000000..1fb8a7da --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-9-after.jsonl @@ -0,0 +1,7 @@ +{"event":"configuration","source":"c658abce7ff26101f1ea75474098fac002e21de9-dirty","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.116864875,"query_seconds":1.585541292,"total_seconds":1.7024061670000001} +{"event":"trial","route":"datafusion","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.788590791,"query_seconds":0.003340125,"total_seconds":0.791930916} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.781381292,"query_seconds":0.002134041,"total_seconds":0.783515333} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.108684333,"query_seconds":1.572484292,"total_seconds":1.681168625} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.112617042,"query_seconds":1.298072125,"total_seconds":1.4106891670000001} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.795374459,"query_seconds":0.002620125,"total_seconds":0.797994584} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-9-before.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-9-before.jsonl new file mode 100644 index 00000000..1cb0bae6 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-9-before.jsonl @@ -0,0 +1,8 @@ +{"event":"configuration","source":"6544dc410cf57addd06e2ba2057762ba713e953c","nodes":1000000,"repeats":3,"query":"MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count","deadline":null,"datafusion_working_memory_bytes":268435456,"datafusion_target_partitions":4,"datafusion_input_partitions":1,"datafusion_output_rows":1,"datafusion_output_bytes":1024,"indexed_memory_limit":null,"spill":false,"boundary":"row graph fixture excluded; cold representation per trial; preparation includes index or Arrow conversion and engine registration; query includes parsing, planning, execution and portable output; teardown excluded; no equivalent full resource envelope or automatic routing threshold claimed"} +{"event":"trial","route":"indexed","trial":0,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.1129445,"query_seconds":1.622433333,"total_seconds":1.735377833} +{"event":"trial","route":"datafusion","trial":0,"status":"error","error":"Resources exhausted: Memory Exhausted while SpillPool (DiskManager is disabled)"} +{"event":"trial","route":"datafusion","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.774947583,"query_seconds":0.002728,"total_seconds":0.777675583} +{"event":"trial","route":"indexed","trial":1,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.094147958,"query_seconds":1.521060792,"total_seconds":1.61520875} +{"event":"trial","route":"indexed","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.102742583,"query_seconds":1.279228166,"total_seconds":1.381970749} +{"event":"trial","route":"datafusion","trial":2,"status":"pass","expected":62500,"columns":["count"],"rows":[[{"type":"int","value":62500}]],"preparation_seconds":0.82108325,"query_seconds":0.003140833,"total_seconds":0.824224083} +{"event":"error","error":"one or more trials did not pass; all outcomes retained"} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-exits.txt b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-exits.txt new file mode 100644 index 00000000..151c1d9d --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/oom-exits.txt @@ -0,0 +1,20 @@ +0 before exit 0 +0 after exit 0 +1 after exit 0 +1 before exit 0 +2 before exit 0 +2 after exit 0 +3 after exit 0 +3 before exit 0 +4 before exit 0 +4 after exit 0 +5 after exit 0 +5 before exit 0 +6 before exit 0 +6 after exit 0 +7 after exit 0 +7 before exit 0 +8 before exit 1 +8 after exit 0 +9 after exit 0 +9 before exit 1 diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/route-exit.txt b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/route-exit.txt new file mode 100644 index 00000000..7d386f94 --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/route-exit.txt @@ -0,0 +1 @@ +route exit 0 diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/route.jsonl b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/route.jsonl new file mode 100644 index 00000000..fb68169e --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/route.jsonl @@ -0,0 +1,218 @@ +{"event":"configuration","source":"c658abce7ff26101f1ea75474098fac002e21de9-dirty","repeats":5,"sizes":[1000,3000,10000,30000,100000,300000,1000000],"queries":["MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count LIMIT 1","MATCH (n:N) WHERE n.bucket >= 14 RETURN id(n) AS id ORDER BY id LIMIT 10","MATCH (n) RETURN n.bucket AS bucket, count(*) AS count ORDER BY bucket LIMIT 20"],"datafusion_working_memory_bytes":1073741824,"datafusion_target_partitions":4,"batch_rows":8192,"spill":false,"max_intermediate_bytes":4294967296,"boundary":"warm representations captured once per size; trial includes bounded admission, route planning, execution and portable output; both routes share one policy"} +{"event":"capture","nodes":1000,"seconds":0.001150708} +{"event":"trial","nodes":1000,"query":0,"route":"Reference","trial":0,"status":"pass","seconds":0.002050041,"rows":1} +{"event":"trial","nodes":1000,"query":0,"route":"DataFusion","trial":0,"status":"pass","seconds":0.012315083,"rows":1} +{"event":"trial","nodes":1000,"query":0,"route":"DataFusion","trial":1,"status":"pass","seconds":0.000892083,"rows":1} +{"event":"trial","nodes":1000,"query":0,"route":"Reference","trial":1,"status":"pass","seconds":0.001001625,"rows":1} +{"event":"trial","nodes":1000,"query":0,"route":"Reference","trial":2,"status":"pass","seconds":0.000853917,"rows":1} +{"event":"trial","nodes":1000,"query":0,"route":"DataFusion","trial":2,"status":"pass","seconds":0.0010175,"rows":1} +{"event":"trial","nodes":1000,"query":0,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000583125,"rows":1} +{"event":"trial","nodes":1000,"query":0,"route":"Reference","trial":3,"status":"pass","seconds":0.0008655,"rows":1} +{"event":"trial","nodes":1000,"query":0,"route":"Reference","trial":4,"status":"pass","seconds":0.000771042,"rows":1} +{"event":"trial","nodes":1000,"query":0,"route":"DataFusion","trial":4,"status":"pass","seconds":0.0009835,"rows":1} +{"event":"trial","nodes":1000,"query":1,"route":"Reference","trial":0,"status":"pass","seconds":0.000881875,"rows":10} +{"event":"trial","nodes":1000,"query":1,"route":"DataFusion","trial":0,"status":"pass","seconds":0.001925459,"rows":10} +{"event":"trial","nodes":1000,"query":1,"route":"DataFusion","trial":1,"status":"pass","seconds":0.000839709,"rows":10} +{"event":"trial","nodes":1000,"query":1,"route":"Reference","trial":1,"status":"pass","seconds":0.00122,"rows":10} +{"event":"trial","nodes":1000,"query":1,"route":"Reference","trial":2,"status":"pass","seconds":0.000809625,"rows":10} +{"event":"trial","nodes":1000,"query":1,"route":"DataFusion","trial":2,"status":"pass","seconds":0.000494375,"rows":10} +{"event":"trial","nodes":1000,"query":1,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000549459,"rows":10} +{"event":"trial","nodes":1000,"query":1,"route":"Reference","trial":3,"status":"pass","seconds":0.001092917,"rows":10} +{"event":"trial","nodes":1000,"query":1,"route":"Reference","trial":4,"status":"pass","seconds":0.000787042,"rows":10} +{"event":"trial","nodes":1000,"query":1,"route":"DataFusion","trial":4,"status":"pass","seconds":0.000390083,"rows":10} +{"event":"trial","nodes":1000,"query":2,"route":"Reference","trial":0,"status":"pass","seconds":0.001715042,"rows":16} +{"event":"trial","nodes":1000,"query":2,"route":"DataFusion","trial":0,"status":"pass","seconds":0.001742583,"rows":16} +{"event":"trial","nodes":1000,"query":2,"route":"DataFusion","trial":1,"status":"pass","seconds":0.00075925,"rows":16} +{"event":"trial","nodes":1000,"query":2,"route":"Reference","trial":1,"status":"pass","seconds":0.002050792,"rows":16} +{"event":"trial","nodes":1000,"query":2,"route":"Reference","trial":2,"status":"pass","seconds":0.001847708,"rows":16} +{"event":"trial","nodes":1000,"query":2,"route":"DataFusion","trial":2,"status":"pass","seconds":0.000902708,"rows":16} +{"event":"trial","nodes":1000,"query":2,"route":"DataFusion","trial":3,"status":"pass","seconds":0.001020625,"rows":16} +{"event":"trial","nodes":1000,"query":2,"route":"Reference","trial":3,"status":"pass","seconds":0.001638958,"rows":16} +{"event":"trial","nodes":1000,"query":2,"route":"Reference","trial":4,"status":"pass","seconds":0.001674833,"rows":16} +{"event":"trial","nodes":1000,"query":2,"route":"DataFusion","trial":4,"status":"pass","seconds":0.000765958,"rows":16} +{"event":"capture","nodes":3000,"seconds":0.00134475} +{"event":"trial","nodes":3000,"query":0,"route":"Reference","trial":0,"status":"pass","seconds":0.003337208,"rows":1} +{"event":"trial","nodes":3000,"query":0,"route":"DataFusion","trial":0,"status":"pass","seconds":0.000735042,"rows":1} +{"event":"trial","nodes":3000,"query":0,"route":"DataFusion","trial":1,"status":"pass","seconds":0.000547875,"rows":1} +{"event":"trial","nodes":3000,"query":0,"route":"Reference","trial":1,"status":"pass","seconds":0.002743792,"rows":1} +{"event":"trial","nodes":3000,"query":0,"route":"Reference","trial":2,"status":"pass","seconds":0.002157084,"rows":1} +{"event":"trial","nodes":3000,"query":0,"route":"DataFusion","trial":2,"status":"pass","seconds":0.000583125,"rows":1} +{"event":"trial","nodes":3000,"query":0,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000799833,"rows":1} +{"event":"trial","nodes":3000,"query":0,"route":"Reference","trial":3,"status":"pass","seconds":0.00222875,"rows":1} +{"event":"trial","nodes":3000,"query":0,"route":"Reference","trial":4,"status":"pass","seconds":0.002029667,"rows":1} +{"event":"trial","nodes":3000,"query":0,"route":"DataFusion","trial":4,"status":"pass","seconds":0.00054225,"rows":1} +{"event":"trial","nodes":3000,"query":1,"route":"Reference","trial":0,"status":"pass","seconds":0.002269458,"rows":10} +{"event":"trial","nodes":3000,"query":1,"route":"DataFusion","trial":0,"status":"pass","seconds":0.00069175,"rows":10} +{"event":"trial","nodes":3000,"query":1,"route":"DataFusion","trial":1,"status":"pass","seconds":0.000506125,"rows":10} +{"event":"trial","nodes":3000,"query":1,"route":"Reference","trial":1,"status":"pass","seconds":0.002371,"rows":10} +{"event":"trial","nodes":3000,"query":1,"route":"Reference","trial":2,"status":"pass","seconds":0.002078333,"rows":10} +{"event":"trial","nodes":3000,"query":1,"route":"DataFusion","trial":2,"status":"pass","seconds":0.000514708,"rows":10} +{"event":"trial","nodes":3000,"query":1,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000375875,"rows":10} +{"event":"trial","nodes":3000,"query":1,"route":"Reference","trial":3,"status":"pass","seconds":0.002429458,"rows":10} +{"event":"trial","nodes":3000,"query":1,"route":"Reference","trial":4,"status":"pass","seconds":0.002164042,"rows":10} +{"event":"trial","nodes":3000,"query":1,"route":"DataFusion","trial":4,"status":"pass","seconds":0.000549916,"rows":10} +{"event":"trial","nodes":3000,"query":2,"route":"Reference","trial":0,"status":"pass","seconds":0.004490291,"rows":16} +{"event":"trial","nodes":3000,"query":2,"route":"DataFusion","trial":0,"status":"pass","seconds":0.000868334,"rows":16} +{"event":"trial","nodes":3000,"query":2,"route":"DataFusion","trial":1,"status":"pass","seconds":0.000603,"rows":16} +{"event":"trial","nodes":3000,"query":2,"route":"Reference","trial":1,"status":"pass","seconds":0.004510834,"rows":16} +{"event":"trial","nodes":3000,"query":2,"route":"Reference","trial":2,"status":"pass","seconds":0.004024083,"rows":16} +{"event":"trial","nodes":3000,"query":2,"route":"DataFusion","trial":2,"status":"pass","seconds":0.000900875,"rows":16} +{"event":"trial","nodes":3000,"query":2,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000903375,"rows":16} +{"event":"trial","nodes":3000,"query":2,"route":"Reference","trial":3,"status":"pass","seconds":0.004553958,"rows":16} +{"event":"trial","nodes":3000,"query":2,"route":"Reference","trial":4,"status":"pass","seconds":0.004431208,"rows":16} +{"event":"trial","nodes":3000,"query":2,"route":"DataFusion","trial":4,"status":"pass","seconds":0.001017041,"rows":16} +{"event":"capture","nodes":10000,"seconds":0.003809459} +{"event":"trial","nodes":10000,"query":0,"route":"Reference","trial":0,"status":"pass","seconds":0.009893916,"rows":1} +{"event":"trial","nodes":10000,"query":0,"route":"DataFusion","trial":0,"status":"pass","seconds":0.0009905,"rows":1} +{"event":"trial","nodes":10000,"query":0,"route":"DataFusion","trial":1,"status":"pass","seconds":0.000500417,"rows":1} +{"event":"trial","nodes":10000,"query":0,"route":"Reference","trial":1,"status":"pass","seconds":0.008274709,"rows":1} +{"event":"trial","nodes":10000,"query":0,"route":"Reference","trial":2,"status":"pass","seconds":0.008086334,"rows":1} +{"event":"trial","nodes":10000,"query":0,"route":"DataFusion","trial":2,"status":"pass","seconds":0.000718417,"rows":1} +{"event":"trial","nodes":10000,"query":0,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000452708,"rows":1} +{"event":"trial","nodes":10000,"query":0,"route":"Reference","trial":3,"status":"pass","seconds":0.007773333,"rows":1} +{"event":"trial","nodes":10000,"query":0,"route":"Reference","trial":4,"status":"pass","seconds":0.008086708,"rows":1} +{"event":"trial","nodes":10000,"query":0,"route":"DataFusion","trial":4,"status":"pass","seconds":0.0007625,"rows":1} +{"event":"trial","nodes":10000,"query":1,"route":"Reference","trial":0,"status":"pass","seconds":0.008264708,"rows":10} +{"event":"trial","nodes":10000,"query":1,"route":"DataFusion","trial":0,"status":"pass","seconds":0.000968667,"rows":10} +{"event":"trial","nodes":10000,"query":1,"route":"DataFusion","trial":1,"status":"pass","seconds":0.000605666,"rows":10} +{"event":"trial","nodes":10000,"query":1,"route":"Reference","trial":1,"status":"pass","seconds":0.00872325,"rows":10} +{"event":"trial","nodes":10000,"query":1,"route":"Reference","trial":2,"status":"pass","seconds":0.007762875,"rows":10} +{"event":"trial","nodes":10000,"query":1,"route":"DataFusion","trial":2,"status":"pass","seconds":0.000980041,"rows":10} +{"event":"trial","nodes":10000,"query":1,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000628042,"rows":10} +{"event":"trial","nodes":10000,"query":1,"route":"Reference","trial":3,"status":"pass","seconds":0.008280333,"rows":10} +{"event":"trial","nodes":10000,"query":1,"route":"Reference","trial":4,"status":"pass","seconds":0.008497875,"rows":10} +{"event":"trial","nodes":10000,"query":1,"route":"DataFusion","trial":4,"status":"pass","seconds":0.00096125,"rows":10} +{"event":"trial","nodes":10000,"query":2,"route":"Reference","trial":0,"status":"pass","seconds":0.020606875,"rows":16} +{"event":"trial","nodes":10000,"query":2,"route":"DataFusion","trial":0,"status":"pass","seconds":0.001239042,"rows":16} +{"event":"trial","nodes":10000,"query":2,"route":"DataFusion","trial":1,"status":"pass","seconds":0.000832334,"rows":16} +{"event":"trial","nodes":10000,"query":2,"route":"Reference","trial":1,"status":"pass","seconds":0.019303208,"rows":16} +{"event":"trial","nodes":10000,"query":2,"route":"Reference","trial":2,"status":"pass","seconds":0.017458,"rows":16} +{"event":"trial","nodes":10000,"query":2,"route":"DataFusion","trial":2,"status":"pass","seconds":0.001057291,"rows":16} +{"event":"trial","nodes":10000,"query":2,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000641959,"rows":16} +{"event":"trial","nodes":10000,"query":2,"route":"Reference","trial":3,"status":"pass","seconds":0.0169605,"rows":16} +{"event":"trial","nodes":10000,"query":2,"route":"Reference","trial":4,"status":"pass","seconds":0.017379416,"rows":16} +{"event":"trial","nodes":10000,"query":2,"route":"DataFusion","trial":4,"status":"pass","seconds":0.001079958,"rows":16} +{"event":"capture","nodes":30000,"seconds":0.014663625} +{"event":"trial","nodes":30000,"query":0,"route":"Reference","trial":0,"status":"pass","seconds":0.034941666,"rows":1} +{"event":"trial","nodes":30000,"query":0,"route":"DataFusion","trial":0,"status":"pass","seconds":0.000940292,"rows":1} +{"event":"trial","nodes":30000,"query":0,"route":"DataFusion","trial":1,"status":"pass","seconds":0.0005825,"rows":1} +{"event":"trial","nodes":30000,"query":0,"route":"Reference","trial":1,"status":"pass","seconds":0.026978208,"rows":1} +{"event":"trial","nodes":30000,"query":0,"route":"Reference","trial":2,"status":"pass","seconds":0.026954208,"rows":1} +{"event":"trial","nodes":30000,"query":0,"route":"DataFusion","trial":2,"status":"pass","seconds":0.000937292,"rows":1} +{"event":"trial","nodes":30000,"query":0,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000512292,"rows":1} +{"event":"trial","nodes":30000,"query":0,"route":"Reference","trial":3,"status":"pass","seconds":0.027328625,"rows":1} +{"event":"trial","nodes":30000,"query":0,"route":"Reference","trial":4,"status":"pass","seconds":0.026671,"rows":1} +{"event":"trial","nodes":30000,"query":0,"route":"DataFusion","trial":4,"status":"pass","seconds":0.000855416,"rows":1} +{"event":"trial","nodes":30000,"query":1,"route":"Reference","trial":0,"status":"pass","seconds":0.027946208,"rows":10} +{"event":"trial","nodes":30000,"query":1,"route":"DataFusion","trial":0,"status":"pass","seconds":0.001033583,"rows":10} +{"event":"trial","nodes":30000,"query":1,"route":"DataFusion","trial":1,"status":"pass","seconds":0.000840584,"rows":10} +{"event":"trial","nodes":30000,"query":1,"route":"Reference","trial":1,"status":"pass","seconds":0.027969875,"rows":10} +{"event":"trial","nodes":30000,"query":1,"route":"Reference","trial":2,"status":"pass","seconds":0.028017,"rows":10} +{"event":"trial","nodes":30000,"query":1,"route":"DataFusion","trial":2,"status":"pass","seconds":0.0011725,"rows":10} +{"event":"trial","nodes":30000,"query":1,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000818958,"rows":10} +{"event":"trial","nodes":30000,"query":1,"route":"Reference","trial":3,"status":"pass","seconds":0.028425042,"rows":10} +{"event":"trial","nodes":30000,"query":1,"route":"Reference","trial":4,"status":"pass","seconds":0.028443584,"rows":10} +{"event":"trial","nodes":30000,"query":1,"route":"DataFusion","trial":4,"status":"pass","seconds":0.000986083,"rows":10} +{"event":"trial","nodes":30000,"query":2,"route":"Reference","trial":0,"status":"pass","seconds":0.065309292,"rows":16} +{"event":"trial","nodes":30000,"query":2,"route":"DataFusion","trial":0,"status":"pass","seconds":0.001110416,"rows":16} +{"event":"trial","nodes":30000,"query":2,"route":"DataFusion","trial":1,"status":"pass","seconds":0.000715875,"rows":16} +{"event":"trial","nodes":30000,"query":2,"route":"Reference","trial":1,"status":"pass","seconds":0.059280042,"rows":16} +{"event":"trial","nodes":30000,"query":2,"route":"Reference","trial":2,"status":"pass","seconds":0.056493292,"rows":16} +{"event":"trial","nodes":30000,"query":2,"route":"DataFusion","trial":2,"status":"pass","seconds":0.00109,"rows":16} +{"event":"trial","nodes":30000,"query":2,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000659792,"rows":16} +{"event":"trial","nodes":30000,"query":2,"route":"Reference","trial":3,"status":"pass","seconds":0.057644708,"rows":16} +{"event":"trial","nodes":30000,"query":2,"route":"Reference","trial":4,"status":"pass","seconds":0.056197416,"rows":16} +{"event":"trial","nodes":30000,"query":2,"route":"DataFusion","trial":4,"status":"pass","seconds":0.001130375,"rows":16} +{"event":"capture","nodes":100000,"seconds":0.057476667} +{"event":"trial","nodes":100000,"query":0,"route":"Reference","trial":0,"status":"pass","seconds":0.126999042,"rows":1} +{"event":"trial","nodes":100000,"query":0,"route":"DataFusion","trial":0,"status":"pass","seconds":0.000989125,"rows":1} +{"event":"trial","nodes":100000,"query":0,"route":"DataFusion","trial":1,"status":"pass","seconds":0.00061975,"rows":1} +{"event":"trial","nodes":100000,"query":0,"route":"Reference","trial":1,"status":"pass","seconds":0.097522625,"rows":1} +{"event":"trial","nodes":100000,"query":0,"route":"Reference","trial":2,"status":"pass","seconds":0.089181958,"rows":1} +{"event":"trial","nodes":100000,"query":0,"route":"DataFusion","trial":2,"status":"pass","seconds":0.001018667,"rows":1} +{"event":"trial","nodes":100000,"query":0,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000904375,"rows":1} +{"event":"trial","nodes":100000,"query":0,"route":"Reference","trial":3,"status":"pass","seconds":0.090198375,"rows":1} +{"event":"trial","nodes":100000,"query":0,"route":"Reference","trial":4,"status":"pass","seconds":0.090577208,"rows":1} +{"event":"trial","nodes":100000,"query":0,"route":"DataFusion","trial":4,"status":"pass","seconds":0.000959042,"rows":1} +{"event":"trial","nodes":100000,"query":1,"route":"Reference","trial":0,"status":"pass","seconds":0.095939125,"rows":10} +{"event":"trial","nodes":100000,"query":1,"route":"DataFusion","trial":0,"status":"pass","seconds":0.001142125,"rows":10} +{"event":"trial","nodes":100000,"query":1,"route":"DataFusion","trial":1,"status":"pass","seconds":0.000940583,"rows":10} +{"event":"trial","nodes":100000,"query":1,"route":"Reference","trial":1,"status":"pass","seconds":0.09546,"rows":10} +{"event":"trial","nodes":100000,"query":1,"route":"Reference","trial":2,"status":"pass","seconds":0.095357666,"rows":10} +{"event":"trial","nodes":100000,"query":1,"route":"DataFusion","trial":2,"status":"pass","seconds":0.001174667,"rows":10} +{"event":"trial","nodes":100000,"query":1,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000798042,"rows":10} +{"event":"trial","nodes":100000,"query":1,"route":"Reference","trial":3,"status":"pass","seconds":0.096302833,"rows":10} +{"event":"trial","nodes":100000,"query":1,"route":"Reference","trial":4,"status":"pass","seconds":0.095826291,"rows":10} +{"event":"trial","nodes":100000,"query":1,"route":"DataFusion","trial":4,"status":"pass","seconds":0.00109475,"rows":10} +{"event":"trial","nodes":100000,"query":2,"route":"Reference","trial":0,"status":"pass","seconds":0.219252375,"rows":16} +{"event":"trial","nodes":100000,"query":2,"route":"DataFusion","trial":0,"status":"pass","seconds":0.001277125,"rows":16} +{"event":"trial","nodes":100000,"query":2,"route":"DataFusion","trial":1,"status":"pass","seconds":0.001087292,"rows":16} +{"event":"trial","nodes":100000,"query":2,"route":"Reference","trial":1,"status":"pass","seconds":0.205806042,"rows":16} +{"event":"trial","nodes":100000,"query":2,"route":"Reference","trial":2,"status":"pass","seconds":0.192145208,"rows":16} +{"event":"trial","nodes":100000,"query":2,"route":"DataFusion","trial":2,"status":"pass","seconds":0.001223375,"rows":16} +{"event":"trial","nodes":100000,"query":2,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000817125,"rows":16} +{"event":"trial","nodes":100000,"query":2,"route":"Reference","trial":3,"status":"pass","seconds":0.195364542,"rows":16} +{"event":"trial","nodes":100000,"query":2,"route":"Reference","trial":4,"status":"pass","seconds":0.1974075,"rows":16} +{"event":"trial","nodes":100000,"query":2,"route":"DataFusion","trial":4,"status":"pass","seconds":0.001192708,"rows":16} +{"event":"capture","nodes":300000,"seconds":0.218629416} +{"event":"trial","nodes":300000,"query":0,"route":"Reference","trial":0,"status":"pass","seconds":0.350275792,"rows":1} +{"event":"trial","nodes":300000,"query":0,"route":"DataFusion","trial":0,"status":"pass","seconds":0.001529916,"rows":1} +{"event":"trial","nodes":300000,"query":0,"route":"DataFusion","trial":1,"status":"pass","seconds":0.001168291,"rows":1} +{"event":"trial","nodes":300000,"query":0,"route":"Reference","trial":1,"status":"pass","seconds":0.282387334,"rows":1} +{"event":"trial","nodes":300000,"query":0,"route":"Reference","trial":2,"status":"pass","seconds":0.288045375,"rows":1} +{"event":"trial","nodes":300000,"query":0,"route":"DataFusion","trial":2,"status":"pass","seconds":0.001405708,"rows":1} +{"event":"trial","nodes":300000,"query":0,"route":"DataFusion","trial":3,"status":"pass","seconds":0.001185458,"rows":1} +{"event":"trial","nodes":300000,"query":0,"route":"Reference","trial":3,"status":"pass","seconds":0.282740917,"rows":1} +{"event":"trial","nodes":300000,"query":0,"route":"Reference","trial":4,"status":"pass","seconds":0.281708875,"rows":1} +{"event":"trial","nodes":300000,"query":0,"route":"DataFusion","trial":4,"status":"pass","seconds":0.00140775,"rows":1} +{"event":"trial","nodes":300000,"query":1,"route":"Reference","trial":0,"status":"pass","seconds":0.300184208,"rows":10} +{"event":"trial","nodes":300000,"query":1,"route":"DataFusion","trial":0,"status":"pass","seconds":0.001464583,"rows":10} +{"event":"trial","nodes":300000,"query":1,"route":"DataFusion","trial":1,"status":"pass","seconds":0.001158459,"rows":10} +{"event":"trial","nodes":300000,"query":1,"route":"Reference","trial":1,"status":"pass","seconds":0.301391375,"rows":10} +{"event":"trial","nodes":300000,"query":1,"route":"Reference","trial":2,"status":"pass","seconds":0.30277375,"rows":10} +{"event":"trial","nodes":300000,"query":1,"route":"DataFusion","trial":2,"status":"pass","seconds":0.001426292,"rows":10} +{"event":"trial","nodes":300000,"query":1,"route":"DataFusion","trial":3,"status":"pass","seconds":0.001176708,"rows":10} +{"event":"trial","nodes":300000,"query":1,"route":"Reference","trial":3,"status":"pass","seconds":0.30126075,"rows":10} +{"event":"trial","nodes":300000,"query":1,"route":"Reference","trial":4,"status":"pass","seconds":0.303956708,"rows":10} +{"event":"trial","nodes":300000,"query":1,"route":"DataFusion","trial":4,"status":"pass","seconds":0.001579083,"rows":10} +{"event":"trial","nodes":300000,"query":2,"route":"Reference","trial":0,"status":"pass","seconds":0.663294125,"rows":16} +{"event":"trial","nodes":300000,"query":2,"route":"DataFusion","trial":0,"status":"pass","seconds":0.0015255,"rows":16} +{"event":"trial","nodes":300000,"query":2,"route":"DataFusion","trial":1,"status":"pass","seconds":0.001032583,"rows":16} +{"event":"trial","nodes":300000,"query":2,"route":"Reference","trial":1,"status":"pass","seconds":0.653253,"rows":16} +{"event":"trial","nodes":300000,"query":2,"route":"Reference","trial":2,"status":"pass","seconds":0.60173375,"rows":16} +{"event":"trial","nodes":300000,"query":2,"route":"DataFusion","trial":2,"status":"pass","seconds":0.001443791,"rows":16} +{"event":"trial","nodes":300000,"query":2,"route":"DataFusion","trial":3,"status":"pass","seconds":0.000947459,"rows":16} +{"event":"trial","nodes":300000,"query":2,"route":"Reference","trial":3,"status":"pass","seconds":0.597833333,"rows":16} +{"event":"trial","nodes":300000,"query":2,"route":"Reference","trial":4,"status":"pass","seconds":0.599393,"rows":16} +{"event":"trial","nodes":300000,"query":2,"route":"DataFusion","trial":4,"status":"pass","seconds":0.001469833,"rows":16} +{"event":"capture","nodes":1000000,"seconds":1.055499125} +{"event":"trial","nodes":1000000,"query":0,"route":"Reference","trial":0,"status":"pass","seconds":1.262426042,"rows":1} +{"event":"trial","nodes":1000000,"query":0,"route":"DataFusion","trial":0,"status":"pass","seconds":0.002937709,"rows":1} +{"event":"trial","nodes":1000000,"query":0,"route":"DataFusion","trial":1,"status":"pass","seconds":0.002234667,"rows":1} +{"event":"trial","nodes":1000000,"query":0,"route":"Reference","trial":1,"status":"pass","seconds":0.953652583,"rows":1} +{"event":"trial","nodes":1000000,"query":0,"route":"Reference","trial":2,"status":"pass","seconds":0.970394125,"rows":1} +{"event":"trial","nodes":1000000,"query":0,"route":"DataFusion","trial":2,"status":"pass","seconds":0.002467417,"rows":1} +{"event":"trial","nodes":1000000,"query":0,"route":"DataFusion","trial":3,"status":"pass","seconds":0.002238917,"rows":1} +{"event":"trial","nodes":1000000,"query":0,"route":"Reference","trial":3,"status":"pass","seconds":0.973694875,"rows":1} +{"event":"trial","nodes":1000000,"query":0,"route":"Reference","trial":4,"status":"pass","seconds":0.994395292,"rows":1} +{"event":"trial","nodes":1000000,"query":0,"route":"DataFusion","trial":4,"status":"pass","seconds":0.002173958,"rows":1} +{"event":"trial","nodes":1000000,"query":1,"route":"Reference","trial":0,"status":"pass","seconds":1.07066925,"rows":10} +{"event":"trial","nodes":1000000,"query":1,"route":"DataFusion","trial":0,"status":"pass","seconds":0.021220583,"rows":10} +{"event":"trial","nodes":1000000,"query":1,"route":"DataFusion","trial":1,"status":"pass","seconds":0.007289291,"rows":10} +{"event":"trial","nodes":1000000,"query":1,"route":"Reference","trial":1,"status":"pass","seconds":1.098933458,"rows":10} +{"event":"trial","nodes":1000000,"query":1,"route":"Reference","trial":2,"status":"pass","seconds":1.129205208,"rows":10} +{"event":"trial","nodes":1000000,"query":1,"route":"DataFusion","trial":2,"status":"pass","seconds":0.002843917,"rows":10} +{"event":"trial","nodes":1000000,"query":1,"route":"DataFusion","trial":3,"status":"pass","seconds":0.002551416,"rows":10} +{"event":"trial","nodes":1000000,"query":1,"route":"Reference","trial":3,"status":"pass","seconds":1.058206334,"rows":10} +{"event":"trial","nodes":1000000,"query":1,"route":"Reference","trial":4,"status":"pass","seconds":1.056596916,"rows":10} +{"event":"trial","nodes":1000000,"query":1,"route":"DataFusion","trial":4,"status":"pass","seconds":0.003040959,"rows":10} +{"event":"trial","nodes":1000000,"query":2,"route":"Reference","trial":0,"status":"pass","seconds":2.4923315,"rows":16} +{"event":"trial","nodes":1000000,"query":2,"route":"DataFusion","trial":0,"status":"pass","seconds":0.002276,"rows":16} +{"event":"trial","nodes":1000000,"query":2,"route":"DataFusion","trial":1,"status":"pass","seconds":0.001903,"rows":16} +{"event":"trial","nodes":1000000,"query":2,"route":"Reference","trial":1,"status":"pass","seconds":2.139240959,"rows":16} +{"event":"trial","nodes":1000000,"query":2,"route":"Reference","trial":2,"status":"pass","seconds":1.9982467910000001,"rows":16} +{"event":"trial","nodes":1000000,"query":2,"route":"DataFusion","trial":2,"status":"pass","seconds":0.002275708,"rows":16} +{"event":"trial","nodes":1000000,"query":2,"route":"DataFusion","trial":3,"status":"pass","seconds":0.001804208,"rows":16} +{"event":"trial","nodes":1000000,"query":2,"route":"Reference","trial":3,"status":"pass","seconds":1.990349625,"rows":16} +{"event":"trial","nodes":1000000,"query":2,"route":"Reference","trial":4,"status":"pass","seconds":1.997358792,"rows":16} +{"event":"trial","nodes":1000000,"query":2,"route":"DataFusion","trial":4,"status":"pass","seconds":0.002291042,"rows":16} diff --git a/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/workspace-gate-quegee.txt b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/workspace-gate-quegee.txt new file mode 100644 index 00000000..6b5f402c --- /dev/null +++ b/benchmarks/arrow-pipelines/evidence/cypher-route-capitola-20260914/workspace-gate-quegee.txt @@ -0,0 +1,186 @@ +Workspace gate on quegee (x86_64 Debian 13, 16 cores), 2026-09-14 +Base a134e05 plus this commit's changes, with an uncommitted lbug git pin to ladybug-rust 47bfd12e (not in this commit). +Command: LBUG_LOCALIZE_BUNDLED_SYMBOLS=1 cargo test --workspace --all-features --no-fail-fast; then cargo clippy --workspace --all-features --all-targets -- -D warnings +rustc 1.98.1 (48a229cea 2026-09-01) + + Running unittests src/lib.rs (target/debug/deps/grust_algorithm_procedures-97a05717b7e99c02) +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/cypher.rs (target/debug/deps/cypher-fda7a076462413a5) +test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/preparation.rs (target/debug/deps/preparation-72552f25bdca3e37) +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running unittests src/lib.rs (target/debug/deps/grust_algorithms-088f3926b8c4c452) +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/arrow_results.rs (target/debug/deps/arrow_results-e777ef02b03d9776) +test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/degree.rs (target/debug/deps/degree-ad8badbef6ae616a) +test result_allocation_obeys_the_shared_memory_envelope ... ok +test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s + Running tests/kernels.rs (target/debug/deps/kernels-0599a8284d1d3188) +test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s + Running tests/projection.rs (target/debug/deps/projection-11d2b0ae4abe6a99) +test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/reachability_oracle.rs (target/debug/deps/reachability_oracle-f1f014b38b421430) +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s + Running tests/weighted.rs (target/debug/deps/weighted-dfaa52caab2ecdbd) +test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s + Running unittests src/lib.rs (target/debug/deps/grust_arrow-aba632ed50a08ece) +test result: ok. 78 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/roundtrip.rs (target/debug/deps/roundtrip-5333c0c7893ca93c) +test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running unittests src/lib.rs (target/debug/deps/grust_beads-d4bbe0f7aad2d821) +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running unittests src/main.rs (target/debug/deps/grust_beads-8c3ef1c59b4d8433) +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/load.rs (target/debug/deps/load-5e01247a15cfe4c2) +test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running unittests src/lib.rs (target/debug/deps/grust_cocoindex-99d03bc4ec722b10) +test result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/public_export.rs (target/debug/deps/public_export-533c85e8a8406428) +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running unittests src/lib.rs (target/debug/deps/grust_core-62af2a096599bbb9) +test result: ok. 81 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s + Running unittests src/lib.rs (target/debug/deps/grust_cypher-8cebec2c85e79fb3) +test result: ok. 825 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.36s + Running tests/gql_conformance.rs (target/debug/deps/gql_conformance-0841ab163f613596) +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/indexed_memory.rs (target/debug/deps/indexed_memory-1a0dd0f20230550c) +test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/indexed_memory_lean.rs (target/debug/deps/indexed_memory_lean-89fd9bc5622ac925) +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s + Running tests/integer_ordering.rs (target/debug/deps/integer_ordering-da66265ab27f4e25) +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/procedure_snapshot_materialization.rs (target/debug/deps/procedure_snapshot_materialization-718b3a9161049032) +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/read_conformance.rs (target/debug/deps/read_conformance-369232f1e770e03d) +test result: ok. 38 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.07s + Running tests/write_golden.rs (target/debug/deps/write_golden-9eee97afecc0fddd) +test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running unittests src/lib.rs (target/debug/deps/grust_datafusion-5728151f18c8d585) +test result: ok. 57 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.52s + Running unittests src/lib.rs (target/debug/deps/grust_falkor-a4a690e233795384) +test result: ok. 12 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running unittests src/lib.rs (target/debug/deps/grust-5456a1a8187940da) +test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s + Running tests/algorithm_snapshots.rs (target/debug/deps/algorithm_snapshots-3abb538a6e21f736) +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s + Running tests/datafusion_cypher.rs (target/debug/deps/datafusion_cypher-599165cb10461e25) +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s + Running tests/indexed_read.rs (target/debug/deps/indexed_read-daae3a29921769a2) +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running unittests src/lib.rs (target/debug/deps/grust_helix-1ce34f666b9c6459) +test result: ok. 21 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.09s + Running unittests src/lib.rs (target/debug/deps/grust_ladybug-113d849d9ae6cf9d) +test result: ok. 18 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.88s + Running unittests src/lib.rs (target/debug/deps/grust_lancedb-fd91b6e692b57534) +test result: ok. 32 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 42.11s + Running unittests src/lib.rs (target/debug/deps/grust_memory-0495574203ab8fb8) +test result: ok. 38 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s + Running unittests src/lib.rs (target/debug/deps/grust_pggraph-024538e94f0bf429) +test result: ok. 2 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.01s + Running unittests src/lib.rs (target/debug/deps/grust_postgres-50e8bee05f53d5f6) +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running unittests src/lib.rs (target/debug/deps/grust_postgres_core-0f2d0dbc2eaba564) +test result: ok. 18 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/pg_cypher_live.rs (target/debug/deps/pg_cypher_live-56a1a25f8f9f14cd) +test result: ok. 0 passed; 0 failed; 5 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/scalar_count_sql.rs (target/debug/deps/scalar_count_sql-54b652f959708f3a) +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s + Running unittests src/lib.rs (target/debug/deps/grust_postgres_pgq-0e5c35f137346a89) +test result: ok. 5 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running unittests src/lib.rs (target/debug/deps/grust_procedures-5dcb295f6eb17158) +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/contracts.rs (target/debug/deps/contracts-14891690ca7b1ee3) +test result: ok. 23 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running unittests src/lib.rs (target/debug/deps/grust_sail-4539d18f6605596b) +test result: ok. 65 passed; 0 failed; 29 ignored; 0 measured; 0 filtered out; finished in 0.02s + Running tests/not_null_live.rs (target/debug/deps/not_null_live-dacf60d224e747f1) +test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/temp_view_live.rs (target/debug/deps/temp_view_live-eb1e0d8d4382af7b) +test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running unittests src/lib.rs (target/debug/deps/grust_sql_core-84780f100c21b9bc) +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running unittests src/lib.rs (target/debug/deps/grust_surreal-d33207a709b3eb7d) +test result: ok. 46 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.08s + Running unittests src/lib.rs (target/debug/deps/grust_turso-fa8f62207072ac87) +test result: ok. 18 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.63s + Running tests/exact_scalar_strings.rs (target/debug/deps/exact_scalar_strings-7e9e77c8c8564270) +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.39s + Running tests/guarded_commit.rs (target/debug/deps/guarded_commit-065cfe7d6bc5f0e7) +test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.03s + Running tests/parallel_edges.rs (target/debug/deps/parallel_edges-2b9d02b985d72145) +test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.16s + Running tests/read_pushdown_oracle.rs (target/debug/deps/read_pushdown_oracle-043a0822876863b2) +test result: ok. 22 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.26s + Running tests/relationship_uniqueness.rs (target/debug/deps/relationship_uniqueness-dc9c9c7532ac44b0) +test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.38s + Running tests/scalar_count.rs (target/debug/deps/scalar_count-ed2e7f95d0c111c1) +test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.83s + Running tests/transaction_atomic.rs (target/debug/deps/transaction_atomic-0338c66e7f24b8df) +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.16s + Running tests/turso_read_query.rs (target/debug/deps/turso_read_query-e4023bb09678121a) +test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.53s + Running unittests src/lib.rs (target/debug/deps/querygraph_memory-096506a0be40f215) +test result: ok. 74 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.41s + Running tests/cognition_backend_hardening.rs (target/debug/deps/cognition_backend_hardening-5ad0f41fd77ac101) +test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.04s + Running tests/cognition_bearer_traits.rs (target/debug/deps/cognition_bearer_traits-fea20703fb1e9f8d) +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/cognition_commit.rs (target/debug/deps/cognition_commit-271e1e2a0550e62a) +test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.18s + Running tests/cognition_commit_integrity.rs (target/debug/deps/cognition_commit_integrity-56078f19a383b94e) +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.13s + Running tests/cognition_commit_races.rs (target/debug/deps/cognition_commit_races-d269c546dd184e5e) +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.95s + Running tests/cognition_commit_recovery.rs (target/debug/deps/cognition_commit_recovery-865ee66ae702bf20) +test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.30s + Running tests/cognition_commit_validation.rs (target/debug/deps/cognition_commit_validation-240be13ea80daa98) +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.87s + Running tests/cognition_engine.rs (target/debug/deps/cognition_engine-502ab55f6d14c16d) +test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.45s + Running tests/cognition_entity_guard.rs (target/debug/deps/cognition_entity_guard-48e7af732dbe7d11) +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.24s + Running tests/cognition_job_state.rs (target/debug/deps/cognition_job_state-76ee90830d6d8b9e) +test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.44s + Running tests/cognition_no_change.rs (target/debug/deps/cognition_no_change-f3497603f28ea3e6) +test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.28s + Running tests/cognition_operation.rs (target/debug/deps/cognition_operation-88a1e26f9108b698) +test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/cognition_outbox.rs (target/debug/deps/cognition_outbox-7f7b90ed4e65d6d4) +test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.33s + Running tests/cognition_scheduler_invariants.rs (target/debug/deps/cognition_scheduler_invariants-d4d614cf09836cf7) +test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.20s + Running tests/multi_tenant.rs (target/debug/deps/multi_tenant-19cab96a9e817c7b) +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s + Running tests/sail_cognition_live.rs (target/debug/deps/sail_cognition_live-b636e3f314f04a09) +test result: ok. 0 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.00s + Running tests/turso_persistence.rs (target/debug/deps/turso_persistence-5151985a6059177b) +test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.12s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s +TESTS_EXIT 0 +CLIPPY_EXIT 0 + +Control: the same test build without LBUG_LOCALIZE_BUNDLED_SYMBOLS failed to link with 20 duplicate ZSTD symbols. diff --git a/benchmarks/arrow-pipelines/src/bin/cypher_route.rs b/benchmarks/arrow-pipelines/src/bin/cypher_route.rs new file mode 100644 index 00000000..c385ca44 --- /dev/null +++ b/benchmarks/arrow-pipelines/src/bin/cypher_route.rs @@ -0,0 +1,120 @@ +//! Warm-representation route costs for automatic Cypher routing. +//! One RoutedGraph is captured per size; each trial runs the complete bounded +//! read entrypoint (admission, planning, execution, portable output) on a +//! forced route. Capture time is reported once and excluded from trials. +use grust_core::{Graph, Node, Props, Value}; +use grust_cypher::{CypherParameters, ReadQueryPolicy}; +use grust_datafusion::{ + DataFusionEngine, ExecutionOptions, SpillPolicy, + cypher::{ReadRoute, RouteMode, RoutedGraph}, +}; +use serde_json::json; +use std::{error::Error, num::NonZeroUsize, sync::Arc, time::Duration, time::Instant}; + +type ProfileResult = Result>; +const QUERIES: &[&str] = &[ + "MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count LIMIT 1", + "MATCH (n:N) WHERE n.bucket >= 14 RETURN id(n) AS id ORDER BY id LIMIT 10", + "MATCH (n) RETURN n.bucket AS bucket, count(*) AS count ORDER BY bucket LIMIT 20", +]; + +#[tokio::main(flavor = "multi_thread", worker_threads = 4)] +async fn main() { + if let Err(error) = run().await { + println!("{}", json!({"event":"error","error":error.to_string()})); + std::process::exit(1); + } +} + +async fn run() -> ProfileResult<()> { + let mut args = std::env::args().skip(1); + let repeats: usize = args.next().unwrap_or_else(|| "5".into()).parse()?; + let sizes = args.map(|size| size.parse()).collect::, _>>()?; + let policy = ReadQueryPolicy { + max_graph_nodes: usize::MAX, + max_graph_edges: usize::MAX, + max_graph_bytes: usize::MAX, + max_candidate_work: usize::MAX, + max_intermediate_bytes: 4 << 30, + max_execution_time: Duration::from_secs(600), + ..ReadQueryPolicy::default() + }; + println!( + "{}", + json!({"event":"configuration","source":option_env!("GRUST_PROFILE_SOURCE"), + "repeats":repeats,"sizes":sizes,"queries":QUERIES, + "datafusion_working_memory_bytes":1u64 << 30,"datafusion_target_partitions":4, + "batch_rows":8192,"spill":false,"max_intermediate_bytes":4u64 << 30, + "boundary":"warm representations captured once per size; trial includes bounded admission, route planning, execution and portable output; both routes share one policy"}) + ); + let parameters = CypherParameters::new(); + let mut failed = false; + for size in sizes { + let graph = Arc::new(Graph::new( + (0..size) + .map(|id| { + Node::new( + if id % 2 == 0 { "N" } else { "M" }, + id.to_string(), + Props::from([("bucket".into(), Value::Int((id % 16) as i64))]), + ) + }) + .collect(), + vec![], + )); + let engine = DataFusionEngine::new(ExecutionOptions { + working_memory_bytes: NonZeroUsize::new(1 << 30).unwrap(), + target_partitions: NonZeroUsize::new(4).unwrap(), + batch_rows: NonZeroUsize::new(8192).unwrap(), + spill: SpillPolicy::Disabled, + })?; + let started = Instant::now(); + let routed = RoutedGraph::capture(&engine, graph)?.with_min_datafusion_nodes(0); + println!( + "{}", + json!({"event":"capture","nodes":size,"seconds":started.elapsed().as_secs_f64()}) + ); + for (query_index, query) in QUERIES.iter().enumerate() { + let mut expected = None; + for trial in 0..repeats { + let routes = if trial % 2 == 0 { + [ReadRoute::Reference, ReadRoute::DataFusion] + } else { + [ReadRoute::DataFusion, ReadRoute::Reference] + }; + for route in routes { + let started = Instant::now(); + let outcome = routed + .run_bounded_read_query(query, ¶meters, &policy, RouteMode::Force(route)) + .await; + let seconds = started.elapsed().as_secs_f64(); + let line = match outcome { + Ok(result) => { + let status = match &expected { + None => { + expected = Some(result.table.clone()); + "pass" + } + Some(table) if *table == result.table => "pass", + Some(_) => "mismatch", + }; + failed |= status != "pass"; + json!({"event":"trial","nodes":size,"query":query_index,"route":format!("{route:?}"), + "trial":trial,"status":status,"seconds":seconds,"rows":result.table.rows.len()}) + } + Err(error) => { + failed = true; + json!({"event":"trial","nodes":size,"query":query_index,"route":format!("{route:?}"), + "trial":trial,"status":"error","error":error.to_string()}) + } + }; + println!("{line}"); + } + } + } + } + if failed { + return Err("one or more trials did not pass; all outcomes retained".into()); + } + Ok(()) +} diff --git a/codex-to-codex.md b/codex-to-codex.md index 14f4ff35..5662bb29 100644 --- a/codex-to-codex.md +++ b/codex-to-codex.md @@ -731,3 +731,23 @@ Profile 2b90557 passed release build, Clippy and all 42 oracle checks on Capitol ## 2026-09-14T22:16:28.938749+00:00 — Codex: conversion paired evidence and retained failure Unreleased Arrow conversion 6544dc4 passed 82 native-major tests and Clippy. Initial end-to-end run retained one 256 MiB DataFusion memory exhaustion; all smaller cases passed. Paired original/changed binaries then passed 36 checks, with lower measured preparation cost for this fixture. Evidence retains both runs; failure not declared resolved. All own Cargo/profile processes are terminal. Investigation continues on Capitola; other benchmark pins/hosts remain unchanged. + +## 2026-09-14T23:00:04Z — strain benchmark (Claude Code): FINISHED + +My LanceDB pass on quegee has ended (RUST_REACH8_DONE 22:58:23Z). No run of mine is active here now; quegee is clear for you. hn-shard.service was restored by the tier window. I will append here before starting anything new on this host. + +## 2026-09-14T23:04:00Z — Claude Code (automatic Cypher routing): FINISHED acknowledged; host use + +Read strain FINISHED/RUST_REACH8_DONE: quegee is clear. Continuing Codex's automatic Cypher-to-DataFusion goal on local branch claude/auto-datafusion-route (unpushed, from a134e05). Capitola: isolated clone ~/src/grust-auto-route, at most four nice Cargo jobs; Codex checkouts untouched. Quegee: I will run workspace gates here next unless you reserve it below. Scope: automatic routing for single-node scans only; relationship joins stay on the reference executor. Candidate fix for the retained 256 MiB failure: zero-copy partition splitting so no round-robin repartition charges shared parent buffers; not yet qualified. No benchmark pins or other hosts changed. + +## 2026-09-14T23:33:52Z — Claude Code (automatic Cypher routing): Ladybug findings for the strain harness + +For the strain owner, from running grust-ladybug in the full workspace gate on quegee: +1. No crates.io lbug contains ladybug-rust PR #33. lbug 0.20.4's .cargo_vcs_info.json names 7afc780e (dirty), the commit before merge 47bfd12e. Testing uses a local, uncommitted git pin to 47bfd12e (its Cargo.toml says 0.18.0, so [patch.crates-io] against ^0.20.2 is silently ignored). +2. The switch is proven: with LBUG_LOCALIZE_BUNDLED_SYMBOLS=1, cargo test --workspace --all-features links; the same build without it fails with 20 duplicate ZSTD symbols. Built alone, -p grust-ladybug never links zstd-sys, so single-crate runs cannot show the collision. +3. Each Ladybug database reserves max_db_size (8 TiB default) of address space; parallel unit tests intermittently failed "Mmap for size 8796093022208 failed" on x86-64. Branch claude/auto-datafusion-route adds LadybugConfig::max_db_bytes (None = engine default in production; 16 GiB under cfg(test), as lbug's own tests do). adversarial-graph/src/backends.rs:466 builds LadybugConfig with every field listed, so it will need "max_db_bytes: None," or "..Default::default()" when it moves to this Grust. I have not edited adversarial-graph. +Quegee: my final full workspace gate and Clippy are running now, twelve nice jobs. No benchmark pins changed. + +## 2026-09-15T00:04:55Z — Claude Code: taking grust, eigen, quegee and lakecat for the Grust upgrade reruns + +The user has put grust and eigen under this session's control through this file, and lakecat exclusively. Plan: branch grust-best = main + automatic Cypher→DataFusion scan routing (5b5a708) + the lakecat Turso MVCC load commits (edge-source index retired, PASSIVE automatic checkpoints) + MVCC loads with automatic checkpoints off until one closing TRUNCATE. After focused gates the harness is pinned to that revision and the strain tiers are rerun on grust, eigen and quegee. The old-pin ladders on grust (neo4j com-Orkut) and eigen (turso-mvcc cit-Patents) are superseded and will be stopped before rebuilding. Eigen stays idle 02:45–03:20 and 12:45–13:20, 14:45–15:20 UTC for its two newspapers. Every heavy build runs under systemd-run MemoryMax with at most 4 jobs; no two builds overlap on a host. Capitola is not used (it goes off at 04:00 UTC); this session winds down by then. diff --git a/crates/grust-cypher/src/lib.rs b/crates/grust-cypher/src/lib.rs index 70821ada..056cec8f 100644 --- a/crates/grust-cypher/src/lib.rs +++ b/crates/grust-cypher/src/lib.rs @@ -47,8 +47,8 @@ pub use read::{ pub use read_budget::MAX_RANGE_ITEMS; pub use read_policy::{ PreparedReadRequest, ReadQueryPolicy, run_bounded_read_query, run_bounded_read_query_indexed, - run_bounded_read_query_on_snapshot, run_bounded_read_query_with_registry, validate_read_query, - validate_read_query_with_registry, + run_bounded_read_query_on_snapshot, run_bounded_read_query_with_registry, + run_prepared_read_query_indexed, validate_read_query, validate_read_query_with_registry, }; pub mod semantics; pub mod session; diff --git a/crates/grust-cypher/src/read_policy.rs b/crates/grust-cypher/src/read_policy.rs index 91b60690..eecc87af 100644 --- a/crates/grust-cypher/src/read_policy.rs +++ b/crates/grust-cypher/src/read_policy.rs @@ -217,6 +217,28 @@ fn run_bounded_read_query_with_executor( BoundedInput::Graph(graph) => request.check_graph(graph)?, BoundedInput::Indexed(index) => request.check_index(index)?, } + run_prepared(&request, execute) +} + +/// Execute an already admitted request on a typed index, keeping its original +/// deadline, policy and parameters. Automatic routers use this after another +/// executor declined before execution, so a declined route never resets the +/// deadline or repeats query admission. Input size is checked against `index`. +pub fn run_prepared_read_query_indexed( + request: &PreparedReadRequest<'_>, + index: &TypedGraphIndex, +) -> Result { + request.check_index(index)?; + run_prepared(request, |query| { + execute_read_query_indexed(index, query, request.parameters()) + }) +} + +fn run_prepared( + request: &PreparedReadRequest<'_>, + execute: impl FnOnce(&Query) -> Result, +) -> Result { + let policy = request.policy(); let limits = ReadExecutionBudgetLimits { max_candidate_work: policy.max_candidate_work, max_intermediate_bytes: policy.max_intermediate_bytes, diff --git a/crates/grust-datafusion/src/cypher/collect.rs b/crates/grust-datafusion/src/cypher/collect.rs index f79db762..1b8a26ea 100644 --- a/crates/grust-datafusion/src/cypher/collect.rs +++ b/crates/grust-datafusion/src/cypher/collect.rs @@ -1,13 +1,18 @@ //! Incremental portable result collection with explicit output limits. use super::{decode_result_batch, decode_result_batch_with_context}; use datafusion::{ + arrow::datatypes::SchemaRef, common::{DataFusionError, Result}, dataframe::DataFrame, + execution::SendableRecordBatchStream, }; use futures::TryStreamExt; use grust_cypher::CypherResultTable; use grust_procedures::ExecutionContext; -use std::io::{self, Write}; +use std::{ + io::{self, Write}, + sync::Arc, +}; /// Collect a typed plan while enforcing cumulative result rows and serialized /// JSON bytes (`{"columns":...,"rows":...}`, matching the Cypher read policy). @@ -46,27 +51,43 @@ async fn collect( max_rows: usize, max_output_bytes: usize, execution: Option<&ExecutionContext>, +) -> Result { + let schema = Arc::new(frame.schema().as_arrow().clone()); + collect_stream( + schema, + frame.execute_stream(), + max_rows, + max_output_bytes, + execution, + ) + .await +} + +/// Collect a stream whose planning already happened elsewhere, such as an +/// instrumented physical plan. Output columns come from `schema`; the stream is +/// opened only after the column-name copies have been admitted. +pub(super) async fn collect_stream( + schema: SchemaRef, + stream: impl Future>, + max_rows: usize, + max_output_bytes: usize, + execution: Option<&ExecutionContext>, ) -> Result { if let Some(execution) = execution { - let bytes = frame - .schema() - .fields() - .iter() - .try_fold(0usize, |bytes, field| { - bytes - .checked_add(std::mem::size_of::()) - .and_then(|bytes| bytes.checked_add(field.name().len())) - .ok_or_else(|| { - DataFusionError::Execution("Cypher column copy size overflow".into()) - }) - })?; + let bytes = schema.fields().iter().try_fold(0usize, |bytes, field| { + bytes + .checked_add(std::mem::size_of::()) + .and_then(|bytes| bytes.checked_add(field.name().len())) + .ok_or_else(|| { + DataFusionError::Execution("Cypher column copy size overflow".into()) + }) + })?; execution .charge_cumulative_memory(bytes) .map_err(|error| DataFusionError::External(Box::new(error)))?; } let mut table = CypherResultTable { - columns: frame - .schema() + columns: schema .fields() .iter() .map(|field| field.name().clone()) @@ -86,7 +107,7 @@ async fn collect( counter .write_all(b",\"rows\":[]}") .map_err(|e| error(e.to_string()))?; - let mut stream = frame.execute_stream().await?; + let mut stream = stream.await?; while let Some(batch) = stream.try_next().await? { if batch.num_rows() > max_rows.saturating_sub(table.rows.len()) { return Err(DataFusionError::Execution(format!( diff --git a/crates/grust-datafusion/src/cypher/mod.rs b/crates/grust-datafusion/src/cypher/mod.rs index 76562ba2..afefc6fb 100644 --- a/crates/grust-datafusion/src/cypher/mod.rs +++ b/crates/grust-datafusion/src/cypher/mod.rs @@ -1,4 +1,4 @@ -//! Typed Cypher expression lowering. Query routing is not yet implemented. +//! Typed Cypher lowering, explicit execution and automatic bounded-read routing. mod aggregate; pub use aggregate::lower_aggregate_with_bindings; mod bindings; @@ -15,6 +15,11 @@ pub use collect::{collect_result, collect_result_with_context}; pub use planner::{PlanKind, QueryPlan}; pub use result::{decode_result_batch, decode_result_batch_with_context}; mod relationship; +mod route; +pub use route::{ + DEFAULT_MIN_DATAFUSION_NODES, ReadRoute, RouteDecline, RouteExplain, RouteMode, RoutedGraph, + RoutedReadResult, +}; mod scan; mod trail; pub use pattern::plan_relationship_scan; @@ -227,3 +232,6 @@ mod execute_tests; #[cfg(test)] mod trail_tests; + +#[cfg(test)] +mod route_tests; diff --git a/crates/grust-datafusion/src/cypher/route.rs b/crates/grust-datafusion/src/cypher/route.rs new file mode 100644 index 00000000..530733fe --- /dev/null +++ b/crates/grust-datafusion/src/cypher/route.rs @@ -0,0 +1,514 @@ +//! Automatic route selection for ordinary bounded Cypher reads. +//! +//! [`RoutedGraph`] captures one immutable graph as both a typed index and a +//! DataFusion snapshot, so the two routes can never observe different data. +//! Each read is admitted once through [`PreparedReadRequest`]; whichever route +//! runs keeps that admission, including the original absolute deadline. +//! +//! DataFusion runs only single-node scan plans whose every physical operator is +//! covered by candidate-work and intermediate-byte accounting. The read policy +//! maps onto one [`ExecutionContext`] exactly as the reference executor maps it: +//! `max_candidate_work` bounds scanned rows, `max_intermediate_bytes` bounds +//! cumulative bytes emitted by copying operators plus portable output copies, +//! and the deadline covers planning, execution and output consumption. Output +//! rows/bytes are checked incrementally and again by the shared request. +//! Relationship patterns stay on the reference executor: upstream hash joins +//! build candidate pairs before any hook could charge them. +use super::{GraphSnapshot, PlanKind, QueryPlan, SnapshotStatistics, UnsupportedScan}; +use crate::DataFusionEngine; +use datafusion::{ + arrow::{datatypes::SchemaRef, record_batch::RecordBatch}, + common::{DataFusionError, Result as DataFusionResult, tree_node::TreeNodeRecursion}, + execution::{SendableRecordBatchStream, TaskContext, context::SessionContext}, + physical_expr::PhysicalExpr, + physical_plan::{ + DisplayAs, DisplayFormatType, ExecutionPlan, PlanProperties, displayable, execute_stream, + execution_plan::replace_children_if_necessary, stream::RecordBatchStreamAdapter, + }, +}; +use futures::StreamExt; +use grust_arrow::{ArrowGraph, ArrowGraphTables}; +use grust_core::{Graph, TypedGraphIndex, prelude::Result}; +use grust_cypher::{ + CypherParameters, CypherResultTable, PreparedReadRequest, ReadQueryPolicy, gql_execution, +}; +use grust_procedures::{ExecutionContext, ExecutionLimits, ProcedureError}; +use std::{fmt, sync::Arc}; + +/// Smallest captured node table for which automatic selection prefers the +/// DataFusion route. The `cypher_route` profile (warm representations, three +/// scan shapes) found the typed index faster for a count at 1,000 nodes and +/// DataFusion faster for every shape from 3,000 nodes up. Callers with other +/// workloads or hosts can override it with [`RoutedGraph::with_min_datafusion_nodes`]. +pub const DEFAULT_MIN_DATAFUSION_NODES: usize = 3_000; + +/// Physical operators whose outputs the accounting wrapper can charge. Any other +/// operator after optimization, including every join, declines the route. +const COVERED_OPERATORS: &[&str] = &[ + "DataSourceExec", + "FilterExec", + "ProjectionExec", + "AggregateExec", + "SortExec", + "SortPreservingMergeExec", + "CoalescePartitionsExec", + "CoalesceBatchesExec", + "RepartitionExec", + "GlobalLimitExec", + "LocalLimitExec", + "CooperativeExec", + "EmptyExec", + "PlaceholderRowExec", +]; + +/// Covered operators that emit their input arrays unchanged or as zero-copy +/// slices. Their outputs were already charged where they were produced. +const PASS_THROUGH_OPERATORS: &[&str] = &[ + "CoalescePartitionsExec", + "GlobalLimitExec", + "LocalLimitExec", + "CooperativeExec", +]; + +/// An executor for one bounded read. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum ReadRoute { + /// The typed-index reference executor, which supports the whole read surface. + Reference, + /// The typed DataFusion 55 plan over the captured Arrow snapshot. + DataFusion, +} + +/// How a routed read chooses its executor. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum RouteMode { + /// Select DataFusion only for admitted plans above the size threshold. + Automatic, + /// Require one route, for qualification. Forcing DataFusion fails, without + /// executing anything, when its plan cannot be admitted under the policy. + Force(ReadRoute), +} + +/// Why the DataFusion route was not chosen. Every reason is determined before +/// execution starts; errors after that point propagate and are never rerouted. +#[derive(Clone, Debug, Eq, PartialEq)] +pub enum RouteDecline { + /// The caller forced the reference route. + ReferenceForced, + /// The graph could not be captured as native Arrow tables. + CaptureUnavailable(String), + /// The typed compiler does not support this query shape. + Unsupported { + kind: PlanKind, + reason: UnsupportedScan, + }, + /// Relationship patterns need join candidate accounting that upstream + /// hash joins do not expose. + RelationshipJoin, + /// The captured node table is smaller than the automatic threshold. + BelowThreshold { node_rows: usize, threshold: usize }, + /// Scanning every node would exceed the policy's candidate-work limit. + CandidateWork { required: usize, limit: usize }, + /// The optimized physical plan contains an operator without accounting. + UncoveredOperator(String), +} + +impl fmt::Display for RouteDecline { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::ReferenceForced => write!(f, "reference route forced"), + Self::CaptureUnavailable(reason) => write!(f, "Arrow capture unavailable: {reason}"), + Self::Unsupported { kind, reason } => { + write!(f, "unsupported {kind:?} shape ({reason:?})") + } + Self::RelationshipJoin => { + write!(f, "relationship joins lack candidate-work accounting") + } + Self::BelowThreshold { + node_rows, + threshold, + } => write!( + f, + "{node_rows} nodes is below the {threshold}-node threshold" + ), + Self::CandidateWork { required, limit } => write!( + f, + "scanning {required} nodes exceeds the {limit} candidate-work limit" + ), + Self::UncoveredOperator(name) => write!(f, "operator {name} has no accounting"), + } + } +} + +/// The selected route and the evidence behind it, reported without executing. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct RouteExplain { + pub chosen: ReadRoute, + /// Why DataFusion was not chosen; `None` when it was. + pub datafusion_declined: Option, + /// Exact capture counts, including the added ordinal payload. Capture + /// happened once when the [`RoutedGraph`] was built, not per query. + pub capture: Option, + /// Optimized physical plan, when the typed compiler produced one. + pub physical_plan: Option, +} + +/// A completed routed read. +#[derive(Clone, Debug, PartialEq)] +pub struct RoutedReadResult { + pub table: CypherResultTable, + pub explain: RouteExplain, +} + +/// One immutable graph captured for both executors. +pub struct RoutedGraph { + index: TypedGraphIndex, + snapshot: std::result::Result, + context: SessionContext, + min_datafusion_nodes: usize, +} + +impl RoutedGraph { + /// Build the typed index and the Arrow snapshot from the same graph. A graph + /// whose values have no native Arrow column still gets the reference route; + /// the capture failure is reported by every explain. + pub fn capture(engine: &DataFusionEngine, graph: Arc) -> Result { + let snapshot = ArrowGraph::from_graph(&graph) + .and_then(|arrow| { + let (nodes, edges) = arrow.into_tables(); + ArrowGraphTables::try_new(nodes, edges) + }) + .map_err(|error| error.to_string()) + .and_then(|tables| { + GraphSnapshot::try_new(engine, tables).map_err(|error| error.to_string()) + }); + Ok(Self { + index: TypedGraphIndex::new(graph)?, + snapshot, + context: engine.context().clone(), + min_datafusion_nodes: DEFAULT_MIN_DATAFUSION_NODES, + }) + } + + /// Replace the automatic size threshold. Zero considers every admitted plan. + pub fn with_min_datafusion_nodes(mut self, nodes: usize) -> Self { + self.min_datafusion_nodes = nodes; + self + } + + /// The reference executor's typed index. + pub fn index(&self) -> &TypedGraphIndex { + &self.index + } + + /// The captured DataFusion snapshot, if capture succeeded. + pub fn snapshot(&self) -> Option<&GraphSnapshot> { + self.snapshot.as_ref().ok() + } + + /// Admit a bounded read and report the route it would take, without executing. + pub async fn explain( + &self, + query_text: &str, + parameters: &CypherParameters, + policy: &ReadQueryPolicy, + mode: RouteMode, + ) -> Result { + let request = PreparedReadRequest::new(query_text, parameters, policy)?; + request.check_index(&self.index)?; + Ok(self.select(&request, mode).await?.1) + } + + /// Run a bounded read on the selected route under the complete read policy. + /// The reference route is synchronous and runs on the calling task. + pub async fn run_bounded_read_query( + &self, + query_text: &str, + parameters: &CypherParameters, + policy: &ReadQueryPolicy, + mode: RouteMode, + ) -> Result { + let request = PreparedReadRequest::new(query_text, parameters, policy)?; + request.check_index(&self.index)?; + let (plan, explain) = self.select(&request, mode).await?; + let table = match plan { + None => grust_cypher::run_prepared_read_query_indexed(&request, &self.index)?, + Some(plan) => self.execute(&request, plan).await?, + }; + Ok(RoutedReadResult { table, explain }) + } + + async fn select( + &self, + request: &PreparedReadRequest<'_>, + mode: RouteMode, + ) -> Result<(Option>, RouteExplain)> { + let mut explain = RouteExplain { + chosen: ReadRoute::Reference, + datafusion_declined: None, + capture: self.snapshot().map(GraphSnapshot::statistics), + physical_plan: None, + }; + let decline = |mut explain: RouteExplain, reason: RouteDecline| { + if mode == RouteMode::Force(ReadRoute::DataFusion) { + return Err(gql_execution(format!( + "DataFusion route unavailable: {reason}" + ))); + } + explain.datafusion_declined = Some(reason); + Ok((None, explain)) + }; + if mode == RouteMode::Force(ReadRoute::Reference) { + return decline(explain, RouteDecline::ReferenceForced); + } + let snapshot = match &self.snapshot { + Ok(snapshot) => snapshot, + Err(reason) => { + return decline(explain, RouteDecline::CaptureUnavailable(reason.clone())); + } + }; + let frame = match snapshot + .plan(request.query(), &self.context, request.parameters()) + .map_err(|error| gql_execution(describe(&error)))? + { + QueryPlan::Unsupported { kind, reason } => { + return decline(explain, RouteDecline::Unsupported { kind, reason }); + } + QueryPlan::Supported { + kind: PlanKind::RelationshipScan, + .. + } => return decline(explain, RouteDecline::RelationshipJoin), + QueryPlan::Supported { + kind: PlanKind::NodeScan, + frame, + } => frame, + }; + let node_rows = snapshot.statistics().node_rows; + if mode == RouteMode::Automatic && node_rows < self.min_datafusion_nodes { + return decline( + explain, + RouteDecline::BelowThreshold { + node_rows, + threshold: self.min_datafusion_nodes, + }, + ); + } + let limit = request.policy().max_candidate_work; + if node_rows > limit { + return decline( + explain, + RouteDecline::CandidateWork { + required: node_rows, + limit, + }, + ); + } + let plan = frame + .create_physical_plan() + .await + .map_err(|error| gql_execution(describe(&error)))?; + explain.physical_plan = Some(displayable(plan.as_ref()).indent(true).to_string()); + if let Some(name) = uncovered_operator(&plan) { + return decline(explain, RouteDecline::UncoveredOperator(name)); + } + explain.chosen = ReadRoute::DataFusion; + Ok((Some(plan), explain)) + } + + async fn execute( + &self, + request: &PreparedReadRequest<'_>, + plan: Arc, + ) -> Result { + let policy = request.policy(); + let execution = ExecutionContext::new(ExecutionLimits { + memory_bytes: policy.max_intermediate_bytes, + work_units: policy.max_candidate_work, + batch_rows: 1024, + deadline: Some(request.deadline()), + }) + .map_err(|error| gql_execution(error.to_string()))?; + let schema: SchemaRef = plan.schema(); + let run = async { + let plan = instrument(plan, &execution)?; + super::collect::collect_stream( + schema, + std::future::ready(execute_stream(plan, self.context.task_ctx())), + policy.max_result_rows, + policy.max_output_bytes, + Some(&execution), + ) + .await + }; + let table = crate::run_cancellable(&execution, run) + .await + .map_err(|error| gql_execution(describe(&error)))?; + request.check_output(&table)?; + Ok(table) + } +} + +/// Operator name without a variant suffix, e.g. `SortExec(TopK)` -> `SortExec`. +fn operator_kind(plan: &Arc) -> &str { + let name = plan.name(); + name.split_once('(').map_or(name, |(kind, _)| kind) +} + +fn uncovered_operator(plan: &Arc) -> Option { + if !COVERED_OPERATORS.contains(&operator_kind(plan)) { + return Some(plan.name().to_owned()); + } + plan.children().into_iter().find_map(uncovered_operator) +} + +/// Wrap every leaf in candidate-row charging and every copying operator in +/// cumulative byte charging. Operators are only wrapped, never replaced, so +/// the optimized plan and its upstream kernels run unchanged. +fn instrument( + plan: Arc, + execution: &ExecutionContext, +) -> DataFusionResult> { + let children = plan + .children() + .into_iter() + .map(|child| instrument(Arc::clone(child), execution)) + .collect::>>()?; + let charge = if children.is_empty() { + Some(Charge::CandidateRows) + } else if PASS_THROUGH_OPERATORS.contains(&operator_kind(&plan)) { + None + } else { + Some(Charge::CopiedBytes) + }; + let plan = if children.is_empty() { + plan + } else { + replace_children_if_necessary(plan, children)? + }; + Ok(match charge { + None => plan, + Some(charge) => Arc::new(AccountedExec { + input: plan, + charge, + execution: execution.clone(), + }), + }) +} + +#[derive(Clone, Copy, Debug)] +enum Charge { + /// Rows produced by a scan are candidate work. + CandidateRows, + /// Arrays produced by an operator are cumulative intermediate copies, + /// measured by their sliced contents rather than shared parent buffers. + CopiedBytes, +} + +impl Charge { + fn apply(self, execution: &ExecutionContext, batch: &RecordBatch) -> DataFusionResult<()> { + let charged = match self { + Self::CandidateRows => execution.charge_work(batch.num_rows()), + Self::CopiedBytes => execution.charge_cumulative_memory(slice_bytes(batch)?), + }; + charged.map_err(|error| DataFusionError::External(Box::new(error))) + } +} + +fn slice_bytes(batch: &RecordBatch) -> DataFusionResult { + batch.columns().iter().try_fold(0_usize, |total, column| { + let bytes = column.to_data().get_slice_memory_size()?; + total + .checked_add(bytes) + .ok_or_else(|| DataFusionError::Execution("intermediate byte count overflow".into())) + }) +} + +/// Charges each emitted batch against the query's shared execution context +/// before passing it on. Shared across partitions through one context. +#[derive(Debug)] +struct AccountedExec { + input: Arc, + charge: Charge, + execution: ExecutionContext, +} + +impl DisplayAs for AccountedExec { + fn fmt_as(&self, _format: DisplayFormatType, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "AccountedExec: {:?}", self.charge) + } +} + +impl ExecutionPlan for AccountedExec { + fn name(&self) -> &str { + "AccountedExec" + } + + fn properties(&self) -> &Arc { + self.input.properties() + } + + fn children(&self) -> Vec<&Arc> { + vec![&self.input] + } + + fn apply_expressions( + &self, + _f: &mut dyn FnMut(&Arc) -> DataFusionResult, + ) -> DataFusionResult { + Ok(TreeNodeRecursion::Continue) + } + + fn with_new_children( + self: Arc, + children: Vec>, + ) -> DataFusionResult> { + let [input] = <[_; 1]>::try_from(children).map_err(|_| { + DataFusionError::Internal("AccountedExec requires exactly one child".into()) + })?; + Ok(Arc::new(Self { + input, + charge: self.charge, + execution: self.execution.clone(), + })) + } + + fn execute( + &self, + partition: usize, + context: Arc, + ) -> DataFusionResult { + let stream = self.input.execute(partition, context)?; + let schema = stream.schema(); + let (charge, execution) = (self.charge, self.execution.clone()); + Ok(Box::pin(RecordBatchStreamAdapter::new( + schema, + stream.map(move |batch| { + let batch = batch?; + charge.apply(&execution, &batch)?; + Ok(batch) + }), + ))) + } +} + +/// Translate a DataFusion failure into the reference executor's messages for +/// the same policy outcomes, so callers see one bounded-read contract. +fn describe(error: &DataFusionError) -> String { + if let DataFusionError::External(inner) = error.find_root() + && let Some(error) = inner.downcast_ref::() + { + return match error { + ProcedureError::DeadlineExceeded => "bounded read execution timed out".into(), + ProcedureError::BudgetExceeded { + resource: "work", .. + } => format!( + "bounded read candidate-work units ({error}) while executing DataFusion scan" + ), + ProcedureError::BudgetExceeded { .. } => format!( + "bounded read cumulative intermediate bytes ({error}) while executing DataFusion scan" + ), + error => format!("bounded read DataFusion execution failed: {error}"), + }; + } + format!("bounded read DataFusion execution failed: {error}") +} diff --git a/crates/grust-datafusion/src/cypher/route_tests.rs b/crates/grust-datafusion/src/cypher/route_tests.rs new file mode 100644 index 00000000..63df8ccd --- /dev/null +++ b/crates/grust-datafusion/src/cypher/route_tests.rs @@ -0,0 +1,309 @@ +use super::*; +use crate::{DataFusionEngine, ExecutionOptions, SpillPolicy}; +use grust_arrow::{ArrowGraph, ArrowGraphTables}; +use grust_core::{Edge, Graph, Node, Props}; +use grust_cypher::ReadQueryPolicy; +use std::sync::Arc; + +const NODES: usize = 20_000; + +fn engine(working_memory_bytes: usize) -> DataFusionEngine { + DataFusionEngine::new(ExecutionOptions { + working_memory_bytes: working_memory_bytes.try_into().unwrap(), + target_partitions: 4.try_into().unwrap(), + batch_rows: 1024.try_into().unwrap(), + spill: SpillPolicy::Disabled, + }) + .unwrap() +} + +fn graph() -> Arc { + let nodes = (0..NODES) + .map(|id| { + let mut props = Props::from([("bucket".into(), Value::Int((id % 16) as i64))]); + if id % 3 == 0 { + props.insert("tag".into(), Value::String(format!("t{}", id % 5))); + } + Node::new(if id % 2 == 0 { "N" } else { "M" }, id.to_string(), props) + }) + .collect(); + let edges = (0..100) + .map(|id| Edge::new("E", id.to_string(), (id + 1).to_string(), Props::new())) + .collect(); + Arc::new(Graph::new(nodes, edges)) +} + +fn policy() -> ReadQueryPolicy { + ReadQueryPolicy { + max_candidate_work: 10_000_000, + max_result_rows: 50, + ..ReadQueryPolicy::default() + } +} + +const SCANS: &[&str] = &[ + "MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count LIMIT 1", + "MATCH (n) WHERE n.bucket >= 14 RETURN id(n) AS id ORDER BY id LIMIT 7", + "MATCH (n:M) RETURN n.bucket AS bucket, count(*) AS count ORDER BY bucket LIMIT 20", + "MATCH (n) WHERE n.tag IS NULL RETURN count(*) AS count LIMIT 1", + "MATCH (n {tag: 't2'}) RETURN DISTINCT n.bucket AS bucket ORDER BY bucket LIMIT 20", + "MATCH (n:N) RETURN max(n.bucket) AS high, min(n.tag) AS low LIMIT 1", +]; + +#[tokio::test] +async fn automatic_scans_match_reference_results() { + let engine = engine(64 << 20); + let routed = RoutedGraph::capture(&engine, graph()) + .unwrap() + .with_min_datafusion_nodes(0); + let parameters = CypherParameters::new(); + for text in SCANS { + let expected = grust_cypher::run_bounded_read_query_indexed( + routed.index(), + text, + ¶meters, + &policy(), + ) + .unwrap(); + let automatic = routed + .run_bounded_read_query(text, ¶meters, &policy(), RouteMode::Automatic) + .await + .unwrap(); + assert_eq!(automatic.explain.chosen, ReadRoute::DataFusion, "{text}"); + assert_eq!(automatic.explain.datafusion_declined, None, "{text}"); + assert!(automatic.explain.physical_plan.is_some(), "{text}"); + assert_eq!(automatic.table, expected, "{text}"); + let reference = routed + .run_bounded_read_query( + text, + ¶meters, + &policy(), + RouteMode::Force(ReadRoute::Reference), + ) + .await + .unwrap(); + assert_eq!(reference.explain.chosen, ReadRoute::Reference); + assert_eq!( + reference.explain.datafusion_declined, + Some(RouteDecline::ReferenceForced) + ); + assert_eq!(reference.table, expected, "{text}"); + } +} + +#[tokio::test] +async fn joins_and_unsupported_shapes_stay_on_reference() { + let engine = engine(64 << 20); + let routed = RoutedGraph::capture(&engine, graph()) + .unwrap() + .with_min_datafusion_nodes(0); + let parameters = CypherParameters::new(); + let join = "MATCH (a)-[:E]->(b) RETURN count(*) AS count LIMIT 1"; + let result = routed + .run_bounded_read_query(join, ¶meters, &policy(), RouteMode::Automatic) + .await + .unwrap(); + assert_eq!(result.explain.chosen, ReadRoute::Reference); + assert_eq!( + result.explain.datafusion_declined, + Some(RouteDecline::RelationshipJoin) + ); + assert_eq!(result.table.rows, vec![vec![Value::Int(100)]]); + let forced = routed + .run_bounded_read_query( + join, + ¶meters, + &policy(), + RouteMode::Force(ReadRoute::DataFusion), + ) + .await + .unwrap_err(); + assert!(forced.to_string().contains("route unavailable"), "{forced}"); + let optional = routed + .explain( + "OPTIONAL MATCH (n:N) RETURN count(*) AS count LIMIT 1", + ¶meters, + &policy(), + RouteMode::Automatic, + ) + .await + .unwrap(); + assert_eq!(optional.chosen, ReadRoute::Reference); + assert!(matches!( + optional.datafusion_declined, + Some(RouteDecline::Unsupported { .. }) + )); + // Invalid Cypher is an error, never a route decline. + assert!( + routed + .run_bounded_read_query( + "MATCH (n) RETURN absent.x AS x LIMIT 1", + ¶meters, + &policy(), + RouteMode::Automatic + ) + .await + .is_err() + ); +} + +#[tokio::test] +async fn threshold_and_candidate_work_decline_before_execution() { + let engine = engine(64 << 20); + const { assert!(DEFAULT_MIN_DATAFUSION_NODES < NODES) }; + let routed = RoutedGraph::capture(&engine, graph()) + .unwrap() + .with_min_datafusion_nodes(NODES + 1); + let parameters = CypherParameters::new(); + let text = SCANS[0]; + let explain = routed + .explain(text, ¶meters, &policy(), RouteMode::Automatic) + .await + .unwrap(); + assert_eq!(explain.chosen, ReadRoute::Reference); + assert_eq!( + explain.datafusion_declined, + Some(RouteDecline::BelowThreshold { + node_rows: NODES, + threshold: NODES + 1 + }) + ); + assert_eq!(explain.capture.unwrap().node_rows, NODES); + let routed = routed.with_min_datafusion_nodes(0); + let tight = ReadQueryPolicy { + max_candidate_work: NODES - 1, + ..policy() + }; + let explain = routed + .explain(text, ¶meters, &tight, RouteMode::Automatic) + .await + .unwrap(); + assert_eq!( + explain.datafusion_declined, + Some(RouteDecline::CandidateWork { + required: NODES, + limit: NODES - 1 + }) + ); + let exact = ReadQueryPolicy { + max_candidate_work: NODES, + ..policy() + }; + let result = routed + .run_bounded_read_query(text, ¶meters, &exact, RouteMode::Automatic) + .await + .unwrap(); + assert_eq!(result.explain.chosen, ReadRoute::DataFusion); +} + +#[tokio::test] +async fn intermediate_bytes_are_enforced_across_partitions() { + let engine = engine(64 << 20); + let routed = RoutedGraph::capture(&engine, graph()) + .unwrap() + .with_min_datafusion_nodes(0); + let parameters = CypherParameters::new(); + // Filtering keeps half the nodes, so operators emit far more than 4 KiB + // before the final LIMIT, even though the result is seven rows. + let small = ReadQueryPolicy { + max_intermediate_bytes: 4096, + ..policy() + }; + let error = routed + .run_bounded_read_query( + "MATCH (n:N) RETURN id(n) AS id ORDER BY id LIMIT 7", + ¶meters, + &small, + RouteMode::Force(ReadRoute::DataFusion), + ) + .await + .unwrap_err(); + assert!( + error.to_string().contains("cumulative intermediate bytes"), + "{error}" + ); +} + +#[tokio::test] +async fn capture_failure_keeps_the_reference_route() { + let engine = engine(64 << 20); + let nodes = (0..3) + .map(|id| { + Node::new( + "N", + id.to_string(), + // Mixed property types have no single native Arrow column. + Props::from([( + "mixed".into(), + if id == 0 { + Value::String("zero".into()) + } else { + Value::Int(id) + }, + )]), + ) + }) + .collect(); + let routed = RoutedGraph::capture(&engine, Arc::new(Graph::new(nodes, vec![]))) + .unwrap() + .with_min_datafusion_nodes(0); + let result = routed + .run_bounded_read_query( + "MATCH (n:N) RETURN count(*) AS count LIMIT 1", + &CypherParameters::new(), + &policy(), + RouteMode::Automatic, + ) + .await + .unwrap(); + assert_eq!(result.explain.chosen, ReadRoute::Reference); + assert!(matches!( + result.explain.datafusion_declined, + Some(RouteDecline::CaptureUnavailable(_)) + )); + assert_eq!(result.table.rows, vec![vec![Value::Int(3)]]); +} + +#[tokio::test] +async fn single_large_batches_are_split_without_round_robin() { + let rows = 4 * 8192 * 3; + let nodes = (0..rows) + .map(|id| { + Node::new( + "N", + id.to_string(), + Props::from([("bucket".into(), Value::Int((id % 16) as i64))]), + ) + }) + .collect(); + // A pool smaller than four copies of the whole batch: the former + // round-robin plan charged every queued slice the full parent buffers. + let graph = Arc::new(Graph::new(nodes, vec![])); + let (nodes, edges) = ArrowGraph::from_graph(&graph).unwrap().into_tables(); + let whole = nodes.batches()[0].get_array_memory_size(); + let engine = engine(whole * 3); + let snapshot = + GraphSnapshot::try_new(&engine, ArrowGraphTables::try_new(nodes, edges).unwrap()).unwrap(); + let query = grust_cypher::parser::parse_query( + "MATCH (n:N) WHERE n.bucket = 3 RETURN count(*) AS count", + ) + .unwrap(); + let frame = || match snapshot + .plan(&query, engine.context(), &CypherParameters::new()) + .unwrap() + { + QueryPlan::Supported { frame, .. } => frame, + QueryPlan::Unsupported { .. } => panic!("scan must be supported"), + }; + let plan = frame().create_physical_plan().await.unwrap(); + let text = datafusion::physical_plan::displayable(plan.as_ref()) + .indent(true) + .to_string(); + assert!(!text.contains("RoundRobinBatch"), "{text}"); + // Repeat enough times that the former scheduling-dependent failure would + // have been likely to appear at least once. + for _ in 0..20 { + let table = collect_result(frame(), 1, 1024).await.unwrap(); + assert_eq!(table.rows, vec![vec![Value::Int((rows / 16) as i64)]]); + } +} diff --git a/crates/grust-datafusion/src/lib.rs b/crates/grust-datafusion/src/lib.rs index efd7c39d..36a1d233 100644 --- a/crates/grust-datafusion/src/lib.rs +++ b/crates/grust-datafusion/src/lib.rs @@ -170,14 +170,55 @@ impl DataFusionEngine { } fn table_provider(&self, table: ArrowTable) -> Result> { let schema = table.schema(); - let batches = table.into_batches(); - let count = self.partitions.get().min(batches.len().max(1)); + let partitions = partition_batches(table.into_batches(), self.partitions.get()); + Ok(Arc::new(MemTable::try_new(schema, partitions)?)) + } +} + +/// Smallest slice worth a separate partition when splitting one large batch. +const MIN_SLICE_ROWS: usize = 8192; + +/// Distribute batches over at most `target` partitions. Large tables with fewer +/// batches than partitions are split into contiguous zero-copy slices, so the +/// planner does not add round-robin repartitioning above the scan. Upstream +/// repartitioning charges every queued slice the full size of its shared parent +/// buffers; many slices of one large batch could exhaust the working-memory +/// pool although nothing was copied. Row order within each partition is kept. +fn partition_batches( + batches: Vec, + target: usize, +) -> Vec> { + let rows = batches.iter().map(|batch| batch.num_rows()).sum::(); + if batches.len() >= target || rows < target.saturating_mul(MIN_SLICE_ROWS) { + let count = target.min(batches.len().max(1)); let mut partitions = (0..count).map(|_| Vec::new()).collect::>(); for (i, batch) in batches.into_iter().enumerate() { partitions[i % count].push(batch); } - Ok(Arc::new(MemTable::try_new(schema, partitions)?)) + return partitions; + } + let per_partition = rows.div_ceil(target); + let mut partitions = (0..target).map(|_| Vec::new()).collect::>(); + let (mut index, mut filled) = (0, 0); + for batch in batches { + let mut offset = 0; + while offset < batch.num_rows() { + let capacity = if index + 1 == target { + usize::MAX + } else { + per_partition - filled + }; + let take = capacity.min(batch.num_rows() - offset); + partitions[index].push(batch.slice(offset, take)); + offset += take; + filled += take; + if filled == per_partition && index + 1 < target { + index += 1; + filled = 0; + } + } } + partitions } #[cfg(test)] diff --git a/crates/grust-ladybug/Cargo.toml b/crates/grust-ladybug/Cargo.toml index 69369ab8..03a27bdd 100644 --- a/crates/grust-ladybug/Cargo.toml +++ b/crates/grust-ladybug/Cargo.toml @@ -15,6 +15,7 @@ default = ["arrow"] arrow = ["dep:arrow", "lbug/arrow", "dep:grust-arrow"] [dependencies] +tempfile = { workspace = true } grust-arrow = { version = "0.20.0", path = "../grust-arrow", default-features = false, features = ["arrow-55"], optional = true } arrow = { version = "55", optional = true } async-trait = { workspace = true } @@ -25,3 +26,7 @@ serde_json = { workspace = true } [dev-dependencies] tempfile = { workspace = true } tokio = { workspace = true } + +[[example]] +name = "copy_bench" +required-features = ["arrow"] diff --git a/crates/grust-ladybug/examples/copy_bench.rs b/crates/grust-ladybug/examples/copy_bench.rs new file mode 100644 index 00000000..7dcd354f --- /dev/null +++ b/crates/grust-ladybug/examples/copy_bench.rs @@ -0,0 +1,294 @@ +//! Times the ways the `lbug` crate can bulk-load a graph, on one synthetic +//! graph, so the adapter's load path can be chosen from a measurement: +//! +//! A. what `grust-ladybug` does today: register Arrow batches as a scratch +//! table and `COPY t FROM (MATCH (x:scratch) RETURN …)`; +//! B. `COPY t FROM scratch` over the registered table directly, the shape +//! of Python's `COPY t FROM $df` (may be rejected: reported, not fatal); +//! C. `COPY t FROM 'file.csv'`, the engine's native file path; +//! D. C again with INT64 primary keys and no properties, the CSR-parquet +//! shape the maintainer loads cit-Patents from in seconds; +//! E. the CSR registration (`create_arrow_rel_table_csr`) with INT64 keys, +//! copied through MATCH and directly. +//! +//! VARIANTS=ABCDE cargo run --release -p grust-ladybug --example copy_bench -- [nodes] [edges] +//! +//! Prints one line per variant: rows, seconds, rows per second, or the +//! engine's error text. +use std::{fs::File, io::Write, sync::Arc, time::Instant}; + +use arrow::{ + array::{ArrayRef, Int64Array, StringArray, UInt64Array}, + datatypes::{DataType, Field, Schema}, + record_batch::RecordBatch, +}; + +fn arg(i: usize, default: usize) -> usize { + std::env::args() + .nth(i) + .and_then(|s| s.parse().ok()) + .unwrap_or(default) +} + +fn strings(name: &str, values: &[String]) -> (Field, ArrayRef) { + ( + Field::new(name, DataType::Utf8, false), + Arc::new(StringArray::from_iter_values( + values.iter().map(String::as_str), + )), + ) +} + +fn ints(name: &str, values: &[i64]) -> (Field, ArrayRef) { + ( + Field::new(name, DataType::Int64, false), + Arc::new(Int64Array::from(values.to_vec())), + ) +} + +/// CSR node offsets: the engine wants UINT64 row offsets, not key values. +fn uints(name: &str, values: &[i64]) -> (Field, ArrayRef) { + ( + Field::new(name, DataType::UInt64, false), + Arc::new(UInt64Array::from( + values.iter().map(|v| *v as u64).collect::>(), + )), + ) +} + +fn batch(cols: Vec<(Field, ArrayRef)>) -> RecordBatch { + let (fields, arrays): (Vec<_>, Vec<_>) = cols.into_iter().unzip(); + RecordBatch::try_new(Arc::new(Schema::new(fields)), arrays).expect("batch") +} + +fn q(conn: &lbug::Connection<'_>, sql: &str) -> Result<(), String> { + conn.query(sql).map(drop).map_err(|e| e.to_string()) +} + +/// Runs `f`, prints the variant's rate or the error it returned. +fn time(label: &str, rows: usize, f: impl FnOnce() -> Result<(), String>) { + let wanted = std::env::var("VARIANTS").unwrap_or_else(|_| "ABCDE".into()); + if !wanted.contains(&label[..1]) { + println!("{label:58} skipped (VARIANTS={wanted})"); + return; + } + let t = Instant::now(); + match f() { + Ok(()) => { + let s = t.elapsed().as_secs_f64(); + println!( + "{label:58} {rows:>9} rows {s:8.2} s {:>10.0} rows/s", + rows as f64 / s + ); + } + Err(e) => println!("{label:58} ERROR {}", e.lines().next().unwrap_or("")), + } +} + +fn main() { + let n = arg(1, 1_000_000); + let m = arg(2, 5_000_000); + let dir = tempfile::tempdir().expect("tempdir"); + let db = lbug::Database::new(dir.path().join("db"), lbug::SystemConfig::default()).expect("db"); + let conn = lbug::Connection::new(&db).expect("conn"); + println!( + "lbug {} · {n} nodes, {m} edges · db in {}", + env!("CARGO_PKG_VERSION"), + dir.path().display() + ); + + // ---- the data: string ids like the adapter's, and their integer twins + let ids: Vec = (0..n).map(|i| format!("n{i}")).collect(); + let empty: Vec = vec!["{}".to_string(); n]; + let from_i: Vec = (0..m).map(|i| ((i * 7919) % n) as i64).collect(); + let to_i: Vec = (0..m).map(|i| ((i * 104_729 + 13) % n) as i64).collect(); + let from_s: Vec = from_i.iter().map(|i| format!("n{i}")).collect(); + let to_s: Vec = to_i.iter().map(|i| format!("n{i}")).collect(); + let eids: Vec = (0..m).map(|i| format!("e{i}")).collect(); + let eprops: Vec = vec!["{}".to_string(); m]; + + // ---- string-keyed tables, the adapter's schema + for t in ["N", "N2", "N3"] { + q( + &conn, + &format!("CREATE NODE TABLE {t}(id STRING, props STRING, PRIMARY KEY(id));"), + ) + .unwrap(); + } + for (e, t) in [("E", "N"), ("E2", "N2"), ("E3", "N3")] { + q( + &conn, + &format!("CREATE REL TABLE {e}(FROM {t} TO {t}, id STRING, props STRING);"), + ) + .unwrap(); + } + + // A. registered Arrow + COPY FROM (MATCH …): the adapter today + let node_batch = batch(vec![strings("id", &ids), strings("props", &empty)]); + time("A nodes: arrow table, COPY FROM (MATCH …)", n, || { + conn.create_arrow_table("s_nodes", &[node_batch.clone()]) + .map_err(|e| e.to_string())?; + let r = q( + &conn, + "COPY N FROM (MATCH (x:s_nodes) RETURN x.id, x.props);", + ); + conn.drop_arrow_table("s_nodes") + .map_err(|e| e.to_string())?; + r + }); + let rel_batch = batch(vec![ + strings("from", &from_s), + strings("to", &to_s), + strings("id", &eids), + strings("props", &eprops), + ]); + time("A edges: arrow rel table, COPY FROM (MATCH …)", m, || { + conn.create_arrow_rel_table("s_rels", &[rel_batch.clone()], "N", "N") + .map_err(|e| e.to_string())?; + let r = q( + &conn, + "COPY E FROM (MATCH (a:N)-[r:s_rels]->(b:N) RETURN a.id, b.id, r.id, r.props);", + ); + conn.drop_arrow_table("s_rels").map_err(|e| e.to_string())?; + r + }); + + // B. COPY t FROM directly + time("B nodes: arrow table, COPY FROM s_nodes", n, || { + conn.create_arrow_table("s_nodes", &[node_batch.clone()]) + .map_err(|e| e.to_string())?; + let r = q(&conn, "COPY N2 FROM s_nodes;"); + conn.drop_arrow_table("s_nodes") + .map_err(|e| e.to_string())?; + r + }); + time("B edges: arrow rel table, COPY FROM s_rels", m, || { + conn.create_arrow_rel_table("s_rels", &[rel_batch.clone()], "N2", "N2") + .map_err(|e| e.to_string())?; + let r = q(&conn, "COPY E2 FROM s_rels;"); + conn.drop_arrow_table("s_rels").map_err(|e| e.to_string())?; + r + }); + + // C. CSV files, string keys + let nodes_csv = dir.path().join("nodes.csv"); + let rels_csv = dir.path().join("rels.csv"); + { + let mut f = File::create(&nodes_csv).unwrap(); + for id in &ids { + writeln!(f, "{id},{{}}").unwrap(); + } + let mut f = File::create(&rels_csv).unwrap(); + for i in 0..m { + writeln!(f, "{},{},{},{{}}", from_s[i], to_s[i], eids[i]).unwrap(); + } + } + time("C nodes: COPY FROM 'nodes.csv' (STRING keys)", n, || { + q(&conn, &format!("COPY N3 FROM '{}';", nodes_csv.display())) + }); + time("C edges: COPY FROM 'rels.csv' (STRING keys)", m, || { + q(&conn, &format!("COPY E3 FROM '{}';", rels_csv.display())) + }); + + // D. CSV files, INT64 keys, no properties: the CSR-parquet shape + q(&conn, "CREATE NODE TABLE NI(id INT64, PRIMARY KEY(id));").unwrap(); + for e in ["EI", "EI2", "EI3"] { + q(&conn, &format!("CREATE REL TABLE {e}(FROM NI TO NI);")).unwrap(); + } + let nodes_i_csv = dir.path().join("nodes_i.csv"); + let rels_i_csv = dir.path().join("rels_i.csv"); + { + let mut f = File::create(&nodes_i_csv).unwrap(); + for i in 0..n { + writeln!(f, "{i}").unwrap(); + } + let mut f = File::create(&rels_i_csv).unwrap(); + for i in 0..m { + writeln!(f, "{},{}", from_i[i], to_i[i]).unwrap(); + } + } + time("D nodes: COPY FROM 'nodes_i.csv' (INT64 keys)", n, || { + q(&conn, &format!("COPY NI FROM '{}';", nodes_i_csv.display())) + }); + time( + "D edges: COPY FROM 'rels_i.csv' (INT64 keys, no props)", + m, + || q(&conn, &format!("COPY EI FROM '{}';", rels_i_csv.display())), + ); + + // E. CSR registration with INT64 keys: indices sorted by source, indptr offsets + let mut order: Vec = (0..m).collect(); + order.sort_by_key(|&i| from_i[i]); + let indices: Vec = order.iter().map(|&i| to_i[i]).collect(); + let mut indptr: Vec = vec![0; n + 1]; + for &i in &order { + indptr[from_i[i] as usize + 1] += 1; + } + for i in 0..n { + indptr[i + 1] += indptr[i]; + } + let indices_batch = batch(vec![uints("to", &indices)]); + let indptr_batch = batch(vec![uints("offset", &indptr)]); + time( + "E edges: CSR arrow rel table, COPY FROM (MATCH …)", + m, + || { + conn.create_arrow_rel_table_csr( + "s_csr", + &[indices_batch.clone()], + &[indptr_batch.clone()], + "NI", + "NI", + "to", + ) + .map_err(|e| e.to_string())?; + let r = q( + &conn, + "COPY EI2 FROM (MATCH (a:NI)-[r:s_csr]->(b:NI) RETURN a.id, b.id);", + ); + conn.drop_arrow_table("s_csr").map_err(|e| e.to_string())?; + r + }, + ); + time("E edges: CSR arrow rel table, COPY FROM s_csr", m, || { + conn.create_arrow_rel_table_csr( + "s_csr", + &[indices_batch.clone()], + &[indptr_batch.clone()], + "NI", + "NI", + "to", + ) + .map_err(|e| e.to_string())?; + let r = q(&conn, "COPY EI3 FROM s_csr;"); + conn.drop_arrow_table("s_csr").map_err(|e| e.to_string())?; + r + }); + + // sanity: what landed + for (t, want) in [ + ("N", n), + ("N2", n), + ("N3", n), + ("NI", n), + ("E", m), + ("E2", m), + ("E3", m), + ("EI", m), + ("EI2", m), + ("EI3", m), + ] { + let pat = if t.starts_with('E') { + format!("MATCH ()-[r:{t}]->() RETURN count(r);") + } else { + format!("MATCH (x:{t}) RETURN count(x);") + }; + let got = conn + .query(&pat) + .ok() + .and_then(|mut r| r.next()) + .map(|row| format!("{:?}", row[0])) + .unwrap_or_else(|| "?".into()); + println!("count {t:4} = {got} (want {want})"); + } +} diff --git a/crates/grust-ladybug/src/bulk_load.rs b/crates/grust-ladybug/src/bulk_load.rs index 19df632b..8c133d0d 100644 --- a/crates/grust-ladybug/src/bulk_load.rs +++ b/crates/grust-ladybug/src/bulk_load.rs @@ -1,32 +1,32 @@ -//! Bulk loading through registered Arrow tables. +//! Bulk loading through the engine's file `COPY`. //! //! One `MERGE` per node or edge costs the engine a full statement each, //! measured at about 50 ms of CPU for a relationship `MERGE … SET` on a 4-vCPU -//! host, so a 200,000-edge slice took five hours. The `lbug` crate can register -//! Arrow record batches as tables; `COPY … FROM (MATCH …)` over such a table -//! loads the same 100,000 nodes in a quarter of a second and 112,000 edges in -//! eight. +//! host, so a 200,000-edge slice took five hours. Registering the rows as an +//! Arrow table and running `COPY … FROM (MATCH …)` over it fixed the nodes +//! (a million rows a second) but not the edges: that probe is superlinear in +//! the engine, 4,100 edges a second at 500,000 and unfinished after two hours +//! at five million, which is why cit-Patents never loaded. The engine's own +//! `COPY … FROM 'file.csv'` takes the same rows, STRING keys and JSON props +//! included, at 1.3 million edges a second (five million in 3.9 s; 2.2 million +//! nodes a second), so every bulk write goes through a temporary CSV file. +//! The `lbug` crate has no equivalent of Python's `COPY … FROM $df`, and the +//! binder rejects `COPY … FROM `. //! //! `put_graph` keeps its upsert semantics: rows whose node id, or whose //! `(from, to)` pair for a relationship table, already exists go through the //! per-row `MERGE` path exactly as before; every other row is copied. Within //! one load the last row for an id or pair wins, which is what a sequence of -//! `MERGE`s would leave behind. +//! `MERGE`s would leave behind. An empty target table skips the read-back of +//! existing keys, so a fresh load costs nothing beyond the copy. -use std::{ - collections::{BTreeMap, HashSet}, - sync::atomic::AtomicU64, -}; +use std::collections::{BTreeMap, HashSet}; use arrow::record_batch::RecordBatch; use grust_core::prelude::*; use super::{LadybugGraphStore, ladybug_error, props_to_string}; -/// A registered Arrow table's name is unique per process so two loads on the -/// same connection never collide. -static SCRATCH_TABLES: AtomicU64 = AtomicU64::new(0); - impl LadybugGraphStore { /// Write `nodes` into `table`, copying rows whose id is new and merging /// the rest. Returns how many rows were written. @@ -36,7 +36,11 @@ impl LadybugGraphStore { table: &str, nodes: &[&Node], ) -> Result { - let existing = self.existing_node_ids(conn, table)?; + let existing = if Self::count(conn, &format!("MATCH (n:{table}) RETURN count(n);"))? == 0 { + HashSet::new() + } else { + self.existing_node_ids(conn, table)? + }; let mut fresh: BTreeMap<&str, &Node> = BTreeMap::new(); for node in nodes { if existing.contains(node.id.as_str()) { @@ -57,7 +61,7 @@ impl LadybugGraphStore { ("id", ids.clone()), ("props", props.iter().map(String::as_str).collect()), ])?; - self.copy_from_arrow_nodes(conn, table, batch)?; + self.copy_from_csv(conn, table, &batch)?; self.bulk_write_node_index(conn, table, &fresh)?; Ok(nodes.len()) } @@ -72,7 +76,15 @@ impl LadybugGraphStore { to_table: &str, edges: &[&Edge], ) -> Result { - let existing = self.existing_edge_pairs(conn, rel_table, from_table, to_table)?; + let existing = if Self::count( + conn, + &format!("MATCH ()-[r:{rel_table}]->() RETURN count(r);"), + )? == 0 + { + HashSet::new() + } else { + self.existing_edge_pairs(conn, rel_table, from_table, to_table)? + }; let mut fresh: BTreeMap<(&str, &str), &Edge> = BTreeMap::new(); for edge in edges { let pair = (edge.from.as_str(), edge.to.as_str()); @@ -99,35 +111,59 @@ impl LadybugGraphStore { ("id", ids.iter().map(String::as_str).collect()), ("props", props.iter().map(String::as_str).collect()), ])?; - let scratch = scratch_table_name(&self.config.table_prefix); - conn.create_arrow_rel_table(&scratch, &[batch], from_table, to_table) - .map_err(ladybug_error)?; - let copy = Self::exec( - conn, - &format!( - "COPY {rel_table} FROM (MATCH (a:{from_table})-[r:{scratch}]->(b:{to_table}) RETURN a.id, b.id, r.id, r.props);" - ), - ); - conn.drop_arrow_table(&scratch).map_err(ladybug_error)?; - copy?; + // Positional: FROM, TO, then the rel table's own columns. + self.copy_from_csv(conn, rel_table, &batch)?; Ok(edges.len()) } - fn copy_from_arrow_nodes( + /// Writes `batch` to a temporary CSV and runs the engine's `COPY` over it, + /// columns positional. The file lives only for the statement. + fn copy_from_csv( &self, conn: &lbug::Connection<'_>, table: &str, - batch: RecordBatch, + batch: &RecordBatch, ) -> Result<()> { - let scratch = scratch_table_name(&self.config.table_prefix); - conn.create_arrow_table(&scratch, &[batch]) - .map_err(ladybug_error)?; - let copy = Self::exec( + let io = |err: std::io::Error| { + GrustError::Backend(format!("Ladybug bulk copy file error: {err}")) + }; + let dir = tempfile::Builder::new() + .prefix("grust-ladybug-copy-") + .tempdir() + .map_err(io)?; + let path = dir.path().join("rows.csv"); + { + let file = std::fs::File::create(&path).map_err(io)?; + let mut writer = arrow::csv::WriterBuilder::new() + .with_header(false) + .build(file); + writer.write(batch).map_err(|err| { + GrustError::Serialization(format!("Ladybug bulk copy CSV error: {err}")) + })?; + } + let path = path.display().to_string(); + if path.contains('\'') { + return Err(GrustError::Backend(format!( + "Ladybug bulk copy: temp path has a quote: {path}" + ))); + } + Self::exec( conn, - &format!("COPY {table} FROM (MATCH (n:{scratch}) RETURN n.id, n.props);"), - ); - conn.drop_arrow_table(&scratch).map_err(ladybug_error)?; - copy + &format!( + "COPY {table} FROM '{path}' (HEADER=false, DELIM=',', QUOTE='\"', ESCAPE='\"');" + ), + ) + } + + /// One integer from a `RETURN count(…)` query. + fn count(conn: &lbug::Connection<'_>, query: &str) -> Result { + let mut rows = conn.query(query).map_err(ladybug_error)?; + match rows.next().as_deref() { + Some([lbug::Value::Int64(n), ..]) => Ok(*n), + other => Err(GrustError::Serialization(format!( + "Ladybug count returned {other:?}" + ))), + } } /// The metadata index gets one row per new node, copied the same way. @@ -145,17 +181,7 @@ impl LadybugGraphStore { ("label", labels), ("table_name", vec![table; fresh.len()]), ])?; - let scratch = scratch_table_name(&self.config.table_prefix); - conn.create_arrow_table(&scratch, &[batch]) - .map_err(ladybug_error)?; - let copy = Self::exec( - conn, - &format!( - "COPY {node_index} FROM (MATCH (n:{scratch}) RETURN n.id, n.kind, n.label, n.table_name);" - ), - ); - conn.drop_arrow_table(&scratch).map_err(ladybug_error)?; - copy + self.copy_from_csv(conn, &node_index, &batch) } fn existing_node_ids( @@ -192,13 +218,87 @@ impl LadybugGraphStore { } } -fn scratch_table_name(prefix: &str) -> String { - let n = SCRATCH_TABLES.fetch_add(1, std::sync::atomic::Ordering::Relaxed); - format!("{prefix}_arrow_in_{}_{n}", std::process::id()) -} - /// A record batch of non-null UTF-8 columns. fn string_batch(columns: &[(&str, Vec<&str>)]) -> Result { grust_arrow::v55::string_batch(columns) .map_err(|err| GrustError::Serialization(format!("Ladybug Arrow batch error: {err}"))) } + +#[cfg(test)] +mod tests { + use super::super::LadybugGraphStore; + use grust_core::prelude::*; + use serde_json::json; + + fn props(pairs: &[(&str, serde_json::Value)]) -> Props { + pairs + .iter() + .map(|(k, v)| (k.to_string(), Value::from(v.clone()))) + .collect() + } + + /// Values that break naive CSV: commas, double quotes, newlines, a + /// backslash, a quote-only string, an empty string, unicode. + #[test] + fn bulk_copy_round_trips_hostile_props_and_then_upserts() -> Result<()> { + let rt = tokio::runtime::Runtime::new().unwrap(); + rt.block_on(async { + let store = LadybugGraphStore::in_memory()?; + let nasty = props(&[ + ("comma", json!("a,b")), + ("quote", json!("say \"hi\"")), + ("newline", json!("line1\nline2")), + ("backslash", json!("c:\\path")), + ("only_quote", json!("\"")), + ("empty", json!("")), + ("unicode", json!("naïve 日本 🚀")), + ("nested", json!({"k": ["x,y", "\"z\""]})), + ]); + let nodes: Vec = (0..3) + .map(|i| Node::new("Thing", format!("n{i}"), nasty.clone())) + .collect(); + let edges = vec![ + Edge::new("REL", "n0", "n1", nasty.clone()), + Edge::new("REL", "n1", "n2", props(&[("plain", json!(1))])), + ]; + let report = store.put_graph(&Graph::new(nodes, edges)).await?; + assert_eq!((report.nodes, report.edges), (3, 2)); + // Reads add the id into the props; compare the stored values only. + let mut got = store.get_node(&NodeId::new("n1")).await?.expect("n1"); + got.props.remove("id"); + assert_eq!(got.props, nasty); + let out = store + .get_edges(EdgeQuery { + from: Some(NodeId::new("n0")), + ..Default::default() + }) + .await?; + assert_eq!(out.len(), 1); + assert_eq!(out[0].props, nasty); + + // A second load of the same ids takes the merge path (the tables + // are no longer empty) and must not duplicate anything. + // A put carries its edges' endpoints, as every caller's batches do. + let again = Graph::new( + vec![ + Node::new("Thing", "n0", nasty.clone()), + Node::new("Thing", "n1", props(&[("v", json!(2))])), + ], + vec![Edge::new("REL", "n0", "n1", props(&[("v", json!(2))]))], + ); + store.put_graph(&again).await?; + let mut got = store.get_node(&NodeId::new("n1")).await?.expect("n1"); + got.props.remove("id"); + assert_eq!(got.props, props(&[("v", json!(2))])); + let out = store + .get_edges(EdgeQuery { + from: Some(NodeId::new("n0")), + ..Default::default() + }) + .await?; + assert_eq!(out.len(), 1, "upsert must not add a parallel edge"); + assert_eq!(out[0].props, props(&[("v", json!(2))])); + Ok(()) + }) + } +} diff --git a/crates/grust-ladybug/src/lib.rs b/crates/grust-ladybug/src/lib.rs index 407c04fe..6587fe3f 100644 --- a/crates/grust-ladybug/src/lib.rs +++ b/crates/grust-ladybug/src/lib.rs @@ -34,8 +34,18 @@ pub struct LadybugConfig { /// concurrent writers queue on it. On, the engine resolves conflicting /// write transactions and a losing writer gets its error back. pub concurrent_writes: bool, + /// Cap on the database size in bytes. `None` keeps the engine's default of + /// 8 TiB, which the engine reserves as virtual address space per open + /// database; on x86-64 Linux (128 TiB of user address space) that allows + /// only about fifteen databases open at once in one process. + pub max_db_bytes: Option, } +/// Unit tests open many databases in parallel threads; like Ladybug's own +/// test configuration, they cap each at 16 GiB so reservations cannot exhaust +/// the process address space. +const TEST_MAX_DB_BYTES: u64 = 16 * 1024 * 1024 * 1024; + impl Default for LadybugConfig { fn default() -> Self { Self { @@ -45,6 +55,7 @@ impl Default for LadybugConfig { query_timeout_ms: None, buffer_pool_bytes: None, concurrent_writes: false, + max_db_bytes: cfg!(test).then_some(TEST_MAX_DB_BYTES), } } } @@ -116,11 +127,14 @@ struct RelTable { impl LadybugGraphStore { pub fn new(config: LadybugConfig) -> Result { - let system = match config.buffer_pool_bytes { - Some(bytes) => lbug::SystemConfig::default().buffer_pool_size(bytes), - None => lbug::SystemConfig::default(), + let mut system = + lbug::SystemConfig::default().enable_multi_writes(config.concurrent_writes); + if let Some(bytes) = config.buffer_pool_bytes { + system = system.buffer_pool_size(bytes); + } + if let Some(bytes) = config.max_db_bytes { + system = system.max_db_size(bytes); } - .enable_multi_writes(config.concurrent_writes); let db = match &config.path { LadybugPath::InMemory => lbug::Database::in_memory(system), LadybugPath::Directory(path) => lbug::Database::new(path, system), diff --git a/crates/grust-sql-core/src/lib.rs b/crates/grust-sql-core/src/lib.rs index bfa6c57e..b5edafce 100644 --- a/crates/grust-sql-core/src/lib.rs +++ b/crates/grust-sql-core/src/lib.rs @@ -46,6 +46,14 @@ pub trait GraphSqlDialect { None } + /// Whether the universal bootstrap creates `_edges_from_idx` on + /// `from_id`. The edge primary key already leads with `from_id`, so a + /// dialect whose planner answers source lookups from that key (Turso's + /// does) can skip the index and the write it costs on every edge. + fn edge_source_index(&self) -> bool { + true + } + fn commit_transaction(&self) -> &'static str { "COMMIT" } @@ -121,8 +129,7 @@ pub fn universal_bootstrap_sql( props {props_type} NOT NULL DEFAULT {props_default},{identity_column} PRIMARY KEY (from_id, label, to_id{identity_key}) ); - CREATE INDEX IF NOT EXISTS {edge_from_idx} ON {edges_table}(from_id); - CREATE INDEX IF NOT EXISTS {edge_to_idx} ON {edges_table}(to_id); + {edge_from_index}CREATE INDEX IF NOT EXISTS {edge_to_idx} ON {edges_table}(to_id); CREATE INDEX IF NOT EXISTS {node_label_idx} ON {nodes_table}(label);", nodes_table = tables.nodes, edges_table = tables.edges, @@ -136,7 +143,15 @@ pub fn universal_bootstrap_sql( .unwrap_or_default(), props_type = dialect.props_column_type(), props_default = dialect.empty_props_default(), - edge_from_idx = quote_ident(&format!("{table_prefix}_edges_from_idx")), + edge_from_index = if dialect.edge_source_index() { + format!( + "CREATE INDEX IF NOT EXISTS {} ON {}(from_id);\n ", + quote_ident(&format!("{table_prefix}_edges_from_idx")), + tables.edges + ) + } else { + String::new() + }, edge_to_idx = quote_ident(&format!("{table_prefix}_edges_to_idx")), node_label_idx = quote_ident(&format!("{table_prefix}_nodes_label_idx")), ) diff --git a/crates/grust-turso/Cargo.toml b/crates/grust-turso/Cargo.toml index 420586b4..6e44d1c8 100644 --- a/crates/grust-turso/Cargo.toml +++ b/crates/grust-turso/Cargo.toml @@ -12,6 +12,8 @@ categories.workspace = true [features] default = [] sync = ["turso/sync"] +# Turso's own default allocator, as the global allocator of the whole binary. +mimalloc = ["turso/mimalloc"] [dependencies] async-trait = { workspace = true } diff --git a/crates/grust-turso/examples/bulk_load.rs b/crates/grust-turso/examples/bulk_load.rs index cfa59794..2f71ad8e 100644 --- a/crates/grust-turso/examples/bulk_load.rs +++ b/crates/grust-turso/examples/bulk_load.rs @@ -28,6 +28,10 @@ struct Args { mode: TursoJournalMode, path: String, snap: Option, + wal_load: bool, + load_threads: usize, + batch: usize, + round_groups: usize, } fn parse_args() -> Args { @@ -41,6 +45,10 @@ fn parse_args() -> Args { .display() .to_string(), snap: None, + wal_load: false, + load_threads: 1, + batch: 500, + round_groups: 0, }; let mut it = std::env::args().skip(1); while let Some(flag) = it.next() { @@ -58,6 +66,16 @@ fn parse_args() -> Args { } "--path" => args.path = value, "--snap" => args.snap = Some(value), + "--load-threads" => args.load_threads = value.parse().expect("--load-threads"), + "--batch" => args.batch = value.parse().expect("--batch"), + "--round-groups" => args.round_groups = value.parse().expect("--round-groups"), + "--wal-load" => { + args.wal_load = match value.as_str() { + "on" => true, + "off" => false, + other => panic!("unknown wal-load {other}"), + } + } other => panic!("unknown flag {other}"), } } @@ -190,7 +208,7 @@ impl Csr { } } -#[tokio::main(flavor = "current_thread")] +#[tokio::main(flavor = "multi_thread")] async fn main() -> Result<()> { let args = parse_args(); let started = Instant::now(); @@ -211,11 +229,14 @@ async fn main() -> Result<()> { let store = TursoGraphStore::connect(TursoConfig { path: args.path.clone(), table_prefix: "ag".to_string(), - batch_size: 500, + batch_size: args.batch, journal_mode: args.mode, }) .await?; store.bootstrap().await?; + store.set_bulk_load_via_wal(args.wal_load); + store.set_mvcc_load_parallelism(args.load_threads); + store.set_mvcc_load_round_groups(args.round_groups); let load = Instant::now(); for graph in csr.node_chunks(args.chunk.max(1)) { diff --git a/crates/grust-turso/examples/concurrent_writes.rs b/crates/grust-turso/examples/concurrent_writes.rs new file mode 100644 index 00000000..06bd07c0 --- /dev/null +++ b/crates/grust-turso/examples/concurrent_writes.rs @@ -0,0 +1,195 @@ +//! Hot-node write contention, shaped like the adversarial-graph A4 scenario: +//! `--writers` tasks each attach `--per` new edges to one hub vertex, every +//! write a `put_node` then a `put_edge` on the writer's own store handle. +//! +//! ```text +//! cargo run --release -p grust-turso --example concurrent_writes -- \ +//! [--writers 16] [--per 200] [--mode wal|mvcc] [--handles separate|shared] \ +//! [--path /tmp/concurrent.db] +//! ``` +//! +//! `separate` opens every writer with `TursoGraphStore::connect` on the same +//! path (one `turso::Database` each, as the harness does); `shared` opens +//! them with `connect_shared` (one database, one connection per writer). +//! Prints accepted writes, typed conflicts, other errors, wall time and +//! accepted writes per second, then checks the hub's final out-degree. + +use grust_core::prelude::*; +use grust_turso::{TursoConfig, TursoGraphStore, TursoJournalMode, TursoSynchronous}; +use std::sync::Arc; +use std::time::Instant; + +const HUB: &str = "hub"; + +struct Args { + writers: usize, + per: usize, + mode: TursoJournalMode, + shared: bool, + sync: Option, + group: bool, + path: String, +} + +fn parse_args() -> Args { + let mut args = Args { + writers: 16, + per: 200, + mode: TursoJournalMode::Mvcc, + shared: false, + sync: None, + group: false, + path: std::env::temp_dir() + .join("grust-turso-concurrent.db") + .display() + .to_string(), + }; + let mut it = std::env::args().skip(1); + while let Some(flag) = it.next() { + let value = it.next().unwrap_or_else(|| panic!("{flag} needs a value")); + match flag.as_str() { + "--writers" => args.writers = value.parse().expect("--writers"), + "--per" => args.per = value.parse().expect("--per"), + "--mode" => { + args.mode = match value.as_str() { + "wal" => TursoJournalMode::Wal, + "mvcc" => TursoJournalMode::Mvcc, + other => panic!("unknown mode {other}"), + } + } + "--handles" => { + args.shared = match value.as_str() { + "shared" => true, + "separate" => false, + other => panic!("unknown handles {other}"), + } + } + "--sync" => { + args.sync = Some(match value.as_str() { + "full" => TursoSynchronous::Full, + "normal" => TursoSynchronous::Normal, + "off" => TursoSynchronous::Off, + other => panic!("unknown sync {other}"), + }) + } + "--group" => { + args.group = match value.as_str() { + "on" => true, + "off" => false, + other => panic!("unknown group {other}"), + } + } + "--path" => args.path = value, + other => panic!("unknown flag {other}"), + } + } + args +} + +fn is_conflict(err: &GrustError) -> bool { + let msg = err.to_string().to_ascii_lowercase(); + msg.contains("conflict") || msg.contains("busy") || msg.contains("locked") +} + +#[tokio::main(flavor = "multi_thread")] +async fn main() -> Result<()> { + let args = parse_args(); + for suffix in ["", "-wal", "-shm", "-log"] { + let _ = std::fs::remove_file(format!("{}{suffix}", args.path)); + } + let config = TursoConfig { + path: args.path.clone(), + table_prefix: "ag".to_string(), + batch_size: 500, + journal_mode: args.mode, + }; + let mut base = TursoGraphStore::connect(config.clone()).await?; + base.bootstrap().await?; + if args.group { + base = base.with_group_commit().await?; + } + base.put_node(&Node::new("Node", HUB, Props::new())).await?; + + let opened = Instant::now(); + let mut stores = Vec::with_capacity(args.writers); + for _ in 0..args.writers { + let store = if args.shared { + base.connect_shared().await? + } else { + TursoGraphStore::connect(config.clone()).await? + }; + if let Some(mode) = args.sync { + store.set_synchronous(mode).await?; + } + stores.push(Arc::new(store)); + } + let open_s = opened.elapsed().as_secs_f64(); + + let barrier = Arc::new(tokio::sync::Barrier::new(args.writers)); + let started = Instant::now(); + let mut tasks = Vec::with_capacity(args.writers); + for (w, store) in stores.into_iter().enumerate() { + let barrier = barrier.clone(); + let per = args.per; + tasks.push(tokio::spawn(async move { + let (mut accepted, mut conflicts, mut other) = (0usize, 0usize, Vec::new()); + barrier.wait().await; + for i in 0..per { + let target = format!("hot-{w}-{i}"); + let outcome = match store + .put_node(&Node::new("Node", target.clone(), Props::new())) + .await + { + Ok(_) => store + .put_edge(&Edge::new("EDGE", HUB, target, Props::new())) + .await + .map(|_| ()), + Err(e) => Err(e), + }; + match outcome { + Ok(()) => accepted += 1, + Err(e) if is_conflict(&e) => conflicts += 1, + Err(e) => other.push(e.to_string()), + } + } + (accepted, conflicts, other) + })); + } + let (mut accepted, mut conflicts, mut other) = (0usize, 0usize, Vec::new()); + for task in tasks { + let (a, c, o) = task.await.expect("writer task"); + accepted += a; + conflicts += c; + other.extend(o); + } + let wall = started.elapsed().as_secs_f64(); + let degree = base + .get_edges(EdgeQuery { + from: Some(NodeId::new(HUB)), + to: None, + label: None, + }) + .await? + .len(); + println!( + "group={} sync={:?} mode={:?} handles={} writers={} per={} open={:.2}s wall={:.2}s accepted={} conflicts={} other={} writes/s={:.0} hub_degree={} consistent={}", + args.group, + args.sync, + args.mode, + if args.shared { "shared" } else { "separate" }, + args.writers, + args.per, + open_s, + wall, + accepted, + conflicts, + other.len(), + accepted as f64 / wall, + degree, + degree == accepted + ); + for e in other.iter().take(3) { + println!(" other error: {e}"); + } + Ok(()) +} diff --git a/crates/grust-turso/src/lib.rs b/crates/grust-turso/src/lib.rs index b9e1e70c..fca651a5 100644 --- a/crates/grust-turso/src/lib.rs +++ b/crates/grust-turso/src/lib.rs @@ -5,7 +5,7 @@ use grust_cypher::pushdown::{NoTypeHints, SqlDialect, StrOp, combine_union, plan use grust_cypher::{CypherParameters, CypherResultTable}; use grust_sql_core::{GraphSqlDialect, UniversalTableRefs}; use std::sync::Arc; -use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; mod guarded_commit; @@ -17,8 +17,9 @@ pub enum TursoJournalMode { Wal, /// Multi-version concurrency control (`PRAGMA journal_mode = mvcc`), which /// enables `BEGIN CONCURRENT` concurrent writers. MVCC is a database-*header* - /// mode, so it only takes effect on a **fresh** database — an existing WAL - /// database is not converted (`connect` errors if the mode cannot be applied). + /// mode; Turso 0.7.2 converts an existing WAL database when it is opened + /// in this mode, and a live store can switch between the two + /// (`set_bulk_load_via_wal`). `connect` errors if the mode is not applied. Mvcc, } @@ -32,6 +33,92 @@ impl TursoJournalMode { } } +/// `PRAGMA synchronous` for a connection. Turso's default is `Full`: every +/// commit fsyncs (under MVCC, while holding the global commit lock). +/// `Normal` skips the per-commit fsync and still fsyncs at checkpoints, so a +/// crash can lose the commits since the last checkpoint; `Off` never fsyncs. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum TursoSynchronous { + Off, + Normal, + Full, +} + +impl TursoSynchronous { + fn pragma_value(self) -> &'static str { + match self { + TursoSynchronous::Off => "OFF", + TursoSynchronous::Normal => "NORMAL", + TursoSynchronous::Full => "FULL", + } + } +} + +/// One statement waiting for a group commit, and where to send its outcome. +struct GroupJob { + sql: String, + reply: tokio::sync::oneshot::Sender>, +} + +/// The MVCC group committer: a task that owns one connection and commits +/// whatever statements are waiting as one transaction. +struct GroupCommit { + jobs: tokio::sync::mpsc::UnboundedSender, +} + +/// Most statements one group commit carries; a larger backlog waits for the +/// next commit. +const GROUP_COMMIT_MAX_STATEMENTS: usize = 256; + +impl GroupCommit { + fn spawn(committer: TursoGraphStore) -> Self { + let (jobs, mut queue) = tokio::sync::mpsc::unbounded_channel::(); + tokio::spawn(async move { + while let Some(first) = queue.recv().await { + let mut batch = vec![first]; + while batch.len() < GROUP_COMMIT_MAX_STATEMENTS { + match queue.try_recv() { + Ok(job) => batch.push(job), + Err(_) => break, + } + } + let statements: Vec = batch.iter().map(|job| job.sql.clone()).collect(); + if batch.len() > 1 + && committer + .execute_transaction(&statements, true) + .await + .is_ok() + { + for job in batch { + let _ = job.reply.send(Ok(())); + } + continue; + } + // One statement, or a batch that failed as a whole: each job + // runs as its own transaction with the usual conflict retry, + // so only the failing statement's caller sees the error. + for job in batch { + let outcome = committer + .execute_transaction(std::slice::from_ref(&job.sql), true) + .await; + let _ = job.reply.send(outcome); + } + } + }); + Self { jobs } + } + + async fn submit(&self, sql: String) -> Result<()> { + let (reply, outcome) = tokio::sync::oneshot::channel(); + self.jobs + .send(GroupJob { sql, reply }) + .map_err(|_| GrustError::Backend("Turso group committer has stopped".to_string()))?; + outcome + .await + .map_err(|_| GrustError::Backend("Turso group committer dropped a write".to_string()))? + } +} + #[derive(Clone, Debug)] pub struct TursoConfig { pub path: String, @@ -80,6 +167,18 @@ pub struct TursoGraphStore { /// cannot accidentally absorb a concurrent read or write on the same /// connection. connection_gate: tokio::sync::Mutex<()>, + /// MVCC group commit shared by every handle opened with `connect_shared` + /// from a store that called `with_group_commit`. + group: Option>, + /// When set, an MVCC `put_graph` switches the database to WAL for the + /// load and back to MVCC afterwards (see `set_bulk_load_via_wal`). + bulk_load_via_wal: AtomicBool, + /// Concurrent writers an MVCC `put_graph` uses (see + /// `set_mvcc_load_parallelism`); 1 loads on this connection alone. + mvcc_load_parallelism: AtomicUsize, + /// Commit groups a parallel load writer runs between round checkpoints + /// (see `set_mvcc_load_round_groups`); 0 means `MVCC_PARALLEL_ROUND_GROUPS`. + mvcc_load_round_groups: AtomicUsize, /// Set before an explicit transaction begins and cleared only after its /// commit or rollback finishes. If a future is cancelled while the gate is /// held, the next caller recovers the abandoned transaction first. @@ -108,6 +207,53 @@ impl TursoGraphStore { index_cache: std::sync::Mutex::new(None), connection_gate: tokio::sync::Mutex::new(()), transaction_needs_rollback: AtomicBool::new(false), + group: None, + bulk_load_via_wal: AtomicBool::new(false), + mvcc_load_parallelism: AtomicUsize::new(1), + mvcc_load_round_groups: AtomicUsize::new(0), + }; + store.apply_journal_mode().await?; + store + .execute_discarding_rows(&format!("PRAGMA cache_size = -{PAGE_CACHE_KIB}")) + .await?; + Ok(store) + } + + /// Another handle on this store's open database: a new connection that + /// shares the same `turso::Database` (its MVCC store and page cache) + /// instead of opening the file again. Concurrent writers in one process + /// should use this; `connect` on the same path builds a separate database + /// object. + pub async fn connect_shared(&self) -> Result { + let db = match &self._db { + TursoDatabase::Local(db) => db.clone(), + #[cfg(feature = "sync")] + TursoDatabase::Synced(_) => { + return Err(GrustError::Unsupported( + "connect_shared is not available on a synced Turso store".to_string(), + )); + } + }; + let conn = db.connect().map_err(|err| { + GrustError::Backend(format!("failed to connect to Turso database: {err}")) + })?; + let store = Self { + config: self.config.clone(), + _db: TursoDatabase::Local(db), + conn, + index_cache: std::sync::Mutex::new(None), + connection_gate: tokio::sync::Mutex::new(()), + transaction_needs_rollback: AtomicBool::new(false), + // Handles share the store's committer; the committer's own + // connection is opened before it exists and so commits directly. + group: self.group.clone(), + bulk_load_via_wal: AtomicBool::new(self.bulk_load_via_wal.load(Ordering::Acquire)), + mvcc_load_parallelism: AtomicUsize::new( + self.mvcc_load_parallelism.load(Ordering::Acquire), + ), + mvcc_load_round_groups: AtomicUsize::new( + self.mvcc_load_round_groups.load(Ordering::Acquire), + ), }; store.apply_journal_mode().await?; store @@ -131,8 +277,7 @@ impl TursoGraphStore { .await?; if got.as_deref() != Some(want) { return Err(GrustError::Backend(format!( - "requested Turso journal_mode = {want} but the database reports {got:?}; \ - MVCC must be set on a fresh database (an existing WAL database cannot be converted)" + "requested Turso journal_mode = {want} but the database reports {got:?}" ))); } Ok(()) @@ -174,6 +319,100 @@ impl TursoGraphStore { } } + /// The first column of the first row of `sql` as an integer. + async fn query_scalar_i64(&self, sql: &str) -> Result { + let _gate = self.lock_connection().await?; + let mut rows = self + .conn + .query(sql, ()) + .await + .map_err(|err| GrustError::Backend(format!("Turso query failed: {err}: {sql}")))?; + let row = rows + .next() + .await + .map_err(|err| GrustError::Backend(format!("Turso row read failed: {err}: {sql}")))? + .ok_or_else(|| GrustError::Backend(format!("Turso returned no row: {sql}")))?; + match row.get_value(0) { + Ok(turso::Value::Integer(value)) => Ok(value), + other => Err(GrustError::Backend(format!( + "Turso returned {other:?} where an integer was expected: {sql}" + ))), + } + } + + /// Turn on MVCC group commit for this store and every handle later opened + /// from it with `connect_shared`. Single-statement writes (`put_node`, + /// `put_edge`, and the other one-statement mutations) from all of those + /// handles are handed to one committer, which runs whatever is waiting + /// as one `BEGIN CONCURRENT … COMMIT`: under `synchronous = FULL` that + /// is one fsync for the batch instead of one per write, and each caller + /// returns only after the commit that carries its statement. A batch + /// that fails is retried one statement at a time, so a failing or + /// conflicting write fails only its own caller. WAL stores are returned + /// unchanged. Must be called from within a Tokio runtime. + pub async fn with_group_commit(mut self) -> Result { + if self.config.journal_mode != TursoJournalMode::Mvcc || self.group.is_some() { + return Ok(self); + } + let committer = self.connect_shared().await?; + self.group = Some(Arc::new(GroupCommit::spawn(committer))); + Ok(self) + } + + /// Set `PRAGMA synchronous` on this store's connection. Each handle has + /// its own connection, so set it on every handle that should use it. + pub async fn set_synchronous(&self, mode: TursoSynchronous) -> Result<()> { + self.execute_discarding_rows(&format!("PRAGMA synchronous = {}", mode.pragma_value())) + .await + } + + /// Set Turso's engine-level group commit (`PRAGMA mvcc_group_commit`). + /// Store-wide: one setting covers every connection on the database, + /// including handles from `connect_shared` and parallel load writers. + /// + /// Turso 0.8 (upstream `main`) has this and enables it by default; there + /// it batches concurrent single-statement commits into one fsync, and + /// [`Self::with_group_commit`] on top of it measured ~19% slower, so on + /// 0.8 leave this on and skip the client-side committer. Turso 0.7.2 has + /// no such pragma and returns an error; on 0.7.2 the client-side + /// committer is the only group commit and is worth ~6.6x on hot-node + /// writes. Bulk loads are unaffected either way (measured within drift). + pub async fn set_mvcc_group_commit(&self, enabled: bool) -> Result<()> { + let value = if enabled { "on" } else { "off" }; + self.execute_discarding_rows(&format!("PRAGMA mvcc_group_commit = {value}")) + .await + } + + /// Run this MVCC store's `put_graph` calls through WAL: each switches the + /// database to `journal_mode = wal`, loads in one transaction (WAL's load + /// path, several times faster than MVCC's, which keeps every index entry + /// of every row version in memory), checkpoints, and switches back to + /// MVCC, also when the load fails. Journal mode is database-wide, so the + /// caller must not run other writes on the database while a `put_graph` + /// is in flight; a bulk load before serving is the intended use. A no-op + /// for WAL stores. Handles opened later with `connect_shared` inherit it. + pub fn set_bulk_load_via_wal(&self, enabled: bool) { + self.bulk_load_via_wal.store(enabled, Ordering::Release); + } + + /// Load an MVCC store's `put_graph` with `writers` concurrent connections + /// (`BEGIN CONCURRENT` transactions on disjoint slices of the nodes, then + /// of the edges), so the per-row MVCC index work runs on several cores; + /// WAL allows one writer. 1 (the default) loads on this connection alone. + /// Needs a multi-threaded Tokio runtime to use more than one core. + pub fn set_mvcc_load_parallelism(&self, writers: usize) { + self.mvcc_load_parallelism + .store(writers.max(1), Ordering::Release); + } + + /// Commit groups each parallel load writer runs between the checkpoints + /// that bound the load's resident row versions. Fewer, larger rounds mean + /// fewer checkpoints and more memory; 0 restores + /// `MVCC_PARALLEL_ROUND_GROUPS`. + pub fn set_mvcc_load_round_groups(&self, groups: usize) { + self.mvcc_load_round_groups.store(groups, Ordering::Release); + } + pub async fn in_memory() -> Result { Self::connect(TursoConfig::default()).await } @@ -207,6 +446,10 @@ impl TursoGraphStore { index_cache: std::sync::Mutex::new(None), connection_gate: tokio::sync::Mutex::new(()), transaction_needs_rollback: AtomicBool::new(false), + group: None, + bulk_load_via_wal: AtomicBool::new(false), + mvcc_load_parallelism: AtomicUsize::new(1), + mvcc_load_round_groups: AtomicUsize::new(0), }) } @@ -295,6 +538,13 @@ impl TursoGraphStore { async fn execute_data(&self, sql: &str) -> Result<()> { match self.config.journal_mode { TursoJournalMode::Wal => self.execute(sql).await, + TursoJournalMode::Mvcc if self.group.is_some() => { + self.invalidate_snapshot(); + let group = self.group.as_ref().expect("checked above"); + let result = group.submit(sql.to_string()).await; + self.invalidate_snapshot(); + result + } TursoJournalMode::Mvcc => { self.execute_concurrent(std::slice::from_ref(&sql.to_string())) .await @@ -357,6 +607,14 @@ impl TursoGraphStore { ))); } if concurrent && is_mvcc_conflict(&err) && attempt < MAX_ATTEMPTS { + // Back off before retrying: a Busy here means another + // writer holds the commit or checkpoint lock, and an + // immediate BEGIN CONCURRENT would spin against it. + // The workspace tokio has no timer, so yield to the + // runtime once per attempt so far instead of sleeping. + for _ in 0..attempt { + tokio::task::yield_now().await; + } continue; } return Err(err); @@ -423,6 +681,24 @@ impl TursoGraphStore { Ok(!columns.is_empty() && !columns.iter().any(|c| c == "identity_key")) } + /// Whether the retired `_edges_from_idx` is still in the schema. + async fn edges_from_index_exists_unlocked(&self) -> Result { + let sql = format!( + "SELECT 1 FROM sqlite_schema WHERE type = 'index' AND name = {}", + sql_str(&format!("{}_edges_from_idx", self.config.table_prefix)) + ); + let mut rows = self.conn.query(&sql, ()).await.map_err(|err| { + GrustError::Backend(format!("Turso schema query failed: {err}: {sql}")) + })?; + Ok(rows + .next() + .await + .map_err(|err| { + GrustError::Backend(format!("Turso schema row read failed: {err}: {sql}")) + })? + .is_some()) + } + fn edges_table(&self) -> String { quote_ident(&format!("{}_edges", self.config.table_prefix)) } @@ -463,9 +739,35 @@ fn prepared_upsert_edges_sql(table: &str, rows: usize) -> String { } impl TursoGraphStore { - /// `put_graph`: the rows of `graph` as prepared multi-row upserts. + /// `put_graph`: the rows of `graph` as prepared multi-row upserts, loaded + /// with foreign keys off, like the Memory reference: an edge whose + /// endpoint node is not stored is kept, and the endpoint exists only as + /// that edge's end. The schema's `REFERENCES nodes(id) ON DELETE CASCADE` + /// stays in force for every other write on this connection, and cascade + /// fires on the connection that deletes, so serving semantics are + /// unchanged. Before this, only the parallel MVCC writers loaded without + /// the check (they never had `PRAGMA foreign_keys` set), so whether a + /// dangling edge was accepted depended on the writer count; now every + /// path agrees, and WAL and single-writer loads skip the two endpoint + /// probes per edge that parallel loads already skipped. async fn put_graph_rows(&self, graph: &Graph) -> Result { + // Set outside any transaction: SQLite ignores the pragma inside one. + let foreign_keys = self.query_scalar_i64("PRAGMA foreign_keys").await?; + self.execute_discarding_rows("PRAGMA foreign_keys = OFF") + .await?; + let loaded = self.put_graph_rows_unchecked(graph).await; + self.execute_discarding_rows(&format!("PRAGMA foreign_keys = {foreign_keys}")) + .await?; + loaded + } + + async fn put_graph_rows_unchecked(&self, graph: &Graph) -> Result { let rows = self.config.batch_size.max(1); + if self.config.journal_mode == TursoJournalMode::Mvcc + && self.bulk_load_via_wal.load(Ordering::Acquire) + { + return self.put_graph_rows_via_wal(graph, rows).await; + } if self.config.journal_mode == TursoJournalMode::Mvcc { // MVCC keeps every row version of an open transaction in memory // until it commits, so one BEGIN CONCURRENT around a whole load @@ -474,29 +776,198 @@ impl TursoGraphStore { // this way. Committing every MVCC_LOAD_COMMIT_STATEMENTS batches // keeps each transaction small; a load is no longer all-or-nothing // under MVCC, and a failure leaves the groups committed before it. - let group = MVCC_LOAD_COMMIT_STATEMENTS * rows; - for chunk in graph.nodes.chunks(group) { - self.load_transaction(chunk, &[], true, rows).await?; - } - for chunk in graph.edges.chunks(group) { - self.load_transaction(&[], chunk, true, rows).await?; - } + // + // Under the default synchronous = FULL every MVCC commit fsyncs + // the logical log while holding Turso's global commit lock. The + // load runs at NORMAL, which skips the per-commit fsync; the + // TRUNCATE checkpoint below still fsyncs, so the load is durable + // when put_graph returns. The connection's previous setting is + // restored whether or not the load succeeds. + let synchronous = self.query_scalar_i64("PRAGMA synchronous").await?; + self.execute_discarding_rows("PRAGMA synchronous = NORMAL") + .await?; + let loaded = self.put_mvcc_parallel(graph, rows).await; + self.execute_discarding_rows(&format!("PRAGMA synchronous = {synchronous}")) + .await?; + loaded?; } else { self.load_transaction(&graph.nodes, &graph.edges, false, rows) .await?; - // One transaction leaves the whole load in the write-ahead log, - // and every read until the next checkpoint pays to look through - // it; checkpointing here keeps that cost inside the load interval. - // MVCC mode has no equivalent the engine allows by default. - self.execute_discarding_rows("PRAGMA wal_checkpoint(TRUNCATE)") + } + // End every load in a TRUNCATE checkpoint, inside the load interval. + // WAL: one transaction leaves the whole load in the write-ahead log, + // and every read until the next checkpoint pays to look through it. + // MVCC: the automatic checkpoints leave up to a threshold's worth of + // the load in the logical log and its row versions in memory; this + // folds them into the database file, so the load returns durable in + // the B-tree and with the in-memory store drained. + self.execute_discarding_rows("PRAGMA wal_checkpoint(TRUNCATE)") + .await?; + Ok(LoadReport { + nodes: graph.nodes.len(), + edges: graph.edges.len(), + }) + } + + /// An MVCC store's `put_graph` run through WAL (`set_bulk_load_via_wal`): + /// switch to WAL, load everything in one transaction, checkpoint, switch + /// back to MVCC whether or not the load succeeded. + async fn put_graph_rows_via_wal(&self, graph: &Graph, rows: usize) -> Result { + self.switch_journal_mode("wal").await?; + let loaded = async { + self.load_transaction(&graph.nodes, &graph.edges, false, rows) .await?; + self.execute_discarding_rows("PRAGMA wal_checkpoint(TRUNCATE)") + .await } + .await; + let back = self.switch_journal_mode("mvcc").await; + loaded?; + back?; Ok(LoadReport { nodes: graph.nodes.len(), edges: graph.edges.len(), }) } + /// Set the database's journal mode and confirm the engine reports it. + async fn switch_journal_mode(&self, want: &str) -> Result<()> { + let got = self + .query_scalar_text(&format!("PRAGMA journal_mode = {want}")) + .await?; + if got.as_deref() == Some(want) { + Ok(()) + } else { + Err(GrustError::Backend(format!( + "Turso did not switch journal_mode to {want}; it reports {got:?}" + ))) + } + } + + /// The MVCC load with `mvcc_load_parallelism` writers: nodes, then + /// edges, each split into that many contiguous slices, every slice loaded + /// on its own shared connection and task. + /// + /// Automatic checkpoints cannot run while another writer's transaction + /// is open (they fail Busy and are skipped), so free-running writers kept + /// every row version resident: 14–15 GB at 5 M edges against 1.7 GB for + /// one writer. The writers therefore advance in rounds of at most + /// `MVCC_PARALLEL_ROUND_GROUPS` commit groups each; after every round, + /// with nothing in flight, one TRUNCATE checkpoint folds the round into + /// the database file. The first error is returned after its round ends. + async fn put_mvcc_parallel(&self, graph: &Graph, rows: usize) -> Result<()> { + let writers = self.mvcc_load_parallelism.load(Ordering::Acquire); + if writers <= 1 || graph.nodes.len() + graph.edges.len() < 2 * rows { + return self.put_mvcc_groups(graph, rows).await; + } + let mut handles = Vec::with_capacity(writers); + for _ in 0..writers { + let handle = self.connect_shared().await?; + handle + .execute_discarding_rows("PRAGMA synchronous = NORMAL") + .await?; + handle + .execute_discarding_rows(&format!( + "PRAGMA cache_size = -{MVCC_LOAD_WRITER_CACHE_KIB}" + )) + .await?; + handles.push(Arc::new(handle)); + } + // Turso runs an inline garbage-collection pass on the commit path + // every mvcc_gc_threshold new row versions; a bulk load leaves nothing + // worth collecting, and the round checkpoints below drain the versions + // anyway. The store's own threshold is restored after the load. + let gc = self.query_scalar_i64("PRAGMA mvcc_gc_threshold").await?; + self.execute_discarding_rows("PRAGMA mvcc_gc_threshold = -1") + .await?; + let loaded = self.put_mvcc_parallel_phases(graph, rows, &handles).await; + self.execute_discarding_rows(&format!("PRAGMA mvcc_gc_threshold = {gc}")) + .await?; + loaded + } + + /// The rounds of a parallel MVCC load: nodes, then edges, each writer + /// taking one slice per round, with a checkpoint between rounds. + async fn put_mvcc_parallel_phases( + &self, + graph: &Graph, + rows: usize, + handles: &[Arc], + ) -> Result<()> { + let writers = handles.len(); + let slices = |len: usize| -> Vec> { + let per = len.div_ceil(writers).max(1); + (0..len) + .step_by(per) + .map(|s| s..(s + per).min(len)) + .collect() + }; + let groups = match self.mvcc_load_round_groups.load(Ordering::Acquire) { + 0 => MVCC_PARALLEL_ROUND_GROUPS, + n => n, + }; + let round = groups * MVCC_LOAD_COMMIT_STATEMENTS * rows; + for phase in [false, true] { + let len = if phase { + graph.edges.len() + } else { + graph.nodes.len() + }; + let ranges = slices(len); + let mut offset = 0; + loop { + let mut tasks = Vec::new(); + for (range, handle) in ranges.iter().zip(handles.iter().cloned()) { + let start = range.start + offset; + if start >= range.end { + continue; + } + let part_range = start..(start + round).min(range.end); + let part = if phase { + Graph::new(Vec::new(), graph.edges[part_range].to_vec()) + } else { + Graph::new(graph.nodes[part_range].to_vec(), Vec::new()) + }; + tasks.push(tokio::spawn(async move { + handle.put_mvcc_groups(&part, rows).await + })); + } + if tasks.is_empty() { + break; + } + let mut first_err = None; + for task in tasks { + let outcome = task.await.map_err(|err| { + GrustError::Backend(format!("Turso parallel load task failed: {err}")) + }); + if let Err(err) = outcome.and_then(|r| r) { + first_err.get_or_insert(err); + } + } + if let Some(err) = first_err { + return Err(err); + } + self.execute_discarding_rows("PRAGMA wal_checkpoint(TRUNCATE)") + .await?; + offset += round; + } + } + Ok(()) + } + + /// The MVCC load body: nodes, then edges, each committed in groups of + /// `MVCC_LOAD_COMMIT_STATEMENTS` batches. + async fn put_mvcc_groups(&self, graph: &Graph, rows: usize) -> Result<()> { + let group = MVCC_LOAD_COMMIT_STATEMENTS * rows; + for chunk in graph.nodes.chunks(group) { + self.load_transaction(chunk, &[], true, rows).await?; + } + for chunk in graph.edges.chunks(group) { + self.load_transaction(&[], chunk, true, rows).await?; + } + Ok(()) + } + /// Upsert `nodes` then `edges` in one transaction on the shared /// connection, with the same gate, rollback and MVCC retry handling as /// [`Self::execute_transaction`]. @@ -541,6 +1012,14 @@ impl TursoGraphStore { ))); } if concurrent && is_mvcc_conflict(&err) && attempt < MAX_ATTEMPTS { + // Back off before retrying: a Busy here means another + // writer holds the commit or checkpoint lock, and an + // immediate BEGIN CONCURRENT would spin against it. + // The workspace tokio has no timer, so yield to the + // runtime once per attempt so far instead of sleeping. + for _ in 0..attempt { + tokio::task::yield_now().await; + } continue; } return Err(err); @@ -561,7 +1040,6 @@ impl TursoGraphStore { self.invalidate_snapshot(); let nodes_table = self.nodes_table(); for chunk in nodes.chunks(rows) { - let sql = prepared_upsert_nodes_sql(&nodes_table, chunk.len()); let mut params = Vec::with_capacity(chunk.len() * 3); for node in chunk { params.push(turso::Value::Text(node.id.as_str().to_string())); @@ -570,11 +1048,11 @@ impl TursoGraphStore { &node.props, )?)); } + let sql = prepared_upsert_nodes_sql(&nodes_table, chunk.len()); self.execute_prepared_unlocked(&sql, params).await?; } let edges_table = self.edges_table(); for chunk in edges.chunks(rows) { - let sql = prepared_upsert_edges_sql(&edges_table, chunk.len()); let mut params = Vec::with_capacity(chunk.len() * 6); for edge in chunk { params.push(match &edge.id { @@ -589,6 +1067,7 @@ impl TursoGraphStore { )?)); params.push(turso::Value::Text(edge_identity_key(edge))); } + let sql = prepared_upsert_edges_sql(&edges_table, chunk.len()); self.execute_prepared_unlocked(&sql, params).await?; } Ok(()) @@ -702,7 +1181,16 @@ impl GraphAdminStore for TursoGraphStore { )) .await?; } - self.execute_unlocked(&sql).await + self.execute_unlocked(&sql).await?; + // Stores bootstrapped before the edge-source index was retired still + // carry it: every plan already reads source lookups from the edge + // primary key, so dropping it changes no answer and ends its cost. + if self.edges_from_index_exists_unlocked().await? { + let idx = quote_ident(&format!("{}_edges_from_idx", self.config.table_prefix)); + self.execute_unlocked(&format!("DROP INDEX IF EXISTS {idx}")) + .await?; + } + Ok(()) } async fn clear(&self) -> Result<()> { @@ -839,6 +1327,14 @@ impl CypherMutationExecutor for TursoGraphStore { ))); } if concurrent && is_mvcc_conflict(&err) && attempt < MAX_ATTEMPTS { + // Back off before retrying: a Busy here means another + // writer holds the commit or checkpoint lock, and an + // immediate BEGIN CONCURRENT would spin against it. + // The workspace tokio has no timer, so yield to the + // runtime once per attempt so far instead of sleeping. + for _ in 0..attempt { + tokio::task::yield_now().await; + } continue; } return Err(err); @@ -919,6 +1415,15 @@ pub const PAGE_CACHE_KIB: u64 = 1024 * 1024; /// rows about ten thousand rows at a time. pub const MVCC_LOAD_COMMIT_STATEMENTS: usize = 20; +/// Commit groups each parallel MVCC load writer runs between the checkpoints +/// that bound the load's resident row versions (see `put_mvcc_parallel`). +pub const MVCC_PARALLEL_ROUND_GROUPS: usize = 5; + +/// Page cache for one parallel load writer's connection. A writer streams +/// rows in and re-reads almost nothing, so it does not need the store's +/// `PAGE_CACHE_KIB`; eight writers at that size budget 8 GiB of cache alone. +pub const MVCC_LOAD_WRITER_CACHE_KIB: u64 = 128 * 1024; + pub fn upsert_nodes_sql(table: &str, nodes: &[Node]) -> Result { TursoDialect.upsert_nodes_sql(table, nodes) } @@ -1447,6 +1952,14 @@ impl GraphSqlDialect for TursoDialect { )) } + /// The edge primary key `(from_id, label, to_id, identity_key)` leads with + /// `from_id`, and Turso's planner reads every source lookup and traversal + /// hop from it even when `_edges_from_idx` exists; that index + /// only cost a write per edge (a quarter of an MVCC load, a sixth of WAL). + fn edge_source_index(&self) -> bool { + false + } + fn edge_identity_column(&self) -> Option<&'static str> { Some("identity_key") } diff --git a/crates/grust-turso/src/tests.rs b/crates/grust-turso/src/tests.rs index 3b5a36d6..f4ea44b0 100644 --- a/crates/grust-turso/src/tests.rs +++ b/crates/grust-turso/src/tests.rs @@ -28,6 +28,11 @@ fn bootstrap_creates_universal_turso_tables() { assert!(sql.contains("props TEXT NOT NULL")); assert!(sql.contains("identity_key text NOT NULL DEFAULT ''")); assert!(sql.contains("PRIMARY KEY (from_id, label, to_id, identity_key)")); + assert!( + !sql.contains("edges_from_idx"), + "Turso reads source lookups from the edge primary key" + ); + assert!(sql.contains("CREATE INDEX IF NOT EXISTS \"grust_edges_to_idx\"")); } #[test] @@ -717,3 +722,226 @@ async fn prepared_bulk_load_writes_exactly_the_rows_of_the_sql_text_upserts() { } } } + +/// A load large enough to span many commit groups, with repeated keys and +/// every kind of edge identity, over a `rewriting_graph` prefix. +fn bulk_rewriting_graph() -> Graph { + let base = rewriting_graph(); + let mut nodes = base.nodes; + let mut edges = base.edges; + for i in 0..300 { + nodes.push(Node::new("N", format!("b{i}"), weight(i))); + } + for i in 0..300i64 { + for k in 1..=6i64 { + let to = format!("b{}", (i * 7 + k * 13) % 300); + let edge = Edge::new("E", format!("b{i}"), to, weight(i * 10 + k)); + edges.push(match k % 4 { + 0 => edge, + 1 => edge.with_id(format!("e{i}-{k}")), + 2 => edge.with_id(format!("id:e{i}")), + _ => edge.with_id(""), + }); + } + } + // Re-put a slice of the nodes and edges with new props: the last row wins. + for i in (0..300).step_by(17) { + nodes.push(Node::new("M", format!("b{i}"), weight(-i))); + edges.push( + Edge::new( + "E", + format!("b{i}"), + format!("b{}", (i * 7 + 13) % 300), + weight(-i), + ) + .with_id(format!("e{i}-1")), + ); + } + Graph::new(nodes, edges) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn mvcc_bulk_load_stores_the_rows_of_the_wal_load_and_keeps_concurrent_writers() { + let dir = tempfile::tempdir().unwrap(); + let graph = bulk_rewriting_graph(); + let config = |name: &str, journal_mode| TursoConfig { + path: dir.path().join(name).to_string_lossy().into_owned(), + journal_mode, + batch_size: 7, + ..TursoConfig::default() + }; + + let wal = TursoGraphStore::connect(config("wal.db", TursoJournalMode::Wal)) + .await + .unwrap(); + wal.bootstrap().await.unwrap(); + wal.put_graph(&graph).await.unwrap(); + let expected = raw_rows(&wal).await; + + let mvcc_config = config("mvcc.db", TursoJournalMode::Mvcc); + let mvcc = TursoGraphStore::connect(mvcc_config.clone()).await.unwrap(); + mvcc.bootstrap().await.unwrap(); + // A tiny checkpoint threshold makes the automatic checkpoints run in the + // middle of the load, between commit groups, as they do in a large one. + mvcc.execute_discarding_rows("PRAGMA mvcc_checkpoint_threshold = 8192") + .await + .unwrap(); + // Load in two calls, as the harness does: nodes, then edges. + mvcc.put_graph(&Graph::new(graph.nodes.clone(), Vec::new())) + .await + .unwrap(); + mvcc.put_graph(&Graph::new(Vec::new(), graph.edges.clone())) + .await + .unwrap(); + let mode = mvcc.query_scalar_text("PRAGMA journal_mode").await.unwrap(); + assert_eq!(mode.as_deref(), Some("mvcc"), "the load stays an MVCC load"); + assert_eq!( + mvcc.query_scalar_i64("PRAGMA synchronous").await.unwrap(), + 2, + "a load runs at synchronous = NORMAL and restores FULL afterwards" + ); + assert_eq!(raw_rows(&mvcc).await, expected, "MVCC rows equal WAL rows"); + assert!( + expected.iter().any(|row| row.len() == 6 + && row[5] + .as_deref() + .is_some_and(|key| key.starts_with("id:id:"))), + "the identity keys of prefix-like ids are part of the comparison" + ); + drop(mvcc); + + // The load ended in a TRUNCATE checkpoint: a fresh connection reads every + // row, and the store still takes concurrent BEGIN CONCURRENT writers. + let reopened = TursoGraphStore::connect(mvcc_config.clone()).await.unwrap(); + reopened.bootstrap().await.unwrap(); + assert_eq!(raw_rows(&reopened).await, expected, "rows survive a reopen"); + let second = TursoGraphStore::connect(mvcc_config.clone()).await.unwrap(); + let writers = [reopened, second].map(|store| { + tokio::spawn(async move { + for i in 0..20 { + store + .put_edge(&Edge::new("W", "b0", format!("b{i}"), Props::new())) + .await + .expect("concurrent put"); + } + }) + }); + for writer in writers { + writer.await.unwrap(); + } + let verify = TursoGraphStore::connect(mvcc_config).await.unwrap(); + let written = verify + .get_edges(EdgeQuery { + from: Some(NodeId::new("b0")), + label: Some(Label::new("W")), + ..Default::default() + }) + .await + .unwrap(); + assert_eq!(written.len(), 20, "both writers' edges land once each"); +} + +async fn index_names(store: &TursoGraphStore) -> Vec> { + store + .run_text_rows( + "SELECT name FROM sqlite_schema WHERE type = 'index' ORDER BY name", + 1, + ) + .await + .unwrap() + .concat() +} + +#[tokio::test] +async fn bootstrap_omits_the_edge_source_index_and_drops_a_legacy_one() { + let store = TursoGraphStore::in_memory().await.unwrap(); + store.bootstrap().await.unwrap(); + let names = index_names(&store).await; + assert!(!names.contains(&Some("grust_edges_from_idx".to_string()))); + assert!(names.contains(&Some("grust_edges_to_idx".to_string()))); + + store + .execute("CREATE INDEX grust_edges_from_idx ON grust_edges(from_id)") + .await + .unwrap(); + store.bootstrap().await.unwrap(); + let names = index_names(&store).await; + assert!(!names.contains(&Some("grust_edges_from_idx".to_string()))); +} + +/// Feasibility probe: can a live MVCC store, with a group committer and a +/// shared handle open, switch to WAL for a bulk load and back to MVCC? +/// Records what the engine does; asserts correctness only when it switches. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_live_mvcc_store_switched_to_wal_and_back() { + let dir = tempfile::tempdir().unwrap(); + let store = TursoGraphStore::connect(TursoConfig { + path: dir.path().join("switch.db").display().to_string(), + table_prefix: "s".to_string(), + batch_size: 500, + journal_mode: TursoJournalMode::Mvcc, + }) + .await + .unwrap(); + store.bootstrap().await.unwrap(); + let store = store.with_group_commit().await.unwrap(); + let other = Arc::new(store.connect_shared().await.unwrap()); + + let to_wal = store.query_scalar_text("PRAGMA journal_mode = wal").await; + eprintln!("MVCC -> WAL on a live store: {to_wal:?}"); + if !matches!(to_wal.as_ref().map(|m| m.as_deref()), Ok(Some("wal"))) { + return; + } + let nodes: Vec = (0..50) + .map(|i| Node::new("Node", format!("w{i}"), Props::new())) + .collect(); + store + .execute(&upsert_nodes_sql(&store.nodes_table(), &nodes).unwrap()) + .await + .unwrap(); + let back = store.query_scalar_text("PRAGMA journal_mode = mvcc").await; + eprintln!("WAL -> MVCC on a live store: {back:?}"); + assert!( + matches!(back.as_ref().map(|m| m.as_deref()), Ok(Some("mvcc"))), + "a store that switched to WAL must switch back" + ); + + for i in 0..50 { + assert!( + other + .get_node(&NodeId::new(format!("w{i}"))) + .await + .unwrap() + .is_some(), + "row w{i} written under WAL reads back on another handle" + ); + } + let writer = { + let other = other.clone(); + tokio::spawn(async move { + for i in 0..20 { + other + .put_node(&Node::new("Node", format!("c{i}"), Props::new())) + .await + .unwrap(); + } + }) + }; + for i in 0..20 { + store + .put_node(&Node::new("Node", format!("d{i}"), Props::new())) + .await + .unwrap(); + } + writer.await.unwrap(); + for id in (0..20).flat_map(|i| [format!("c{i}"), format!("d{i}")]) { + assert!( + store + .get_node(&NodeId::new(id.clone())) + .await + .unwrap() + .is_some(), + "concurrent MVCC write {id} landed after switching back" + ); + } +} diff --git a/crates/grust-turso/tests/dangling_edges.rs b/crates/grust-turso/tests/dangling_edges.rs new file mode 100644 index 00000000..95a03b16 --- /dev/null +++ b/crates/grust-turso/tests/dangling_edges.rs @@ -0,0 +1,92 @@ +//! `put_graph` accepts an edge whose endpoint node is not stored, the way the +//! Memory reference does, on every load path alike: WAL, a single MVCC +//! writer, and parallel MVCC writers. Foreign keys are switched off for the +//! load only; a single-edge write afterwards still meets the schema's +//! `REFERENCES nodes(id)`, so serving stays strict. + +use grust_core::prelude::*; +use grust_turso::{TursoConfig, TursoGraphStore, TursoJournalMode}; + +/// 300 nodes, a star of edges out of `n0`, and one edge to a node that is +/// never stored. Big enough that a store with several writers takes the +/// parallel path (it needs at least `2 * batch_size` rows). +fn graph_with_a_dangling_edge() -> Graph { + let nodes: Vec = (0..300) + .map(|i| Node::new("Node", format!("n{i}"), Props::new())) + .collect(); + let mut edges: Vec = (1..300) + .map(|i| Edge::new("EDGE", "n0", format!("n{i}"), Props::new())) + .collect(); + edges.push(Edge::new("EDGE", "n0", "ghost", Props::new())); + Graph::new(nodes, edges) +} + +async fn load_and_check(mode: TursoJournalMode, writers: usize, dir: &tempfile::TempDir) { + let store = TursoGraphStore::connect(TursoConfig { + path: dir + .path() + .join(format!("dangling-{mode:?}-{writers}.db")) + .display() + .to_string(), + table_prefix: "d".to_string(), + batch_size: 50, + journal_mode: mode, + }) + .await + .unwrap(); + store.bootstrap().await.unwrap(); + store.set_mvcc_load_parallelism(writers); + + let report = store + .put_graph(&graph_with_a_dangling_edge()) + .await + .unwrap_or_else(|e| panic!("{mode:?} x{writers}: load with a dangling edge failed: {e}")); + assert_eq!( + report.edges, 300, + "{mode:?} x{writers}: every edge reported" + ); + + let out = store + .get_edges(EdgeQuery { + from: Some(NodeId::new("n0")), + to: None, + label: None, + }) + .await + .unwrap(); + assert_eq!( + out.len(), + 300, + "{mode:?} x{writers}: the dangling edge is stored too" + ); + assert!( + out.iter().any(|e| e.to.as_str() == "ghost"), + "{mode:?} x{writers}: the edge to the unstored node reads back" + ); + assert!( + store + .get_node(&NodeId::new("ghost")) + .await + .unwrap() + .is_none(), + "{mode:?} x{writers}: the endpoint exists only as an edge end" + ); + + // Foreign keys are back on: a single-edge write to a missing node is + // refused, as before this change. + let strict = store + .put_edge(&Edge::new("EDGE", "n0", "ghost2", Props::new())) + .await; + assert!( + strict.is_err(), + "{mode:?} x{writers}: serving writes still enforce REFERENCES nodes(id)" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_dangling_edge_loads_the_same_on_every_path() { + let dir = tempfile::tempdir().unwrap(); + load_and_check(TursoJournalMode::Wal, 1, &dir).await; + load_and_check(TursoJournalMode::Mvcc, 1, &dir).await; + load_and_check(TursoJournalMode::Mvcc, 4, &dir).await; +} diff --git a/crates/grust-turso/tests/fresh_load_duplicates.rs b/crates/grust-turso/tests/fresh_load_duplicates.rs new file mode 100644 index 00000000..d05cf9b6 --- /dev/null +++ b/crates/grust-turso/tests/fresh_load_duplicates.rs @@ -0,0 +1,86 @@ +//! A load into empty tables takes the plain-INSERT path; duplicate keys +//! inside that load must still end as one row per key, through the upsert +//! replay. Both journal modes, since both take the same batch path. + +use grust_core::prelude::*; +use grust_turso::{TursoConfig, TursoGraphStore, TursoJournalMode}; + +async fn load_with_duplicates(mode: TursoJournalMode, dir: &tempfile::TempDir) { + let store = TursoGraphStore::connect(TursoConfig { + path: dir + .path() + .join(format!("dup-{mode:?}.db")) + .display() + .to_string(), + table_prefix: "d".to_string(), + batch_size: 50, + journal_mode: mode, + }) + .await + .unwrap(); + store.bootstrap().await.unwrap(); + + // 300 distinct ids, each appearing twice, and the second copy carries the + // props the store must end up with. + let mut nodes = Vec::new(); + for round in 0..2 { + for i in 0..300 { + let mut props = Props::new(); + props.insert("round".to_string(), Value::from(round as i64)); + nodes.push(Node::new("Node", format!("n{i}"), props)); + } + } + let mut edges = Vec::new(); + for round in 0..2 { + for i in 1..300 { + let mut props = Props::new(); + props.insert("round".to_string(), Value::from(round as i64)); + edges.push(Edge::new("EDGE", "n0", format!("n{i}"), props)); + } + } + store.put_graph(&Graph::new(nodes, edges)).await.unwrap(); + + for i in [0usize, 7, 299] { + let node = store + .get_node(&NodeId::new(format!("n{i}"))) + .await + .unwrap() + .expect("node stored once"); + assert_eq!( + node.props.get("round"), + Some(&Value::from(1i64)), + "{mode:?}: the last write of n{i} wins" + ); + } + let out = store + .get_edges(EdgeQuery { + from: Some(NodeId::new("n0")), + to: None, + label: None, + }) + .await + .unwrap(); + assert_eq!(out.len(), 299, "{mode:?}: one edge per key, not two"); + assert!( + out.iter() + .all(|e| e.props.get("round") == Some(&Value::from(1i64))), + "{mode:?}: the last write of each edge wins" + ); + + // A second load, now that the tables are not empty, still merges. + store + .put_graph(&Graph::new( + vec![Node::new("Node", "n0", Props::new())], + Vec::new(), + )) + .await + .unwrap(); + assert!(store.get_node(&NodeId::new("n0")).await.unwrap().is_some()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_fresh_load_with_duplicate_keys_stores_one_row_per_key() { + let dir = tempfile::tempdir().unwrap(); + load_with_duplicates(TursoJournalMode::Wal, &dir).await; + load_with_duplicates(TursoJournalMode::Mvcc, &dir).await; +} diff --git a/crates/grust-turso/tests/group_commit.rs b/crates/grust-turso/tests/group_commit.rs new file mode 100644 index 00000000..d797b90c --- /dev/null +++ b/crates/grust-turso/tests/group_commit.rs @@ -0,0 +1,123 @@ +//! MVCC group commit: concurrent single-statement writes from shared handles +//! are all applied exactly once, and a failing statement fails only its own +//! caller. + +use grust_core::prelude::*; +use grust_turso::{TursoConfig, TursoGraphStore, TursoJournalMode}; +use std::sync::Arc; + +async fn mvcc_store(dir: &tempfile::TempDir) -> TursoGraphStore { + let store = TursoGraphStore::connect(TursoConfig { + path: dir.path().join("group.db").display().to_string(), + table_prefix: "g".to_string(), + batch_size: 500, + journal_mode: TursoJournalMode::Mvcc, + }) + .await + .unwrap(); + store.bootstrap().await.unwrap(); + store.with_group_commit().await.unwrap() +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn group_commit_applies_every_concurrent_write_once() { + let dir = tempfile::tempdir().unwrap(); + let base = mvcc_store(&dir).await; + base.put_node(&Node::new("Node", "hub", Props::new())) + .await + .unwrap(); + + let writers = 8; + let per = 50; + let mut tasks = Vec::new(); + for w in 0..writers { + let store = Arc::new(base.connect_shared().await.unwrap()); + tasks.push(tokio::spawn(async move { + for i in 0..per { + let id = format!("n-{w}-{i}"); + store + .put_node(&Node::new("Node", id.clone(), Props::new())) + .await + .unwrap(); + store + .put_edge(&Edge::new("EDGE", "hub", id, Props::new())) + .await + .unwrap(); + } + })); + } + for task in tasks { + task.await.unwrap(); + } + + let edges = base + .get_edges(EdgeQuery { + from: Some(NodeId::new("hub")), + to: None, + label: None, + }) + .await + .unwrap(); + assert_eq!( + edges.len(), + writers * per, + "every acknowledged edge is stored" + ); + let mut targets: Vec<_> = edges.iter().map(|e| e.to.as_str().to_string()).collect(); + targets.sort(); + targets.dedup(); + assert_eq!(targets.len(), writers * per, "no write is applied twice"); + + // The writes survive a fresh connection to the same file. + drop(base); + let reopened = TursoGraphStore::connect(TursoConfig { + path: dir.path().join("group.db").display().to_string(), + table_prefix: "g".to_string(), + batch_size: 500, + journal_mode: TursoJournalMode::Mvcc, + }) + .await + .unwrap(); + let edges = reopened + .get_edges(EdgeQuery { + from: Some(NodeId::new("hub")), + to: None, + label: None, + }) + .await + .unwrap(); + assert_eq!(edges.len(), writers * per, "the writes are durable"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn group_commit_failure_is_reported_to_its_own_caller_only() { + let dir = tempfile::tempdir().unwrap(); + let base = mvcc_store(&dir).await; + let good = Arc::new(base.connect_shared().await.unwrap()); + let good_task = { + let good = good.clone(); + tokio::spawn(async move { + for i in 0..20 { + good.put_node(&Node::new("Node", format!("ok-{i}"), Props::new())) + .await + .unwrap(); + } + }) + }; + // An edge to endpoints that do not exist violates the edge table's + // foreign keys only if they are enforced on the committer's connection; + // either way the good writer's nodes must all land. + let _ = base + .put_edge(&Edge::new("EDGE", "missing-a", "missing-b", Props::new())) + .await; + good_task.await.unwrap(); + for i in 0..20 { + assert!( + base.get_node(&NodeId::new(format!("ok-{i}"))) + .await + .unwrap() + .is_some(), + "node ok-{i} from the other writer landed" + ); + } +} diff --git a/crates/grust-turso/tests/parallel_load.rs b/crates/grust-turso/tests/parallel_load.rs new file mode 100644 index 00000000..98f385e0 --- /dev/null +++ b/crates/grust-turso/tests/parallel_load.rs @@ -0,0 +1,90 @@ +//! `set_mvcc_load_parallelism`: an MVCC `put_graph` split across concurrent +//! writers stores exactly the rows of the graph, once each, and leaves the +//! store serving ordinary writes. + +use grust_core::prelude::*; +use grust_turso::{TursoConfig, TursoGraphStore, TursoJournalMode}; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_parallel_mvcc_load_stores_every_row_once() { + let dir = tempfile::tempdir().unwrap(); + let store = TursoGraphStore::connect(TursoConfig { + path: dir.path().join("parallel.db").display().to_string(), + table_prefix: "p".to_string(), + batch_size: 100, + journal_mode: TursoJournalMode::Mvcc, + }) + .await + .unwrap(); + store.bootstrap().await.unwrap(); + store.set_mvcc_load_parallelism(4); + + let nodes: Vec = (0..3000) + .map(|i| Node::new("Node", format!("n{i}"), Props::new())) + .collect(); + let edges: Vec = (0..5000) + .map(|i| { + Edge::new( + "EDGE", + format!("n{}", i % 3000), + format!("n{}", (i * 7 + 1) % 3000), + Props::new(), + ) + }) + .collect(); + let mut expected: Vec<(String, String)> = edges + .iter() + .map(|e| (e.from.as_str().to_string(), e.to.as_str().to_string())) + .collect(); + expected.sort(); + expected.dedup(); + + // Vertices, then edges, as the harness loads. + store + .put_graph(&Graph::new(nodes, Vec::new())) + .await + .unwrap(); + store + .put_graph(&Graph::new(Vec::new(), edges)) + .await + .unwrap(); + + for i in (0..3000).step_by(97) { + assert!( + store + .get_node(&NodeId::new(format!("n{i}"))) + .await + .unwrap() + .is_some(), + "node n{i} landed" + ); + } + let mut stored = Vec::new(); + for i in 0..3000 { + for e in store + .get_edges(EdgeQuery { + from: Some(NodeId::new(format!("n{i}"))), + to: None, + label: None, + }) + .await + .unwrap() + { + stored.push((e.from.as_str().to_string(), e.to.as_str().to_string())); + } + } + stored.sort(); + assert_eq!(stored, expected, "every edge stored exactly once"); + + store + .put_node(&Node::new("Node", "after", Props::new())) + .await + .unwrap(); + assert!( + store + .get_node(&NodeId::new("after")) + .await + .unwrap() + .is_some() + ); +} diff --git a/crates/grust-turso/tests/wal_bulk_load.rs b/crates/grust-turso/tests/wal_bulk_load.rs new file mode 100644 index 00000000..c8ae8a37 --- /dev/null +++ b/crates/grust-turso/tests/wal_bulk_load.rs @@ -0,0 +1,69 @@ +//! `set_bulk_load_via_wal`: an MVCC store's `put_graph` loads through WAL and +//! the store is back in MVCC afterwards (concurrent BEGIN CONCURRENT writers, +//! which WAL would refuse, all land). + +use grust_core::prelude::*; +use grust_turso::{TursoConfig, TursoGraphStore, TursoJournalMode}; +use std::sync::Arc; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn an_mvcc_bulk_load_through_wal_keeps_every_row_and_mvcc_writers() { + let dir = tempfile::tempdir().unwrap(); + let store = TursoGraphStore::connect(TursoConfig { + path: dir.path().join("walload.db").display().to_string(), + table_prefix: "b".to_string(), + batch_size: 500, + journal_mode: TursoJournalMode::Mvcc, + }) + .await + .unwrap(); + store.bootstrap().await.unwrap(); + let store = store.with_group_commit().await.unwrap(); + store.set_bulk_load_via_wal(true); + + let nodes: Vec = (0..1000) + .map(|i| Node::new("Node", format!("n{i}"), Props::new())) + .collect(); + let edges: Vec = (1..1000) + .map(|i| Edge::new("EDGE", "n0", format!("n{i}"), Props::new())) + .collect(); + // Two calls, as the harness loads: vertices, then edges. + store + .put_graph(&Graph::new(nodes, Vec::new())) + .await + .unwrap(); + store + .put_graph(&Graph::new(Vec::new(), edges)) + .await + .unwrap(); + + let hub = EdgeQuery { + from: Some(NodeId::new("n0")), + to: None, + label: None, + }; + assert_eq!(store.get_edges(hub.clone()).await.unwrap().len(), 999); + + let mut tasks = Vec::new(); + for w in 0..4 { + let handle = Arc::new(store.connect_shared().await.unwrap()); + tasks.push(tokio::spawn(async move { + for i in 0..25 { + handle + .put_edge(&Edge::new("EDGE", "n0", format!("n{}", 1 + w * 25 + i), { + let mut p = Props::new(); + p.insert("w".to_string(), Value::from(w as i64)); + p + })) + .await + .unwrap(); + } + })); + } + for task in tasks { + task.await.unwrap(); + } + // The upserts touched existing edges (same keys), so the count is + // unchanged and every one of them succeeded under BEGIN CONCURRENT. + assert_eq!(store.get_edges(hub).await.unwrap().len(), 999); +} diff --git a/crates/grust-turso/tests/wal_to_mvcc.rs b/crates/grust-turso/tests/wal_to_mvcc.rs new file mode 100644 index 00000000..37559c75 --- /dev/null +++ b/crates/grust-turso/tests/wal_to_mvcc.rs @@ -0,0 +1,94 @@ +//! Feasibility probe: can a database loaded in WAL mode be reopened as MVCC? +//! If it can, a bulk load could run at WAL speed and the store then serve +//! MVCC's concurrent writers. The test records what the engine does either +//! way; it only asserts correctness when the reopen succeeds. + +use grust_core::prelude::*; +use grust_turso::{TursoConfig, TursoGraphStore, TursoJournalMode}; +use std::sync::Arc; + +fn config(path: &std::path::Path, mode: TursoJournalMode) -> TursoConfig { + TursoConfig { + path: path.display().to_string(), + table_prefix: "w".to_string(), + batch_size: 500, + journal_mode: mode, + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_wal_loaded_database_reopened_as_mvcc() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("convert.db"); + + let wal = TursoGraphStore::connect(config(&path, TursoJournalMode::Wal)) + .await + .unwrap(); + wal.bootstrap().await.unwrap(); + let nodes: Vec = (0..200) + .map(|i| Node::new("Node", format!("n{i}"), Props::new())) + .collect(); + let edges: Vec = (1..200) + .map(|i| Edge::new("EDGE", "n0", format!("n{i}"), Props::new())) + .collect(); + wal.put_graph(&Graph::new(nodes, edges)).await.unwrap(); + drop(wal); + + let reopened = TursoGraphStore::connect(config(&path, TursoJournalMode::Mvcc)).await; + let mvcc = match reopened { + Err(err) => { + eprintln!("WAL -> MVCC reopen refused: {err}"); + return; + } + Ok(store) => store, + }; + eprintln!("WAL -> MVCC reopen accepted"); + let before = mvcc + .get_edges(EdgeQuery { + from: Some(NodeId::new("n0")), + to: None, + label: None, + }) + .await + .unwrap(); + assert_eq!( + before.len(), + 199, + "every WAL-loaded edge reads back under MVCC" + ); + + let mvcc = Arc::new(mvcc.with_group_commit().await.unwrap()); + let mut tasks = Vec::new(); + for w in 0..4 { + let store = Arc::new(mvcc.connect_shared().await.unwrap()); + tasks.push(tokio::spawn(async move { + for i in 0..25 { + let id = format!("m{w}-{i}"); + store + .put_node(&Node::new("Node", id.clone(), Props::new())) + .await + .unwrap(); + store + .put_edge(&Edge::new("EDGE", "n0", id, Props::new())) + .await + .unwrap(); + } + })); + } + for task in tasks { + task.await.unwrap(); + } + let after = mvcc + .get_edges(EdgeQuery { + from: Some(NodeId::new("n0")), + to: None, + label: None, + }) + .await + .unwrap(); + assert_eq!( + after.len(), + 299, + "concurrent MVCC writes land on the converted store" + ); +} diff --git a/crates/grust/Cargo.toml b/crates/grust/Cargo.toml index 5d70b1f3..91e0dd61 100644 --- a/crates/grust/Cargo.toml +++ b/crates/grust/Cargo.toml @@ -34,6 +34,10 @@ sail = ["dep:grust-sail", "cypher"] surreal = ["dep:grust-surreal"] turso = ["dep:grust-turso"] turso-sync = ["turso", "grust-turso/sync"] +# Turso's own allocator (mimalloc) as the process's global allocator. Off by +# default: a global allocator is the application's decision. Measured +14% +# (Turso 0.8) and +16% (0.7.2) on 4-writer MVCC bulk loads, +9-16% on WAL. +turso-mimalloc = ["turso", "grust-turso/mimalloc"] typed-garde = ["grust-core/typed-garde"] typed-zod-rs = ["typed-garde", "grust-core/typed-zod-rs"] diff --git a/crates/grust/src/lib.rs b/crates/grust/src/lib.rs index 295c6eb9..27928663 100644 --- a/crates/grust/src/lib.rs +++ b/crates/grust/src/lib.rs @@ -43,10 +43,10 @@ pub use grust_cypher::{ ensure_query_uses_graph, execute_cypher_mutation_returning_with_options_on_store, query_graph_selection, run_bounded_read_query, run_bounded_read_query_indexed, run_bounded_read_query_on_snapshot, run_bounded_read_query_with_registry, - run_read_query_on_named_graph, run_read_query_with_registry, sail_cypher_constraints, - sail_cypher_ddl, sail_cypher_mutation_plan, sail_cypher_mutation_plan_with_options, - sail_cypher_mutation_plan_with_return_options, validate_read_query, - validate_read_query_with_registry, + run_prepared_read_query_indexed, run_read_query_on_named_graph, run_read_query_with_registry, + sail_cypher_constraints, sail_cypher_ddl, sail_cypher_mutation_plan, + sail_cypher_mutation_plan_with_options, sail_cypher_mutation_plan_with_return_options, + validate_read_query, validate_read_query_with_registry, }; #[cfg(feature = "falkor")] diff --git a/docs/blog/grust-brine/dist/VERSION.md b/docs/blog/grust-brine/dist/VERSION.md index 05bba2d4..9c2aa322 100644 --- a/docs/blog/grust-brine/dist/VERSION.md +++ b/docs/blog/grust-brine/dist/VERSION.md @@ -1,7 +1,7 @@ blog_name: grust-brine blog_domain: querygraph.ai slug: grust-brine -version_stamp: 0.20.0-90dee0 -built_at: 2026-09-14 +version_stamp: 0.20.0-1369f3 +built_at: 2026-09-17 textpack_file: grust-brine.textpack -textpack_link: grust-brine (0.20.0-90dee0).textpack +textpack_link: grust-brine (0.20.0-1369f3).textpack diff --git a/docs/blog/grust-brine/dist/grust-brine (0.20.0-90dee0).textpack b/docs/blog/grust-brine/dist/grust-brine (0.20.0-1369f3).textpack similarity index 100% rename from docs/blog/grust-brine/dist/grust-brine (0.20.0-90dee0).textpack rename to docs/blog/grust-brine/dist/grust-brine (0.20.0-1369f3).textpack diff --git a/docs/blog/grust-brine/dist/grust-brine.textpack b/docs/blog/grust-brine/dist/grust-brine.textpack index f1fb835c..3df7a046 100644 Binary files a/docs/blog/grust-brine/dist/grust-brine.textpack and b/docs/blog/grust-brine/dist/grust-brine.textpack differ diff --git a/docs/blog/grust-brine/post.md b/docs/blog/grust-brine/post.md index f1f0a409..ac617dcc 100644 --- a/docs/blog/grust-brine/post.md +++ b/docs/blog/grust-brine/post.md @@ -16,10 +16,22 @@ The DataFusion bridge adds controlled snapshot capture that reserves relationshi Combined input-policy capture checks exact serialized input size and preserves the prepared request's original deadline. Caller-owned input buffers, schema metadata and backend authority remain separate concerns. These controls compose with existing cancellation and portable-result admission; they do not establish complete operator-level accounting. +## Turso under strain + +Alongside the release, the Turso adapter was rebuilt for concurrent load and measured with the adversarial-graph strain harness on SNAP graphs up to 117 million edges. This work lives on the `turso-mvcc-concurrency` branch (pull request #6) and is not part of the 0.20.0 crates; the numbers below are that branch against Turso 0.7.2, the pinned engine. + +MVCC loads previously ran one transaction over the whole graph, at about 1,500 edges per second. They now run over several writer connections under `BEGIN CONCURRENT`, advancing in rounds so a checkpoint can run between rounds: four writers load GAP-road at 27,700 edges per second where one writer managed 2,900, with peak memory down from 15 GB to under 7 GB at web-Google scale. A store that must be filled fastest fills through WAL and switches to MVCC afterwards, at WAL speed, 1.5 to 3 times the parallel MVCC load depending on the engine. + +Loads run with foreign keys off on every path, so WAL, single-writer and parallel loads store the same graph and accept dangling edges as the Memory reference does; incremental writes keep their strictness. That alone lifted WAL loads by about 60%. Concurrent single-statement writes can share one durable commit: sixteen writers attaching 200 edges each to one node take 3.2 seconds instead of 21, all 3,200 accepted, with `synchronous = FULL` throughout. + +On one non-burstable host, the MVCC store loads com-Orkut at 16,000 edges per second against Neo4j's 16,800 and WAL's 24,700, accepts every concurrent hot-node write where WAL accepts 61 of 3,200, and traverses faster than Neo4j on every graph; Neo4j is three to five times leaner in memory. Upstream Turso `main` (the coming 0.8) loads WAL 22% faster and serves reads 4 to 24% faster, but its MVCC bulk loads are 15 to 26% slower, in the index-key comparison, which upstream pull request #8385 addresses; on 0.8 the engine's own group commit replaces the client-side one. + +The book chapter "Turso under strain" carries the measurements, the methodology (alternating pairs with CPU steal recorded, because burstable hosts throttle silently), and the recommended configuration per engine version. + ## Qualification and the automatic execution goal Regression coverage includes sliced and nullable arrays, nested children, empty buffers, C Data export lifetime, raw algorithm results, exact admission boundaries and capture failure cleanup. Release qualification records workspace tests, package verification and registry evidence separately from focused tests. Automatic Cypher-to-DataFusion routing remains active work. It requires preserved language and resource-policy contracts, qualified provider and operator behavior, and end-to-end cost evidence including preparation. Existing benchmark pins remain unchanged; correctness qualification supplies no performance comparison by itself. -See the [repository documentation](https://github.com/querygraph/grust), [Arrow architecture](https://github.com/querygraph/grust/blob/main/docs/arrow-pipelines.md), [automatic execution goal](https://github.com/querygraph/grust/blob/main/docs/goals/cypher-datafusion-execution.md), and the [Grust book](https://firstpair.org/book/grust). +See the [repository documentation](https://github.com/querygraph/grust), [Turso concurrency pull request](https://github.com/querygraph/grust/pull/6), [strain results](https://adversari.al/graph/strain/), [Arrow architecture](https://github.com/querygraph/grust/blob/main/docs/arrow-pipelines.md), [automatic execution goal](https://github.com/querygraph/grust/blob/main/docs/goals/cypher-datafusion-execution.md), and the [Grust book](https://firstpair.org/book/grust). diff --git a/docs/book/build/dist/VERSION.md b/docs/book/build/dist/VERSION.md index b56b63e3..c35f255b 100644 --- a/docs/book/build/dist/VERSION.md +++ b/docs/book/build/dist/VERSION.md @@ -4,9 +4,9 @@ author: Alexy Khrabrov title_stem: grust edition: full version: 0.20.0 -version_stamp: 0.20.0-fa1cbbaa -source_commit: fa1cbbaa -built_at: 2026-09-14T21:54:44Z +version_stamp: 0.20.0-3674fc58 +source_commit: 3674fc58 +built_at: 2026-09-17T18:54:09Z toolchain_lock: ../firstpair/publishing/toolchain.lock.json primary_format: typst kindle_name: grust (0.20.0) @@ -16,10 +16,10 @@ epub_file: grust.epub html_file: grust.html html_chapters_dir: grust-chapters html_title: Grust -pdf_link: grust (0.20.0-fa1cbbaa).pdf -epub_link: grust (0.20.0-fa1cbbaa).epub -html_link: grust (0.20.0-fa1cbbaa).html -html_chapters_link: grust (0.20.0-fa1cbbaa)-chapters +pdf_link: grust (0.20.0-3674fc58).pdf +epub_link: grust (0.20.0-3674fc58).epub +html_link: grust (0.20.0-3674fc58).html +html_chapters_link: grust (0.20.0-3674fc58)-chapters mobi_file: grust.mobi pdf_file_typst: grust.pdf -pdf_link_typst: grust (0.20.0-fa1cbbaa).pdf +pdf_link_typst: grust (0.20.0-3674fc58).pdf diff --git a/docs/book/build/dist/grust-chapters/chapter-001.html b/docs/book/build/dist/grust-chapters/chapter-001.html index fb1e48b8..e010b7f8 100644 --- a/docs/book/build/dist/grust-chapters/chapter-001.html +++ b/docs/book/build/dist/grust-chapters/chapter-001.html @@ -5,7 +5,7 @@ - + Grust @@ -112,258 +47,444 @@
-Next: +Next: Previous: Compact snapshots and multigraph persistence
-

15 Graph analytics through one Rust -implementation

-

Acorn adds graph analytics to Grust’s backend-neutral property-graph -API. Applications prepare an explicit immutable snapshot, run a Rust -kernel, and choose Rust, Arrow or ordinary Cypher consumption. -Algorithms do not own a database connection or choose another graph. -Memory and private Turso snapshots demonstrate the same prepared query -against two independently captured sources.

-
-Diagram 8 - -
-

grust-algorithms has no Cypher dependency. -grust-procedures owns signatures, providers, checked -cursors and the shared execution budget. The -grust-algorithm-procedures crate connects them. An -application can register an independent provider without changing the -parser, analyzer or executor. The facade exposes these through -algorithms, cypher and optional -arrow features.

-

15.1 Starting from the facade

-
[dependencies]
-grust = { package = "grust-graph", version = "0.20.0", features = ["algorithms", "cypher", "arrow"] }
-

Register the desired providers once and retain the immutable -registry:

-
use grust::{algorithm_procedures, procedures, CypherParameters};
-
-let mut builder = procedures::RegistryBuilder::default();
-procedures::register_builtins(&mut builder)?;
-algorithm_procedures::register_algorithms(&mut builder)?;
-let registry = builder.build();
-let result = grust::run_read_query_with_registry(
-    &graph,
-    "roads",
-    "CALL grust.algorithms.dijkstra('start', {weightProperty: 'cost'}) \
-     YIELD nodeId, distance RETURN nodeId, distance",
-    &CypherParameters::new(),
-    &registry,
-)?;
-

Here graph is a caller-owned grust::Graph. -A bounded query additionally opts into -ReadQueryPolicy::allow_read_procedures; catalog permission -alone cannot admit analytics. For a backend capture, construct a -LocalSnapshot after access checks, with an explicit graph, -revision and principal. A prepared plan rejects a different graph name -before invoking a provider. The capability pins data; the identity -strings do not grant access.

-

For direct Rust, create an ExecutionContext with -memory/work limits, batch size and optional deadline. -GraphProjection::from_graph accepts label selection, -orientation and a weight policy. from_topology accepts -external IDs and typed edges; from_arrow_batches reads -typed columns directly. Kernel results share the projection, so dropping -the caller’s projection handle cannot invalidate IDs. Runnable source -examples are weighted_paths in -grust-algorithms and custom_procedure in -grust-cypher.

-

15.2 Available operations

-

All procedure names below have the prefix -grust.algorithms.. They call the same kernels as direct -Rust and have typed Arrow result adapters.

+

15 Turso under strain

+

This chapter documents the Turso adapter as it stands on the +turso-mvcc-concurrency branch (pull request #6, head +abea518), which is the adapter the next release ships. +Everything here was measured; where a number depends on the engine +version, the chapter says which version. The pinned engine is Turso +0.7.2 from crates.io. Upstream main (0.8.0-pre.11 at the +time of measurement) is discussed separately, because it changes two of +the conclusions.

+

15.1 Two journal modes, two +contracts

+

TursoConfig::journal_mode selects the concurrency model, +and the choice decides what the store does when several writers touch +the same node.

+

TursoJournalMode::Wal is Turso’s write-ahead log: one +writer at a time. It is the faster loader by a wide margin, and under +contention it does the honest single-writer thing: it accepts one +writer’s transaction and refuses the rest with a typed busy error. A +refusal is not a lost write, so a WAL store passes the harness’s +hot-node scenario, but it does so by accepting between 20 and 120 of +3,200 attempted writes.

+

TursoJournalMode::Mvcc enables multi-version concurrency +control (PRAGMA journal_mode = mvcc). Writes run inside +BEGIN CONCURRENT transactions with bounded conflict retry, +so concurrent writers make progress. On every graph measured, an MVCC +store accepted all 3,200 hot-node writes. It loads more slowly, and it +holds more memory while loading, because every row version of an open +transaction stays resident until it commits.

+

Turso 0.7.2 converts between the modes on a live database. A database +loaded in WAL mode and reopened as MVCC is accepted; a live MVCC store +switches to WAL and back with PRAGMA journal_mode. +tests/wal_to_mvcc.rs and +tests/wal_bulk_load.rs pin both directions, and both pass +against upstream main as well. The earlier documentation +that said an existing WAL database is not converted was wrong for 0.7.2 +and has been removed.

+

Choose by workload. A store that is loaded once and then read is a +WAL store. A store that takes concurrent writes after loading is an MVCC +store, and the rest of this chapter is about making its load cost +bearable.

+

15.2 Loading an MVCC store

+

put_graph on an MVCC store no longer runs one +transaction around the whole graph. That design kept every row version +of the load in memory until the end and retried all of it on a conflict; +the strain harness measured about 1,500 edges per second that way. Loads +now commit in groups of MVCC_LOAD_COMMIT_STATEMENTS +batches, so a failure leaves the groups committed before it, and an MVCC +put_graph is no longer all-or-nothing. WAL keeps its +whole-load transaction.

+

15.2.1 Parallel writers

+

set_mvcc_load_parallelism(writers) runs the load over +writers connections, each opened through +connect_shared(), each taking a disjoint slice of the nodes +and then of the edges under its own BEGIN CONCURRENT +transactions. Turso executes one connection on one thread, so this is +how a load uses more than one core; it needs a multi-threaded Tokio +runtime. The harness default is four writers.

+

Free-running writers defeat Turso’s automatic checkpoints: a +checkpoint needs the checkpoint lock, every open transaction holds it in +read mode, and with several writers there is always one open, so the +checkpoint returns busy and row versions accumulate. A web-Google load +reached 14 to 15 GB that way. The writers therefore advance in rounds of +MVCC_PARALLEL_ROUND_GROUPS commit groups; between rounds, +with nothing in flight, one TRUNCATE checkpoint folds the +round into the database file. set_mvcc_load_round_groups +adjusts the round size; smaller rounds trade time for memory, and the +sweep found no speed to be gained from larger ones.

+

The effect, on one host with four writers, against the single-writer +loader:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
graphedgesone writerfour writers
web-Google5.1 M16,013 edges/s30,985
cit-Patents16.5 M5,52626,661
soc-Pokec30.6 M4,69826,385
GAP-road57.7 M2,92827,708
sx-stackoverflow63.5 M2,92913,464
+

Peak memory during a load fell from about 15 GB to 3 to 7 GB at +web-Google scale. Scaling from one to eight writers measured 326, 220, +171 and 142 seconds for web-Google on an 8-vCPU host: about 35% of the +load is serialized, in the engine’s commit path.

+

15.2.2 The fastest fill: through +WAL

+

set_bulk_load_via_wal(true) makes an MVCC store’s +put_graph switch the database to WAL, load everything in +one transaction, checkpoint, and switch back to MVCC, whether or not the +load succeeds. Journal mode is database-wide, so nothing else may write +during the load; a bulk load before serving is the intended use. Handles +opened later with connect_shared inherit the setting.

+

This is exactly WAL speed, and WAL loads are 1.5 to 1.7 times faster +than four MVCC writers on Turso 0.7.2. On upstream main the +gap is wider (see below), and the via-WAL fill is 2.65 to 3.1 times the +parallel MVCC load. The round trip costs nothing measurable over a plain +WAL load at five to seventeen million edges.

+

15.2.3 Foreign keys are off during +a load

+

The Turso schema declares edge endpoints as +REFERENCES nodes(id) ON DELETE CASCADE, and +bootstrap turns PRAGMA foreign_keys on for the +bootstrap connection. Until commit 3f1ed19, that made the +result of a load depend on the writer count: WAL and single-writer MVCC +loads ran on the bootstrap connection and rejected an edge whose +endpoint node was not stored, while parallel MVCC writers, which never +set the pragma, accepted it.

+

put_graph now turns foreign keys off on the bootstrap +connection for the duration of the load, outside any transaction, and +restores the previous setting afterwards, whether or not the load +succeeded. Every path agrees, and the behavior matches the Memory +reference, where a vertex can exist only as an edge endpoint. +ON DELETE CASCADE fires on the connection that deletes, +which keeps the pragma on, so serving semantics do not change: a single +put_edge to a missing node is still refused. +tests/dangling_edges.rs loads a dangling edge through WAL, +one MVCC writer and four MVCC writers, reads it back identically on +each, and checks the restored strictness.

+

The two endpoint probes per edge were costing more than a third of +WAL’s load throughput. With foreign keys off, WAL loads of web-Google +rose from 42.9 k to 70.3 k edges per second on Turso 0.7.2 (about 60%); +the parallel MVCC path, whose writers were already unchecked, did not +move.

+

15.3 Concurrent writes and group +commit

+

Under the default synchronous = FULL, every MVCC commit +fsyncs the logical log while holding Turso’s global commit lock. Sixteen +writers each attaching 200 edges to one hot node took about 22 seconds +on 0.7.2; Neo4j does the same work in three to four.

+

with_group_commit() batches concurrent single-statement +writes into one transaction: one fsync per batch of up to +GROUP_COMMIT_MAX_STATEMENTS statements, and every +acknowledged write is durable when it is acknowledged. It is not a +relaxation of synchronous. Handles from +connect_shared share the store’s committer. On 0.7.2 it +takes that workload from 21.1 to 3.2 seconds on one host, with all 3,200 +writes accepted and no conflicts.

+

set_synchronous exposes the fsync policy per connection. +Bulk loads already run at NORMAL internally (they commit +rarely and end in a TRUNCATE checkpoint that fsyncs, so the +load is durable when put_graph returns), so the setting +matters for single-statement writes, where NORMAL removes +the per-commit fsync at the cost of durability on power loss.

+

15.3.1 Version scoping: this +changes on Turso 0.8

+

Upstream main has engine-level group commit, on by +default (PRAGMA mvcc_group_commit, store-wide). There, +Grust’s client-side committer becomes a second coordination layer and +costs about 19%: the same 16-by-200 workload takes 2.62 seconds with the +engine’s group commit alone and 3.2 seconds with +with_group_commit on top. The client committer’s numbers +are identical on both engine versions, which is the point: it sets the +ceiling.

+

set_mvcc_group_commit(enabled) exposes the pragma. It +errors on 0.7.2, which has no such pragma. The intended policy is: on +0.7.2, use with_group_commit; on 0.8, leave the engine’s +group commit on and do not wrap. The default of +with_group_commit should flip when the pinned engine moves. +The pin stays at 0.7.2 until 0.8.0 is a stable release, because the +facade must not force a pre-release on downstream crates.

+

15.4 Allocator

+

The turso crate enables mimalloc in its +default features, on 0.7.2 and on main: it installs +mimalloc as the #[global_allocator] of the whole process. +grust-turso depends on turso with default +features off, because a library should not choose the process allocator, +and exposes the feature as mimalloc = ["turso/mimalloc"]; +the facade forwards it as turso-mimalloc. So a Grust +application runs Turso on the platform allocator unless it opts in, +while a plain Turso build does not. Measured on four-writer MVCC loads +of web-Google, same host, alternating pairs: +14% on main +and +16% on 0.7.2; on WAL loads +9% and +16%. Every number in this +chapter was taken on the platform allocator; the strain harness has +since made mimalloc its process-wide default and tags such rows +alloc=mimalloc. A deployment that owns its binary should +turn it on.

+

15.5 What was measured

+

The adversarial-graph strain harness loads real SNAP graphs and runs +hot-node fan-out (A1), deep paths (A2), sixteen concurrent writers +against one hub (A4), guarded-commit replay (A7) and an operability +probe (A12), under nine hard gates. Every cell in this section passed +with every gate at zero.

+

15.5.1 Turso against Turso, same +host

+

On quegee (a c5n.4xlarge with no burst credits and less than half a +second of CPU steal per run), the three largest graphs, same binary:

--+++++ - - + + + + + - - + + + + + + + + + + + + - - + + + + + + +
OperationContractgraphMVCC loadWAL loadMVCC A4WAL A4
bfsHop distances from one external source IDcom-Orkut, 117.2 M edges16,046 edges/s24,6663,200/3,200 in 1.82 s61/3,200
soc-LiveJournal1, 69.0 M19,74931,2563,200/3,200 in 4.81 s119/3,200
multiSourceBfsMinimum hop distance from a nonempty source arraysx-stackoverflow, 63.5 M18,50431,2433,200/3,200 in 6.54 s50/3,200
+

WAL loads 1.5 to 1.7 times faster and then refuses 96 to 98% of +concurrent hot-node writes. MVCC with group commit accepts every one. +These WAL rows enforced foreign keys and the parallel MVCC rows did not, +so the load ratio understates WAL; with both sides unchecked, WAL’s lead +is larger. The MVCC rows in this table and the next ran eight writers at +synchronous = NORMAL; their A4 times measure conflict +retries rather than the commit path and are shown for the acceptance +count only. The durable four-writer lane did com-Orkut’s A4 in 5.29 s, +all 3,200 accepted, on a different host.

+

The durable seven-graph MVCC ladder on grust (the harness default: +synchronous = FULL, group commit, four writers) is clean on +all four core families on every whole graph up to com-Orkut. It was the +first MVCC run to reach com-Orkut at all: the single-writer loader would +have taken most of a day.

+

15.5.2 Against Neo4j, same +host

+

Neo4j 5.26 Community over Bolt in a 6 GiB container, quegee, zero +steal:

+ ++++++ + - - + + + + + + - - + + + + - - + + + + - - + + + + - - + + + + - - + + + + - - + + + + - - + + + +
dfsDeterministic reachable-node discovery orderNeo4jTurso MVCCTurso WAL
dijkstraDistances with finite nonnegative weightscom-Orkut load16,800 edges/s16,04624,666
shortestPathsOne selected full shortest path per reachable targetcom-Orkut A1 / A2144.5 s / 4,082 s138.1 s / 1,312 s96.8 s / 943 s
wccWeak components, including isolatescom-Orkut A42.90 s, 3,200/3,2001.82 s, 3,200/3,2001.05 s, 61/3,200
sccDirected strong components with iterative traversalcom-Orkut peak RSS4.9 GB22.5 GB12.4 GB
degreeExact projected arc counts and optional weighted strengthsoc-LiveJournal1 load28,06219,74931,256
pagerankWeighted scores, residual, iterations and convergence statussoc-LiveJournal1 A1 / A241.8 s / 2,501 s29.5 s / 1,065 s19.5 s / 773 s
topologicalSortComplete DAG order or a concrete closed cycle witnesssoc-LiveJournal1 A42.90 s, 3,200/3,2004.81 s, 3,200/3,2001.29 s, 119/3,200
-

projectionStats inspects selected topology. -estimateCsr reports nominal adjacency-buffer upper bounds -from snapshot counts, excluding graph storage, ID maps, original edge -tables, kernel scratch, output and allocator overhead. Selection can -reduce those counts. The estimate is not total memory admission.

-

The common configuration keys are orientation, -nodeLabels, relationshipTypes, -weightProperty and defaultWeight. Unknown keys -fail. Null label arrays select all labels; empty arrays select none. -Edges crossing outside the selected node set are excluded, while -selected isolates remain. Orientations are outgoing, incoming and -undirected. Undirected loops contribute one traversal arc; other -undirected edges contribute two. Parallel edges remain distinct. -Topology algorithms ignore weights after projection validation.

-

Weights must be finite and nonnegative. Integer weights outside -0..=2^53 are rejected rather than rounded. Missing or null -weights fail unless a finite nonnegative default is explicitly supplied. -Unit projections omit weight storage. Dijkstra uses strict improvement -and stable adjacency order for equal costs; zero-weight ties cannot -create predecessor cycles. Reachable cost overflow is an error. -Unreachable distances are infinity in direct Rust and null in -Cypher/Arrow. Component IDs are the external ID at the minimum -projection row in each component.

-

degree returns every selected node, including isolates. -Direct Rust degree(&projection) exposes exact -usize counts and optional f64 strengths. Arrow -columns are nodeId: Utf8, degree: UInt64, and -nullable strength: Float64; Cypher returns an exact checked -integer count and null strength for unweighted projections. No -normalization is implicit.

-
CALL grust.algorithms.degree({orientation: 'incoming', weightProperty: 'cost'})
-YIELD nodeId, degree, strength
-RETURN nodeId, degree, strength
-

Counts include parallel and zero-weight arcs. Undirected loops count -once under this projection contract. Weighted strength sums finite -nonnegative weights; overflow fails explicitly. Negative weights remain -unsupported. Unweighted execution reads existing CSR offsets in O(V); -weighted execution takes O(V+A). Result buffers retain the shared memory -admission, and bounded work-accounting chunks preserve cancellation -polling without a lock per scalar. This contract does not establish -universal degree-centrality compatibility with other engines.

-

PageRank defaults to damping .85, L1 tolerance 1e-8 and 1000 -iterations. Scores start uniformly. Optional personalization controls -teleportation and dangling mass; zero outgoing weight is dangling. -Parallel edges contribute separately. Finite large weights are scaled -before sums. Iteration-limit results explicitly report non-convergence. -Seeds, community algorithms, negative-weight paths, all-pairs/k-shortest -paths, flow, similarity and ML are deferred; the current catalog is not -universal GDS compatibility.

-

15.3 Full paths and incremental -consumption

-

A full path contains source-first external node IDs and cumulative -costs beginning at zero. Original edge ordinals identify the selected -multigraph edges. The zero-hop source path is included; unreachable -targets are omitted.

-
CALL grust.algorithms.shortestPaths('start', {weightProperty: 'cost'})
-YIELD nodeIds, costs
-UNWIND range(0, size(costs) - 1) AS i
-RETURN count(nodeIds[i]), sum(costs[i])
-LIMIT 1
-

This query consumes every actual requested array element. LIMIT -bounds the final aggregate output, not its input. The executor -incrementally handles CALL/YIELD filters, simple WITH, UNWIND, plain -RETURN and ungrouped COUNT/SUM/AVG. Generic array/index aggregation -avoids allocating a binding row for every path element. It does not -recognize a benchmark graph or substitute a closed-form answer. Other -shapes use the existing budgeted materializing executor. Grouping, -sorting, DISTINCT and collecting have no spill implementation and may -exceed admission. Early LIMIT can stop consumption, but a global kernel -may already have computed its result before producing the first -batch.

-

One shared execution context accounts for projection buffers, -scratch, provider results and downstream intermediates. Reservations -stay attached through internal ownership transfers. Temporary consumed -values release live admission; legacy materializing operators retain -cumulative-copy accounting. Cancellation, deadline and work checks span -preparation, kernels, reconstruction and consumption. Execution is -synchronous without prefetch or parallel kernel workers.

-

Borrowed path visitor slices expire when the callback returns. Pull -cursors reuse scratch storage. Owned scalar batches retain admission -through wrapper clones. Brine 0.20.0’s Arrow ownership extension also -retains algorithm-result admission through raw batches, buffer slices -and nested children; shared physical buffers keep the original -reservation without charging each clone again. Full paths and -cycle/order arrays use LargeList offsets with explicit limits. An -individual oversized path fails instead of truncating. Caller-owned -input and externally retained legacy result tables require caller -admission; logical accounting is not a process RSS sandbox for untrusted -provider code.

-

15.4 Discovery, preparation reuse -and execution classes

-

db.procedures describes the final registry generation, -including argument and option defaults, outputs, provider identity, -modes, correlation and computation boundary. Prepared explanations use -the same definitions and consumer classifier. -explain_snapshot also identifies the admitted revision and -principal without running algorithms. These are Rust planning methods; a -literal Cypher EXPLAIN prefix is not added by this extension.

-

Projection reuse is query-scoped and keyed by snapshot, principal, -representation, selection, orientation and weight policy. Correlated -CALL still executes once per incoming row; caching preparation does not -cache algorithm answers. Reverse SCC adjacency is built lazily and -stores only topology, without duplicate weights or edge identities. -Projections retain their selection policy for inspection.

-

The current procedure executor accepts an explicitly selected local -snapshot. Backend-native execution is rejected when unsupported, with no -automatic whole- graph download. Memory and a private Turso database -have snapshot integration proofs, including reads from old captures -after later writes. Arbitrary external writers, remote snapshot -stability and native analytics are not thereby qualified. -Mutation/write-back needs a separate atomic transaction design and -remains absent.

-

Both local direct Rust and ordinary Cypher completed the 65,536-node -full-path chain, consuming 2,147,516,416 entries in each array. Those -single observations have disclosed process envelopes and differ from the -frozen Docker protocol. The repository retains independent small-graph -oracles, source/binary receipts, frozen baseline results and current -qualification status. See the coverage -and resource contracts and benchmark -evidence.

+

Turso wins traversal on every graph. A7, guarded-commit replay, is +not compared: it runs only on stores with a Grust +GraphCommitStore, and the harness drives Neo4j through its +own Cypher without one, which is a gap in the harness path, not in +Neo4j. Load splits by graph: on com-Orkut the MVCC store matches Neo4j’s +load; on soc-LiveJournal1 Neo4j loads 1.4 times faster. The A4 column is +not a durability-matched comparison, because Neo4j commits durably and +these MVCC rows ran at synchronous = NORMAL; in the durable +same-host pairs the strain page counts, Neo4j wins every hot-node-write +pair against the MVCC store and loses five of six against WAL, which +accepts far fewer writes. Neo4j is three to five times leaner in memory +throughout. Neo4j has no rows for the two smallest graphs, and the +cit-Patents deep-path cell is vacuous on every backend (no path reaches +the depth), so it is never cited.

+

15.5.3 Upstream +main

+

Grust compiles and passes all 80 grust-turso tests +unchanged against Turso main at 19710d58d +(0.8.0-pre.11, 1,859 commits past 0.7.2). Measured against 0.7.2 on the +same host with alternating pairs and the engine version asserted in +every probe:

+
    +
  • WAL loads are 22% faster on main; the B-tree write path +improved.
    • +
  • Concurrent single-statement writes are better on main, +as described above.
    • +
  • MVCC bulk loads are slower on main, and the gap grows +with the graph: 5% at one writer and 15% at four on web-Google, 26% at +four on cit-Patents, and 20 to 23% at four writers on sx-stackoverflow, +soc-LiveJournal1 and com-Orkut on the same host. Reads on those stores +are 4 to 24% faster on main; only the write path +regressed.
    • +
  • The cause is not group commit, not checkpointing and not the +allocator; each was ruled out by A/B. A profile puts it in the value +comparison inside the MVCC index-key compare +(types::cmp_in_column and +read_value_serial_type, about 31% of the load on +main against 25.5% on 0.7.2). Upstream pull request #8385 +rewrites exactly that path: against its own base it is 29 to 30% faster +on this load, and under it the index-key share collapses to a single 12% +symbol.
    • +
+

The recommended configuration on main, all of it +existing switches: fill MVCC stores through WAL, serve with the engine’s +group commit and without with_group_commit, build with +turso-mimalloc, and load with foreign keys off, which the +branch already does.

+

15.6 Methodology, because it +decided several numbers

+

Benchmark hosts that are burstable instances (t2 on AWS) +throttle silently once their CPU credits are spent; the throttling shows +up as CPU steal, not as an error. One seven-graph lane ran a second +engine “1.7 times slower” for that reason alone and was discarded. Every +number in this chapter was taken with steal recorded, and every engine +or configuration comparison on a burstable host was run as alternating +pairs within one run so both sides saw the same throttling. Absolute +throughput comes from the non-burstable host only.

+

Hot-node write times at synchronous = NORMAL vary from +1.4 to 5.2 seconds on both engines with no consistent direction; they +measure conflict-retry luck, not the commit path, and are not cited. The +write-contention comparison is the FULL-sync alternating +A/B.

+

/tmp is a RAM filesystem on the benchmark hosts; every +load timing here is on disk. Run-to-run drift on the same host is about +2%, so any effect under about 4% needs alternating pairs, and several +plausible ideas were rejected on that basis: deferring secondary indexes +during a load (slower), a plain INSERT for loads into empty +tables (4% slower), a smaller writer page cache with GC off (flat), +larger rounds (memory, not time), and dropping the edge identity column +from the key (a real 2 to 4%, not worth a stateful dialect).

+

15.7 Limitations

+

The MVCC store holds more memory than the graph: 13 to 23 GB at 60 to +120 million edges with four writers, against 5 to 7 GB for Neo4j. The +via-WAL fill needs exclusive use of the database for the duration of the +load. A parallel MVCC load is not all-or-nothing. Foreign keys are not +enforced during loads, by design. set_mvcc_group_commit is +meaningful only on Turso 0.8. And the engine’s global commit lock, not +Grust, is what caps parallel MVCC loads at roughly 80% of WAL speed on +0.7.2.

diff --git a/docs/book/build/dist/grust-chapters/chapter-018.html b/docs/book/build/dist/grust-chapters/chapter-018.html index 5abe592b..c7040335 100644 --- a/docs/book/build/dist/grust-chapters/chapter-018.html +++ b/docs/book/build/dist/grust-chapters/chapter-018.html @@ -5,8 +5,8 @@ - - Arrow pipelines, ADBC and DataFusion + + Graph analytics through one Rust implementation @@ -47,26 +112,394 @@
+Next: -Previous: Arrow pipelines, ADBC and DataFusion +Previous: Graph analytics through one Rust implementation
-

Conclusion

-

Grust is small by design. Its core model is easy to hold in your -head, and its backend contract is narrow enough that very different -systems can implement it. That is the source of its leverage.

-

The project says: build your graph once, keep the domain model in -Rust, and let the backend translate. Sometimes that translation is a map -scan. Sometimes it is Arrow and LanceDB. Sometimes it is PostgreSQL, -Spark SQL, Redis graph commands, SurrealQL, Helix SDK calls, or -CocoIndex target state. And sometimes it is a capability-secured agent -memory path in which TypeSec guards authority while Grust persists -opaque records and their entity graph durably in Turso.

-

The more Grust grows, the more important that center becomes: a -stable property graph model, a backend-neutral traversal IR, explicit -errors, feature-gated integrations, and enough Rust type structure to -make the right path feel natural.

+

17 Arrow pipelines, ADBC and +DataFusion

+

Grust’s Arrow-based adapters use grust-arrow for common +batch and IPC operations. The graph core and other adapters remain +independent of Arrow. LanceDB and Sail select Arrow 58; the private +Ladybug adapter selects Arrow 55. The default interchange and ADBC +helper use Arrow 59. These modules compile the same implementation +against the native SDK types.

+
+Diagram 9 + +
+

17.1 Native tables and readers

+

ArrowTable retains arbitrary native Arrow columns, +metadata and batch boundaries. Its schema is checked when constructed. +Its into_reader method transfers ownership into the +standard RecordBatchReader interface without concatenating +batches or serializing their contents.

+

BatchReader wraps a standard reader with a row limit and +an input-batch memory limit. It pulls on demand and slices without +copying buffers. Errors terminate the stream after one error result. A +slice can retain its complete original allocation. The memory limit +applies to decoded Arrow arrays, not process RSS or the upstream +decoder’s temporary allocations. ByteLimitWriter separately +bounds encoded output, including schema bytes, before the sink grows +past its limit.

+

The shared IPC functions accept standard readers and caller-owned +sinks. They support multiple batches and schema-only empty streams. IPC +remains an explicit encoding boundary; it is not needed when producer +and consumer already share native Arrow types.

+

17.2 Property-graph +interchange

+

The original ArrowGraph is a validated pair of +scalar-property batches:

+ ++++ + + + + + + + + + + + + + + + + +
TableStructural columns
Nodesnon-null UTF-8 node_id, label
Edgesnon-null UTF-8 source, target, +label; nullable UTF-8 edge_id
+

Property columns use property.<key> and a non-null +Boolean present.<key>. The marker distinguishes a +missing property from an explicit null. Scalar properties support Null, +Boolean, Int64, Float64 and UTF-8, preserving integer precision. Mixed +non-null types and complex graph properties are rejected by this scalar +interchange contract rather than silently coerced.

+
use grust::arrow::ArrowGraph;
+use std::fs::File;
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let graph = grust::Graph::default();
+    let tables = ArrowGraph::from_graph(&graph)?;
+    tables.write_ipc(
+        File::create("nodes.arrow")?,
+        File::create("edges.arrow")?,
+    )?;
+    let restored = ArrowGraph::read_ipc(
+        File::open("nodes.arrow")?,
+        File::open("edges.arrow")?,
+    )?.to_graph()?;
+    assert_eq!(graph, restored);
+    Ok(())
+}
+

This legacy file-format API still stores one batch in each of two +Arrow files. ArrowGraphTables supports the same graph +contract across multiple batches. It validates global node identity and +edge endpoints without building property rows or adjacency. It preserves +isolates, loops, parallel edges and row order. +ArrowGraph::into_tables and +ArrowGraphTables::into_tables expose general Arrow tables +whose readers compose with other consumers.

+

These graph restrictions do not constrain arbitrary +ArrowTable values passed to ADBC or backend-native +registration. Those pipelines retain nested, dictionary, temporal and +extension columns according to the destination’s capabilities.

+

17.3 Database ingestion

+

The shared universal storage layout uses id,label,props +for nodes and key,id,from_id,to_id,label,props for edges. +The property payload preserves Grust’s tagged serialization, including +complex values. Native storage encoders and decoders share the stable +edge-key validation rule. Projection can rename columns without copying +their arrays.

+

LanceDB’s load_arrow takes standard readers in that +storage layout and merges native batches, maintaining typed mirrors. +Sail’s load_arrow accepts its existing SQL column names and +plain-JSON property representation. It reuses identity columns and +performs required property normalization and endpoint-label resolution. +Spark Connect transports the staged batches as IPC. Sail’s graph IPC +convenience loader now uses the same streaming path.

+

Stream graph loads validate a batch before writing it. Nodes precede +edges, and earlier batches can remain committed if a later operation +fails or is cancelled. They do not promise atomic whole-import +replacement. Consult the adapter contract for existing-row upserts, +constraints and typed mirror behavior.

+

Ladybug can register native readers without IPC, and visit query +batches without collecting all output. Registration requires all input +batches simultaneously; a conservative retained-memory bound applies +before registering the table. Registration creates a queryable Arrow +table, not an implicit persisted graph upsert. Persisted graph loading +continues through the adapter’s Arrow-backed COPY path and transaction +policy.

+

17.4 ADBC and extension

+

Enable the facade’s adbc feature, or +grust-arrow’s optional adbc feature, to bind +an Arrow 59 reader to a caller-owned ADBC statement with +grust_arrow::adbc::ingest. ADBC controls connection +lifecycle, target catalog/schema, ingestion mode, transactions and +cancellation. The helper preserves driver errors and unknown +affected-row counts. An ingestion mode such as append is not +reinterpreted as graph upsert.

+

The optional ffi feature exports standard Arrow C +streams through upstream Arrow’s implementation and release callbacks. +Grust adds no unsafe FFI code. Rust callers using the same Arrow major +can share readers directly; IPC is an explicit option for serialized +cross-language or cross-version boundaries.

+

New adapters should reuse the standard reader and shared batch +operations, keeping only their schema and backend semantics locally. +Enable the facade’s datafusion feature for the shared +DataFusion 55 foundation. It registers native Arrow 59 tables and +validated graph catalogs without re-encoding their buffers, accepts +upstream table providers, and exposes DataFrames and streaming SQL +results. Working-memory, parallelism, batch size and spill policy are +explicit. Input buffers and retained results remain caller-owned +admission; the working pool is not a process RSS limit.

+

BlockingReader connects a result stream to synchronous +Arrow and ADBC consumers on an ordinary thread or +spawn_blocking, using a caller-owned live Tokio runtime. It +adds no worker, prefetch queue or IPC boundary. Existing backend SDKs +keep their own compatible Arrow and execution-engine versions. Cypher +lowering and algorithm kernel selection require separate semantic and +performance qualification; this foundation does not automatically change +them.

+

See the repository’s docs/arrow-pipelines.md for +detailed admission, compatibility and testing contracts. Reproducible +pipeline benchmarks measure native slicing, explicit IPC boundaries and +graph validation separately from backend load and algorithm timings.

+

17.5 Typed Cypher execution

+

Combining facade features cypher and +datafusion enables the bridge at +grust::datafusion::cypher. Direct consumers can enable +grust-datafusion feature cypher. It provides +an explicit execution bridge over native Arrow graph tables. +GraphSnapshot::execute accepts Cypher text, parameters and +OutputLimits; it uses the existing parser and semantic +analyzer, builds DataFusion 55 expressions directly, and returns an +ordinary CypherResultTable. No SQL text or intermediate +row-oriented graph is generated. Isopod 0.17.0 adds this explicit +surface. Ordinary Cypher entrypoints do not yet select it automatically, +and it does not implement the complete bounded read policy.

+

17.5.1 Snapshot and +composition

+

GraphSnapshot captures validated immutable node/edge +providers, independent of session catalog replacement. Existing Arrow +buffers remain shared. An additional UInt64 ordinal, costing eight bytes +per edge, supplies physical relationship identity within the snapshot. +Parallel edges and optional/repeated external edge IDs remain distinct. +Backend transaction and authorization identity, input storage and +ordinal allocation remain caller responsibilities.

+

GraphSnapshot::plan selects the node or relationship +compiler from the parsed query shape. QueryPlan records its +PlanKind and either a DataFrame or an unsupported reason. +CypherExecution distinguishes completed execution from an +unsupported query that never executed. Parse, semantic, planning, +execution and output-limit errors propagate without retrying another +executor or snapshot.

+

Applications can also use plan_node_scan, +plan_relationship_scan, and the snapshot’s +directed/undirected relationship operators directly. The shared +ExpressionBindings contract resolves graph variables to +typed physical expressions for scalar and aggregate compilation. Missing +properties become null; unknown variables remain errors. This allows +composition without copying the scalar semantics into each pattern +planner.

+

17.5.2 Qualified language +surface

+

The current compiler supports node scans and fixed-length +relationship paths in either direction or undirected form; labels, +relationship types and scalar inline maps; named, anonymous and repeated +endpoint bindings; WHERE, projection, DISTINCT, count variants/grouping, +integer/string MIN/MAX, node identity, projected ordering, and +literal/parameter pagination. Scalar domains are Boolean, Int64, UTF-8 +and null. Only Boolean true retains a WHERE row.

+

Undirected matching emits both orientations of non-loop edges and +each self-loop once. Repeated endpoints constrain matching to self-loops +using one node join; the undirected form omits its reverse branch. +Anonymous names are generated after semantic analysis and cannot collide +with explicit pattern bindings.

+

Variable-length paths, OPTIONAL MATCH, correlated maps, +floating-point and mixed numeric expressions, arithmetic, additional +aggregates and duplicate projection names still need mappings or result +remapping. Integer SUM requires particular care: preserving an equal +final total does not preserve sequential overflow behavior across +partitions. Unsupported cases are not silently coerced.

+

17.5.3 Output and remaining policy +work

+

decode_result_batch preserves column order/names, row +multiplicity, nulls and exact Int64 values when converting native scalar +batches to portable rows. Types are resolved once per column; +unsupported types fail before allocating rows. Arrow/ADBC consumers can +retain native batches and avoid owned row conversion.

+

collect_result checks cumulative row counts before +decoding each batch and counts exact serialized JSON output bytes +without allocating a JSON buffer. The count includes column metadata, +delimiters, escaping and inter-row commas, including empty results. +Exceeding either limit returns an error with no partial result. One +decoded batch still requires separate memory admission.

+

Mantis 0.19.0’s decode_result_batch_with_context charges +cumulative logical copy bytes before creating portable rows and strings. +It measures the actual Arrow slice, including null-aware UTF-8 lengths, +column names and row/value containers. +collect_result_with_context also charges collection +metadata and moved row containers, and shares cancellation/deadline +control. Controlled GraphSnapshot::execute_with_context now +uses this collector. The charge stays consumed after output is dropped; +it is not a live-heap or allocator-capacity measurement. Arrow input, +validation descriptors and operator allocations remain separate. No +query is automatically routed on this basis.

+

DataFusion’s working-memory pool and these output checks do not +enforce Cypher’s candidate-work, intermediate-copy, query/input or +deadline contracts. Automatic selection remains pending until those +boundaries and measured cost decisions are integrated. The completed +scan profile reports conversion/preparation costs separately; the +parallel-ring path profile also retains preparation and admission +boundaries. Passing explicit execution tests does not establish +backend-wide speed or resource-policy parity.

+

RelationshipPlan::join_trail composes plans through +shared node bindings and excludes physical edge reuse across all joined +parts. Parallel edges remain distinct. Plans must come from the same +captured snapshot; cloned handles retain that identity. Physical columns +are renamed before composition to avoid alias collisions. The parsed +path planner composes these operators for fixed-length paths; +variable-length lowering and path-resource admission remain separate +work.

+

Fixed-length paths reuse the shared RETURN and predicate compilers +after trail composition. Labels, relationship types, inline maps and +repeated nodes constrain the complete path, with physical edge +uniqueness across all segments. Direction may differ per segment. +Variable-length bounds and named path values remain unsupported; the +pinned parallel-ring profile measures two- and three-hop execution with +separate preparation costs; it is not a general routing threshold.

+

17.5.4 Shared cancellation and +deadlines (Ostracod 0.18.0)

+

ExecutionContext::cancelled() is a runtime-independent +notification shared by procedures and relational execution. +grust_datafusion::run_cancellable races an operation with +that notification and the context’s absolute deadline. +GraphSnapshot::execute_with_context applies it through +complete Cypher result consumption. The operation and its owned streams +are dropped on cancellation or deadline; errors are never retried. +Dropping one wrapper unregisters its waiter without cancelling sibling +work. A deadline needs Tokio’s time driver.

+

Control is cooperative: synchronous work within one poll is not +preempted, and providers/kernels must checkpoint during computation. +Wrapping stream creation alone does not control later consumption. These +methods do not charge candidate work or intermediate allocations and do +not establish complete read-policy admission or automatic routing.

+

control_stream retains cancellation and deadline control +for an Arrow stream’s whole lifetime. +DataFusionEngine::execute_stream_with_context controls both +SQL preparation and the returned stream. Batches pass through without +buffer copies, queues or worker tasks. Completion or the first error +immediately drops the provider stream and timer; subsequent polls remain +finished. The controlled stream can feed BlockingReader and +ADBC ingestion directly.

+

17.5.5 Prepared read admission +(Ostracod 0.18.0)

+

grust_cypher::PreparedReadRequest owns the validated +AST, policy and original absolute deadline, borrows the admitted +immutable parameters, and retains any application registry generation. +The bounded reference executor uses its request, graph/index and output +checks. Indexed graph checks reuse the cached exact serialized size; +parameter/output counting does not allocate encoded JSON. Oversized +parameters fail before graph inspection. Preparation does not authorize +or bind a backend snapshot, install execution budgets, or qualify a +DataFusion route. Route-specific candidate/intermediate accounting must +still be enforced.

+

17.5.6 Exact snapshot statistics +(Ostracod 0.18.0)

+

GraphSnapshot::statistics() returns exact node/edge row +counts, original input batch counts and the logical bytes added for +UInt64 relationship ordinals. These are cached during capture and read +in constant time without a provider scan or graph export. Cloned +snapshots keep identical statistics; session catalog replacement cannot +change them. Batch counts do not claim execution parallelism. +Selectivity, join cardinalities, serialized graph size and total memory +are not inferred from these counts. Automatic routing still requires +qualified costs and complete resource admission.

+

17.5.7 Borrowed graph serialization +(Ostracod 0.18.0)

+

ArrowGraphTables::as_serializable_graph() presents the +ordinary Grust graph serde contract directly over native Arrow 55/58/59 +columns. It preserves row and batch order, identity, sorted property +keys, scalar tags, and missing versus null properties. It builds only +per-batch column descriptors, without row graphs, property maps or +string-value copies. The caller selects the serializer and sink; a +bounded writer over std::io::sink() can count exact JSON +bytes without keeping the encoded output. Writer errors propagate and +may leave a prefix in a real sink. This enables native input-size +admission; automatic routing and full query resource-policy integration +remain separate work.

+

17.5.8 Native Arrow input admission +(Ostracod 0.18.0)

+

GraphSnapshot::try_new_with_input_policy checks a +prepared request’s node/edge limits and exact graph JSON size before +capturing native providers. It uses the borrowed Arrow serialization +view and bounded counting writer, with no row graph or encoded JSON +buffer. Row rejection precedes serialization; the original request +deadline also covers capture. +SnapshotStatistics::serialized_graph_bytes is +Some(exact_bytes) after measured capture and +None for ordinary capture, including an unmeasured empty +graph. Clones retain the measurement.

+

PreparedReadRequest::check_serializable_graph shares +these checks with trusted native adapters. +check_measured_graph reuses exact cached measurements for +the same immutable projection; estimates are not admissible substitutes. +Backend snapshot authority, pre-existing input buffers, ordinal +allocation and execution work/intermediates remain separate obligations. +This is input-size admission, not a complete bounded DataFusion executor +or automatic routing.

+

17.5.9 Owned buffer admission +(Brine 0.20.0)

+

retain_buffer_owner attaches an application token to an +immutable Arrow buffer through safe bytes::Bytes ownership. +Native slices, array clones and C Data exports retain that owner without +copying bytes. Existing clones made before attachment do not acquire the +token. Reserve storage before allocation; this helper preserves +ownership and does not itself measure memory.

+

GraphSnapshot::try_new_with_context admits the added +UInt64 ordinal payload and construction work. Reservations stay with the +buffers after the snapshot, provider or result stream disappears. The +combined try_new_with_input_policy_and_context also checks +exact input size and requires an execution deadline no later than the +prepared request’s deadline. Input buffers, provider/schema metadata and +allocator capacity remain outside the logical ordinal-payload bound. +These capture checks do not establish backend authority, full operator +accounting or automatic execution selection.

+

retain_array_owner extends buffer ownership through an +array’s validity bits and nested children. It rebuilds and validates +metadata without copying payload bytes. Arrays with no physical buffers +retain no token; a wrapper remains necessary when metadata-only lifetime +must be admitted. Algorithm result batches use the same mechanism so raw +batch clones and retained child slices keep their original reservation. +New reservations are not charged for each clone.

diff --git a/docs/book/build/dist/grust-chapters/chapter-020.html b/docs/book/build/dist/grust-chapters/chapter-020.html new file mode 100644 index 00000000..662abeb0 --- /dev/null +++ b/docs/book/build/dist/grust-chapters/chapter-020.html @@ -0,0 +1,72 @@ + + + + + + + + + Conclusion + + + + + +

Conclusion

+

Grust is small by design. Its core model is easy to hold in your +head, and its backend contract is narrow enough that very different +systems can implement it. That is the source of its leverage.

+

The project says: build your graph once, keep the domain model in +Rust, and let the backend translate. Sometimes that translation is a map +scan. Sometimes it is Arrow and LanceDB. Sometimes it is PostgreSQL, +Spark SQL, Redis graph commands, SurrealQL, Helix SDK calls, or +CocoIndex target state. And sometimes it is a capability-secured agent +memory path in which TypeSec guards authority while Grust persists +opaque records and their entity graph durably in Turso.

+

The more Grust grows, the more important that center becomes: a +stable property graph model, a backend-neutral traversal IR, explicit +errors, feature-gated integrations, and enough Rust type structure to +make the right path feel natural.

+ + diff --git a/docs/book/build/dist/grust-chapters/index.html b/docs/book/build/dist/grust-chapters/index.html index 152416f3..445963e4 100644 --- a/docs/book/build/dist/grust-chapters/index.html +++ b/docs/book/build/dist/grust-chapters/index.html @@ -5,7 +5,7 @@ - + Grust