Repository navigation
Conversation
Add grust_datafusion::cypher::RoutedGraph: one graph captured as a typed index and an Arrow snapshot, one bounded-read admission per query, and DataFusion execution for single-node scan plans whose every physical operator charges candidate work and intermediate bytes against the caller's ReadQueryPolicy. Joins, unsupported shapes, graphs under 3,000 nodes and plans over a limit stay on the reference executor through the new run_prepared_read_query_indexed, which keeps the original deadline. Split large single-batch Arrow tables into zero-copy partitions so no round-robin repartition charges each queued slice its shared parent buffers; that intermittently exhausted a 256 MiB pool at one million nodes. Add LadybugConfig::max_db_bytes and cap unit-test databases at 16 GiB; parallel tests exhausted x86-64 address space with 8 TiB reservations. Workspace gate on quegee: 1,675 passed, 0 failed, 49 ignored; Clippy clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZ7SjQFbe58VDZs6xR3nCo
…RUNCATE Profiling an MVCC bulk load (pprof, bulk_load example, 1M edges over 50k nodes) put 60% of the time in the upserts' index work in the MVCC store and 31% in the automatic checkpoints that write the same rows into the B-tree. The edge table carried three indexes: the primary key (from_id, label, to_id, identity_key), to_idx, and from_idx on from_id alone. Turso's planner never reads from_idx: source lookups and every traversal hop already use the primary key's from_id prefix, with or without from_idx present (EXPLAIN QUERY PLAN). The index only cost a write per edge. Turso's dialect now reports edge_source_index() = false (a new GraphSqlDialect method, default true, so Postgres keeps its index), the universal bootstrap omits the index for it, and bootstrap drops it from existing Turso stores. Query plans and results are unchanged. MVCC loads now end in PRAGMA wal_checkpoint(TRUNCATE) inside put_graph, as WAL loads already did, so a load returns with its rows in the database file and the in-memory store drained. Under the default blocking automatic checkpoints this costs nothing measurable. Optional mimalloc feature: Turso's own default allocator (turso/mimalloc), which installs it as the binary's global allocator; MVCC loads ran about 9% faster with it. Off by default, because it changes the allocator of every backend in a binary. 1M edges, 4-core host: MVCC 12.7k -> 15.9k edges/s, WAL 43.6k -> 51.5k. New tests: an MVCC bulk load with checkpoints mid-load stores exactly the rows of the WAL load (identity_key included), survives a reopen and takes concurrent BEGIN CONCURRENT writers afterwards; bootstrap omits the index and drops a legacy one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MVCC connections open with turso::Builder:: experimental_mvcc_passive_checkpoint(true), so the automatic checkpoints taken every mvcc_checkpoint_threshold bytes of logical log (4,120,000 by default in Turso 0.7.2) run PASSIVE instead of TRUNCATE and do not block the store's other readers and writers. put_graph still ends in one TRUNCATE checkpoint, so a load returns durable in the database file. WAL connections are unchanged; the option only affects MVCC checkpoints. Measured cost on a single-writer bulk load (bulk_load example, 4-core host, no edge-source index): 5M edges 14.4k edges/s blocking -> 8.7k passive, peak RSS 0.9 GB -> 2.8 GB. PASSIVE reclaims emptied version chains lazily, so the in-memory store and each checkpoint's scan grow during the load, and each put_graph pays its deferred work in the final TRUNCATE (~11 s per 1M-edge call). The option is experimental in Turso 0.7.2. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
put_graph on an MVCC store sets PRAGMA mvcc_checkpoint_threshold = -1 for the load, so no automatic checkpoint writes the rows committed so far into the B-tree mid-load (profiled at about a third of an MVCC load). The load still ends in one TRUNCATE checkpoint, and the store's previous threshold is restored whether or not the load succeeds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sak2FQcdgs5pkL2ruKrSUZ
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sak2FQcdgs5pkL2ruKrSUZ
Turning automatic checkpoints off for an MVCC put_graph kept every row version of the load resident: 13.4 GB at 5M edges against 11.4 GB for Turso's 4 MB default, heading past a 32 GB host well before com-Orkut. Loads now raise the threshold to MVCC_LOAD_CHECKPOINT_BYTES (64 MiB) and restore the store's own threshold afterwards. bulk_load example on quegee, MVCC: 1M edges 11.8k -> 15.4k edges/s (2.31 -> 2.35 GB peak); 5M edges 995 s -> 423 s, 11.4 GB both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sak2FQcdgs5pkL2ruKrSUZ
…IVE" This reverts commit 3e8b13a.
…ver" This reverts commit 0291906.
This reverts commit a26fdff.
Member
Author
|
Harness confirmation on lakecat (4 cores, 15 GB), adversarial-graph
The interim pins with PASSIVE checkpoints did not finish web-Google within 1,800 s (eigen, lakecat), and cit-Patents was OOM-killed at a 13 GiB cap. |
Member
Author
|
turso-wal through the full strain ladder on grust (2xlarge class, 32 GB; harness pin a26fdff, whose WAL code is identical to this branch's). Every graph passed LOAD, A1, A2, A4, A7 and A12; A3 is unsupported as before.
The previously published strain summary had Turso WAL passing every strict gate only up to GAP-road. turso-mvcc on this branch is still running on quegee and eigen. |
Member
Author
|
Full workspace gate on the branch head 5dff9ac (grust host, 8 cores, 4 jobs under a 24 GiB cgroup cap): |
Member
Author
|
Superseded by #6. The three evidence comments here are carried over verbatim to #6 (comment) — including the workspace gate on 🤖 Generated with Claude Code |
alexy
added a commit
that referenced
this pull request
Sep 20, 2026
Documentation only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
alexy
added a commit
that referenced
this pull request
Sep 20, 2026
Main moved under the gate by codex-to-codex.md and ci-local.sh alone, so no crate or benchmark source is unverified. Proposes the refinement that keeps the standing verdict workable: restrict the recheck to code paths, or the task never finishes while the board is active. ACK G5 after G0. Also corrects the credit: the catch came from asking what else the line was being quoted for, not from the check that passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
alexy
added a commit
that referenced
this pull request
Sep 20, 2026
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sak2FQcdgs5pkL2ruKrSUZ
This was referenced Sep 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
grust_datafusion::cypher::RoutedGraphadmits each read once and picks DataFusion automatically for bounded single-node scans on graphs of at least 3,000 nodes. It charges candidate work and intermediate bytes against the caller'sReadQueryPolicy. Joins and unsupported shapes stay on the reference executor, and errors after execution starts are not rerouted. Large single-batch tables are split into zero-copy partitions, which fixed the intermittent 256 MiB memory exhaustion at one million nodes. The commit also addsLadybugConfig::max_db_bytes, capped at 16 GiB in tests.Turso MVCC checkpoints: measured, then reverted
On the real web-Google edge list (
bulk_load --snap, MVCC, quegee):Turning automatic checkpoints off entirely was fast but kept every row version of the load resident: 13.4 GB at 5M edges, growing about 0.4 GB per million. It would not fit com-Orkut on a 32 GB host. PASSIVE, the 64 MiB threshold and checkpoints-off (3e8b13a, 0291906, a26fdff) are therefore reverted in 41f352d, 3d3ba0b and 5dff9ac.
The adversarial-graph harness itself on lakecat, web-Google
turso-mvccLOAD: 0176718 took 570 s (8,950 edges/s, 3.9 GB); this branch took 431 s (11,835 edges/s, 3.85 GB).Test plan
cargo fmt --all -- --checkcargo test -p grust-turso -p grust-sql-coreLBUG_LOCALIZE_BUNDLED_SYMBOLS=1🤖 Generated with Claude Code
https://claude.ai/code/session_01Sak2FQcdgs5pkL2ruKrSUZ