Repository navigation
Views call graph: compact store (diet part 1) - #1359
Merged
Merged
Conversation
…id copies view_bindings, view_file_surfaces and view_dispatch_files store their JSON zlib-compressed (digests stay those of the JSON text). view_unknown_live and view_dispatch_sites are read only by key or in full into sets, so they drop the rowid table that duplicated their primary-key index. Bumps MATERIALIZATION_VERSION to 13, so views databases rebuild once. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…site view_unknown_live's only reader asks which targets are alive; the writers replace a caller file's rows as a whole. Dropping the site ordinal collapses aft's 1.44M rows (340 MB with its key) to about 248k. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…guard Node ids repeat in every ref, edge and index that names a caller or target; the views base for aft's repository spent about 170 MB on 256-bit digests. Ids now hold 16 hex digits (their prefix sorts as the full digest did), and triggers refuse two different symbols with one id instead of merging them. Bumps BUILD_OUTPUT_VERSION so stores rebuild once. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…ished base Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…act-payloads Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…every index's definition Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Views stores spelled every node id as view:symbol:pos:symbol: or view:symbol:top: plus the digest; ids repeat in refs, edges and their indexes. They are now vp or vt plus the same 16 hex digits, which keeps symbol ids before top-level ids and digest order within each class. No statement filters edges by target node or dispatch hints by method name (checked against the store-diet statement census and its recorded plans), so those two indexes are dropped. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…act-payloads Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
A cold build packs every per-file payload; the default level spent noticeable CPU on about 300 MB of JSON per AFT-sized repository. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
… candidate capture for review The JSON-facts capture compares decoded JSON, so it reads the views store's compressed payload form as well as plain text. AFT_STORE_DIET_CANDIDATE_DIR writes a capture that differs from its golden to a review folder; goldens are still never updated by an ordinary run. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…ments Eleven baseline statements now have candidate text: the compact view_unknown_live (no ordinal), the WITHOUT ROWID dispatch-site table, BLOB payload columns, the files schema with collision triggers and dropped indexes, and test statements for the compact node ids. Seven new statements (collision tests, the size probe, INSERT OR IGNORE) carry new-sql ids. Inventories derived; check passes. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Every reply golden (six operations for legacy and views, both wire captures) replays byte for byte. These store-internal snapshots change: - stored shapes: views node ids are the compact vp/vt form; - tie order: ambiguity candidate ids are 16 hex digits, in the same order; - language corpus: one Kotlin fixture's unknown-receiver rows go 6 -> 3 (per-file dedup), node id form as above; - query plans, checked with inventory.py compare-plans: view_dispatch_sites and view_unknown_live searches now use the WITHOUT ROWID primary key instead of an autoindex. Reviewed exceptions: the view_unknown_live delete has one fewer equality column only because ordinal no longer exists; it still searches the whole primary key. New scans are test-only probes (size measurement, pragma_table_info, sqlite_master, the collision test's scoped_name read). No plan used either dropped index. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…oads BUILD_OUTPUT_VERSION becomes v24 for the short node ids on top of main's v23 resolver change. view_unknown_live keeps main's empty-table fast path, streaming per-file rows with INSERT OR IGNORE. The harness census is mapped again on main's re-recorded baseline (including main's own bash replay index change, which its census had not captured); store snapshots are re-accepted after the next harness run. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…ain's baseline Every reply golden (six operations for legacy and views, both wire captures) replays byte for byte on main's re-recorded resolver baseline. Store-internal snapshots change as reviewed before: compact vp/vt node id form, 16-hex-digit tie-order ids in the same order, one Kotlin fixture's unknown-receiver rows 6 -> 3, and WITHOUT ROWID primary-key plans for view_dispatch_sites and view_unknown_live (compare-plans passes; the unknown-live delete still searches its whole primary key; new scans are test-only probes). Two bash_tasks plans now use main's replay index (#1355), which main's own snapshot had not recorded. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
main's stream-versus-reference proof counted rows per call site (4, then 2 after one of two sites is removed). The compact table keeps one row per caller file and protected target, so it holds 2 rows before and after; the parity checks against the row-by-row reference and the cold rebuild are unchanged. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1 of the call-graph store diet: the views call-graph database for AFT's own repository goes from 1,966 MB to 1,049 MB (−47%) and builds about 10–15% faster, with every recorded answer byte-identical. Per the release ruling, this ships with the views-default release, so users rebuild their call graph once.
What changes in the views store
view_bindings,view_file_surfaces,view_dispatch_files) are stored zlib-compressed at the fastest level; the digests are still those of the JSON text. About 300 MB of repetitive JSON becomes about 37 MB.view_unknown_livekeeps one row per caller file and protected target instead of one per call site: its only reader asks which targets are alive. 1.46 M rows / 345 MB become 253 K rows / 30 MB. Main's empty-table streaming path is kept.WITHOUT ROWID, so they don't store a second copy of their key.vp/vtplus 16 hex digits); a short prefix sorts like the full digest, and store triggers refuse two different symbols that share an id instead of letting one replace the other.BUILD_OUTPUT_VERSIONv24 on top of main's v23.Measured (interleaved in one job, 7,331 files, same probe on both sides):
Verified with the store-diet harness (#1344)
inventory.py compare-planspasses against main's baseline plans: the compacted tables are searched through their primary key; the unknown-live delete still uses its whole key; new scans are test-only probes. Store-internal snapshots (node id form, tie-order ids, one Kotlin fixture's liveness count, plans) were reviewed and re-accepted; each change is listed in the commit messages.inventory.py checkpasses).One proof test from main counted unknown-receiver rows per call site; it now asserts the per-file count (2 before and after removing one of two identical sites), with its parity checks against the row-by-row writer and a cold rebuild unchanged.
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.Summary by cubic
Shrinks the views call-graph store by about 47% (1,966 MB to 1,049 MB for AFT's own repository) and builds about 10–15% faster, with every recorded answer byte-identical. Ships with the views-default release, so users rebuild their call graph once.
What changes in the store
view_unknown_livekeeps one row per caller file and protected target instead of one per call site, cutting 1.46 M rows / 345 MB to 253 K rows / 30 MB.WITHOUT ROWID, and node ids hold 64 digest bits (16 hex digits) instead of 256, with triggers refusing two different symbols that share an id.BUILD_OUTPUT_VERSIONbumps to v24.Verified
Written for commit cfaa25c. Summary will update on new commits.