Skip to content

Views call graph: compact store (diet part 1) - #1359

Merged
ualtinok merged 22 commits into
mainfrom
land/views-store-compact-payloads
Nov 3, 2026
Merged

ualtinok merged 22 commits into
mainfrom
land/views-store-compact-payloads

Conversation

@aft-alfonso

@aft-alfonso aft-alfonso Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Part 1 of the call-graph store diet: the views call-graph database for AFT's own repository goes from 1,966 MB to 1,049 MB (−47%) and builds about 10–15% faster, with every recorded answer byte-identical. Per the release ruling, this ships with the views-default release, so users rebuild their call graph once.

What changes in the views store

  • Per-file JSON payloads (view_bindings, view_file_surfaces, view_dispatch_files) are stored zlib-compressed at the fastest level; the digests are still those of the JSON text. About 300 MB of repetitive JSON becomes about 37 MB.
  • view_unknown_live keeps one row per caller file and protected target instead of one per call site: its only reader asks which targets are alive. 1.46 M rows / 345 MB become 253 K rows / 30 MB. Main's empty-table streaming path is kept.
  • The two site tables are WITHOUT ROWID, so they don't store a second copy of their key.
  • Node ids keep 64 bits of their digest (views spell them vp/vt plus 16 hex digits); a short prefix sorts like the full digest, and store triggers refuse two different symbols that share an id instead of letting one replace the other.
  • Two indexes no statement uses are dropped (edges by target node, dispatch hints by method name). BUILD_OUTPUT_VERSION v24 on top of main's v23.

Measured (interleaved in one job, 7,331 files, same probe on both sides):

main this PR
views base 1,966 MB 1,049 MB
materialize 71.0 / 89.9 / 64.8 s 64.0 / 76.4 / 55.9 s
CPU (user) 116 / 130 / 113 s 110 / 122 / 100 s
peak RSS 2.06 GB 1.97 GB

Verified with the store-diet harness (#1344)

  • Every reply golden replays byte for byte: the six operations for the legacy and views stores and both wire captures.
  • inventory.py compare-plans passes against main's baseline plans: the compacted tables are searched through their primary key; the unknown-live delete still uses its whole key; new scans are test-only probes. Store-internal snapshots (node id form, tie-order ids, one Kotlin fixture's liveness count, plans) were reviewed and re-accepted; each change is listed in the commit messages.
  • The SQL census maps each rewritten statement to its baseline id (1,729 statements; inventory.py check passes).
  • 1,819 library tests (call graph, views, zoom, inspect, harness), 111 call-graph and views integration tests, formatting and the Windows check.

One proof test from main counted unknown-receiver rows per call site; it now asserts the per-file count (2 before and after removing one of two identical sites), with its parity checks against the row-by-row writer and a cold rebuild unchanged.


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Shrinks the views call-graph store by about 47% (1,966 MB to 1,049 MB for AFT's own repository) and builds about 10–15% faster, with every recorded answer byte-identical. Ships with the views-default release, so users rebuild their call graph once.

What changes in the store

  • Per-file JSON payloads are stored zlib-compressed at the fastest level; digests stay those of the JSON text. About 300 MB of repetitive JSON becomes about 37 MB.
  • view_unknown_live keeps one row per caller file and protected target instead of one per call site, cutting 1.46 M rows / 345 MB to 253 K rows / 30 MB.
  • Both site tables are WITHOUT ROWID, and node ids hold 64 digest bits (16 hex digits) instead of 256, with triggers refusing two different symbols that share an id.
  • Two unused indexes are dropped, and BUILD_OUTPUT_VERSION bumps to v24.

Verified

  • The store-diet harness replays every reply golden byte for byte; plans and store-internal snapshots were re-accepted.
  • 1,819 library tests and 111 call-graph and views integration tests pass.

Written for commit cfaa25c. Summary will update on new commits.

View guided diff Turn on auto-fix

ualtinok and others added 22 commits November 3, 2026 09:41
…id copies

view_bindings, view_file_surfaces and view_dispatch_files store their JSON
zlib-compressed (digests stay those of the JSON text). view_unknown_live and
view_dispatch_sites are read only by key or in full into sets, so they drop
the rowid table that duplicated their primary-key index. Bumps
MATERIALIZATION_VERSION to 13, so views databases rebuild once.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…site

view_unknown_live's only reader asks which targets are alive; the writers
replace a caller file's rows as a whole. Dropping the site ordinal collapses
aft's 1.44M rows (340 MB with its key) to about 248k.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…guard

Node ids repeat in every ref, edge and index that names a caller or target;
the views base for aft's repository spent about 170 MB on 256-bit digests.
Ids now hold 16 hex digits (their prefix sorts as the full digest did), and
triggers refuse two different symbols with one id instead of merging them.
Bumps BUILD_OUTPUT_VERSION so stores rebuild once.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…ished base

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…act-payloads

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…every index's definition

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Views stores spelled every node id as view:symbol:pos:symbol: or
view:symbol:top: plus the digest; ids repeat in refs, edges and their
indexes. They are now vp or vt plus the same 16 hex digits, which keeps
symbol ids before top-level ids and digest order within each class.
No statement filters edges by target node or dispatch hints by method
name (checked against the store-diet statement census and its recorded
plans), so those two indexes are dropped.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…act-payloads

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
A cold build packs every per-file payload; the default level spent noticeable
CPU on about 300 MB of JSON per AFT-sized repository.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
… candidate capture for review

The JSON-facts capture compares decoded JSON, so it reads the views store's
compressed payload form as well as plain text. AFT_STORE_DIET_CANDIDATE_DIR
writes a capture that differs from its golden to a review folder; goldens
are still never updated by an ordinary run.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…ments

Eleven baseline statements now have candidate text: the compact
view_unknown_live (no ordinal), the WITHOUT ROWID dispatch-site table, BLOB
payload columns, the files schema with collision triggers and dropped
indexes, and test statements for the compact node ids. Seven new
statements (collision tests, the size probe, INSERT OR IGNORE) carry
new-sql ids. Inventories derived; check passes.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Every reply golden (six operations for legacy and views, both wire
captures) replays byte for byte. These store-internal snapshots change:
- stored shapes: views node ids are the compact vp/vt form;
- tie order: ambiguity candidate ids are 16 hex digits, in the same order;
- language corpus: one Kotlin fixture's unknown-receiver rows go 6 -> 3
  (per-file dedup), node id form as above;
- query plans, checked with inventory.py compare-plans: view_dispatch_sites
  and view_unknown_live searches now use the WITHOUT ROWID primary key
  instead of an autoindex. Reviewed exceptions: the view_unknown_live
  delete has one fewer equality column only because ordinal no longer
  exists; it still searches the whole primary key. New scans are test-only
  probes (size measurement, pragma_table_info, sqlite_master, the
  collision test's scoped_name read). No plan used either dropped index.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…oads

BUILD_OUTPUT_VERSION becomes v24 for the short node ids on top of main's
v23 resolver change. view_unknown_live keeps main's empty-table fast path,
streaming per-file rows with INSERT OR IGNORE. The harness census is mapped
again on main's re-recorded baseline (including main's own bash replay
index change, which its census had not captured); store snapshots are
re-accepted after the next harness run.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…ain's baseline

Every reply golden (six operations for legacy and views, both wire
captures) replays byte for byte on main's re-recorded resolver baseline.
Store-internal snapshots change as reviewed before: compact vp/vt node id
form, 16-hex-digit tie-order ids in the same order, one Kotlin fixture's
unknown-receiver rows 6 -> 3, and WITHOUT ROWID primary-key plans for
view_dispatch_sites and view_unknown_live (compare-plans passes; the
unknown-live delete still searches its whole primary key; new scans are
test-only probes). Two bash_tasks plans now use main's replay index
(#1355), which main's own snapshot had not recorded.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
main's stream-versus-reference proof counted rows per call site (4, then 2
after one of two sites is removed). The compact table keeps one row per
caller file and protected target, so it holds 2 rows before and after; the
parity checks against the row-by-row reference and the cold rebuild are
unchanged.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
@ualtinok
ualtinok added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 994ea07 Nov 3, 2026
93 of 94 checks passed
@ualtinok
ualtinok deleted the land/views-store-compact-payloads branch November 3, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant