fix: re-seed the elementary graph cache on access - #1060
Daniel-Wiszowaty wants to merge 1 commit into
Conversation
get_cache/set_cache/setdefault_cache read graph["elementary"], which is written once by init_elementary_graph() from the on-run-start hook. When that seed is absent the accessors return their none default, and the five call sites that chain a method onto the result crash the whole run: 'None' has no attribute 'update' (test materialization, every test) 'None' has no attribute 'get' (metrics / schema snapshot caches) Route the three accessors through a helper that re-seeds the full default structure when the key is missing, so a lost seed degrades instead of raising. init_elementary_graph() now shares the same defaults macro, so there is one source of truth. This is reachable today under dbt-core's v2 (Fusion) parser, where the package's on-run-start hook never executes, so the seed is never written. That hook behaviour is a separate dbt-side problem; this change only stops elementary from crashing when the seed is absent. Refs elementary-data#1058
|
👋 @Daniel-Wiszowaty |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review. 📝 WalkthroughWalkthroughThe change extracts the elementary graph defaults into a reusable macro. Graph initialization and cache access now use these defaults when the elementary graph entry is absent. ChangesElementary graph cache
Priority: ⬇️ Low Estimated code review effort: 2 (Simple) | ~10 minutes Change: Bug fix Merge Risk: ⚪ Minimal · up to The cache accessors now restore the complete default cache shape when the elementary cache is absent. No current merge-blocking risk is identified. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Comment |
|
Filed the hook half of this upstream: dbt-labs/dbt#16310 — That is why the seed is missing here, so if it gets fixed this issue's crash stops being reachable via that path. The re-seeding in #1060 is still worth having as defence in depth, but the hook bug is the one that actually restores Elementary under the v2 parser — Worth noting the v2 parser already reads |
Fixes the crash class in #1058.
Problem
get_cache/set_cache/setdefault_cacheall readgraph["elementary"], which is written exactly once, byinit_elementary_graph()from theon-run-starthook. When that seed is absent the accessors fall through to theirnonedefault, and the five call sites that chain a method onto the result take the whole run down:macros/edr/materializations/test/test.sql:382get_cache("elementary_test_results").update(...)'None' has no attribute 'update'macros/edr/materializations/test/test.sql:54get_cache("elementary_test_failed_row_counts").update(...)macros/edr/tests/test_metric_stability.sql:196get_cache("tables").get("metrics")...'None' has no attribute 'get'macros/edr/data_monitoring/anomaly_detection/store_metrics_in_cache.sql:3get_cache("tables").get("metrics")...macros/edr/data_monitoring/schema_changes/store_schema_snapshot_tables_in_cache.sql:2get_cache("tables").get(...)The first one runs for every test in the project, so a missing seed means every test fails to compile.
Fix
Route the three accessors through one helper that re-seeds the full default structure when the key is absent, and have
init_elementary_graph()share the same defaults macro so there is a single source of truth. A lost seed now degrades (empty cache) instead of raising.This matches the two suggestions on #1058 — it is option (2), applied in one place rather than at each call site, so future chained call sites are safe too.
How this is reachable today
Under dbt-core's v2 (Fusion) parser, the package's
on-run-starthook never executes, so the seed is never written at all. Reproduction on a stockdbt-duckdbproject with elementary 0.26.0 and dbt-core 1.12.4 — same project, same command, only the parser differs:dbt buildreports2 project hookson the classic path and omits them entirely under--use-v2-parser.To be clear about scope: the hooks not running is a dbt-side problem, not something this PR fixes. It also means
on-run-endartifact upload is skipped, which is a separate and larger issue. This PR only stops elementary from crashing when the seed is absent, whatever the reason.Verification
Stock
dbt-duckdbproject, elementary installed as a local package, dbt-core 1.12.4 +dbt-core-experimental-parser2.0.1.Reproducing the exact failing expression from
handle_dbt_test:where
probe_cacheis just{% do elementary.get_cache("elementary_test_results").update({"x": 1}) %}.No regression on a full build:
dbt build(classic)dbt build --use-v2-parserFiles formatted with the repo's
sqlfmt0.29.0 pre-commit hook.I have not run the warehouse integration suite — no access to those targets. Happy to adjust if you would rather see
setdefault_cacheat each call site instead.Summary by CodeRabbit