Skip to content

fix: re-seed the elementary graph cache on access - #1060

Closed
Daniel-Wiszowaty wants to merge 1 commit into
elementary-data:masterfrom
Daniel-Wiszowaty:fix/reseed-elementary-graph-cache
Closed

Daniel-Wiszowaty wants to merge 1 commit into
elementary-data:masterfrom
Daniel-Wiszowaty:fix/reseed-elementary-graph-cache

Conversation

@Daniel-Wiszowaty

@Daniel-Wiszowaty Daniel-Wiszowaty commented Sep 15, 2026

Copy link
Copy Markdown

Fixes the crash class in #1058.

Problem

get_cache / set_cache / setdefault_cache all read graph["elementary"], which is written exactly once, by init_elementary_graph() from the on-run-start hook. When that seed is absent the accessors fall through to their none default, and the five call sites that chain a method onto the result take the whole run down:

call site expression error
macros/edr/materializations/test/test.sql:382 get_cache("elementary_test_results").update(...) 'None' has no attribute 'update'
macros/edr/materializations/test/test.sql:54 get_cache("elementary_test_failed_row_counts").update(...) same
macros/edr/tests/test_metric_stability.sql:196 get_cache("tables").get("metrics")... 'None' has no attribute 'get'
macros/edr/data_monitoring/anomaly_detection/store_metrics_in_cache.sql:3 get_cache("tables").get("metrics")... same
macros/edr/data_monitoring/schema_changes/store_schema_snapshot_tables_in_cache.sql:2 get_cache("tables").get(...) same

The first one runs for every test in the project, so a missing seed means every test fails to compile.

Fix

Route the three accessors through one helper that re-seeds the full default structure when the key is absent, and have init_elementary_graph() share the same defaults macro so there is a single source of truth. A lost seed now degrades (empty cache) instead of raising.

This matches the two suggestions on #1058 — it is option (2), applied in one place rather than at each call site, so future chained call sites are safe too.

How this is reachable today

Under dbt-core's v2 (Fusion) parser, the package's on-run-start hook never executes, so the seed is never written at all. Reproduction on a stock dbt-duckdb project with elementary 0.26.0 and dbt-core 1.12.4 — same project, same command, only the parser differs:

$ dbt run
Elementary: Runtime data: {... "elementary_version": "0.26.0" ...}
1 of 1 START hook: elementary.on-run-start.0 ................... [RUN]
1 of 1 OK hook: elementary.on-run-start.0 ...................... [OK in 0.01s]
1 of 1 START hook: elementary.on-run-end.0 ..................... [RUN]
1 of 1 OK hook: elementary.on-run-end.0 ........................ [OK in 0.32s]

$ dbt run --use-v2-parser
(no hook lines at all — neither hook runs)

dbt build reports 2 project hooks on the classic path and omits them entirely under --use-v2-parser.

To be clear about scope: the hooks not running is a dbt-side problem, not something this PR fixes. It also means on-run-end artifact upload is skipped, which is a separate and larger issue. This PR only stops elementary from crashing when the seed is absent, whatever the reason.

Verification

Stock dbt-duckdb project, elementary installed as a local package, dbt-core 1.12.4 + dbt-core-experimental-parser 2.0.1.

Reproducing the exact failing expression from handle_dbt_test:

# before
$ dbt run-operation probe_cache
[ERROR]: Encountered an error while running operation: Compilation Error
  'None' has no attribute 'update'

# after
$ dbt run-operation probe_cache
PROBE ok, cache now = {'x': 1}

where probe_cache is just {% do elementary.get_cache("elementary_test_results").update({"x": 1}) %}.

No regression on a full build:

before after
dbt build (classic) PASS=35, 2 project hooks PASS=35, 2 project hooks
dbt build --use-v2-parser PASS=33, no hooks PASS=33, no hooks

Files formatted with the repo's sqlfmt 0.29.0 pre-commit hook.

I have not run the warehouse integration suite — no access to those targets. Happy to adjust if you would rather see setdefault_cache at each call site instead.

Summary by CodeRabbit

  • Refactor
    • Standardized initialization of the elementary graph cache to use shared default settings.
    • Preserved existing graph values when already present.
    • Ensured default graph settings are restored when the parser rebuilds the graph.

get_cache/set_cache/setdefault_cache read graph["elementary"], which is
written once by init_elementary_graph() from the on-run-start hook. When
that seed is absent the accessors return their none default, and the five
call sites that chain a method onto the result crash the whole run:

  'None' has no attribute 'update'   (test materialization, every test)
  'None' has no attribute 'get'      (metrics / schema snapshot caches)

Route the three accessors through a helper that re-seeds the full default
structure when the key is missing, so a lost seed degrades instead of
raising. init_elementary_graph() now shares the same defaults macro, so
there is one source of truth.

This is reachable today under dbt-core's v2 (Fusion) parser, where the
package's on-run-start hook never executes, so the seed is never written.
That hook behaviour is a separate dbt-side problem; this change only stops
elementary from crashing when the seed is absent.

Refs elementary-data#1058
@github-actions

Copy link
Copy Markdown
Contributor

👋 @Daniel-Wiszowaty
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in the elementary repository.

@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: e8f48c48-76f8-4a3f-8dee-634b3210c33e

📥 Commits

Reviewing files that changed from the base of the PR and between 9a7e2b5 and 8ea904b.

📒 Files selected for processing (2)
  • macros/edr/tests/on_run_start/init_elementary_graph.sql
  • macros/utils/graph/cache.sql

Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review.


📝 Walkthrough

Walkthrough

The change extracts the elementary graph defaults into a reusable macro. Graph initialization and cache access now use these defaults when the elementary graph entry is absent.

Changes

Elementary graph cache

Layer / File(s) Summary
Centralize elementary graph defaults
macros/edr/tests/on_run_start/init_elementary_graph.sql
The default elementary graph dictionary moves into elementary_graph_defaults(). init_elementary_graph() calls this macro while preserving graph.setdefault behavior.
Apply defaults during cache access
macros/utils/graph/cache.sql
get_elementary_cache() initializes the elementary cache with the shared defaults. set_cache, get_cache, and setdefault_cache use this accessor.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 8ea90

The cache accessors now restore the complete default cache shape when the elementary cache is absent. No current merge-blocking risk is identified.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: restoring the Elementary graph cache when accessed without its seed.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@Daniel-Wiszowaty

Copy link
Copy Markdown
Author

Filed the hook half of this upstream: dbt-labs/dbt#16310--use-v2-parser never executes on-run-start / on-run-end, for root-project and package hooks alike, silently.

That is why the seed is missing here, so if it gets fixed this issue's crash stops being reachable via that path. The re-seeding in #1060 is still worth having as defence in depth, but the hook bug is the one that actually restores Elementary under the v2 parser — on-run-end artifact upload is skipped too, which no change inside this package can work around.

Worth noting the v2 parser already reads on-run-start (it parses CREATE FUNCTION there to register UDFs), so it sees the hooks and just doesn't run them.

@Daniel-Wiszowaty
Daniel-Wiszowaty deleted the fix/reseed-elementary-graph-cache branch September 16, 2026 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant