bundle: delete libs/dyn; configuration is typed structs with structvar refs and locations by denik · Pull Request #6954 · databricks/cli

denik · 2026-10-06T22:39:05Z

Changes

Bundle configuration no longer goes through libs/dyn; the package is deleted.

config.Root is the typed struct plus a structvar side: pure ${...} references in non-string fields and a tree of source locations (one node per YAML value). YAML is decoded straight into the structs, with the same warnings as before.
Mutators read through structvar.View and write with Set/Merge/Override/… instead of converting the whole configuration to a dyn.Value and back. Paths are structpath everywhere, including diag.Diagnostic.Paths.
Variable resolution, ${...} parsing and the resource-reference resolver run on typed values (structvar.Resolve, same semantics and error text as dynvar).
Generate commands write YAML with the new libs/structs/structyaml (identical output); --json and variable-overrides.json use structvar.ParseJSON (exact int64, same error positions).
Diagnostics about values set by mutators fall back to the location of the nearest parent.

Stacked on #6974 (diag.Location), which is stacked on #6980 (structpath diagnostic paths).

Why

Converting the configuration typed↔dyn around mutators dominated command time on large bundles.

./task bench-1k-summary (1000 jobs, wall seconds, DMS= variant), main → this PR:

benchmark	wall	user CPU	peak RSS
validate	12.6s → 1.29s (9.8x)	21.1s → 2.0s	655MB → 183MB
plan	19.2s → 1.77s (10.8x)	25.5s → 2.9s	670MB → 179MB
deploy (1000 creates)	16.4s → 2.33s (7.0x)	33.4s → 8.3s	676MB → 266MB

(deploy/DMS=true at 1000 jobs fails on main too: the test server caps a deployment version at 800 operations.)

Tests

Existing unit and acceptance tests; one golden changed (path quoting above). Parity tests against libs/dyn (decode, view, writes, merge/override, walk) passed on every bundle fixture before dyn was deleted.

This pull request and its description were written by Isaac.

eng-dev-ecosystem-bot · 2026-10-07T00:11:25Z

Integration test report

Commit: 799731f

Run: 37770991564

	Env	🔄flaky	✅pass	🙈skip	Time
✅	aws linux-2core-8gb		276	18	5:29
✅	aws-windows-latest-4core-16gb		278	16	4:02
✅	azure linux-2core-8gb		275	18	5:11
✅	azure-windows-latest-4core-16gb		277	16	4:07
✅	gcp linux-2core-8gb		276	18	5:41
🔄	gcp-windows-latest-4core-16gb	2	276	16	4:36

	Test Name	gcp-windows-latest-4core-16gb
🔄	TestFsCpSourceIsDirectoryButTargetIsFile/dbfs_to_uc-volumes	🔄f
🔄	TestFsCpSourceIsDirectoryButTargetIsFile/uc-volumes_to_local	🔄f

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
4:04	azure-windows-latest-4core-16gb	TestAccept
4:02	gcp-windows-latest-4core-16gb	TestAccept
4:01	aws linux-2core-8gb	TestAccept
4:00	aws-windows-latest-4core-16gb	TestAccept
3:59	gcp linux-2core-8gb	TestAccept
3:54	azure linux-2core-8gb	TestAccept

Move Location from libs/dyn to libs/diag and use diag.Location directly, so libs/diag no longer imports libs/dyn. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

…r refs and locations Bundle configuration is the typed struct plus structvar (pure ${...} references in non-string fields and a tree of source locations). YAML and JSON decode straight into the structs; mutators, validators, variable resolution, configsync and generate work on structvar views and structpath paths. libs/dyn is deleted. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

… test Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

github-actions Bot added DABs DABs related issues PyDABs labels Oct 6, 2026

denik force-pushed the denik/bundle-struct-no-dyn branch from 13fac18 to b2a1c10 Compare October 7, 2026 09:41

github-actions Bot added the AIR Databricks AI Runtime CLI label Oct 7, 2026

denik changed the title ~~bundle: make the typed configuration the source of truth (no per-mutator dyn round trip)~~ bundle: delete libs/dyn; configuration is typed structs with structvar refs and locations Oct 7, 2026

denik mentioned this pull request Oct 7, 2026

diag: own Location (move from libs/dyn) #6974

Open

denik force-pushed the denik/bundle-struct-no-dyn branch from 6ee2caa to 68af62f Compare October 7, 2026 12:40

denik changed the base branch from main to denik/diag-location-structpath October 7, 2026 12:40

denik force-pushed the denik/diag-location-structpath branch from c8121ae to 8b3a26a Compare October 7, 2026 13:23

denik force-pushed the denik/bundle-struct-no-dyn branch from baccaac to 7086a71 Compare October 7, 2026 14:08

denik mentioned this pull request Oct 7, 2026

diag: use structpath for diagnostic paths #6980

Draft

denik force-pushed the denik/diag-location-structpath branch from 30536d2 to 3afe070 Compare October 7, 2026 14:29

denik force-pushed the denik/bundle-struct-no-dyn branch from 7086a71 to 4c4f0b3 Compare October 7, 2026 14:33

denik force-pushed the denik/diag-location-structpath branch from 3afe070 to c759a40 Compare October 7, 2026 14:38

denik force-pushed the denik/bundle-struct-no-dyn branch from 4c4f0b3 to 3bb81b3 Compare October 7, 2026 14:38

denik force-pushed the denik/diag-location-structpath branch 2 times, most recently from 2b8030a to 47be590 Compare October 7, 2026 16:05

denik force-pushed the denik/bundle-struct-no-dyn branch 2 times, most recently from 722c97c to 649bde3 Compare October 7, 2026 16:17

denik force-pushed the denik/diag-location-structpath branch from 45e5b54 to 53d29f7 Compare October 7, 2026 21:05

denik force-pushed the denik/bundle-struct-no-dyn branch from 649bde3 to 67d1321 Compare October 7, 2026 21:11

denik mentioned this pull request Oct 7, 2026

structpath: build computed paths and patterns from parts; lint rule #6984

Draft

denik force-pushed the denik/diag-location-structpath branch from 53d29f7 to 4be88ef Compare October 8, 2026 07:50

denik force-pushed the denik/bundle-struct-no-dyn branch from 67d1321 to d49f829 Compare October 8, 2026 07:52

denik force-pushed the denik/diag-location-structpath branch from 4be88ef to 367bee6 Compare October 8, 2026 11:12

denik force-pushed the denik/bundle-struct-no-dyn branch from d49f829 to b22dd28 Compare October 8, 2026 11:18

denik and others added 3 commits October 8, 2026 13:28

diag: own Location

88e9aa2

Move Location from libs/dyn to libs/diag and use diag.Location directly, so libs/diag no longer imports libs/dyn. Co-authored-by: Isaac <no-reply@databricks.com>

Drop unused dyn import

ab63169

Co-authored-by: Isaac <no-reply@databricks.com>

Drop unused dyn import in validate_target_mode

368b11a

Co-authored-by: Isaac <no-reply@databricks.com>

gofmt imports

801eb52

Co-authored-by: Isaac <no-reply@databricks.com>

denik force-pushed the denik/diag-location-structpath branch from 367bee6 to 801eb52 Compare October 8, 2026 11:28

denik and others added 10 commits October 8, 2026 13:29

Drop unused dyn import in check_dashboards_modified_remotely

c17b428

Co-authored-by: Isaac <no-reply@databricks.com>

structvar: map YAML locations in the YAML source; drop mappedSource

02dd2a7

Co-authored-by: Isaac <no-reply@databricks.com>

Use NewStringKeysSlice for single key paths

b5a42ba

Co-authored-by: Isaac <no-reply@databricks.com>

Build literal diagnostic paths with NewStringKeysSlice

e1d3314

Co-authored-by: Isaac <no-reply@databricks.com>

Restack fixes: structvar grammar for structpath's PureReferenceToPath…

308fdc7

… test Co-authored-by: Isaac <no-reply@databricks.com>

Restore import block after restack

0a0d36e

Co-authored-by: Isaac <no-reply@databricks.com>

Use structpath.NewPath for key/index paths

a31aec4

Co-authored-by: Isaac <no-reply@databricks.com>

acc: name the ids in bundle/run/job-ref-job-id (after #6975)

dbcae89

Co-authored-by: Isaac <no-reply@databricks.com>

Build the target preset path in one NewPath call

799731f

Co-authored-by: Isaac <no-reply@databricks.com>

denik force-pushed the denik/bundle-struct-no-dyn branch from b22dd28 to 799731f Compare October 8, 2026 11:34

denik force-pushed the denik/diag-location-structpath branch from c17b428 to 87a5a1d Compare October 8, 2026 11:57

Navigation Menu

Repository navigation

bundle: delete libs/dyn; configuration is typed structs with structvar refs and locations - #6954

bundle: delete libs/dyn; configuration is typed structs with structvar refs and locations#6954
denik wants to merge 14 commits into
denik/diag-location-structpathfrom
denik/bundle-struct-no-dyn

denik commented Oct 6, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

denik commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

eng-dev-ecosystem-bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

denik commented Oct 6, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Oct 7, 2026 •

edited

Loading