Repository navigation
[AIGTWY-4977] Cover managed skills in CUJ 2, 3 and 6 - #1026
Merged
Merged
Conversation
david-siqi-liu
force-pushed
the
david/cuj-skills-coverage
branch
from
October 7, 2026 17:25
c26f5e4 to
6159a29
Compare
david-siqi-liu
force-pushed
the
david/cuj-skills-coverage
branch
from
October 7, 2026 17:45
6159a29 to
cdb4cbd
Compare
david-siqi-liu
marked this pull request as ready for review
October 7, 2026 17:54
david-siqi-liu
force-pushed
the
david/cuj-skills-coverage
branch
2 times, most recently
from
October 7, 2026 19:18
c23c6a0 to
73fef2f
Compare
david-siqi-liu
force-pushed
the
david/cuj-skills-coverage
branch
from
October 7, 2026 19:26
73fef2f to
c23c6a0
Compare
david-siqi-liu
force-pushed
the
david/cuj-skills-coverage
branch
from
October 7, 2026 19:58
c23c6a0 to
b9aedd7
Compare
david-siqi-liu
requested review from
AarushiShah-db,
lilly-luo and
rohita5l
as code owners
October 7, 2026 19:58
david-siqi-liu
force-pushed
the
david/cuj-skills-coverage
branch
from
October 7, 2026 20:14
b9aedd7 to
c23c6a0
Compare
david-siqi-liu
force-pushed
the
david/cuj-skills-coverage
branch
2 times, most recently
from
October 7, 2026 20:41
dc895c4 to
e56e8ed
Compare
david-siqi-liu
force-pushed
the
david/cuj-skills-coverage
branch
from
October 7, 2026 21:08
e56e8ed to
f5fb31e
Compare
david-siqi-liu
enabled auto-merge
October 7, 2026 23:20
The CUJ plan's skills checks were skipped because Default Storage rejects
skill uploads. The ug_e2e.skills fixtures now live on customer S3 storage,
so the journeys cover them.
- CUJ 2 (skills.names = [ug_e2e.skills.fixture-summary]):
- only that bundle downloads, byte-identical to the live fixture;
- it is listed in /skills without its siblings;
- both agents invoke it, answering with a token that exists only in the
bundle's reference file.
- CUJ 3 (skills.unity_catalog_location = ug_e2e.skills):
- every skill in the schema downloads and is listed;
- the ug_e2e.other_skills decoy stays absent;
- ug status counts the skills;
- both agents invoke one.
- CUJ 6:
- Phase A downloads the named skill once to both roots, with no terminal
launch, since one would make ug own /etc and block phase B's headless
configure;
- Phases B and C drop the selector, and the skill disappears from disk
and /skills, including after relaunch;
- the hand-authored user skill is the positive control that the /skills
view loaded.
Shared helpers are in tests/e2e_cuj/helpers/skills.py.
Co-authored-by: Isaac <no-reply@databricks.com>
david-siqi-liu
force-pushed
the
david/cuj-skills-coverage
branch
from
October 7, 2026 23:46
f5fb31e to
ee953dc
Compare
lilly-luo
approved these changes
Oct 8, 2026
lilly-luo
disabled auto-merge
October 8, 2026 00:49
rohita5l
pushed a commit
to djcelis/unity-gateway
that referenced
this pull request
Oct 8, 2026
…bricks#1032) Companion: databricks#1026 (CUJ 2, 3 and 6 managed skills coverage). Independent PR; merge this one first, since it fixes the Managed config · Claude failure databricks#1026 also sees. Integration tests use a stubbed config on the common managed workspace, and the e2e CUJs stay real and read-only. Until now the managed integration suites mixed live published configs, three different workspaces with their own service principals, and configs built in Python. One of those workspaces (ap-northeast-2) began returning an empty `system.ai` catalog, which turned Managed config · Claude red on every branch. Changes: - Every managed-config integration test runs on the managed workspace (`E2E_ADMIN_WORKSPACE`). It gets its CodingAgentConfig from a checked-in JSON fixture in `tests/fixtures/managed_config/` through `UCODE_MANAGED_CONFIG_STUB`. The fixtures are in the GET shape `ug configure --file` (databricks#866) accepts, so switching to `--file` once databricks#866 lands is mechanical. - The Python config builders and the fetch-then-stub helper are removed. The former live published-config tests (`test_ug_configure_managed.py`) now stub too. The stub records the same `published` outcome and TTL cache, so the TTL test still applies. - The Claude defaults tests no longer target `eng-ml-inference-batch-inference-us-west-2` or `eng-ml-inference-ap-northeast-2`. Their service principals, `UG_MPS_DEFAULTS_CLIENT_SECRET`, `UG_PARENT_SCHEMA_DEFAULTS_CLIENT_SECRET` and the runner's minting code are gone. Both repo secrets can be deleted once this merges. - The `managed` marker is gone. Managed-config integration tests are `managed_fixture`. - The dedicated e2e CUJs (`tests/e2e_cuj/`) are untouched: they keep reading the real published workspace configs. - `tests/AGENTS.md` rule 4, `tests/README.md` and `tests/integration/README.md` are updated. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CUJ plan's skills checks for CUJ 2, 3 and 6 were skipped because Default Storage rejects skill uploads. With skill fixtures on a schema backed by customer S3 storage, the journeys can now cover them.
Changes:
skills.names = ["ug_e2e.skills.fixture-summary"]):/skills, and the schema's other skills are absent;_assert_cuj2_confignow expects the skills selector.skills.unity_catalog_location = "ug_e2e.skills"):ug_e2e.other_skillsdecoy stays absent;ug statuscounts the skills per agent;/etc, and phase B's headless configure would then refuse./skills, including after relaunch.tests/e2e_cuj/helpers/skills.py. Thetests/README.mdcoverage row is updated.fixture_summaryisfixture-summary.One-time workspace setup, done outside the tests on 2026-10-07:
ug_e2e.skillssits on S3-backed storage;READ_VOLUMEon it, whichskill_authz.jsonnetrequires for skill reads and downloads;ug_e2e.skillsandug_e2e.other_skillsrecreated on that storage, holding the four fixture skills;skills.names, and CUJ 3 already publishedunity_catalog_location = ug_e2e.skills.This overlaps Mason's open #1014 and #1015 in
test_cuj2_mps_mcp.py, where this PR only adds one key to_assert_cuj2_config.This pull request and its description were written by Isaac.