Skip to content

[AIGTWY-4977] Cover managed skills in CUJ 2, 3 and 6 - #1026

Merged
lilly-luo merged 1 commit into
mainfrom
david/cuj-skills-coverage
Oct 8, 2026
Merged

lilly-luo merged 1 commit into
mainfrom
david/cuj-skills-coverage

Conversation

@david-siqi-liu

@david-siqi-liu david-siqi-liu commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

The CUJ plan's skills checks for CUJ 2, 3 and 6 were skipped because Default Storage rejects skill uploads. With skill fixtures on a schema backed by customer S3 storage, the journeys can now cover them.

Changes:

  • CUJ 2 (skills.names = ["ug_e2e.skills.fixture-summary"]):
    • only that bundle downloads, byte-identical to the live fixture;
    • both agents list it in /skills, and the schema's other skills are absent;
    • both agents invoke it and answer with a token that exists only in the bundle's reference file.
    • _assert_cuj2_config now expects the skills selector.
  • CUJ 3 (skills.unity_catalog_location = "ug_e2e.skills"):
    • every skill in the schema downloads and is listed;
    • the ug_e2e.other_skills decoy stays absent;
    • ug status counts the skills per agent;
    • both agents invoke a downloaded skill.
  • CUJ 6:
    • Phase A publishes the named skill. It downloads once to both skill roots and is unchanged by a repeat configure.
    • Phase A runs no terminal launch: one would make ug own /etc, and phase B's headless configure would then refuse.
    • Phases B and C drop the selector, and the skill disappears from disk and from each agent's /skills, including after relaunch.
    • The hand-authored user skill must render in the same screen, so the absence checks can't pass on an unloaded list.
    • The user's Claude settings are seeded with the theme the Claude TUI saves on first launch, so only ug's writes show as a diff.
  • Shared helpers live in tests/e2e_cuj/helpers/skills.py. The tests/README.md coverage row is updated.
  • UC skill names reject underscores, so the plan's fixture_summary is fixture-summary.

One-time workspace setup, done outside the tests on 2026-10-07:

  • CUJ 6 metastore (ca-central-1):
    • ug_e2e.skills sits on S3-backed storage;
    • the CUJ SP has READ_VOLUME on it, which skill_authz.jsonnet requires for skill reads and downloads;
    • workspace A publishes the Phase A skills selector.
  • CUJ 2/3 metastore (us-east-1):
    • the same S3-backed storage credential and an external location;
    • ug_e2e.skills and ug_e2e.other_skills recreated on that storage, holding the four fixture skills;
    • CUJ 2 publishes skills.names, and CUJ 3 already published unity_catalog_location = ug_e2e.skills.

This overlaps Mason's open #1014 and #1015 in test_cuj2_mps_mcp.py, where this PR only adds one key to _assert_cuj2_config.

This pull request and its description were written by Isaac.

@david-siqi-liu
david-siqi-liu force-pushed the david/cuj-skills-coverage branch from c26f5e4 to 6159a29 Compare October 7, 2026 17:25
@david-siqi-liu david-siqi-liu changed the title [AIGTWY-4977] Cover the managed skill in CUJ 6 [AIGTWY-4977] Cover managed skills in CUJ 2, 3 and 6 Oct 7, 2026
@david-siqi-liu
david-siqi-liu force-pushed the david/cuj-skills-coverage branch from 6159a29 to cdb4cbd Compare October 7, 2026 17:45
@david-siqi-liu
david-siqi-liu marked this pull request as ready for review October 7, 2026 17:54
@david-siqi-liu
david-siqi-liu force-pushed the david/cuj-skills-coverage branch 2 times, most recently from c23c6a0 to 73fef2f Compare October 7, 2026 19:18
@david-siqi-liu david-siqi-liu changed the title [AIGTWY-4977] Cover managed skills in CUJ 2, 3 and 6 [AIGTWY-4977] Cover the managed skill in CUJ 6 Oct 7, 2026
@david-siqi-liu
david-siqi-liu force-pushed the david/cuj-skills-coverage branch from 73fef2f to c23c6a0 Compare October 7, 2026 19:26
@david-siqi-liu david-siqi-liu changed the title [AIGTWY-4977] Cover the managed skill in CUJ 6 [AIGTWY-4977] Cover managed skills in CUJ 2, 3 and 6 Oct 7, 2026
@david-siqi-liu
david-siqi-liu force-pushed the david/cuj-skills-coverage branch from c23c6a0 to b9aedd7 Compare October 7, 2026 19:58
@david-siqi-liu
david-siqi-liu force-pushed the david/cuj-skills-coverage branch from b9aedd7 to c23c6a0 Compare October 7, 2026 20:14
@david-siqi-liu
david-siqi-liu force-pushed the david/cuj-skills-coverage branch 2 times, most recently from dc895c4 to e56e8ed Compare October 7, 2026 20:41
@david-siqi-liu
david-siqi-liu force-pushed the david/cuj-skills-coverage branch from e56e8ed to f5fb31e Compare October 7, 2026 21:08
The CUJ plan's skills checks were skipped because Default Storage rejects
skill uploads. The ug_e2e.skills fixtures now live on customer S3 storage,
so the journeys cover them.

- CUJ 2 (skills.names = [ug_e2e.skills.fixture-summary]):
  - only that bundle downloads, byte-identical to the live fixture;
  - it is listed in /skills without its siblings;
  - both agents invoke it, answering with a token that exists only in the
    bundle's reference file.
- CUJ 3 (skills.unity_catalog_location = ug_e2e.skills):
  - every skill in the schema downloads and is listed;
  - the ug_e2e.other_skills decoy stays absent;
  - ug status counts the skills;
  - both agents invoke one.
- CUJ 6:
  - Phase A downloads the named skill once to both roots, with no terminal
    launch, since one would make ug own /etc and block phase B's headless
    configure;
  - Phases B and C drop the selector, and the skill disappears from disk
    and /skills, including after relaunch;
  - the hand-authored user skill is the positive control that the /skills
    view loaded.

Shared helpers are in tests/e2e_cuj/helpers/skills.py.

Co-authored-by: Isaac <no-reply@databricks.com>
@david-siqi-liu
david-siqi-liu force-pushed the david/cuj-skills-coverage branch from f5fb31e to ee953dc Compare October 7, 2026 23:46
@lilly-luo
lilly-luo disabled auto-merge October 8, 2026 00:49
@lilly-luo
lilly-luo added this pull request to the merge queue Oct 8, 2026
Merged via the queue into main with commit c943f2a Oct 8, 2026
27 checks passed
@lilly-luo
lilly-luo deleted the david/cuj-skills-coverage branch October 8, 2026 00:53
rohita5l pushed a commit to djcelis/unity-gateway that referenced this pull request Oct 8, 2026
…bricks#1032)

Companion: databricks#1026 (CUJ 2, 3 and 6 managed skills coverage). Independent
PR; merge this one first, since it fixes the Managed config · Claude
failure databricks#1026 also sees.

Integration tests use a stubbed config on the common managed workspace,
and the e2e CUJs stay real and read-only. Until now the managed
integration suites mixed live published configs, three different
workspaces with their own service principals, and configs built in
Python. One of those workspaces (ap-northeast-2) began returning an
empty `system.ai` catalog, which turned Managed config · Claude red on
every branch.

Changes:
- Every managed-config integration test runs on the managed workspace
(`E2E_ADMIN_WORKSPACE`). It gets its CodingAgentConfig from a checked-in
JSON fixture in `tests/fixtures/managed_config/` through
`UCODE_MANAGED_CONFIG_STUB`. The fixtures are in the GET shape `ug
configure --file` (databricks#866) accepts, so switching to `--file` once databricks#866
lands is mechanical.
- The Python config builders and the fetch-then-stub helper are removed.
The former live published-config tests (`test_ug_configure_managed.py`)
now stub too. The stub records the same `published` outcome and TTL
cache, so the TTL test still applies.
- The Claude defaults tests no longer target
`eng-ml-inference-batch-inference-us-west-2` or
`eng-ml-inference-ap-northeast-2`. Their service principals,
`UG_MPS_DEFAULTS_CLIENT_SECRET`,
`UG_PARENT_SCHEMA_DEFAULTS_CLIENT_SECRET` and the runner's minting code
are gone. Both repo secrets can be deleted once this merges.
- The `managed` marker is gone. Managed-config integration tests are
`managed_fixture`.
- The dedicated e2e CUJs (`tests/e2e_cuj/`) are untouched: they keep
reading the real published workspace configs.
- `tests/AGENTS.md` rule 4, `tests/README.md` and
`tests/integration/README.md` are updated.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants