Skip to content

test(playground): add e2e suite for execution identity - #630

Draft
MarioCadenas wants to merge 1 commit into
mainfrom
tests/e2e-execution-identity
Draft

MarioCadenas wants to merge 1 commit into
mainfrom
tests/e2e-execution-identity

Conversation

@MarioCadenas

Copy link
Copy Markdown
Collaborator

Adds an end-to-end suite (the e2e package from TesterArmy) that checks which principal AppKit runs each call as: the app service principal by default, the signed-in user on the on-behalf-of path.

Depends on the execution-identity stack (#592, #594, #595, #597, #601). Without it, appkit.asUser(req) doesn't exist, so this stays a draft until the stack lands.

How it works

  • A minimal test app (apps/dev-playground/e2e/identity/app/) runs in production mode. In development mode a missing user token silently runs as the SP, which would hide the fail-closed checks.
  • Requests carry the headers the Databricks Apps proxy forwards (x-forwarded-access-token, x-forwarded-user, x-forwarded-email).
  • Assertions are deterministic: exact current_user() values, error codes, and tool outputs read from the agent SSE stream. Nothing depends on the model's text.

Coverage

B1, B2, B3 (block and one-call forms), B4, B5, B7, plus the fail-closed and no-deprecation guardrails from the execution-identity test checklist. The mapping table is in e2e/identity/README.md.

Results (run against the stack tip 7b451e32 merged with main in a scratch worktree, against dogfood)

Local mode (your own token, no SP secret): 5 passed, 7 skipped, 0 failed. The 7 skipped tests compare SP vs user identity, which needs a distinct SP, so they skip with an explicit reason instead of passing for nothing. They have not run yet.

Run

cd apps/dev-playground
E2E_USER_TOKEN=$(databricks auth token --profile <profile> | jq -r .access_token) pnpm test:e2e

The other env vars are listed in the README.

Also in this PR

  • @playwright/test bumped from 1.61.0 to 1.63.0, plus playwright 1.63.0, to satisfy the peer range of @e2edev/web. dev-playground's existing Playwright integration tests still pass (29/29).
  • e2e and @e2edev/web pinned exact to the matching canary build. npm says @e2edev/web moved to @e2e-dev/web, but the Databricks npm proxy doesn't serve the new name yet.

Not covered

  • The deprecation check doesn't load genie or serving, which is where the original regression was.
  • agent.act UI steps are not wired yet (no model provider chosen).
  • B6 and Part A need a deployed app or the CLI.

This pull request and its description were written by Isaac.

Adds an e2e (TesterArmy) suite that checks which principal AppKit runs each
call as: the app service principal by default, the signed-in user on the OBO
path. A minimal test app is started in production mode and requests carry the
Databricks Apps proxy headers. All assertions are deterministic.

Covers checklist rows B1, B2, B3, B4, B5, B7 plus the fail-closed and
no-deprecation guardrails. Without an SP secret the suite runs in local mode
and skips the SP-vs-user comparisons with an explicit reason.

Requires the execution-identity stack (#592, #594, #595, #597, #601).

Also bumps @playwright/test to 1.63.0 (peer range of @e2edev/web).

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 37440274781 -R databricks/appkit -n appkit-template-0.82.0-pr.cb63e48-tests-e2e-execution-identity-630 -D appkit-pr-630 \
  && unzip -o "appkit-pr-630/appkit-template-0.82.0-pr.cb63e48-tests-e2e-execution-identity-630.zip" -d "appkit-pr-630" \
  && databricks apps init --template "appkit-pr-630"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant