Skip to content

FEAT: Browse Scenario result parents and children in the GUI (#3602) - #3611

Merged
Roman Lutz (romanlutz) merged 14 commits into
microsoft:mainfrom
FazeelUsmani:gui-scenario-result-hierarchy
Nov 1, 2026
Merged

Roman Lutz (romanlutz) merged 14 commits into
microsoft:mainfrom
FazeelUsmani:gui-scenario-result-hierarchy

Conversation

@FazeelUsmani

@FazeelUsmani Fazeel Usmani (FazeelUsmani) commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes #3602 (scope 10A). Extend the existing Atomic attack groups table with saved parent/child results using the hierarchy endpoints delivered by #3167. The overlapping standalone Result hierarchy section has been removed. There are no backend runtime or accounting changes in this PR's diff against current main.

  • Keep aligned result/objective, result-type, attack-outcome, duration, retry/error, timestamp, and action columns. Indent only the result/objective cell. Distinguish orchestration parents, target-facing results, and unknown legacy roles.
  • A saved result's primary link opens its corresponding existing Chat. Details remains a separate metadata action. Parents without their own conversation expand their saved children instead of inventing a transcript.
  • Retain Scenario context with attackConversationRoutePath and reuse the existing Chat breadcrumb. Conversation switching, URL normalization, and return navigation preserve History/suite state. A full saved-Chat URL reload still exposes the Scenario breadcrumb.
  • Root continuation uses the root cursor; a selected parent's continuation uses its own children_cursor. Preserve absolute child-reference positions, including missing, duplicate, and ancestor-cycle references. Do not invent roots for rootless diagnostic samples.
  • Preserve backend group identities and summary counts while loading, collapsing, and re-expanding rows. The header uses server producer counts rather than the loaded progress-row count. Loading/error states retain explicitly labelled flat execution rows.
  • Retain cached pages across group/global collapse, handle retryable and stale-cursor errors, and reload snapshots after Resume. Reloading a parent also cancels obsolete descendant requests so late responses cannot restore stale pages.
  • Details prefers the selected saved result's objective over a plan seed, while retaining current progress/score metadata.

The gap was saved lineage and direct navigation, not the absence of an existing results table. The consolidation follows the refreshed #3602 scope; the implementation explanation records why the earlier brief allowed an overlapping presentation.

New bounded transcript/evidence rendering remains the separate scope-10B follow-up. This PR reuses the existing saved-Chat loader rather than adding another one.

Side note, not changed here: run-level retries counts each child of a compound attack as a re-attempt of the parent's unit (retry_pressure in pyrit/analytics/scenario_statistics.py), so a 520-step Sequential run shows "Retries: 520". Happy to open an issue if that isn't intended.

Tests and Documentation

Added/updated coverage for consolidation, saved objectives, conversationless parents, direct Chat links, History/suite state, breadcrumb return, run switching, root/child paging, unresolved/cyclic references, late descendant cancellation, collapse caching, error/reload behavior, focus restoration, backend-count independence, and narrow touch/keyboard use. Updated pyrit/backend/README.md to describe the consolidated GUI and distinguish result records, child-reference positions, summary metrics, and existing Chat navigation.

The offline browser fixture stores actual owned conversations with user/assistant messages, not result-only rows. It contains 524 result records and a parent with 520 child-reference positions, including missing and duplicate references. No external model was called by these local browser journeys.

Readiness fixes: integrated main with normal git merge, without rebase or force-push. Full frontend CI exposed two App.labels integration cases whose API mock omitted getRunHierarchy; added the complete hierarchy DTO fixture and retained all saved-label assertions. An existing scenario-history browser case still expected the result row to open Details. Its locator now selects the explicit Open details button, retaining every modal, URL, Chat, History, and breadcrumb assertion. Neither correction changes production behavior or removes assertions/timeouts.

Incoming main #3770 separately replaced the old 50 ms wall-clock hierarchy assertion with a deterministic assertion that exactly one node is mapped, retaining truncation, one projected node, and the exact aggregate total. That already-landed change was explicitly accepted during integration. It checks the one-node mapping budget, not whole-traversal performance. No hierarchy producer implementation was changed here.

Final exact-head CI: all six PR workflows completed successfully at fdede50985dec9b0550ed9880b5fc06b9d283d5a, which includes main 5000620fe939154923e2559a4bd32050d5586ede. The full frontend gate collected 172 passing suites and 4,037 passing tests, with coverage of 94.55% statements, 89.71% branches, 93.7% functions, and 96.37% lines. Frontend lint/type and every browser shard passed. The complete Python matrix, including macOS variants, passed. All 28 required checks were independently revalidated as successful, with approval satisfied and all five review threads resolved.

Workflow Exact-head result
Frontend Tests Passed
build_and_test Passed
coverage Passed
build-book Passed
docker_build Passed
CodeQL Passed

Local evidence: the production build and all 3 real saved-message hierarchy/Chat browser journeys passed at that same final head. The 108 affected registry, progress-read-model, deterministic mapping-budget, and documentation contract/workflow cases passed following the final main integration. Earlier focused integration selections also passed: 494 frontend cases, then 118 frontend cases and two reopened-editor Chat cases; 142 analytics/hierarchy cases; 25 scorer-service cases; 313 CLI cases; 117 shell scorer cases; 231 credential scorer cases plus three unchanged concurrent-message variants; and 19 converter/mapping selections. These selections overlap and are not added into a single total or substituted for final-head CI.

Exact local validation commands and results

Commands use Windows paths. $artifacts is a session report directory. Frontend commands were run from frontend; Python commands were run from the repository root. uv run --no-sync npm.cmd ensures Vite uses this worktree's Python environment rather than the Windows Store alias.

uv sync --locked

Passed after incoming main changed uv.lock.

npm run type-check && npm run lint

Both passed after the latest frontend-changing main integration; both gates also passed in the final exact-head frontend workflow.

npm test -- --ci --runInBand --runTestsByPath src\components\Scenarios\ScenarioRunPage.test.tsx src\components\Scenarios\ScenarioResultHierarchy.test.tsx src\components\Scenarios\scenarioHierarchy.test.ts src\components\Chat\ChatWindow.test.tsx src\App.test.tsx src\utils\routeParams.test.ts --json --outputFile "$artifacts\integration-jest-results.json"

6 suites and 494 cases passed at the first main integration, c8218e0bf2121f676c710b3f0f3e369c2afb8f27.

npm test -- --ci --runInBand --runTestsByPath src\App.labels.test.tsx --json --outputFile "$artifacts\labels-after-results.json"

17 passed after correcting the missing hierarchy API mock. The two original CI failures were independently reproduced first.

npm test -- --ci --runInBand --runTestsByPath src\utils\conversationDraft.test.ts src\utils\routeParams.test.ts src\components\Scenarios\ScenarioResultHierarchy.test.tsx src\components\Scenarios\scenarioHierarchy.test.ts src\App.labels.test.tsx --json --outputFile "$artifacts\latest-main-jest-results.json"
npm test -- --ci --runInBand --runTestsByPath src\components\Chat\ChatWindow.test.tsx --testNamePattern 'text-only video remix' --json --outputFile "$artifacts\latest-main-chat-results.json"

5 suites and 118 cases passed; both selected Chat cases passed. The Chat command did not select 263 other cases.

uv run --no-sync ruff check frontend\e2e\fixtures\scenario_hierarchy_backend.py
uv run --no-sync ruff format --check frontend\e2e\fixtures\scenario_hierarchy_backend.py
uv run --no-sync pytest tests\unit\analytics\test_scenario_statistics.py tests\unit\backend\test_scenario_hierarchy_pagination.py tests\unit\backend\test_scenario_hierarchy_parity.py -q
uv run --no-sync pytest tests\unit\backend\test_scorer_service.py -q

Fixture lint/format passed; 142 analytics/hierarchy cases and 25 scorer-service cases passed at their corresponding earlier integrations.

$env:E2E_FRONTEND_PORT = '18544'
$env:PYRIT_BACKEND_URL = 'http://127.0.0.1:18539'
uv run --no-sync npm.cmd run test:e2e -- scenario-history.spec.ts --project mock --grep 'preserves filtered history' --workers 1 --reporter line --output "$artifacts\history-navigation-e2e-results"

The focused existing browser journey passed after correcting its obsolete Details-row locator.

uv run --no-sync pytest tests\unit\registry\test_registry.py tests\unit\backend\test_scenario_progress_read_model.py tests\unit\backend\test_scenario_hierarchy_parity.py::test_hierarchy_traversal_halts_immediately_on_budget_exhaustion tests\unit\build_scripts\test_docs_build_contract.py tests\unit\build_scripts\test_docs_workflow.py -q --disable-warnings

108 passed after the final main integration, covering the affected registry, scorer-metrics freshness, deterministic mapping-budget, and documentation contracts.

$env:PYRIT_E2E_HIERARCHY_PORT = '18545'
uv run --no-sync npm.cmd run test:e2e:scenario-hierarchy -- --reporter line --output "$artifacts\metrics-docs-e2e-results"

Production build and all 3 real-backend browser journeys passed at fdede50985dec9b0550ed9880b5fc06b9d283d5a. They cover paging through 200/400/520 positions, nested/group/global collapse and cached totals, separate Details/focus, actual saved messages, full Chat URL reload/breadcrumb return, and a 390px touch/keyboard layout.

JupyText was not run locally. Existing Vite configuration/chunk-size and backend model-field warnings remain outside this change.

Updated screenshots

Captured from the 5f9f605da58336ad07565f2910cea1579f41c00d build using isolated, synthetic saved data with real transcript rows. Later integrations changed backend/scorer/CLI/dependency/documentation code or test fixtures, not the pictured frontend runtime. These images are not relabelled as final-head captures. The desktop capture uses a fresh browser page, avoiding an exiting modal in the image.

Desktop (1440 x 1050): one existing attack-group table with aligned compound parents, nested children, unresolved positions, outcomes, timestamps, and separate Details.

Consolidated Atomic attack groups with aligned parent and child rows on desktop

Narrow viewport (390 x 844): scrolled to the Sequential group; the remaining table columns are accessible by horizontal scrolling.

Consolidated Sequential group at 390 pixels with horizontally scrollable result columns

Saved Chat and Scenario breadcrumb (1440 x 1050): the selected Step 5 result's stored request/response, shown after a full URL reload with the existing return breadcrumb.

Actual saved Step 5 messages with the existing Scenario return breadcrumb

…ft#3602)

Add a read-only "Result hierarchy" section to the Scenario run page,
built on the hierarchy endpoints from microsoft#3167.

Roots load in pages with the root cursor. Expanding a result loads its
saved children in pages with that node's own children cursor, so the
two never mix. Children are shown at their absolute position under the
parent, and missing, duplicate or cyclic references stay visible in
place as unresolved. Run and group totals are shown exactly as the
backend reports them and are never counted from loaded rows. Rootless
results are reported, not turned into roots.

A stale cursor (HTTP 400) shows a reload action instead of a generic
error. Details reuse the existing attempt dialog, and fall back to the
clicked node when the progress feed has not delivered that result yet.

Tests: unit tests for child positions and root merging, component tests
for paging, stale cursors, reloads, totals and empty/truncated/rootless
states, and a Playwright suite against a seeded backend
(playwright.scenario-hierarchy.config.ts), added to the CI matrix.
@FazeelUsmani

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree

@romanlutz Roman Lutz (romanlutz) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found three functional issues and one presentation issue to address. Details are in the four inline comments.

… layout (microsoft#3602)

- Cycles across fetched pages: each row now knows the results above it,
  and a fetched child that is one of them stays in its position as an
  unresolved reference instead of an expandable row. A single node
  response can't see the path, so the check lives in the tree.
- Resumed runs: the page passes the current run status in. A run that
  is active again, or whose status changed since the hierarchy loaded,
  shows the snapshot note and Reload hierarchy.
- Details objective: the hierarchy node's objective is used whenever its
  result ID matches the selected attempt, so a progress row for a seed
  missing from the plan no longer hides it.
- Layout: each group is a table with Result / objective, Result type,
  Attack outcome, Duration, Retries and Actions. Nesting indents only the
  first column, so outcomes and numbers stay aligned. Outcomes read
  Success, Failure, Error or Undetermined, and a note explains that rows
  are saved attack results and numbered children are a compound attack's
  component results, not messages or retries.

Tests: paged P -> A -> B -> A cycle, failed-to-resumed run (component and
through the page's Resume action), objective with the progress row
already loaded, and column alignment. Playwright suite updated for the
table.
Extend the existing atomic attack-group table with saved parent/child rows, separate Details, and direct conversation links that retain Scenario and history context. Preserve backend summary counts and cancel stale descendant requests on parent reload. Seed real offline transcripts and cover navigation, paging, collapse, and cached-request regressions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@romanlutz

Copy link
Copy Markdown
Contributor

Implemented the requested consolidation in 455764a38ee1b513e9a18e32092b7131e1aef2a6.

Saved parent/child browsing now extends the existing Atomic attack groups table instead of adding a second results browser. Only the result/objective column is indented. Saved result links open their corresponding Chat directly, while Details remains a separate metadata action. An orchestration parent without its own conversation expands its children rather than inventing a chat. The links retain Scenario context and reuse the existing Chat breadcrumb, including after a full URL reload.

The gap in #3602 was saved lineage and navigation, not the absence of a results table. The original brief acknowledged existing execution rows and shared viewers, but did not clearly require extending that existing surface. That allowed an overlapping presentation. The refreshed issue now makes consolidation and direct saved-Chat navigation explicit.

The existing backend still owns grouping, accounting, and root/child pagination. Loading or collapsing rows does not recalculate its totals. No new transcript loader or scope-10B backend was added.

The screenshots below were captured at the exact commit above using an isolated offline fixture with actual saved user and assistant messages. No external model was called. The narrow table scrolls horizontally to expose the remaining aligned columns.

Consolidated Atomic attack groups with aligned parent and child rows on desktop

Consolidated Sequential group at 390 pixels with horizontally scrollable result columns

Actual saved Step 5 messages with the existing Scenario return breadcrumb

Roman Lutz (romanlutz) and others added 10 commits October 31, 2026 22:36
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use the separate Details button in the existing history/provenance journey now that result rows primarily open saved Chat. Preserve all modal, URL, and breadcrumb assertions and all existing performance limits. Integrate main's dependency-only lock update normally.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate main's explicit config-file validation fix normally. All 313 affected CLI unit cases passed; this introduces no frontend or accounting changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate main's linear-time shell scorer and environment-example updates normally. Validate the complete affected scorer unit suite without changing frontend behavior or performance limits.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate the credential connection-string variants from microsoft#3639. The affected credential scorer suite and unchanged concurrent-message regression cases pass locally.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate microsoft#3766 without changing the scenario hierarchy implementation or its performance limit.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate microsoft#3770, microsoft#3697 and microsoft#3695 normally. As requested, retain the upstream deterministic one-node mapping-budget assertion, truncation and exact totals; do not restore the removed wall-clock benchmark.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@romanlutz
Roman Lutz (romanlutz) merged commit f3a88da into microsoft:main Nov 1, 2026
66 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FEAT GUI: Browse Scenario result parents and children

2 participants