Repository navigation
FEAT: Browse Scenario result parents and children in the GUI (#3602) - #3611
Roman Lutz (romanlutz) merged 14 commits into
Conversation
…ft#3602) Add a read-only "Result hierarchy" section to the Scenario run page, built on the hierarchy endpoints from microsoft#3167. Roots load in pages with the root cursor. Expanding a result loads its saved children in pages with that node's own children cursor, so the two never mix. Children are shown at their absolute position under the parent, and missing, duplicate or cyclic references stay visible in place as unresolved. Run and group totals are shown exactly as the backend reports them and are never counted from loaded rows. Rootless results are reported, not turned into roots. A stale cursor (HTTP 400) shows a reload action instead of a generic error. Details reuse the existing attempt dialog, and fall back to the clicked node when the progress feed has not delivered that result yet. Tests: unit tests for child positions and root merging, component tests for paging, stale cursors, reloads, totals and empty/truncated/rootless states, and a Playwright suite against a seeded backend (playwright.scenario-hierarchy.config.ts), added to the CI matrix.
|
@microsoft-github-policy-service agree |
Roman Lutz (romanlutz)
left a comment
There was a problem hiding this comment.
I found three functional issues and one presentation issue to address. Details are in the four inline comments.
… layout (microsoft#3602) - Cycles across fetched pages: each row now knows the results above it, and a fetched child that is one of them stays in its position as an unresolved reference instead of an expandable row. A single node response can't see the path, so the check lives in the tree. - Resumed runs: the page passes the current run status in. A run that is active again, or whose status changed since the hierarchy loaded, shows the snapshot note and Reload hierarchy. - Details objective: the hierarchy node's objective is used whenever its result ID matches the selected attempt, so a progress row for a seed missing from the plan no longer hides it. - Layout: each group is a table with Result / objective, Result type, Attack outcome, Duration, Retries and Actions. Nesting indents only the first column, so outcomes and numbers stay aligned. Outcomes read Success, Failure, Error or Undetermined, and a note explains that rows are saved attack results and numbered children are a compound attack's component results, not messages or retries. Tests: paged P -> A -> B -> A cycle, failed-to-resumed run (component and through the page's Resume action), objective with the progress row already loaded, and column alignment. Playwright suite updated for the table.
Extend the existing atomic attack-group table with saved parent/child rows, separate Details, and direct conversation links that retain Scenario and history context. Preserve backend summary counts and cancel stale descendant requests on parent reload. Seed real offline transcripts and cover navigation, paging, collapse, and cached-request regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Implemented the requested consolidation in Saved parent/child browsing now extends the existing Atomic attack groups table instead of adding a second results browser. Only the result/objective column is indented. Saved result links open their corresponding Chat directly, while Details remains a separate metadata action. An orchestration parent without its own conversation expands its children rather than inventing a chat. The links retain Scenario context and reuse the existing Chat breadcrumb, including after a full URL reload. The gap in #3602 was saved lineage and navigation, not the absence of a results table. The original brief acknowledged existing execution rows and shared viewers, but did not clearly require extending that existing surface. That allowed an overlapping presentation. The refreshed issue now makes consolidation and direct saved-Chat navigation explicit. The existing backend still owns grouping, accounting, and root/child pagination. Loading or collapsing rows does not recalculate its totals. No new transcript loader or scope-10B backend was added. The screenshots below were captured at the exact commit above using an isolated offline fixture with actual saved user and assistant messages. No external model was called. The narrow table scrolls horizontally to expose the remaining aligned columns. |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use the separate Details button in the existing history/provenance journey now that result rows primarily open saved Chat. Preserve all modal, URL, and breadcrumb assertions and all existing performance limits. Integrate main's dependency-only lock update normally. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate main's explicit config-file validation fix normally. All 313 affected CLI unit cases passed; this introduces no frontend or accounting changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate main's linear-time shell scorer and environment-example updates normally. Validate the complete affected scorer unit suite without changing frontend behavior or performance limits. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate the credential connection-string variants from microsoft#3639. The affected credential scorer suite and unchanged concurrent-message regression cases pass locally. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate microsoft#3766 without changing the scenario hierarchy implementation or its performance limit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate microsoft#3770, microsoft#3697 and microsoft#3695 normally. As requested, retain the upstream deterministic one-node mapping-budget assertion, truncation and exact totals; do not restore the removed wall-clock benchmark. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>



Description
Fixes #3602 (scope 10A). Extend the existing Atomic attack groups table with saved parent/child results using the hierarchy endpoints delivered by #3167. The overlapping standalone Result hierarchy section has been removed. There are no backend runtime or accounting changes in this PR's diff against current main.
attackConversationRoutePathand reuse the existing Chat breadcrumb. Conversation switching, URL normalization, and return navigation preserve History/suite state. A full saved-Chat URL reload still exposes the Scenario breadcrumb.children_cursor. Preserve absolute child-reference positions, including missing, duplicate, and ancestor-cycle references. Do not invent roots for rootless diagnostic samples.The gap was saved lineage and direct navigation, not the absence of an existing results table. The consolidation follows the refreshed #3602 scope; the implementation explanation records why the earlier brief allowed an overlapping presentation.
New bounded transcript/evidence rendering remains the separate scope-10B follow-up. This PR reuses the existing saved-Chat loader rather than adding another one.
Side note, not changed here: run-level
retriescounts each child of a compound attack as a re-attempt of the parent's unit (retry_pressureinpyrit/analytics/scenario_statistics.py), so a 520-step Sequential run shows "Retries: 520". Happy to open an issue if that isn't intended.Tests and Documentation
Added/updated coverage for consolidation, saved objectives, conversationless parents, direct Chat links, History/suite state, breadcrumb return, run switching, root/child paging, unresolved/cyclic references, late descendant cancellation, collapse caching, error/reload behavior, focus restoration, backend-count independence, and narrow touch/keyboard use. Updated
pyrit/backend/README.mdto describe the consolidated GUI and distinguish result records, child-reference positions, summary metrics, and existing Chat navigation.The offline browser fixture stores actual owned conversations with user/assistant messages, not result-only rows. It contains 524 result records and a parent with 520 child-reference positions, including missing and duplicate references. No external model was called by these local browser journeys.
Readiness fixes: integrated main with normal
git merge, without rebase or force-push. Full frontend CI exposed twoApp.labelsintegration cases whose API mock omittedgetRunHierarchy; added the complete hierarchy DTO fixture and retained all saved-label assertions. An existing scenario-history browser case still expected the result row to open Details. Its locator now selects the explicit Open details button, retaining every modal, URL, Chat, History, and breadcrumb assertion. Neither correction changes production behavior or removes assertions/timeouts.Incoming main #3770 separately replaced the old 50 ms wall-clock hierarchy assertion with a deterministic assertion that exactly one node is mapped, retaining truncation, one projected node, and the exact aggregate total. That already-landed change was explicitly accepted during integration. It checks the one-node mapping budget, not whole-traversal performance. No hierarchy producer implementation was changed here.
Final exact-head CI: all six PR workflows completed successfully at
fdede50985dec9b0550ed9880b5fc06b9d283d5a, which includes main5000620fe939154923e2559a4bd32050d5586ede. The full frontend gate collected 172 passing suites and 4,037 passing tests, with coverage of 94.55% statements, 89.71% branches, 93.7% functions, and 96.37% lines. Frontend lint/type and every browser shard passed. The complete Python matrix, including macOS variants, passed. All 28 required checks were independently revalidated as successful, with approval satisfied and all five review threads resolved.Local evidence: the production build and all 3 real saved-message hierarchy/Chat browser journeys passed at that same final head. The 108 affected registry, progress-read-model, deterministic mapping-budget, and documentation contract/workflow cases passed following the final main integration. Earlier focused integration selections also passed: 494 frontend cases, then 118 frontend cases and two reopened-editor Chat cases; 142 analytics/hierarchy cases; 25 scorer-service cases; 313 CLI cases; 117 shell scorer cases; 231 credential scorer cases plus three unchanged concurrent-message variants; and 19 converter/mapping selections. These selections overlap and are not added into a single total or substituted for final-head CI.
Exact local validation commands and results
Commands use Windows paths.
$artifactsis a session report directory. Frontend commands were run fromfrontend; Python commands were run from the repository root.uv run --no-sync npm.cmdensures Vite uses this worktree's Python environment rather than the Windows Store alias.uv sync --lockedPassed after incoming main changed
uv.lock.npm run type-check && npm run lintBoth passed after the latest frontend-changing main integration; both gates also passed in the final exact-head frontend workflow.
6 suites and 494 cases passed at the first main integration,
c8218e0bf2121f676c710b3f0f3e369c2afb8f27.17 passed after correcting the missing hierarchy API mock. The two original CI failures were independently reproduced first.
5 suites and 118 cases passed; both selected Chat cases passed. The Chat command did not select 263 other cases.
Fixture lint/format passed; 142 analytics/hierarchy cases and 25 scorer-service cases passed at their corresponding earlier integrations.
The focused existing browser journey passed after correcting its obsolete Details-row locator.
108 passed after the final main integration, covering the affected registry, scorer-metrics freshness, deterministic mapping-budget, and documentation contracts.
Production build and all 3 real-backend browser journeys passed at
fdede50985dec9b0550ed9880b5fc06b9d283d5a. They cover paging through 200/400/520 positions, nested/group/global collapse and cached totals, separate Details/focus, actual saved messages, full Chat URL reload/breadcrumb return, and a 390px touch/keyboard layout.JupyText was not run locally. Existing Vite configuration/chunk-size and backend model-field warnings remain outside this change.
Updated screenshots
Captured from the
5f9f605da58336ad07565f2910cea1579f41c00dbuild using isolated, synthetic saved data with real transcript rows. Later integrations changed backend/scorer/CLI/dependency/documentation code or test fixtures, not the pictured frontend runtime. These images are not relabelled as final-head captures. The desktop capture uses a fresh browser page, avoiding an exiting modal in the image.Desktop (1440 x 1050): one existing attack-group table with aligned compound parents, nested children, unresolved positions, outcomes, timestamps, and separate Details.
Narrow viewport (390 x 844): scrolled to the Sequential group; the remaining table columns are accessible by horizontal scrolling.
Saved Chat and Scenario breadcrumb (1440 x 1050): the selected Step 5 result's stored request/response, shown after a full URL reload with the existing return breadcrumb.