Skip to content

FEAT GUI: Configure a scenario from selected dataset examples #2759

Open

Description

@romanlutz

Is your feature request related to a problem? Please describe.

Selecting several dataset examples should lead to a configurable scenario, not concatenate unrelated prompts into chat or immediately call a target. Users need to see the exact selection and execution configuration before launching.

Work item 15 of 17 in #2744. Technical prerequisites: the table/selection IDs, the general scorer registry API from #2755/#2990, the exact-selection contract in #2757, and the scenario/default-selection policy in #2760. Keep not ready yet until promoted.

Describe the solution you'd like

Add explicit multi-select to the explorer and a Configure scenario action. Preserve selected IDs across pages, make selected-versus-visible counts clear, and allow reviewing/removing/clearing the selection. Do not turn a page selection into an implicit select-all-matching query.

For one example the existing chat import actions remain available. For multiple examples, use the scenario path rather than concatenation or a queue of chat drafts.

Hand the exact selected IDs and objective inputs to the existing Scanner form/estimate/launch flow through #2757/#2760. Target/scorer selection, preset seeding, dataset form pieces, shared parameters, single-technique editing, and launch confirmation already exist. Do not add a second configuration page. Prefill exactly the selected examples, prompt sending, no additional converters, and a compatible scorer default; collect missing required objectives and explain missing prerequisites.

Reuse ChatScorerPicker in Scenario mode and the configured instances from the existing scorer catalog, not every constructible /api/scorers/types entry. Submit the existing objective_scorer reference (registry_name and identifier_hash), retained by #2757. Keep exact identity checks when a scorer changes or the registry is reinitialized. Show useful descriptions, result-family information, relevant tags/default information, and objective-compatibility/prerequisite guidance.

Current main already projects descriptions, display_type, scorable_types, condition_types, evaluation identity/metrics, and a Chat-specific disabled reason. Core Scenario default resolution also exists. Reuse those fields and policy; add only genuinely missing registry-tag/default information or objective-specific explanations needed by this consumer. The current Scenario picker accepts the true_false family, which alone does not prove coverage of the selected examples' objective conditions. Do not equate Chat eligibility with Scenario objective compatibility or hardcode another compatibility table in the browser.

Explain empty/unusable registries, missing judge-target dependencies, stale references after reinitialization, and missing/incompatible defaults. Offer setup guidance or an explicit valid choice rather than silently substituting a scorer or treating a raw refusal detector as an objective-success scorer.

Preview the selected population and effective configuration before explicit launch confirmation. On launch, reuse the normal scenario queue/progress/history. Preserve explorer state when cancelling or returning. No new persistent named dataset or generated scenario source file is created by opening the form.

Acceptance criteria:

  • Selections survive pagination and can be reviewed/cleared without selecting hidden unrequested examples.
  • Configure scenario performs no target call and uses exact IDs, not a reconstructed dataset filter.
  • Default settings mean prompt sending only, no extra converters, and an explicit compatible scorer.
  • The picker distinguishes configured instances from buildable classes and shows tags, default selection, compatibility, and prerequisites from shared typed backend/core metadata.
  • Missing objectives, invalid/stale selections or scorer names, incompatible presets/defaults, missing dependencies, and empty configuration have clear validation and setup guidance.
  • Preview, estimate, confirmation, and launch agree on the exact selected examples and effective configured scorer; duplicate launch is prevented.
  • Cancel/Back/navigation preserve the user's browse state, and success goes to existing scenario progress/history.
  • Component/routing and any added metadata-projection tests cover configured versus buildable choices, defaults/tags, missing dependencies, stale references, and the full flow using mocked APIs and no privileged datasets.

Describe alternatives you've considered, if relevant

Do not build another launch page with duplicated state, a general scorer-builder GUI, batch chat behavior, or a direct launch from a selection checkbox. Advanced technique/converter controls are item 16; this item should deliver the useful default flow first.

Additional context

This item retains the selected-example consumer's metadata/setup-guidance acceptance from #2755. The general API, configured picker, source-pinned request, and core default handling are already present; the exact explorer selection/handoff and its objective compatibility acceptance are not. Coordinate any remaining projection with #2760 so UI and execution use the same policy.

Start with the Datasets table, frontend\src\components\Scenarios\ScenarioDetail.tsx and its extracted form/estimate/launch boundaries, shared parameter controls, and scenariosApi. Coordinate with #2653 instead of growing a second monolithic configuration component. Follow frontend style/test instructions.

Relationship to scenario dataset generation

This overlaps the scenario dataset plan, whose #3143 is already merged. ScenarioDatasetTable, useScenarioDatasets, shared request/settings, and scenario-owned ANY/SUBSET/FIXED options are shipped. Reuse those form/model pieces rather than add another named-dataset picker, limit contract, or browser-owned policy. Also coordinate existing form decomposition #2653/#3526 and active launch/rerun UI changes instead of creating competing form state.

A scenario source table selects named populations, limits, and fetch policies. This issue instead begins with exact example IDs chosen across explorer pages. Keep those IDs authoritative and reviewable; do not expose source/cap controls that silently replace or resample the selection. Multi-page selection, the configured-scorer picker, missing-objective handling, confirmation, and return-to-explorer state remain this issue's work through #2757/#2760.

Merged generation support (#2904/#3116) means stored generated objectives can participate like other seeds. Explicit scenario refresh is also shipped in #3206. Neither adds generator controls to this form or permits refresh to replace exact selected IDs. Opening/estimating must not generate data; attack and judging targets are not implicit generation targets. New generation controls or additional destructive-refresh UX are optional. This October 31, 2026 refresh leaves prerequisites and order intact.

Metadata

Metadata

Assignees

No one assigned

    Labels

    GUIUmbrella label for all feedback submitted via the Co-PyRIT GUIfeature-requestnot ready yetThis issue needs more definition or is blocked by a pending change.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions