Skip to content

FEAT: Resolve exact dataset example selections for scenarios #2757

Open

Description

@romanlutz

Is your feature request related to a problem? Please describe.

Chosen examples must remain those exact logical examples through preview, estimate and launch. Named-source/filter/cap requests cannot encode that intent, and IDs alone cannot detect changed membership/content.

Part of #2744. Technical prerequisites: completed #2748 browsing identity and #2755 scorer API. #2756 is not a technical dependency. Preferred release still puts provider loading first; keep not ready yet until explicitly promoted.

Describe the solution you'd like

Add the missing bounded typed exact-selection contract and shared resolver. Retain API-provided selection_key plus example_id; group IDs can span datasets, so never widen to another dataset's members.

Define canonical selection freshness from complete membership and relevant stored content/configuration, not IDs or previews alone. A server-provided membership/content fingerprint or equivalent validated manifest must detect removed/added members and changes to values, roles/sequence, media references and objective conditions. Resolve canonical records from memory, never arbitrary browser paths/content.

Exact mode is mutually exclusive with source/filter/sampling/fetch overrides that alter its population. Do not apply default five-item caps, resample, create a persistent selection dataset or regenerate missing records. Invalid/duplicate/stale/missing/unsupported selections need explicit errors, not dropped or replaced rows.

Reuse estimate/run models and objective_scorer: RegisteredComponentReference with registry_name and identifier_hash. Keep stale-instance checks. This issue owns selected inputs/resolution, not a new picker/default policy/compatibility table. #2760 and shared scorer/core policy own selected-population execution compatibility. Retain explicit objective inputs for examples that need them.

Acceptance criteria:

  • N dataset-scoped references resolve to N complete examples with roles/order, source IDs/hashes, typed conditions, media and provenance.
  • Same-ID changed membership/content, cross-dataset group IDs, duplicates/missing references and conflicting modes are tested.
  • More than five selected examples are not capped; a supported request bound errors rather than truncates.
  • Preview/estimate/launch use the same effective selection/freshness and exact explicit scorer. Stale references require refresh or an explicit new choice.
  • Resolution/estimates are read-only: no provider calls, generation, writes, legacy template-file loading or premature rendering.
  • Existing named-dataset/scorer behavior remains compatible; focused resolver/contract tests use portable existing memory APIs, no migrations.

Describe alternatives you've considered, if relevant

No copied persistent dataset, entire-dataset browser transfer, serialized live Python objects, name-only scorer field or DTO-specific scorer rules. #2760 is the later consumer, not a prerequisite for completing this contract.

Additional context

Start with catalog scenario models, ScenarioConfigurationResolver, SeedRecord, dataset-scoped browse reads and inline DatasetAttackConfiguration groups. Inspection records are not already execution-ready components; preserve configuration as configuration and validate supported shapes at setup.

Scorer source resolution, defaults and condition helpers already exist. Refresh from #3206 must invalidate stale selections, not replace them. Reuse normal scenario/result persistence. Scope narrowed and technical dependencies corrected November 4, 2026.

Metadata

Metadata

Assignees

No one assigned

    Labels

    GUIUmbrella label for all feedback submitted via the Co-PyRIT GUIfeature-requestnot ready yetThis issue needs more definition or is blocked by a pending change.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions