FEAT GUI: Expose built-in dataset provider metadata #2754

Is your feature request related to a problem? Please describe.

The explorer should help users discover built-in datasets that are not loaded yet. Provider names alone do not explain modalities, size, variants, sources, or whether credentials and gated access are required.

Work item 9 of 17 in #2744. Technical prerequisites: the dataset identity/summary contract and catalog from items 1 and 3. This starts the provider-loading milestone after item 8; keep not ready yet until promoted.

Describe the solution you'd like

Extend the existing dataset catalog with the richer metadata-only built-in-provider descriptors it still lacks. GET /api/datasets?loaded_only=false already combines registered provider names and stored datasets, with loaded, provider_available, stored origins, logical-example/piece/objective counts, and selection_key. Do not introduce another catalog or repeat those fields under new names.

The standalone Datasets page currently uses the loaded-only catalog. Add the provider-descriptor/loaded-versus-available integration to that existing surface while preserving its routes, opaque selection keys, browse state, and count semantics.

• Distinguish in-memory datasets from available providers and expose a loaded/available filter without confusing name collisions or partial/variant loads.
• Include available description, source URL, license/provenance, modality, harm categories, size information, and supported variants/options. Missing information remains unknown.
• Expose only documented, supported non-secret load options. Use registered provider identities; do not accept arbitrary Python paths, code, or unvalidated kwargs from clients.
• Describe credential and external gated-access prerequisites without returning credential values. A configured credential is not proof that its account has dataset access.
• Keep enumeration side-effect free: no dataset materialization, downloads, model calls, or credential-dependent fetches merely to display cards.

Acceptance criteria:

• Loaded-only, provider-only, and combined views distinguish provenance and availability correctly.
• Known provider metadata is displayed without confusing an estimate or coarse size category with an exact loaded count.
• Missing metadata or an unsupported load option does not break the whole catalog.
• Gated/public examples can be described without real credentials or remote fetches.
• Existing name-list clients remain compatible and metadata/read-only tests are included.

Describe alternatives you've considered, if relevant

Avoid a hardcoded frontend dataset list, a second provider registry, fetching every dataset for metadata, or a generic code-execution/loader endpoint. This issue does not load datasets; item 10 adds that operation.

Additional context

Start with SeedDatasetProvider, SeedDatasetMetadata, provider metadata, and the dataset backend service/models. Preserve the framework rule that downstream consumers read seeds from memory after loading. Coordinate count provenance with #2735 and follow dataset/Python/model/test instructions.

Relationship to scenario dataset generation

Coordinate with the scenario dataset plan. #3143 is merged, including shared catalog counts/provider availability, aggregate stored origins, and scenario source metadata. Reuse those shipped projections. The genuine additions here are richer built-in-provider descriptions, supported non-secret options, access prerequisites, and their integration into the existing explorer, not another discovery path.

Stored origin and provider availability are different facts. Current seed details already expose SeedRecord.origin; a dataset can contain multiple origins or unknown provenance. Do not label a dataset generated/local/remote solely from its name or registered provider. TargetObjectiveProvider from #2904 is intentionally excluded from automatic registration/discovery; configured-provider support in #3116 does not make it a normal unloaded built-in card. Do not enumerate or call generation targets to fill metadata, or infer that a stored generated dataset can be regenerated from its provenance alone.

Generator configuration UI and richer generic generation capabilities remain optional follow-ups. Scope refreshed against main 791e12c6e5e4302e7c1211af6af8973aa6fe1f37 on October 31, 2026. These links do not promote this item or change the existing sequence.