FEAT GUI: Load built-in datasets into memory on request #2753

Is your feature request related to a problem? Please describe.

Users need an explicit way to load a built-in dataset from the explorer. Some loads are long-running, gated, or asset-heavy, and a failed/repeated request must not misrepresent the dataset as fully loaded or create duplicates.

Work item 10 of 17 in #2744. Technical prerequisite: item 9's registered provider identity and supported-option contract. Keep not ready yet until promoted.

Describe the solution you'd like

Expose the explorer-specific explicit Load operation and status workflow over the existing registered-provider/preparation and CentralMemory persistence APIs. Scenario loading/preparation already exists; add only the adapter and operation behavior that the standalone explorer needs, not another provider lifecycle.

• Accept only a known provider and its supported, validated non-secret options. Do not accept arbitrary URLs, modules, code, or credential values.
• Expose a load-operation identifier and status so the GUI can handle long-running work without blocking navigation or the API event loop. Reuse existing runtime/service patterns instead of building a durable job platform.
• Report useful stages such as loading and persisting rather than invented percentage progress. Define terminal success/failure, partial-load reporting, and honest behavior if process-local status is lost after restart.
• Prevent duplicate simultaneous submissions for the same provider/options. Define retry/already-loaded behavior using existing persistence/deduplication behavior; do not silently overwrite user data or reload a different variant under the same identity.
• Respect application authorization, use the configured server credentials, and make shared-memory scope explicit. Distinguish missing credentials, gated-access denial, invalid options, upstream/network failure, and persistence failure without leaking secrets.

Acceptance criteria:

• Public and gated mock providers load through the existing provider-to-memory path.
• Browsing/GET requests never trigger this operation.
• Concurrent duplicate requests and retries have documented, tested behavior.
• Failure after partial work is not reported as complete success.
• Loaded summaries/rows become queryable through the existing explorer APIs.
• Tests use fake providers/local media and cover status, failures, authorization, and invalid requests without network datasets or keys.

Describe alternatives you've considered, if relevant

No new memory tables/migrations, separate dataset store, browser credential handling, arbitrary loader execution, or mandatory durable queue. Do not promise resumable/cancellable provider downloads unless the provider actually supports them.

Additional context

Start with the dataset backend service, SeedDatasetProvider, existing memory dataset insertion APIs, and current backend asynchronous operation/error conventions. Keep blocking third-party work off async request paths and follow Python/dataset/database/test instructions.

Relationship to scenario dataset generation

The scenario dataset plan already supplies preparation through merged #2956, configured providers through #3116, and shared source/fetch settings and GUI controls through merged #3143. Main also has /api/scenarios/catalog/{name}/datasets/load, scenariosApi.loadDatasets, and useScenarioDatasets. Reuse the existing validation, preparation, persistence, and error handling; do not run a scenario or build another provider lifecycle merely to load explorer data.

What remains here is a known-provider, validated-option operation usable without a Scenario form, with explicit status/duplicate/retry/partial-failure semantics for the explorer. The scenario-scoped missing-only Load action and final count response do not deliver that whole operation/status/access journey.

Keep this issue scoped to known built-in providers and validated non-secret options. A configured Python generation provider is not permission for the REST endpoint to accept arbitrary live provider objects, model instructions, or credentials. Dynamic generator configuration in the GUI is a separate optional follow-up, not part of this Load operation.

The merged if_missing path reuses stored named data, including manual edits. Explicit always scenario refresh is also merged in #3206, not future backend work. Ordinary explorer Load/retry must not silently become regeneration, replacement, or a destructive refresh fallback. Preserve the duplicate-submission, partial-failure, and access-guidance requirements above. This October 31, 2026 scope refresh changes no sequencing or readiness.