Skip to content

[Feature] Endpoint presets#4028

Draft
peterschmidt85 wants to merge 3 commits into
masterfrom
endpoint-presets
Draft

[Feature] Endpoint presets#4028
peterschmidt85 wants to merge 3 commits into
masterfrom
endpoint-presets

Conversation

@peterschmidt85

@peterschmidt85 peterschmidt85 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Problem

Creating a reliable model service requires selecting a model variant, serving configuration, and
hardware, then validating that combination. The result cannot currently be saved and reused as a
local endpoint recipe.

Change

  • Add agent-assisted endpoint preset creation and local preset storage.
  • Add commands to list, apply, and delete endpoint presets.
  • Add endpoint configuration, documentation, and focused coverage for the new lifecycle.

See the Endpoints concept page for the proposed UX.

WIP

  • High: For a base-model request, the supervisor accepts final_report.json.model without checking
    which repo/path the final service command loads. A preset can therefore advertise a different
    model from the one it deploys.
  • High: The supervisor accepts final_report.json.context_length without verifying it against the
    final service. An overstated value can make the recipe match endpoint requirements it does not
    satisfy.
  • High: final_report.json.service_yaml is required for success but is never parsed or used. Invalid
    YAML currently has no effect on preset creation.
  • High: The supervisor does not compare the final run with endpoint placement constraints such as
    fleets, backends, regions, spot policy, and maximum price. Preset creation can succeed after the
    agent violates the endpoint request.
  • High: Cleanup discovers submitted runs from agent-maintained submissions.jsonl. A run submitted
    before the journal entry is written can remain running and continue consuming resources.
  • High: The recipe is saved before run cleanup finishes. If cleanup fails, the command reports
    failure but leaves the recipe in the local store.
  • High: Claude stream-JSON is read with asyncio's default 64 KiB line limit. A larger tool result
    terminates preset creation with ValueError.
  • High: Preset serialization treats any environment key or value containing token, key,
    secret, or password as secret. Benign values such as TOKENIZERS_PARALLELISM=false are
    changed into unresolved environment references.
  • High: The agent can save mutable service images such as vllm/vllm-openai:latest. Reusing the
    preset can then run serving code different from what its validation evidence covered.
  • Medium: If the final report names a run that does not exist, the verifier raises an internal
    AttributeError instead of a useful CLI error.
  • Medium: Presets are persistent files under ~/.dstack/presets, but the stored format has no version
    or migration contract for future dstack releases.
  • Medium: Saved benchmark evidence can omit the final run UUID even though the supervisor knows it.
    Reused run names then make the evidence ambiguous.
  • Medium: Preset creation reads the authentication token from the private api.client._token field,
    coupling the command to an internal client implementation detail.
  • Medium: Local preset save, list, and delete operations have no cross-process lock. Concurrent CLI
    commands can race despite individual file replacement being atomic.
  • Medium: Preset directories replace / and \\ with --. Arbitrary base names can collide and can
    still contain filename characters invalid on Windows.
  • Low: Automatic apply selects the first recipe with available offers, ordered effectively by
    recipe ID. It has no defined performance, price, or quality selection policy.
  • Low: Multi-group preset listing prints the first group's resources on the recipe row and then
    prints that group again, producing duplicated and potentially confusing output.

AI assistance

AI assistance was used for implementation, tests, documentation, and E2E investigation.

@peterschmidt85 peterschmidt85 changed the title Add endpoint preset creation and reuse [Feature] Endpoint presets Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant