[Bug]: 500 server error when re-applying replica groups service

Steps to reproduce

Have an old replica group service submitted before Set explicit GPU defaults in ResourcesSpec and improve default GPU vendor selection #3573
Re-apply the same service with the same name – the server will fail with 500 and the following log:

  Top-level `resources` is not allowed when `replicas` is a list. Specify `resources` in each replica group instead. (type=value_error)
__root__
  Missing configuration (type=value_error)

Actual behaviour

The problem is the old run can no longer pass the replica groups check because default resources changed in #3573

dstack/src/dstack/_internal/core/models/configurations.py

Lines 1011 to 1018 in 8657913

    
           resources = values.get("resources") 
        
           default_resources = ResourcesSpec() 
        
           if resources and resources.dict() != default_resources.dict(): 
        
               raise ValueError( 
        
                   "Top-level `resources` is not allowed when `replicas` is a list. " 
        
                   "Specify `resources` in each replica group instead." 
        
               )

Expected behaviour

Although we can handle the error, the check itself is very fragile since it depends on default ResourcesSpec() staying the same.

dstack version

master

Server logs

File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/server/routers/runs.py", line 132, in get_plan
    run_plan = await runs.get_plan(
               ^^^^^^^^^^^^^^^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/server/services/runs/__init__.py", line 347, in get_plan
    current_resource = await get_run_by_name(
                       ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/server/services/runs/__init__.py", line 297, in get_run_by_name
    return run_model_to_run(run_model, return_in_api=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/server/services/runs/__init__.py", line 757, in run_model_to_run
    run_spec = get_run_spec(run_model)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/server/services/runs/__init__.py", line 117, in get_run_spec
    return RunSpec.__response__.parse_raw(run_model.run_spec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/main.py", line 572, in pydantic.main.BaseModel.parse_raw
  File "pydantic/main.py", line 549, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 364, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 2 validation errors for RunSpecResponse
configuration -> ServiceConfigurationResponse -> __root__
  Top-level `resources` is not allowed when `replicas` is a list. Specify `resources` in each replica group instead. (type=value_error)
__root__
  Missing configuration (type=value_error)

Additional information

No response

It seems services with replica groups not working at all after #3573 – not only re-apply. When applying a service I get CLI error:

Traceback (most recent call last):
  File "/Users/r4victor/Projects/dstack/dstack/.venv/bin/dstack", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/cli/main.py", line 95, in main
    args.func(args)
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/cli/commands/apply.py", line 98, in _command
    configurator.apply_configuration(
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/cli/services/configurators/run.py", line 105, in apply_configuration
    run_plan = self.api.runs.get_run_plan(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/api/_public/runs.py", line 520, in get_run_plan
    run_plan = self._api_client.runs.get_plan(self._project, run_spec)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/api/server/_runs.py", line 80, in get_plan
    return parse_obj_as(RunPlan.__response__, resp.json())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/tools.py", line 38, in pydantic.tools.parse_obj_as
  File "pydantic/main.py", line 364, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 2 validation errors for ParsingModel[RunPlanResponse]
__root__ -> effective_run_spec -> configuration -> ServiceConfigurationResponse -> __root__
  Top-level `resources` is not allowed when `replicas` is a list. Specify `resources` in each replica group instead. (type=value_error)
__root__ -> effective_run_spec -> __root__
  Missing configuration (type=value_error)

Apparently, this is because server modifies the resources by setting gpu vendor but the replica group check relies on default resources being unmodified.

Navigation Menu

Repository navigation

[Bug]: 500 server error when re-applying replica groups service #3676

Steps to reproduce

Actual behaviour

Expected behaviour

dstack version

Server logs

Additional information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	resources = values.get("resources")

	default_resources = ResourcesSpec()
	if resources and resources.dict() != default_resources.dict():
	raise ValueError(
	"Top-level `resources` is not allowed when `replicas` is a list. "
	"Specify `resources` in each replica group instead."
	)

[Bug]: 500 server error when re-applying replica groups service #3676

Description

Steps to reproduce

Actual behaviour

Expected behaviour

dstack version

Server logs

Additional information

Activity

r4victor commented on Mar 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions