feat(config): consolidate all time knobs into settings.timeouts; make --timeout a real run watchdog

.pre-commit-config.yaml

@@ -58,7 +58,7 @@ repos:
         entry: uv run --no-sync python scripts/regenerate_templates.py
         language: system
         pass_filenames: false
-        files: ^(src/inference_endpoint/config/(schema\.py|templates/.*)|src/inference_endpoint/endpoint_client/config\.py|scripts/regenerate_templates\.py)$
+        files: ^(src/inference_endpoint/config/((schema|audit|model_params|datasets|settings|timeouts)\.py|templates/.*)|src/inference_endpoint/endpoint_client/config\.py|scripts/regenerate_templates\.py)$
 
       - id: add-license-header
         name: Add license headers

docs/CLI_DESIGN.md

@@ -67,7 +67,7 @@ Both paths produce the **same subclass with the same defaults**. A YAML file wit
 
 2. **Auto-selects subclass.** `type: "offline"` → `OfflineBenchmarkConfig`, `type: "online"` → `OnlineBenchmarkConfig`, others → base `BenchmarkConfig`.
 
-3. **Optional CLI overrides.** `--timeout` and `--mode` applied via `config.with_updates(...)` which re-runs validators.
+3. **Optional CLI overrides.** `--timeout` maps into `settings.timeouts.run_timeout_s` via `with_updates(...)` (re-runs validators); `--mode` selects the `TestMode` passed to the runner and never touches the config object.
 
 ### Why subclasses?
 
@@ -131,7 +131,6 @@ Validation is layered, executing in order:
  1. cyclopts        → required args? unknown flags?
  2. Pydantic fields → type coercion, ge/le constraints
  3. Sub-model validators:
-    ├── RuntimeConfig._validate_durations    → max >= min duration
     ├── LoadPattern._validate_completeness   → poisson needs qps, concurrency needs target
     └── HTTPClientConfig._workers_not_zero   → num_workers != 0
  4. BenchmarkConfig._resolve_and_validate:
@@ -204,5 +203,5 @@ class HTTPClientConfig(WithUpdatesMixin, BaseModel):
 `BenchmarkConfig` is frozen. Use `with_updates()` to produce new instances with re-validation:
 
 ```python
-config = config.with_updates(timeout=300, datasets=["new_data.jsonl"])
+config = config.with_updates(report_dir="results/run1", datasets=["new_data.jsonl"])
 ```

docs/CLI_QUICK_REFERENCE.md

@@ -96,17 +96,15 @@ Flag names shown as `--full.dotted.path --alias`. Both forms work.
 - `--model-params.max-new-tokens --max-output-tokens` - Max output tokens (default: 1024)
 - `--model-params.osl-distribution.min --min-output-tokens` - Min output tokens (default: 1)
 - `--model-params.streaming --streaming` - Streaming mode: auto/on/off (default: auto)
-- `--runtime.min-duration-ms --duration` - Min duration: ms default, or with suffix (600s, 10m) (default: 600000)
-- `--runtime.n-samples-to-issue --num-samples` - Explicit sample count override
+- `--runtime.n-samples-to-issue --num-samples` - Explicit sample count (omit to issue the dataset once — the default)
 - `--client.num-workers --workers` - HTTP workers (-1=auto, default: -1)
 - `--client.max-connections --max-connections` - Max TCP connections (-1=unlimited)
 - `--endpoint-config.api-key --api-key` - API authentication
 - `--endpoint-config.api-type --api-type` - API type: openai/sglang (default: openai)
 - `--report-dir` - Report output directory
-  Note: applies to CLI-driven `benchmark offline` / `benchmark online`; `benchmark from-config`
-  does not expose a CLI override for `report_dir`. Set it in the YAML only if you need to control
-  the output location; otherwise a default report directory is used.
-- `--timeout` - Global timeout in seconds
+  Note: `benchmark from-config` also accepts `--report-dir` as an override of the YAML value;
+  when neither is set a default report directory is used.
+- `--timeout` - Whole-run watchdog in seconds (off by default). If it fires, the run is aborted, the report is marked INTERRUPTED, and the process exits non-zero.
 - `--enable-cpu-affinity / --no-cpu-affinity` - NUMA-aware CPU pinning (default: true)
 - `--no-early-stopping` - opt out of the MLPerf early-stopping percentile estimates in `result_summary.json` (default: on; see [early_stopping.md](early_stopping.md))
 
@@ -118,6 +116,67 @@ Flag names shown as `--full.dotted.path --alias`. Both forms work.
 
 **All other schema fields** are accessible via dotted paths (e.g., `--model-params.temperature`, `--model-params.top-k`, `--runtime.scheduler-random-seed`). Run `--help` to see the full list.
 
+## Time Knobs
+
+All give-up deadlines live under `settings.timeouts`; the only workload duration is
+`settings.runtime.max_duration_ms`; endpoint-client worker lifecycle timeouts are client
+internals under `settings.client`. `null`/unset means "wait indefinitely" (or "off") everywhere.
+
+Where every knob acts over the life of a run:
+
+```text
+run_benchmark ── run_timeout_s deadline captured here ─────────────────────────────┐
+│                                                                                  │
+├─ setup: dataset + tokenizer load          (counts against run_timeout_s)         │
+├─ launch metrics/event-logger services     ── service_ready_timeout_s             │
+├─ start endpoint-client workers            ── client.worker_initialization_timeout│
+│                                                                                  │
+├─ WARMUP       issue ──────────┤ drain ─┤  ── warmup_drain_timeout_s              │
+│                                                                                  │
+├─ PERFORMANCE  issue ──────────┤ drain ─┤                                         │
+│               │               │        └─ performance_drain_timeout_s            │
+│               └───────────────┴─ max_duration_ms caps ISSUING only; reaching it  │
+│                                  ends the phase NORMALLY (valid report) and      │
+│                                  SKIPS the drain — the two never run together    │
+│                                                                                  │
+├─ ACCURACY     issue ──────────┤ drain ─┤  ── accuracy_drain_timeout_s            │
+│                                                                                  │
+├─ metrics drain (tokenize buffered ISL/OSL)── metrics_drain_timeout_s             │
+│                                              (expiry FAILS the run:              │
+│                                              complete: false + non-zero exit)    │
+├─ worker shutdown                          ── client.worker_graceful_shutdown_wait│
+│                                              then client.worker_force_kill_timeout
+└─ finalize: score accuracy, write artifacts                                       │
+                                                                                   │
+ run_timeout_s (whole-run watchdog) ───────────────────────────────────────────────┘
+ firing at ANY point above aborts the run: report marked INTERRUPTED, non-zero exit
+```
+
+| YAML path                                       | CLI flag                                 | Semantics                                                                                                                                                                        |
+| ----------------------------------------------- | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `settings.runtime.max_duration_ms`              | `--runtime.max-duration-ms`              | Caps performance-phase issuing (ms, or suffix: `600s`, `10m`); reaching it ends the phase NORMALLY — the report stays valid — and skips the performance drain                    |
+| `settings.timeouts.run_timeout_s`               | `--timeout`                              | Whole-run watchdog over everything above; firing aborts the entire run — report marked INTERRUPTED, non-zero exit                                                                |
+| `settings.timeouts.service_ready_timeout_s`     | `--service-ready-timeout`                | Wait for the metrics-aggregator/event-logger services to become ready (default 30)                                                                                               |
+| `settings.timeouts.warmup_drain_timeout_s`      | `--warmup-drain-timeout`                 | Bound on in-flight warmup requests after the warmup phase ends (default 240)                                                                                                     |
+| `settings.timeouts.performance_drain_timeout_s` | `--performance-drain-timeout`            | Bound on in-flight performance requests after the phase stops issuing (default: wait indefinitely)                                                                               |
+| `settings.timeouts.accuracy_drain_timeout_s`    | `--accuracy-drain-timeout`               | Bound on in-flight accuracy requests after the phase ends (default: wait indefinitely)                                                                                           |
+| `settings.timeouts.metrics_drain_timeout_s`     | `--metrics-drain-timeout`                | Budget for the metrics aggregator to finish tokenizing buffered samples after the run ends (default: wait indefinitely); expiring fails the run with `complete: false` artifacts |
+| `settings.client.worker_initialization_timeout` | `--client.worker-initialization-timeout` | Wait for endpoint-client worker processes to start (default 60)                                                                                                                  |
+| `settings.client.worker_graceful_shutdown_wait` | `--client.worker-graceful-shutdown-wait` | Post-run wait for workers to exit gracefully (default 0.5)                                                                                                                       |
+| `settings.client.worker_force_kill_timeout`     | `--client.worker-force-kill-timeout`     | Wait after the graceful window before force-killing workers (default 0.5)                                                                                                        |
+
+How the knobs compose:
+
+1. **`--num-samples` / dataset-once defines the work.** An explicit `runtime.n_samples_to_issue`
+   sets the sample count; omitting it issues the performance dataset once.
+2. **`runtime.max_duration_ms` caps performance-phase issuing** and ends the phase normally —
+   remaining samples are not issued, in-flight requests are abandoned (no drain), the report is
+   valid. It does not bound the drain: issuing and draining are consecutive, never concurrent.
+3. **Per-phase drain timeouts bound the post-phase wait** for requests still in flight after a
+   phase stops issuing on its own.
+4. **`timeouts.run_timeout_s` is the only total-wall-time bound** (setup, every phase, every
+   drain) — firing aborts the run, marks the report INTERRUPTED, and exits non-zero.
+
 ## Environment Variables
 
 **In YAML files** — use `${VAR}` or `${VAR:-default}` syntax:
@@ -224,14 +283,13 @@ inference-endpoint benchmark online \
   --report-dir production_report \
   -v
 
-# Or with duration (calculates samples from target_qps * duration)
+# Without --num-samples, the dataset is issued once (the default)
 inference-endpoint benchmark online \
   --endpoints https://api.production.com \
   --model Qwen/Qwen3-8B \
   --dataset prod_queries.jsonl \
   --load-pattern poisson \
   --target-qps 100 \
-  --duration 5m \
   --workers 16 \
   --report-dir production_report \
   -v
@@ -248,8 +306,8 @@ inference-endpoint init submission
 # 3. Run (YAML mode)
 inference-endpoint benchmark from-config \
   --config submission_template.yaml
-# Note: from-config only accepts --config, --timeout, and --mode via CLI.
-# Set report_dir in the YAML if you need a specific output location.
+# from-config accepts --config, --timeout, --mode, --accuracy-only, and
+# --report-dir; everything else comes from the YAML.
 ```
 
 ### Validate First
@@ -290,8 +348,7 @@ datasets:
 
 settings:
   runtime:
-    min_duration_ms: 600000 # 10 minutes
-    n_samples_to_issue: null # Optional: explicit sample count (null = auto-calculate)
+    n_samples_to_issue: null # Optional: explicit sample count (null = issue the dataset once)
     scheduler_random_seed: 42 # For Poisson/distribution sampling
     dataloader_random_seed: 42 # For dataset shuffling
   load_pattern:
@@ -331,8 +388,8 @@ Note: For submission configs, `model_params.name` is optional when `submission_r
 
 **Sample Count Control:**
 
-- Priority: `--num-samples` > calculated (target_qps × duration) > dataset size
-- Default duration: 600000ms (10 minutes)
+- `--num-samples` sets an explicit sample count; without it the dataset is issued once
+- Behavior change: bare configs (no `--num-samples`) now run the dataset once instead of deriving 10 minutes' worth of samples from the target QPS
 
 **Mode Requirements:**
 

docs/DEVELOPMENT.md

@@ -201,7 +201,7 @@ docs/short-description
 
 ## YAML Config Templates
 
-Config templates in `src/inference_endpoint/config/templates/` are auto-generated from schema defaults. When you change `config/schema.py`, regenerate them:
+Config templates in `src/inference_endpoint/config/templates/` are auto-generated from schema defaults. When you change any config schema module (`config/schema.py` and its sibling domain modules), regenerate them:
 
 ```bash
 uv run python scripts/regenerate_templates.py

docs/LOCAL_TESTING.md

@@ -74,8 +74,7 @@ Waiting for 5 responses...
 uv run inference-endpoint -v benchmark offline \
   --endpoints http://localhost:8765 \
   --model Qwen/Qwen3-8B \
-  --dataset tests/assets/datasets/dummy_1k.jsonl \
-  --duration 0
+  --dataset tests/assets/datasets/dummy_1k.jsonl
 
 # Production test with custom params and report generation
 uv run inference-endpoint -v benchmark offline \
@@ -97,7 +96,7 @@ Loading: dummy_1k.jsonl
 Loaded 1000 samples
 Mode: TestMode.PERF, QPS: 10.0, Responses: False
 Streaming: disabled (auto, offline mode)
-Min Duration: 0.0s, Expected samples: 1000
+Expected samples: 1000
 Scheduler: MaxThroughputScheduler (pattern: max_throughput)
 Connecting: http://localhost:8765
 Running...
@@ -115,7 +114,6 @@ uv run inference-endpoint -v benchmark online \
   --endpoints http://localhost:8765 \
   --model Qwen/Qwen3-8B \
   --dataset tests/assets/datasets/dummy_1k.jsonl \
-  --duration 0 \
   --load-pattern poisson \
   --target-qps 100 \
   --report-dir online_benchmark_report
@@ -128,7 +126,7 @@ Loading: dummy_1k.jsonl
 Loaded 1000 samples
 Mode: TestMode.PERF, QPS: 100.0, Responses: False
 Streaming: enabled (auto, online mode)
-Min Duration: 0.0s, Expected samples: 1000
+Expected samples: 1000
 Scheduler: PoissonDistributionScheduler (pattern: poisson)
 Connecting: http://localhost:8765
 Running...
@@ -311,9 +309,8 @@ uv run inference-endpoint benchmark online \
 
 **Sample Count Control:**
 
-- Use `--duration 0` when you want a local test to stop after exhausting the dataset instead of running for the default timed duration
-- Sample priority: `--num-samples` > dataset size (when `--duration 0`) > calculated (target_qps × duration)
-- Default duration: 600000ms (10 minutes)
+- By default (no `--num-samples`) a run stops after issuing the dataset once
+- Use `--num-samples` for an explicit sample count
 
 **Testing & Debugging:**
 

docs/async_utils/services/metrics_aggregator/DESIGN.md

@@ -122,7 +122,8 @@ COMPLETE event ─► trigger.fire ─► queue.enqueue(text, on_count)        [
 
 `--drain-timeout` and `--tokenizer-workers` have service-side defaults (`0`
 and `2`) so the service is launchable by hand without tuning knobs, but
-`config/schema.py` is the single source of truth: the benchmark always
+the config schema is the single source of truth (`settings.timeouts.metrics_drain_timeout_s`
+in `config/timeouts.py`, `settings.metrics_tokenizer_workers` in `config/settings.py`): the benchmark always
 forwards the schema values (`--metrics-drain-timeout`,
 `--metrics-tokenizer-workers`), overriding these defaults in normal runs.
 

docs/compliance_audit_plan.md

@@ -523,14 +523,12 @@ Two scenarios must be covered: **Offline** (`max_throughput`) and **SingleStream
 > catches a crashed run — but the examples default to equal for the clearest, least-contentious
 > comparison.
 
-> **`min_duration` is not a duration floor (current limitation).** The load-generator stop
-> check (`session.py`) halts a phase on **sample count** or **`max_duration_ms`** only;
-> `min_duration_ms` merely _derives_ a count when no explicit count is set. Because TEST04
-> drives an explicit `samples` count, each phase stops at `samples` and `min_duration_ms` is
-> **not** honored as a "run for at least 10 minutes" floor. MLCommons' 10-minute compliance
-> minimum therefore is **not** enforced today; combining a count floor with a duration floor
-> ("AND-semantics") is future work. Set `samples` large enough that each phase reaches a
-> stable throughput on its own.
+> **No duration floor (current limitation).** Runs are count-driven: the load-generator stop
+> check (`session.py`) halts a phase on **sample count** or **`runtime.max_duration_ms`**
+> only, and TEST04 drives explicit `samples` / `audit_samples` counts. MLCommons' 10-minute
+> compliance minimum therefore is **not** enforced today; combining a count floor with a
+> duration floor ("AND-semantics") is future work. Set `samples` large enough that each phase
+> reaches a stable throughput on its own.
 
 Both scenarios ship as committed configs (see also
 [`compliance/audit_test/README.md`](../src/inference_endpoint/compliance/audit_test/README.md)):