Skip to content

LIVE_JSON_CONTRACT.md

Latest commit

 

History

History
152 lines (125 loc) · 7.25 KB

LIVE_JSON_CONTRACT.md

File metadata and controls

152 lines (125 loc) · 7.25 KB
Raw

--live-json structured telemetry contract (nexus-loadtest/live/v1)

loadtest --live-json emits newline-delimited JSON (NDJSON) on stdout — one self-describing event per line — instead of the human progress table. It lets an orchestrator consume live/structured telemetry (per-interval samples, error classification, run outcome) rather than scraping a human-readable line, which is brittle and loses TTFT + failure detail.

Default behavior is unchanged: without --live-json, the human table prints exactly as before.

Output & stream rules

  • stdout carries ONLY NDJSON. Every line is one complete JSON object; each line parses independently. The human report + file-path lines move to stderr.
  • stderr carries diagnostics only (banner, FD warnings, the final text report).
  • No secrets, ever. No event contains an Authorization header, bearer token, virtual key, request body, or prompt content. Events carry metrics + labels only.
  • Exit code: 0 = the load tool ran to completion (even if the target failed every request — that is reported in the summary, not the exit code); non-zero = the load tool itself could not run (bad config, unreachable from the start).

CLI

loadtest --config <profile> --live-json \
  [--run-id <id>] [--target-name <name>] [--target-mode <mode>] \
  [--target <url>] [--model <m>] [--vk <vk>] [--stages '@2000:300s']

--run-id / --target-name / --target-mode are labels only, stamped on every event so a consumer can attribute a multiplexed stream. They do not affect the load run. (--vk still sets the bearer token; it is never echoed in output.)

To emit per-interval sample events the profile must set a positive live_interval (e.g. "1s") and the stage must last at least two intervals; otherwise samples are suppressed (as with the human progress line).

Common envelope

Every event carries:

field type meaning
schema string always "nexus-loadtest/live/v1"
type string run_started | sample | error_summary | run_finished | run_failed
run_id string the --run-id label (may be empty)
ts string RFC3339Nano UTC emit time
target object { name, base_url, mode } — descriptive labels

Event types

run_started (once, first)

{ "schema":"nexus-loadtest/live/v1", "type":"run_started", "run_id":"...", "ts":"...",
  "target":{"name":"nexus-hooks-on","base_url":"http://...","mode":"hooks-on"},
  "config":{ "rps":2000, "duration_s":300, "concurrency":20, "streaming":false, "stages":1 } }

config echoes the requested shape (peak offered rate, total duration, peak concurrency, whether any scenario streams, stage count).

sample (per live_interval)

{ "type":"sample", "stage":1, "target":{...}, "phase":"measured",
  "metrics":{ "rps":1998.4, "lat_p50_ms":8.5, "lat_p99_ms":68.0, "ttft_p95_ms":22.0,
              "ok_pct":100.0, "error_rate":0.0 } }

The rolling-window observation for the last interval. ok_pct = 100 − err_pct; error_rate is a fraction.

phase is "warmup" or "measured", and is omitted when the window cannot be assigned to one. It sits at the top level, beside metrics, not inside it — it describes when the window was taken, not a measurement of the system under test, and consumers read it off the event.

The label applies to the whole window, [elapsed − live_interval, elapsed]:

window position phase
entirely within warmup "warmup"
entirely after warmup "measured"
straddling the boundary omitted
warmup is 0s "measured" (no warm window exists)

A straddling window contains both warm-up and steady-state requests and is honestly neither, so it carries no label rather than a confident wrong one — the field exists so a consumer can EXCLUDE warm-up, and a mislabelled window defeats that. Treat an absent phase as unknown; do not default it to "measured" and do not infer it from a neighbouring sample.

Note this marker is for interval-level analysis only. Stage summaries already exclude warm-up from steady-state figures, so run_finished percentiles are unaffected by it.

error_summary (per stage, only when the stage had failures)

{ "type":"error_summary", "target":{...},
  "errors":{ "total":250, "http_401":250, "http_403":0, "http_404":0, "http_429":0,
             "http_5xx":0, "timeouts":0, "connection_errors":0,
             "generator_port_exhaustion":0, "generator_fd_exhaustion":0, "other":0 },
  "classification_hint":"auth", "generator_side":false }

Every count is observed (status codes + error classes). classification_hint is the dominant bucket, one of: auth (401/403), route (404), upstream (5xx), capacity (429 or generator exhaustion), timeout, connection, unknown. It is a hint from client-side evidence, not a root cause — the tool never asserts a cause it cannot see. A clean stage emits no error_summary.

Honesty boundary — generator_side. "cannot assign requested address" (ephemeral-port exhaustion) and "too many open files" are this load tool's limits, not the gateway's. When such a bucket dominates, generator_side:true and classification_hint:"capacity". A consumer MUST map generator_side:true to a generator/rig limit (e.g. loadgen_limited) — never to a gateway failure like failed_capacity. Blaming the gateway for the generator's ceiling is the exact false-signal this flag exists to prevent.

run_finished (once, last, on a clean end)

{ "type":"run_finished", "status":"completed", "target":{...},
  "summary":{ "rps":2000.0, "lat_p50_ms":8.0, "lat_p99_ms":69.0, "ttft_p95_ms":24.0,
              "ok_pct":100.0, "error_rate":0.0, "requests_total":600000, "requests_ok":600000 } }

status is "completed" (natural end) or "interrupted" (SIGINT/SIGTERM — a clean early stop). Both mean the tool worked; how the target did is in the summary (ok_pct / error_rate). A 100%-failing target still yields status:"completed" with ok_pct:0.

run_failed (the load tool itself could not run)

{ "type":"run_failed", "status":"failed", "target":{...},
  "failure":{ "class":"configuration", "reason":"load config: ...",
              "generator_side":false, "http_status":null },
  "last_sample":{ ... }|null }

Emitted only for a runner-level fatal (bad profile, target unreachable from the start), distinct from a target that merely returned errors. class follows the same evidence rules (configuration for a setup/wiring failure the tool hit). The process also exits non-zero.

Versioning

schema is nexus-loadtest/live/v1. Additive fields (new metrics, new error buckets) do not bump the version — consumers must ignore unknown fields. A breaking change (renamed/removed field, changed semantics) bumps to v2.

Consumer guidance

  • Parse line-by-line; ignore blank lines and unknown types / fields.
  • Treat run_finished status:completed ok_pct:0 as the target failed, not as a tool failure — read error_summary/classification_hint for the why.
  • Honor generator_side:true — it is a rig limit, not a gateway verdict.
  • Never expect secrets; never log the raw stream expecting to find auth material.