stream-contract

stream-contract is a dependency-free, offline linter for captured OpenAI-style streaming responses. It checks the wire transcript as an ordered state machine, then reconstructs visible text and tool-call arguments for a human or CI job.

Problem

Individual data: lines can all be valid JSON while the stream is still broken: a sequence number can go backwards, a text delta can arrive after its done event, a tool name can arrive in a later chunk, or a proxy can truncate the terminal event. Client SDKs often surface those failures as an empty response, retry, or timeout.

stream-contract answers the earlier question: what exactly was wrong with this captured stream? It never calls a provider, executes model output, or uploads a trace.

Quick start

Python 3.10+ is required. The runtime has no third-party dependencies.

git clone https://github.com/thisbejim/stream-contract.git
cd stream-contract
python -m pip install .
stream-contract check fixtures/responses-valid.sse --profile responses

The shortest useful workflow is a file or a shell pipeline:

cat production-capture.sse | stream-contract check - --profile responses
stream-contract check run.jsonl --profile chat --format json > stream-report.json

Example output:

PASS fixtures/responses-valid.sse (responses, sse)
events: 9
text: 'Hello from a fixture.'
terminal: response.completed
metadata: sequence_count=9
findings: none

On a broken capture, diagnostics include stable codes and source locations:

FAIL fixtures/responses-broken.sse (responses, sse)
events: 4
findings:
  ERROR   sse.invalid_event: invalid JSON: Expecting ',' delimiter (line 1, column 107) (event 4, line 10)
  WARNING responses.sequence_gap: sequence_number jumps from 0 to 2 (event 2, line 4, sequence_number)
  ERROR   responses.delta_without_part: output delta references unopened content part (event 3, line 7)
  ERROR   responses.missing_terminal: stream has no response.completed, response.incomplete, response.failed, or error event

Exit codes are script-friendly:

• 0 — no errors (warnings are reported but do not fail by default);
• 1 — a contract error, or a warning when --strict is used;
• 2 — invalid arguments or unreadable input.

What it checks

Responses API profile

--profile responses checks the documented Responses event stream, including:

• SSE framing, JSON decoding, and optional [DONE] handling;
• integer sequence_number ordering and gaps;
• response.created/response.completed (or incomplete/failed) lifecycle;
• output-item and content-part add/done pairing;
• text, refusal, and reasoning delta accumulation;
• function-call argument fragments, required names, and final JSON validity;
• events after a terminal event and provider extension warnings.

Chat Completions profile

--profile chat checks OpenAI-compatible Chat Completions chunks, including:

• chunk IDs, choices, delta, and finish reasons;
• text reconstruction and usage-only chunks;
• [DONE] presence and ordering;
• tool-call accumulation by choice/tool index, including a late function.name fragment;
• final tool argument JSON validity.

The checker does not claim full provider compatibility. It validates the documented subset above and makes unsupported/unknown event types visible as warnings.

Input formats

SSE is the default and supports LF, CRLF, comments/keep-alives, multi-line data: fields, event: names, and captures that end without a final newline. Use --input-format sse to make that choice explicit.

JSONL is useful after a logger has stripped SSE framing: one JSON event object per line, or a small { "event": "...", "data": "{...}" } envelope. Use --input-format jsonl or let auto detect it.

--format json emits schema version 1 with status, stable finding objects, the reconstructed text, tool calls, terminal event, and profile metadata. This is suitable for CI annotations or a regression artifact.

Why this tool exists

The OpenAI Responses streaming reference defines sequence numbers and event lifecycles, but it does not provide a local capture linter. Real compatibility bugs include a tool name arriving after an arguments fragment (Vercel AI #14722), malformed JSON being dropped until a stream hangs (OpenAI Codex #31148), and a proxy combining two JSON objects into one SSE event (sub2api #1471).

Endpoint smoke tests such as CompatCanary and the OpenAI gpt-oss compatibility test are useful when a server is running. Parsers such as llm-sse normalize events for an application. stream-contract fills the smaller gap between them: a deterministic, local report over the exact bytes a client saw.

Privacy and safety

All processing is local. There is no account, API key, network request, telemetry, dashboard, or hosted database. Prompts, responses, tool definitions, and traces never leave the machine. Model-generated strings are parsed as data; no tool calls, shell commands, imports, archives, or URLs are executed.

Sanitize captures before committing them. The fixture corpus contains only synthetic data.

Development

python -m venv .venv
.venv/bin/python -m pip install -e '.[dev]'
.venv/bin/python -m pytest
.venv/bin/ruff check .
.venv/bin/python -m compileall -q src tests

The tests run entirely from checked-in fixtures. No provider credentials or network access are required.

Limitations

This is a wire-contract diagnostic, not an LLM quality evaluator, endpoint runner, SDK replacement, load tester, or compatibility certification suite. It does not calculate a Git diff, infer missing events, or decide whether a model's answer is good. Provider-specific extensions are intentionally reported rather than guessed at.

License

MIT. See LICENSE.