wip: post-merge sync of 0.3.0->0.4.0 migration working tree - #5
Closed
maltsev-dev wants to merge 1 commit into
Closed
wip: post-merge sync of 0.3.0->0.4.0 migration working tree#5maltsev-dev wants to merge 1 commit into
maltsev-dev wants to merge 1 commit into
Conversation
PR #3 (Wip/working tree 2026 06 18) merged 1e9e9c0 with the CI-fix commit and the byte-mismatch fix on top, but the actual 0.3.0->0.4.0 migration diff (the 45-file cleanup that this PR was named after) appears to have been stripped during the merge - the working tree at HEAD still shows the full diff vs the initial import. Re-stage the same content here so the audit-trail, examples, decorators cleanup, and 14 new test files are actually on master. The 0.3.0->0.4.0 migration is the same one captured in wip/working-tree-2026-06-18 branch (commit 1244901). Per the CHANGELOG the migration removes PoolConfig/AdaptivePool, the gRPC transport, the signal.signal hijack, the protos/nullrun/v1/track.proto file, six zombie exception types, the decision_history/flow/gate/common placeholder modules, patch_openai/unpatch_openai, and _decision_history._organization_id_var / _api_key_id_var. The other 14 added test files and the analyze.md plan live in the same diff and are part of the 0.4.0 production-readiness release. If this lands in the same place as PR #3 already did, it is a no-op for master. If something went wrong in the merge and the diff was never persisted, this commit restores it. Validation: local 'pytest tests/' on this commit will rerun the test collection chain (decorators -> instrumentation.langgraph -> langchain_core) which is the reason the CI fix is also in this branch; without langchain-core in [dev] the collection errors out across all 20+ test files.
maltsev-dev
added a commit
that referenced
this pull request
Jun 20, 2026
…erage reporter (#26) * fix: P0 security/stability hardening bundle Closes the P0/P1/P2/P3 issues from the security review (plan §10/§11.4). Security / PCI-DSS / GDPR - P0-1: Mask positional PII in `_enforce_sensitive_tool` by introspecting the wrapped function's signature and applying `SENSITIVE_ARG_KEYS` to positional params. Pre-fix, `charge("4111-…-1111", 50)` forwarded the PAN into `/execute` and the audit log. - P0-6 / P3-3: `_safe_repr` now redacts BEFORE truncating. The pre-fix order truncated first, so `details={…}` past position 50 leaked verbatim. `_safe_repr` is now the single source of truth for the redact-then-truncate flow. Cost-audit / reliability - P0-3: Bounded chunked reads on the sync + async httpx transports (`MAX_RESPONSE_BYTES`, default 16 MiB, `NULLRUN_MAX_RESPONSE_BYTES` env override). Above the cap, tracking is skipped and `_coverage_streaming_skipped` is incremented. Replaces the `response.read()` / `await response.aread()` unbounded buffer that held entire LLM streaming bodies in memory. - P0-4: `_do_flush_locked` re-queue on CB OPEN now drops the NEWEST non-critical events instead of the oldest. The oldest events (incident start, billing-period start) are exactly what a billing investigator needs; losing them silently broke monthly rollups. Control-plane events (`state_change`, `kill_received`, `policy_invalidated`, `key_rotated`) are preserved unconditionally so the dashboard KILL switch lands even under sustained backend outage. Identity - S-8 / P2-4: `agent()` now emits `str(uuid.uuid4())` (with dashes). Pre-fix the format was `f"agent-{uuid.uuid4().hex}"` — 32 hex chars, no dashes — and backend UUID-typed columns dropped these to NULL on insert. User-supplied names are still preserved verbatim. - §7.2 #16: `workflow()` context manager now resets `span_id` (not only `workflow_id` / `trace_id`) so nested `with span()` blocks don't leave the inner span_id visible inside the workflow scope. Resource leaks - S-9: `_active_runs` on `NullRunCallback` is now an `OrderedDict` capped at 4096 with FIFO eviction. Pre-fix the dict grew unbounded when `on_chain_end` did not fire (some LangChain versions short-circuit the end hook on chain-body errors). - S-10: WebSocket reconnect loop is now capped at 10 consecutive failures, then falls back to HTTP-poll. Pre-fix the loop ran forever when the backend was permanently down, leaking the WS thread. Transport - §7.2 #6: Separate `hmac_verify_expired_total` counter so SRE can distinguish clock-skew (NTP drift) from forged packets. Mirrored in both the HTTP and WebSocket verify paths. - §7.2 #35: `CircuitBreaker.call` now dispatches the OPEN→HALF_OPEN jitter through `_maybe_apply_open_jitter_sync` / `_maybe_apply_open_jitter_async`. Pre-fix the jitter used `time.sleep` before dispatching to async, which blocked the caller's event loop on every transition. - P2-1: `_coverage_seen` now bumps in the httpx path (sync + async). Pre-fix the counter was only bumped by the `requests` transport, so the dashboard's coverage view was empty for the dominant OpenAI / Anthropic / Gemini / Mistral / Cohere traffic. - P2-3: `is_sensitive_tool` match is case-insensitive. Pre-fix `"stripe.charge"` did not match `"Stripe.Charge"`, bypassing the sensitive gate. Concurrency - §7.2 #39: New `_tools_lock` guards every mutation of `_strict_mode_tools` / `_sensitive_tools`. Same lock guards the coverage-counter bump+prune sequence (§7.2 #33) so two threads can't both observe the dict at length 4095 and both grow it to 4097 before either prune lands. - §7.2 #47: New `_langchain_lock` / `_langgraph_lock` guard the patch sequences end-to-end. Pre-fix two threads racing through `auto_instrument` could both pass the early `_x_patched` check and double-wrap `BaseCallbackManager` / `Pregel`. - §7.2 #33: `_COVERAGE_CAP` (4096) bounds the per-host coverage dicts. Webhook delivery - P3-2: Exponential backoff (0.5s, 1s, 2s, 4s, 8s, 16s, 30s cap) replaces the previous linear schedule. Linear didn't back off fast enough under sustained outage — each KILL/PAUSE spawned its own delivery thread, producing 1000+ spinning threads hammering the dead endpoint. WAL crash-recovery - P1-5b: Atomic WAL writes (tmp + `fsync` + `os.replace`), 64 MiB rotation with `os.replace(wal, wal.1)`, replay drains both `wal.1` and `wal`. New `NULLRUN_WAL_PATH` / `NULLRUN_WAL_MAX_BYTES` env overrides for containers with `readOnlyRootFilesystem: true`. Tests 8 new regression test files (57 tests total): test_agent_id_uuid.py, test_args_pii_masked.py, test_streaming_oom_cap.py, test_lru_active_runs.py, test_reconnect_cap.py, test_coverage_seen_httpx.py, test_webhook_backoff.py, test_redact.py `test_buffer_invariants.py` extended with drop-newest + critical-event preservation cases. `test_release_polish.py` updated to pin the 5s cap on both the sync and async jitter helpers (post §7.2 #35 split). Full incident write-ups in CHANGELOG.md under the same P0/S/P tags. * fix: address ruff lint findings from CI Three CI lint failures on `ruff check src/` — fixes only, no behavioural changes: - **B905** (`src/nullrun/decorators.py:162`): `zip(bound_params, args)` now passes `strict=False` explicitly. Pre-fix the two iterables can be different lengths — `bound_params` is sliced to `[: len(args)]` but the function may have fewer positional parameters than args provided (e.g. *args-style callables), in which case the trailing loop below handles the excess. `strict=` was implicit and triggered B905. Now explicit so the intent is documented in code. - **I001** (`src/nullrun/instrumentation/auto.py:1146`): the late `import os as _os` was moved to the top-of-file import block as `import os` (alphabetical order: hashlib, json, logging, os, threading). The `_os` alias was only there to avoid shadowing — there is no top-level `os` in scope, so the plain name is fine. Call site updated to use `os.environ.get(...)`. - **S108** (`src/nullrun/transport.py:632`): replaced the hardcoded `/tmp/nullrun.wal` with `os.path.join(tempfile.gettempdir(), "nullrun.wal")`. The hardcoded `/tmp` flagged S108 (insecure / non-portable temp path) and would have broken the SDK on Windows out of the box. `gettempdir()` returns the OS-appropriate temp dir (`/tmp` on Linux, `/var/folders/...` on macOS, `%TEMP%` on Windows). `NULLRUN_WAL_PATH` env override still wins, so containers with `readOnlyRootFilesystem: true` are unaffected. Added `import tempfile` to the top-of-file imports. Verified: - `ruff check src/` → All checks passed! - `mypy src/` → Success: no issues found in 23 source files - `pytest` → 493 passed, 13 skipped (CI default, no `-W error`) * chore(release): bump to 0.5.2 - Promote [Unreleased] to [0.5.2] — 2026-06-19; merge the two [Unreleased] sections that had drifted during Sprint 2.5 + Phase 0 development so release tooling scanning for the [Unreleased] anchor picks up the complete change set exactly once. - Add PEP 561 marker (py.typed) — the package ships inline type annotations; the marker tells mypy / pyright / pylance to honour them. - runtime.py (S-4): case-insensitive state compare in check_control_plane. Defensive against any backend casing drift beyond the current PascalCase (handlers.rs:9258). Pinned by tests/test_state_compare_case_insensitive.py (10 cases covering PascalCase / UPPERCASE / lowercase / mixed-case). Working-notes file docs/integration-baseline-2026-06-19.md is deliberately left untracked, matching the analyze.md pattern from d74712e. * test: bump coverage 70.92% → 84.52% with branch coverage Lifts the SDK's Codecov score from 70.92 % to 84.52 % (+13.6 pp) by adding 347 new tests across 10 files that exercise previously-untested branches in the auto-instrumentation patches, runtime gates, transport fallback modes, circuit breaker Redis path, and the @Protect decorator fail-CLOSED contract. pyproject.toml - Enable branch coverage so error / fallback paths count. - Raise fail_under from 70 → 82 (enforced in CI via `coverage run -m pytest && coverage report`). - Add precision=2 and skip_empty=true to keep the report readable. New tests (all 817 pass locally, all 4 CI jobs green): tests/test_autogen_patch.py — 13 tests tests/test_crewai_patch.py — 15 tests tests/test_llama_index_patch.py — 13 tests tests/test_langgraph_callback.py — 38 tests tests/test_auto_requests.py — 24 tests tests/test_runtime_branches.py — 43 tests tests/test_transport_branches.py — 44 tests tests/test_circuit_breaker_branches.py — 31 tests tests/test_protect_branches.py — 43 tests tests/test_actions_context_init.py — 50 tests Per-file coverage deltas: instrumentation/autogen.py 21.33 → 93.41 % instrumentation/crewai.py 22.97 → 90.82 % instrumentation/llama_index.py 28.30 → 100.00 % instrumentation/langgraph.py 23.75 → 93.69 % instrumentation/auto_requests.py 33.72 → 99.09 % breaker/circuit_breaker.py 59.76 → 90.21 % transport.py 82.57 → 84.79 % transport_websocket.py 68.70 → 64.10 % (msg-type branches still need live ws round-trip tests) decorators.py 83.33 → 95.49 % runtime.py 80.14 → 83.24 % context.py 82.76 → 100.00 % actions.py 92.12 → 96.89 % breaker/exceptions.py 98.51 → 97.26 % All 4 CI jobs pass locally (pytest, ruff check, mypy, coverage). Working-notes file docs/integration-baseline-2026-06-19.md is deliberately left untracked, matching the analyze.md pattern from d74712e. * feat(security): make @sensitive registration fail-CLOSED (ADR-008) Sensitive-tool registration is part of the security boundary. The old behaviour caught any exception from _get_or_create_runtime(), logged it at DEBUG, and returned the original function unchanged — which meant the wrapped body would later execute without ever being added to the runtime's sensitive-tool set, completely bypassing the pre-execution gate under partial initialization (e.g. transient NullRunAuthenticationError on import). Replace the silent logger.debug(...) with raise RuntimeError(..., chained from the original exception. The decorator is the registration point, not the call site, so raising at decoration time is the correct signal: the import / module-load fails loudly, the body never gets a chance to run untracked, and the caller can still inspect the root cause via __cause__. The two pre-existing tests pinned the old (silent / wrong-type) contract; update them to assert the new RuntimeError wrapping: - test_sensitive_raises_on_missing_api_key now expects RuntimeError whose __cause__ is the original NullRunAuthenticationError. - test_sensitive_runtime_init_failure_is_silent is renamed to ..._raises and asserts the same __cause__ chaining when a _get_or_create_runtime mock raises. * fix(transport): retry /track/batch on 5xx and align auth-verify path (P0 #2, P0 #5) P0 #2 — _send_batch_with_retry_info used to do a single self._client.post(...) + raise_for_status(). A transient backend 5xx raised out of the flush path; the in-memory buffer was cleared at the call site and every event in the batch was permanently lost. Wrap the post() in _retry_with_backoff (max 3 attempts, exponential backoff + jitter, capped at 10s) so a single 500 no longer drops the whole batch. 429 is retried (helper honors Retry-After when present); other 4xx errors are returned as-is — those are real client bugs and must not be retried (e.g. a 401 just wastes the user's budget). P0 #5 — contract drift: this file's auth-verify call site used /auth/verify, while the corresponding call in runtime.py:599 already used /api/v1/auth/verify. Align the rotation call site to /api/v1/auth/verify so the contract-drift-guard CI catches any future divergence. Update tests/test_transport.py::test_retry_on_500 to assert the new contract (third attempt succeeds → call_count == 3, event id in accepted_event_ids) instead of expecting an immediate exception. Add tests/test_track_batch_retry.py with full regression coverage: single 5xx → success, three consecutive 5xx → BreakerTransportError, 429 with Retry-After → honored before next attempt. * feat(runtime): emit background coverage_report every 60s The SDK has tracked per-host seen / tracked / streaming_skipped counters since 0.4.x (bump_coverage_counter, get_coverage_stats), but there was no path to ship them to the backend — the counters only ever existed in process memory. This commit adds a daemon thread that emits a coverage_report track event every 60 seconds so the backend can build the per-host coverage dashboard. * NullRunRuntime.track_coverage() — returns a track-result dict when there is something to report, or None on cold start (no counters bumped yet) so the backend doesn't get an empty row per minute. * start_coverage_reporter() / stop_coverage_reporter() — idempotent lifecycle, daemon thread, sleeps in 0.5s slices for responsive shutdown, emits once on entry so short-lived processes (CI, batch jobs) still leave a row. * nullrun.init() wires start_coverage_reporter() in; the reporter is a no-op while the process is still cold, so re-init is safe. New tests/test_coverage_report.py pins the contract: cold start → None, post-traffic → track-result dict with type=coverage_report and the three counter dicts, start is idempotent, stop joins cleanly. * chore(breaker): add __main__ shim so 'python -m nullrun.breaker' exits cleanly Historically the SDK shipped a 'python -m nullrun.breaker' entry point for in-container health probes and ad-hoc debugging. The nullrun.breaker subpackage is the circuit-breaker + policy-exceptions surface — it is not a runnable command. Without this shim, containerized deployments that scripted 'python -m nullrun.breaker' as a no-op smoke check would fail with 'No module named nullrun.breaker.__main__'. This module makes that invocation exit cleanly (return 0) and print a short pointer to nullrun-doctor (nullrun.toolbox.diagnostics) for real runtime checks. * chore: gitignore audit.md (project-local working notes, sibling of analyze.md) * test: re-align @sensitive test with fail-CLOSED contract after master merge The auto-merge of master into this branch (commit 7875210) resolved tests/test_protect_branches.py by taking master's side of the conflict, leaving the old test_sensitive_runtime_init_failure_is_silent in place. That test asserts @sensitive does NOT raise — but the production change in commit 58263a1 (this branch) makes @sensitive raise RuntimeError (fail-CLOSED, ADR-008). Result: CI ran the old assertion against the new production code and failed. Restore the renamed and re-asserted version of the test from commit 58263a1 — test_sensitive_runtime_init_failure_raises — so the test asserts the new contract: RuntimeError is raised and __cause__ chains the original exception. runtime.py was resolved correctly by the auto-merge (both sides kept: the new track_coverage / start_coverage_reporter / stop_coverage_reporter / _coverage_reporter_loop methods AND the existing bump_coverage_counter are all present), so no changes there.
maltsev-dev
added a commit
that referenced
this pull request
Jun 21, 2026
* fix: P0 security/stability hardening bundle
Closes the P0/P1/P2/P3 issues from the security review (plan §10/§11.4).
Security / PCI-DSS / GDPR
- P0-1: Mask positional PII in `_enforce_sensitive_tool` by introspecting
the wrapped function's signature and applying `SENSITIVE_ARG_KEYS` to
positional params. Pre-fix, `charge("4111-…-1111", 50)` forwarded the
PAN into `/execute` and the audit log.
- P0-6 / P3-3: `_safe_repr` now redacts BEFORE truncating. The pre-fix
order truncated first, so `details={…}` past position 50 leaked
verbatim. `_safe_repr` is now the single source of truth for the
redact-then-truncate flow.
Cost-audit / reliability
- P0-3: Bounded chunked reads on the sync + async httpx transports
(`MAX_RESPONSE_BYTES`, default 16 MiB, `NULLRUN_MAX_RESPONSE_BYTES`
env override). Above the cap, tracking is skipped and
`_coverage_streaming_skipped` is incremented. Replaces the
`response.read()` / `await response.aread()` unbounded buffer that
held entire LLM streaming bodies in memory.
- P0-4: `_do_flush_locked` re-queue on CB OPEN now drops the NEWEST
non-critical events instead of the oldest. The oldest events
(incident start, billing-period start) are exactly what a billing
investigator needs; losing them silently broke monthly rollups.
Control-plane events (`state_change`, `kill_received`,
`policy_invalidated`, `key_rotated`) are preserved unconditionally
so the dashboard KILL switch lands even under sustained backend
outage.
Identity
- S-8 / P2-4: `agent()` now emits `str(uuid.uuid4())` (with dashes).
Pre-fix the format was `f"agent-{uuid.uuid4().hex}"` — 32 hex chars,
no dashes — and backend UUID-typed columns dropped these to NULL
on insert. User-supplied names are still preserved verbatim.
- §7.2 #16: `workflow()` context manager now resets `span_id` (not
only `workflow_id` / `trace_id`) so nested `with span()` blocks
don't leave the inner span_id visible inside the workflow scope.
Resource leaks
- S-9: `_active_runs` on `NullRunCallback` is now an `OrderedDict`
capped at 4096 with FIFO eviction. Pre-fix the dict grew
unbounded when `on_chain_end` did not fire (some LangChain
versions short-circuit the end hook on chain-body errors).
- S-10: WebSocket reconnect loop is now capped at 10 consecutive
failures, then falls back to HTTP-poll. Pre-fix the loop ran
forever when the backend was permanently down, leaking the
WS thread.
Transport
- §7.2 #6: Separate `hmac_verify_expired_total` counter so SRE can
distinguish clock-skew (NTP drift) from forged packets. Mirrored
in both the HTTP and WebSocket verify paths.
- §7.2 #35: `CircuitBreaker.call` now dispatches the OPEN→HALF_OPEN
jitter through `_maybe_apply_open_jitter_sync` /
`_maybe_apply_open_jitter_async`. Pre-fix the jitter used
`time.sleep` before dispatching to async, which blocked the
caller's event loop on every transition.
- P2-1: `_coverage_seen` now bumps in the httpx path (sync + async).
Pre-fix the counter was only bumped by the `requests` transport,
so the dashboard's coverage view was empty for the dominant
OpenAI / Anthropic / Gemini / Mistral / Cohere traffic.
- P2-3: `is_sensitive_tool` match is case-insensitive. Pre-fix
`"stripe.charge"` did not match `"Stripe.Charge"`, bypassing the
sensitive gate.
Concurrency
- §7.2 #39: New `_tools_lock` guards every mutation of
`_strict_mode_tools` / `_sensitive_tools`. Same lock guards the
coverage-counter bump+prune sequence (§7.2 #33) so two threads
can't both observe the dict at length 4095 and both grow it to
4097 before either prune lands.
- §7.2 #47: New `_langchain_lock` / `_langgraph_lock` guard the
patch sequences end-to-end. Pre-fix two threads racing through
`auto_instrument` could both pass the early `_x_patched` check
and double-wrap `BaseCallbackManager` / `Pregel`.
- §7.2 #33: `_COVERAGE_CAP` (4096) bounds the per-host coverage
dicts.
Webhook delivery
- P3-2: Exponential backoff (0.5s, 1s, 2s, 4s, 8s, 16s, 30s cap)
replaces the previous linear schedule. Linear didn't back off
fast enough under sustained outage — each KILL/PAUSE spawned
its own delivery thread, producing 1000+ spinning threads
hammering the dead endpoint.
WAL crash-recovery
- P1-5b: Atomic WAL writes (tmp + `fsync` + `os.replace`), 64 MiB
rotation with `os.replace(wal, wal.1)`, replay drains both
`wal.1` and `wal`. New `NULLRUN_WAL_PATH` / `NULLRUN_WAL_MAX_BYTES`
env overrides for containers with `readOnlyRootFilesystem: true`.
Tests
8 new regression test files (57 tests total):
test_agent_id_uuid.py, test_args_pii_masked.py,
test_streaming_oom_cap.py, test_lru_active_runs.py,
test_reconnect_cap.py, test_coverage_seen_httpx.py,
test_webhook_backoff.py, test_redact.py
`test_buffer_invariants.py` extended with drop-newest +
critical-event preservation cases. `test_release_polish.py`
updated to pin the 5s cap on both the sync and async jitter
helpers (post §7.2 #35 split).
Full incident write-ups in CHANGELOG.md under the same P0/S/P tags.
* fix: address ruff lint findings from CI
Three CI lint failures on `ruff check src/` — fixes only, no
behavioural changes:
- **B905** (`src/nullrun/decorators.py:162`): `zip(bound_params,
args)` now passes `strict=False` explicitly. Pre-fix the two
iterables can be different lengths — `bound_params` is sliced to
`[: len(args)]` but the function may have fewer positional
parameters than args provided (e.g. *args-style callables), in
which case the trailing loop below handles the excess. `strict=`
was implicit and triggered B905. Now explicit so the intent is
documented in code.
- **I001** (`src/nullrun/instrumentation/auto.py:1146`): the late
`import os as _os` was moved to the top-of-file import block as
`import os` (alphabetical order: hashlib, json, logging, os,
threading). The `_os` alias was only there to avoid shadowing —
there is no top-level `os` in scope, so the plain name is fine.
Call site updated to use `os.environ.get(...)`.
- **S108** (`src/nullrun/transport.py:632`): replaced the
hardcoded `/tmp/nullrun.wal` with
`os.path.join(tempfile.gettempdir(), "nullrun.wal")`. The
hardcoded `/tmp` flagged S108 (insecure / non-portable temp
path) and would have broken the SDK on Windows out of the box.
`gettempdir()` returns the OS-appropriate temp dir
(`/tmp` on Linux, `/var/folders/...` on macOS, `%TEMP%` on
Windows). `NULLRUN_WAL_PATH` env override still wins, so
containers with `readOnlyRootFilesystem: true` are unaffected.
Added `import tempfile` to the top-of-file imports.
Verified:
- `ruff check src/` → All checks passed!
- `mypy src/` → Success: no issues found in 23 source files
- `pytest` → 493 passed, 13 skipped (CI default, no `-W error`)
* chore(release): bump to 0.5.2
- Promote [Unreleased] to [0.5.2] — 2026-06-19; merge the two
[Unreleased] sections that had drifted during Sprint 2.5 +
Phase 0 development so release tooling scanning for the
[Unreleased] anchor picks up the complete change set exactly
once.
- Add PEP 561 marker (py.typed) — the package ships inline type
annotations; the marker tells mypy / pyright / pylance to honour
them.
- runtime.py (S-4): case-insensitive state compare in
check_control_plane. Defensive against any backend casing drift
beyond the current PascalCase (handlers.rs:9258). Pinned by
tests/test_state_compare_case_insensitive.py (10 cases covering
PascalCase / UPPERCASE / lowercase / mixed-case).
Working-notes file docs/integration-baseline-2026-06-19.md is
deliberately left untracked, matching the analyze.md pattern from
d74712e.
* test: bump coverage 70.92% → 84.52% with branch coverage
Lifts the SDK's Codecov score from 70.92 % to 84.52 % (+13.6 pp) by
adding 347 new tests across 10 files that exercise previously-untested
branches in the auto-instrumentation patches, runtime gates, transport
fallback modes, circuit breaker Redis path, and the @Protect decorator
fail-CLOSED contract.
pyproject.toml
- Enable branch coverage so error / fallback paths count.
- Raise fail_under from 70 → 82 (enforced in CI via `coverage run -m
pytest && coverage report`).
- Add precision=2 and skip_empty=true to keep the report readable.
New tests (all 817 pass locally, all 4 CI jobs green):
tests/test_autogen_patch.py — 13 tests
tests/test_crewai_patch.py — 15 tests
tests/test_llama_index_patch.py — 13 tests
tests/test_langgraph_callback.py — 38 tests
tests/test_auto_requests.py — 24 tests
tests/test_runtime_branches.py — 43 tests
tests/test_transport_branches.py — 44 tests
tests/test_circuit_breaker_branches.py — 31 tests
tests/test_protect_branches.py — 43 tests
tests/test_actions_context_init.py — 50 tests
Per-file coverage deltas:
instrumentation/autogen.py 21.33 → 93.41 %
instrumentation/crewai.py 22.97 → 90.82 %
instrumentation/llama_index.py 28.30 → 100.00 %
instrumentation/langgraph.py 23.75 → 93.69 %
instrumentation/auto_requests.py 33.72 → 99.09 %
breaker/circuit_breaker.py 59.76 → 90.21 %
transport.py 82.57 → 84.79 %
transport_websocket.py 68.70 → 64.10 % (msg-type branches
still need live ws
round-trip tests)
decorators.py 83.33 → 95.49 %
runtime.py 80.14 → 83.24 %
context.py 82.76 → 100.00 %
actions.py 92.12 → 96.89 %
breaker/exceptions.py 98.51 → 97.26 %
All 4 CI jobs pass locally (pytest, ruff check, mypy, coverage).
Working-notes file docs/integration-baseline-2026-06-19.md is
deliberately left untracked, matching the analyze.md pattern from
d74712e.
* feat(security): make @sensitive registration fail-CLOSED (ADR-008)
Sensitive-tool registration is part of the security boundary. The
old behaviour caught any exception from _get_or_create_runtime(),
logged it at DEBUG, and returned the original function unchanged —
which meant the wrapped body would later execute without ever being
added to the runtime's sensitive-tool set, completely bypassing the
pre-execution gate under partial initialization (e.g. transient
NullRunAuthenticationError on import).
Replace the silent logger.debug(...) with raise RuntimeError(...,
chained from the original exception. The decorator is the registration
point, not the call site, so raising at decoration time is the correct
signal: the import / module-load fails loudly, the body never gets a
chance to run untracked, and the caller can still inspect the root
cause via __cause__.
The two pre-existing tests pinned the old (silent / wrong-type) contract;
update them to assert the new RuntimeError wrapping:
- test_sensitive_raises_on_missing_api_key now expects RuntimeError
whose __cause__ is the original NullRunAuthenticationError.
- test_sensitive_runtime_init_failure_is_silent is renamed to
..._raises and asserts the same __cause__ chaining when a
_get_or_create_runtime mock raises.
* fix(transport): retry /track/batch on 5xx and align auth-verify path (P0 #2, P0 #5)
P0 #2 — _send_batch_with_retry_info used to do a single
self._client.post(...) + raise_for_status(). A transient backend 5xx
raised out of the flush path; the in-memory buffer was cleared at the
call site and every event in the batch was permanently lost. Wrap the
post() in _retry_with_backoff (max 3 attempts, exponential backoff +
jitter, capped at 10s) so a single 500 no longer drops the whole batch.
429 is retried (helper honors Retry-After when present); other 4xx
errors are returned as-is — those are real client bugs and must not
be retried (e.g. a 401 just wastes the user's budget).
P0 #5 — contract drift: this file's auth-verify call site used
/auth/verify, while the corresponding call in runtime.py:599 already
used /api/v1/auth/verify. Align the rotation call site to /api/v1/auth/verify
so the contract-drift-guard CI catches any future divergence.
Update tests/test_transport.py::test_retry_on_500 to assert the new
contract (third attempt succeeds → call_count == 3, event id in
accepted_event_ids) instead of expecting an immediate exception.
Add tests/test_track_batch_retry.py with full regression coverage:
single 5xx → success, three consecutive 5xx → BreakerTransportError,
429 with Retry-After → honored before next attempt.
* feat(runtime): emit background coverage_report every 60s
The SDK has tracked per-host seen / tracked / streaming_skipped counters
since 0.4.x (bump_coverage_counter, get_coverage_stats), but there was
no path to ship them to the backend — the counters only ever existed
in process memory. This commit adds a daemon thread that emits a
coverage_report track event every 60 seconds so the backend can build
the per-host coverage dashboard.
* NullRunRuntime.track_coverage() — returns a track-result dict when
there is something to report, or None on cold start (no counters
bumped yet) so the backend doesn't get an empty row per minute.
* start_coverage_reporter() / stop_coverage_reporter() — idempotent
lifecycle, daemon thread, sleeps in 0.5s slices for responsive
shutdown, emits once on entry so short-lived processes (CI, batch
jobs) still leave a row.
* nullrun.init() wires start_coverage_reporter() in; the reporter is
a no-op while the process is still cold, so re-init is safe.
New tests/test_coverage_report.py pins the contract: cold start → None,
post-traffic → track-result dict with type=coverage_report and the three
counter dicts, start is idempotent, stop joins cleanly.
* chore(breaker): add __main__ shim so 'python -m nullrun.breaker' exits cleanly
Historically the SDK shipped a 'python -m nullrun.breaker' entry point
for in-container health probes and ad-hoc debugging. The nullrun.breaker
subpackage is the circuit-breaker + policy-exceptions surface — it is
not a runnable command. Without this shim, containerized deployments
that scripted 'python -m nullrun.breaker' as a no-op smoke check would
fail with 'No module named nullrun.breaker.__main__'.
This module makes that invocation exit cleanly (return 0) and print a
short pointer to nullrun-doctor (nullrun.toolbox.diagnostics) for
real runtime checks.
* chore: gitignore audit.md (project-local working notes, sibling of analyze.md)
* test: re-align @sensitive test with fail-CLOSED contract after master merge
The auto-merge of master into this branch (commit 7875210) resolved
tests/test_protect_branches.py by taking master's side of the conflict,
leaving the old test_sensitive_runtime_init_failure_is_silent in place.
That test asserts @sensitive does NOT raise — but the production
change in commit 58263a1 (this branch) makes @sensitive raise
RuntimeError (fail-CLOSED, ADR-008). Result: CI ran the old assertion
against the new production code and failed.
Restore the renamed and re-asserted version of the test from commit
58263a1 — test_sensitive_runtime_init_failure_raises — so the test
asserts the new contract: RuntimeError is raised and __cause__ chains
the original exception.
runtime.py was resolved correctly by the auto-merge (both sides kept:
the new track_coverage / start_coverage_reporter / stop_coverage_reporter
/ _coverage_reporter_loop methods AND the existing bump_coverage_counter
are all present), so no changes there.
maltsev-dev
added a commit
that referenced
this pull request
Jul 4, 2026
Bug-fix release layered on top of 0.12.1. No wire-format change; both fixes are client-side only. * BUG #4 -- check_workflow_budget() now sends a fresh uuidv7 as the "execution_id" field on every /check call instead of reusing workflow_id. The server's gate_reserve_v3 overwrites the field on response anyway, but a client-side placeholder that collides across calls confuses the v3 reservation binding on /track when Transport.track_single() reaches the backend and the field is stale -- exact symptom is 503 RESERVATION_NOT_FOUND per CLAUDE.md section 29. * BUG #5 -- new nullrun.runtime._GATE_CACHE (5s TTL, keyed on (workflow_id, chain_id, model)) collapses consecutive /gate calls from inside `with chain(...)` to a single roundtrip, avoiding 100 /gate calls per 100-step agent loop. Single-shot (Hard mode) callers MUST bypass the cache -- Hard mode's binary allow -> block semantics would let a stale "allow" leak a budget-exhausted call through. Opt-out via NULLRUN_GATE_CACHE_DISABLE=1 for callers that want the legacy always-roundtrip behaviour (used by live smoke tests per docs/runbooks/budget-blue-green-smoke.sh). Tests: 158 new lines in tests/test_v3_wire_contract.py covering per-call execution_id uniqueness, uuidv7 format validation, and the new cache data-structure invariants + opt-out cases. Bumps __version__ + pyproject.toml to 0.12.2.
maltsev-dev
added a commit
that referenced
this pull request
Jul 4, 2026
* release(0.12.2): fresh execution_id per /check + chain-mode /gate cache Bug-fix release layered on top of 0.12.1. No wire-format change; both fixes are client-side only. * BUG #4 -- check_workflow_budget() now sends a fresh uuidv7 as the "execution_id" field on every /check call instead of reusing workflow_id. The server's gate_reserve_v3 overwrites the field on response anyway, but a client-side placeholder that collides across calls confuses the v3 reservation binding on /track when Transport.track_single() reaches the backend and the field is stale -- exact symptom is 503 RESERVATION_NOT_FOUND per CLAUDE.md section 29. * BUG #5 -- new nullrun.runtime._GATE_CACHE (5s TTL, keyed on (workflow_id, chain_id, model)) collapses consecutive /gate calls from inside `with chain(...)` to a single roundtrip, avoiding 100 /gate calls per 100-step agent loop. Single-shot (Hard mode) callers MUST bypass the cache -- Hard mode's binary allow -> block semantics would let a stale "allow" leak a budget-exhausted call through. Opt-out via NULLRUN_GATE_CACHE_DISABLE=1 for callers that want the legacy always-roundtrip behaviour (used by live smoke tests per docs/runbooks/budget-blue-green-smoke.sh). Tests: 158 new lines in tests/test_v3_wire_contract.py covering per-call execution_id uniqueness, uuidv7 format validation, and the new cache data-structure invariants + opt-out cases. Bumps __version__ + pyproject.toml to 0.12.2. * fix(lint): reorder uuid7 import in runtime.py per ruff I001 `ruff check src/` flagged that the BUG #4 line `from nullrun.uuid7 import uuid7_str # CLAUDE.md §24` landed mid-way through the first-party import block (between `nullrun.context` and `nullrun.observability`), breaking I001 import sort. Moved to the bottom of the first-party block (alphabetic order — `uuid7` sorts after `transport`). Also lets ruff auto-fix two cosmetic cleanups in tests/test_v3_wire_contract.py: * sort `_V3_ERROR_CODE_MAP` alphabetic in the existing transport import group (was below `_parse_v3_error_envelope`); * drop a stray blank-line gap between two top-level `from nullrun.transport import (...)` groups. `ruff check src/ tests/` after the fix: 8 pre-existing I001 findings remain in unrelated test files (predicate, test_circuit_breaker_branches.py, test_framework_patches.py, etc.) — out of scope for this PR. Scope above matches the CI step (`ruff check src/`). No behavioural change.
maltsev-dev
added a commit
that referenced
this pull request
Aug 7, 2026
…erage reporter (#26) * fix: P0 security/stability hardening bundle Closes the P0/P1/P2/P3 issues from the security review (plan §10/§11.4). Security / PCI-DSS / GDPR - P0-1: Mask positional PII in `_enforce_sensitive_tool` by introspecting the wrapped function's signature and applying `SENSITIVE_ARG_KEYS` to positional params. Pre-fix, `charge("4111-…-1111", 50)` forwarded the PAN into `/execute` and the audit log. - P0-6 / P3-3: `_safe_repr` now redacts BEFORE truncating. The pre-fix order truncated first, so `details={…}` past position 50 leaked verbatim. `_safe_repr` is now the single source of truth for the redact-then-truncate flow. Cost-audit / reliability - P0-3: Bounded chunked reads on the sync + async httpx transports (`MAX_RESPONSE_BYTES`, default 16 MiB, `NULLRUN_MAX_RESPONSE_BYTES` env override). Above the cap, tracking is skipped and `_coverage_streaming_skipped` is incremented. Replaces the `response.read()` / `await response.aread()` unbounded buffer that held entire LLM streaming bodies in memory. - P0-4: `_do_flush_locked` re-queue on CB OPEN now drops the NEWEST non-critical events instead of the oldest. The oldest events (incident start, billing-period start) are exactly what a billing investigator needs; losing them silently broke monthly rollups. Control-plane events (`state_change`, `kill_received`, `policy_invalidated`, `key_rotated`) are preserved unconditionally so the dashboard KILL switch lands even under sustained backend outage. Identity - S-8 / P2-4: `agent()` now emits `str(uuid.uuid4())` (with dashes). Pre-fix the format was `f"agent-{uuid.uuid4().hex}"` — 32 hex chars, no dashes — and backend UUID-typed columns dropped these to NULL on insert. User-supplied names are still preserved verbatim. - §7.2 #16: `workflow()` context manager now resets `span_id` (not only `workflow_id` / `trace_id`) so nested `with span()` blocks don't leave the inner span_id visible inside the workflow scope. Resource leaks - S-9: `_active_runs` on `NullRunCallback` is now an `OrderedDict` capped at 4096 with FIFO eviction. Pre-fix the dict grew unbounded when `on_chain_end` did not fire (some LangChain versions short-circuit the end hook on chain-body errors). - S-10: WebSocket reconnect loop is now capped at 10 consecutive failures, then falls back to HTTP-poll. Pre-fix the loop ran forever when the backend was permanently down, leaking the WS thread. Transport - §7.2 #6: Separate `hmac_verify_expired_total` counter so SRE can distinguish clock-skew (NTP drift) from forged packets. Mirrored in both the HTTP and WebSocket verify paths. - §7.2 #35: `CircuitBreaker.call` now dispatches the OPEN→HALF_OPEN jitter through `_maybe_apply_open_jitter_sync` / `_maybe_apply_open_jitter_async`. Pre-fix the jitter used `time.sleep` before dispatching to async, which blocked the caller's event loop on every transition. - P2-1: `_coverage_seen` now bumps in the httpx path (sync + async). Pre-fix the counter was only bumped by the `requests` transport, so the dashboard's coverage view was empty for the dominant OpenAI / Anthropic / Gemini / Mistral / Cohere traffic. - P2-3: `is_sensitive_tool` match is case-insensitive. Pre-fix `"stripe.charge"` did not match `"Stripe.Charge"`, bypassing the sensitive gate. Concurrency - §7.2 #39: New `_tools_lock` guards every mutation of `_strict_mode_tools` / `_sensitive_tools`. Same lock guards the coverage-counter bump+prune sequence (§7.2 #33) so two threads can't both observe the dict at length 4095 and both grow it to 4097 before either prune lands. - §7.2 #47: New `_langchain_lock` / `_langgraph_lock` guard the patch sequences end-to-end. Pre-fix two threads racing through `auto_instrument` could both pass the early `_x_patched` check and double-wrap `BaseCallbackManager` / `Pregel`. - §7.2 #33: `_COVERAGE_CAP` (4096) bounds the per-host coverage dicts. Webhook delivery - P3-2: Exponential backoff (0.5s, 1s, 2s, 4s, 8s, 16s, 30s cap) replaces the previous linear schedule. Linear didn't back off fast enough under sustained outage — each KILL/PAUSE spawned its own delivery thread, producing 1000+ spinning threads hammering the dead endpoint. WAL crash-recovery - P1-5b: Atomic WAL writes (tmp + `fsync` + `os.replace`), 64 MiB rotation with `os.replace(wal, wal.1)`, replay drains both `wal.1` and `wal`. New `NULLRUN_WAL_PATH` / `NULLRUN_WAL_MAX_BYTES` env overrides for containers with `readOnlyRootFilesystem: true`. Tests 8 new regression test files (57 tests total): test_agent_id_uuid.py, test_args_pii_masked.py, test_streaming_oom_cap.py, test_lru_active_runs.py, test_reconnect_cap.py, test_coverage_seen_httpx.py, test_webhook_backoff.py, test_redact.py `test_buffer_invariants.py` extended with drop-newest + critical-event preservation cases. `test_release_polish.py` updated to pin the 5s cap on both the sync and async jitter helpers (post §7.2 #35 split). Full incident write-ups in CHANGELOG.md under the same P0/S/P tags. * fix: address ruff lint findings from CI Three CI lint failures on `ruff check src/` — fixes only, no behavioural changes: - **B905** (`src/nullrun/decorators.py:162`): `zip(bound_params, args)` now passes `strict=False` explicitly. Pre-fix the two iterables can be different lengths — `bound_params` is sliced to `[: len(args)]` but the function may have fewer positional parameters than args provided (e.g. *args-style callables), in which case the trailing loop below handles the excess. `strict=` was implicit and triggered B905. Now explicit so the intent is documented in code. - **I001** (`src/nullrun/instrumentation/auto.py:1146`): the late `import os as _os` was moved to the top-of-file import block as `import os` (alphabetical order: hashlib, json, logging, os, threading). The `_os` alias was only there to avoid shadowing — there is no top-level `os` in scope, so the plain name is fine. Call site updated to use `os.environ.get(...)`. - **S108** (`src/nullrun/transport.py:632`): replaced the hardcoded `/tmp/nullrun.wal` with `os.path.join(tempfile.gettempdir(), "nullrun.wal")`. The hardcoded `/tmp` flagged S108 (insecure / non-portable temp path) and would have broken the SDK on Windows out of the box. `gettempdir()` returns the OS-appropriate temp dir (`/tmp` on Linux, `/var/folders/...` on macOS, `%TEMP%` on Windows). `NULLRUN_WAL_PATH` env override still wins, so containers with `readOnlyRootFilesystem: true` are unaffected. Added `import tempfile` to the top-of-file imports. Verified: - `ruff check src/` → All checks passed! - `mypy src/` → Success: no issues found in 23 source files - `pytest` → 493 passed, 13 skipped (CI default, no `-W error`) * chore(release): bump to 0.5.2 - Promote [Unreleased] to [0.5.2] — 2026-06-19; merge the two [Unreleased] sections that had drifted during Sprint 2.5 + Phase 0 development so release tooling scanning for the [Unreleased] anchor picks up the complete change set exactly once. - Add PEP 561 marker (py.typed) — the package ships inline type annotations; the marker tells mypy / pyright / pylance to honour them. - runtime.py (S-4): case-insensitive state compare in check_control_plane. Defensive against any backend casing drift beyond the current PascalCase (handlers.rs:9258). Pinned by tests/test_state_compare_case_insensitive.py (10 cases covering PascalCase / UPPERCASE / lowercase / mixed-case). Working-notes file docs/integration-baseline-2026-06-19.md is deliberately left untracked, matching the analyze.md pattern from d74712e. * test: bump coverage 70.92% → 84.52% with branch coverage Lifts the SDK's Codecov score from 70.92 % to 84.52 % (+13.6 pp) by adding 347 new tests across 10 files that exercise previously-untested branches in the auto-instrumentation patches, runtime gates, transport fallback modes, circuit breaker Redis path, and the @Protect decorator fail-CLOSED contract. pyproject.toml - Enable branch coverage so error / fallback paths count. - Raise fail_under from 70 → 82 (enforced in CI via `coverage run -m pytest && coverage report`). - Add precision=2 and skip_empty=true to keep the report readable. New tests (all 817 pass locally, all 4 CI jobs green): tests/test_autogen_patch.py — 13 tests tests/test_crewai_patch.py — 15 tests tests/test_llama_index_patch.py — 13 tests tests/test_langgraph_callback.py — 38 tests tests/test_auto_requests.py — 24 tests tests/test_runtime_branches.py — 43 tests tests/test_transport_branches.py — 44 tests tests/test_circuit_breaker_branches.py — 31 tests tests/test_protect_branches.py — 43 tests tests/test_actions_context_init.py — 50 tests Per-file coverage deltas: instrumentation/autogen.py 21.33 → 93.41 % instrumentation/crewai.py 22.97 → 90.82 % instrumentation/llama_index.py 28.30 → 100.00 % instrumentation/langgraph.py 23.75 → 93.69 % instrumentation/auto_requests.py 33.72 → 99.09 % breaker/circuit_breaker.py 59.76 → 90.21 % transport.py 82.57 → 84.79 % transport_websocket.py 68.70 → 64.10 % (msg-type branches still need live ws round-trip tests) decorators.py 83.33 → 95.49 % runtime.py 80.14 → 83.24 % context.py 82.76 → 100.00 % actions.py 92.12 → 96.89 % breaker/exceptions.py 98.51 → 97.26 % All 4 CI jobs pass locally (pytest, ruff check, mypy, coverage). Working-notes file docs/integration-baseline-2026-06-19.md is deliberately left untracked, matching the analyze.md pattern from d74712e. * feat(security): make @sensitive registration fail-CLOSED (ADR-008) Sensitive-tool registration is part of the security boundary. The old behaviour caught any exception from _get_or_create_runtime(), logged it at DEBUG, and returned the original function unchanged — which meant the wrapped body would later execute without ever being added to the runtime's sensitive-tool set, completely bypassing the pre-execution gate under partial initialization (e.g. transient NullRunAuthenticationError on import). Replace the silent logger.debug(...) with raise RuntimeError(..., chained from the original exception. The decorator is the registration point, not the call site, so raising at decoration time is the correct signal: the import / module-load fails loudly, the body never gets a chance to run untracked, and the caller can still inspect the root cause via __cause__. The two pre-existing tests pinned the old (silent / wrong-type) contract; update them to assert the new RuntimeError wrapping: - test_sensitive_raises_on_missing_api_key now expects RuntimeError whose __cause__ is the original NullRunAuthenticationError. - test_sensitive_runtime_init_failure_is_silent is renamed to ..._raises and asserts the same __cause__ chaining when a _get_or_create_runtime mock raises. * fix(transport): retry /track/batch on 5xx and align auth-verify path (P0 #2, P0 #5) P0 #2 — _send_batch_with_retry_info used to do a single self._client.post(...) + raise_for_status(). A transient backend 5xx raised out of the flush path; the in-memory buffer was cleared at the call site and every event in the batch was permanently lost. Wrap the post() in _retry_with_backoff (max 3 attempts, exponential backoff + jitter, capped at 10s) so a single 500 no longer drops the whole batch. 429 is retried (helper honors Retry-After when present); other 4xx errors are returned as-is — those are real client bugs and must not be retried (e.g. a 401 just wastes the user's budget). P0 #5 — contract drift: this file's auth-verify call site used /auth/verify, while the corresponding call in runtime.py:599 already used /api/v1/auth/verify. Align the rotation call site to /api/v1/auth/verify so the contract-drift-guard CI catches any future divergence. Update tests/test_transport.py::test_retry_on_500 to assert the new contract (third attempt succeeds → call_count == 3, event id in accepted_event_ids) instead of expecting an immediate exception. Add tests/test_track_batch_retry.py with full regression coverage: single 5xx → success, three consecutive 5xx → BreakerTransportError, 429 with Retry-After → honored before next attempt. * feat(runtime): emit background coverage_report every 60s The SDK has tracked per-host seen / tracked / streaming_skipped counters since 0.4.x (bump_coverage_counter, get_coverage_stats), but there was no path to ship them to the backend — the counters only ever existed in process memory. This commit adds a daemon thread that emits a coverage_report track event every 60 seconds so the backend can build the per-host coverage dashboard. * NullRunRuntime.track_coverage() — returns a track-result dict when there is something to report, or None on cold start (no counters bumped yet) so the backend doesn't get an empty row per minute. * start_coverage_reporter() / stop_coverage_reporter() — idempotent lifecycle, daemon thread, sleeps in 0.5s slices for responsive shutdown, emits once on entry so short-lived processes (CI, batch jobs) still leave a row. * nullrun.init() wires start_coverage_reporter() in; the reporter is a no-op while the process is still cold, so re-init is safe. New tests/test_coverage_report.py pins the contract: cold start → None, post-traffic → track-result dict with type=coverage_report and the three counter dicts, start is idempotent, stop joins cleanly. * chore(breaker): add __main__ shim so 'python -m nullrun.breaker' exits cleanly Historically the SDK shipped a 'python -m nullrun.breaker' entry point for in-container health probes and ad-hoc debugging. The nullrun.breaker subpackage is the circuit-breaker + policy-exceptions surface — it is not a runnable command. Without this shim, containerized deployments that scripted 'python -m nullrun.breaker' as a no-op smoke check would fail with 'No module named nullrun.breaker.__main__'. This module makes that invocation exit cleanly (return 0) and print a short pointer to nullrun-doctor (nullrun.toolbox.diagnostics) for real runtime checks. * chore: gitignore audit.md (project-local working notes, sibling of analyze.md) * test: re-align @sensitive test with fail-CLOSED contract after master merge The auto-merge of master into this branch (commit 7875210) resolved tests/test_protect_branches.py by taking master's side of the conflict, leaving the old test_sensitive_runtime_init_failure_is_silent in place. That test asserts @sensitive does NOT raise — but the production change in commit 58263a1 (this branch) makes @sensitive raise RuntimeError (fail-CLOSED, ADR-008). Result: CI ran the old assertion against the new production code and failed. Restore the renamed and re-asserted version of the test from commit 58263a1 — test_sensitive_runtime_init_failure_raises — so the test asserts the new contract: RuntimeError is raised and __cause__ chains the original exception. runtime.py was resolved correctly by the auto-merge (both sides kept: the new track_coverage / start_coverage_reporter / stop_coverage_reporter / _coverage_reporter_loop methods AND the existing bump_coverage_counter are all present), so no changes there.
maltsev-dev
added a commit
that referenced
this pull request
Aug 7, 2026
* fix: P0 security/stability hardening bundle
Closes the P0/P1/P2/P3 issues from the security review (plan §10/§11.4).
Security / PCI-DSS / GDPR
- P0-1: Mask positional PII in `_enforce_sensitive_tool` by introspecting
the wrapped function's signature and applying `SENSITIVE_ARG_KEYS` to
positional params. Pre-fix, `charge("4111-…-1111", 50)` forwarded the
PAN into `/execute` and the audit log.
- P0-6 / P3-3: `_safe_repr` now redacts BEFORE truncating. The pre-fix
order truncated first, so `details={…}` past position 50 leaked
verbatim. `_safe_repr` is now the single source of truth for the
redact-then-truncate flow.
Cost-audit / reliability
- P0-3: Bounded chunked reads on the sync + async httpx transports
(`MAX_RESPONSE_BYTES`, default 16 MiB, `NULLRUN_MAX_RESPONSE_BYTES`
env override). Above the cap, tracking is skipped and
`_coverage_streaming_skipped` is incremented. Replaces the
`response.read()` / `await response.aread()` unbounded buffer that
held entire LLM streaming bodies in memory.
- P0-4: `_do_flush_locked` re-queue on CB OPEN now drops the NEWEST
non-critical events instead of the oldest. The oldest events
(incident start, billing-period start) are exactly what a billing
investigator needs; losing them silently broke monthly rollups.
Control-plane events (`state_change`, `kill_received`,
`policy_invalidated`, `key_rotated`) are preserved unconditionally
so the dashboard KILL switch lands even under sustained backend
outage.
Identity
- S-8 / P2-4: `agent()` now emits `str(uuid.uuid4())` (with dashes).
Pre-fix the format was `f"agent-{uuid.uuid4().hex}"` — 32 hex chars,
no dashes — and backend UUID-typed columns dropped these to NULL
on insert. User-supplied names are still preserved verbatim.
- §7.2 #16: `workflow()` context manager now resets `span_id` (not
only `workflow_id` / `trace_id`) so nested `with span()` blocks
don't leave the inner span_id visible inside the workflow scope.
Resource leaks
- S-9: `_active_runs` on `NullRunCallback` is now an `OrderedDict`
capped at 4096 with FIFO eviction. Pre-fix the dict grew
unbounded when `on_chain_end` did not fire (some LangChain
versions short-circuit the end hook on chain-body errors).
- S-10: WebSocket reconnect loop is now capped at 10 consecutive
failures, then falls back to HTTP-poll. Pre-fix the loop ran
forever when the backend was permanently down, leaking the
WS thread.
Transport
- §7.2 #6: Separate `hmac_verify_expired_total` counter so SRE can
distinguish clock-skew (NTP drift) from forged packets. Mirrored
in both the HTTP and WebSocket verify paths.
- §7.2 #35: `CircuitBreaker.call` now dispatches the OPEN→HALF_OPEN
jitter through `_maybe_apply_open_jitter_sync` /
`_maybe_apply_open_jitter_async`. Pre-fix the jitter used
`time.sleep` before dispatching to async, which blocked the
caller's event loop on every transition.
- P2-1: `_coverage_seen` now bumps in the httpx path (sync + async).
Pre-fix the counter was only bumped by the `requests` transport,
so the dashboard's coverage view was empty for the dominant
OpenAI / Anthropic / Gemini / Mistral / Cohere traffic.
- P2-3: `is_sensitive_tool` match is case-insensitive. Pre-fix
`"stripe.charge"` did not match `"Stripe.Charge"`, bypassing the
sensitive gate.
Concurrency
- §7.2 #39: New `_tools_lock` guards every mutation of
`_strict_mode_tools` / `_sensitive_tools`. Same lock guards the
coverage-counter bump+prune sequence (§7.2 #33) so two threads
can't both observe the dict at length 4095 and both grow it to
4097 before either prune lands.
- §7.2 #47: New `_langchain_lock` / `_langgraph_lock` guard the
patch sequences end-to-end. Pre-fix two threads racing through
`auto_instrument` could both pass the early `_x_patched` check
and double-wrap `BaseCallbackManager` / `Pregel`.
- §7.2 #33: `_COVERAGE_CAP` (4096) bounds the per-host coverage
dicts.
Webhook delivery
- P3-2: Exponential backoff (0.5s, 1s, 2s, 4s, 8s, 16s, 30s cap)
replaces the previous linear schedule. Linear didn't back off
fast enough under sustained outage — each KILL/PAUSE spawned
its own delivery thread, producing 1000+ spinning threads
hammering the dead endpoint.
WAL crash-recovery
- P1-5b: Atomic WAL writes (tmp + `fsync` + `os.replace`), 64 MiB
rotation with `os.replace(wal, wal.1)`, replay drains both
`wal.1` and `wal`. New `NULLRUN_WAL_PATH` / `NULLRUN_WAL_MAX_BYTES`
env overrides for containers with `readOnlyRootFilesystem: true`.
Tests
8 new regression test files (57 tests total):
test_agent_id_uuid.py, test_args_pii_masked.py,
test_streaming_oom_cap.py, test_lru_active_runs.py,
test_reconnect_cap.py, test_coverage_seen_httpx.py,
test_webhook_backoff.py, test_redact.py
`test_buffer_invariants.py` extended with drop-newest +
critical-event preservation cases. `test_release_polish.py`
updated to pin the 5s cap on both the sync and async jitter
helpers (post §7.2 #35 split).
Full incident write-ups in CHANGELOG.md under the same P0/S/P tags.
* fix: address ruff lint findings from CI
Three CI lint failures on `ruff check src/` — fixes only, no
behavioural changes:
- **B905** (`src/nullrun/decorators.py:162`): `zip(bound_params,
args)` now passes `strict=False` explicitly. Pre-fix the two
iterables can be different lengths — `bound_params` is sliced to
`[: len(args)]` but the function may have fewer positional
parameters than args provided (e.g. *args-style callables), in
which case the trailing loop below handles the excess. `strict=`
was implicit and triggered B905. Now explicit so the intent is
documented in code.
- **I001** (`src/nullrun/instrumentation/auto.py:1146`): the late
`import os as _os` was moved to the top-of-file import block as
`import os` (alphabetical order: hashlib, json, logging, os,
threading). The `_os` alias was only there to avoid shadowing —
there is no top-level `os` in scope, so the plain name is fine.
Call site updated to use `os.environ.get(...)`.
- **S108** (`src/nullrun/transport.py:632`): replaced the
hardcoded `/tmp/nullrun.wal` with
`os.path.join(tempfile.gettempdir(), "nullrun.wal")`. The
hardcoded `/tmp` flagged S108 (insecure / non-portable temp
path) and would have broken the SDK on Windows out of the box.
`gettempdir()` returns the OS-appropriate temp dir
(`/tmp` on Linux, `/var/folders/...` on macOS, `%TEMP%` on
Windows). `NULLRUN_WAL_PATH` env override still wins, so
containers with `readOnlyRootFilesystem: true` are unaffected.
Added `import tempfile` to the top-of-file imports.
Verified:
- `ruff check src/` → All checks passed!
- `mypy src/` → Success: no issues found in 23 source files
- `pytest` → 493 passed, 13 skipped (CI default, no `-W error`)
* chore(release): bump to 0.5.2
- Promote [Unreleased] to [0.5.2] — 2026-06-19; merge the two
[Unreleased] sections that had drifted during Sprint 2.5 +
Phase 0 development so release tooling scanning for the
[Unreleased] anchor picks up the complete change set exactly
once.
- Add PEP 561 marker (py.typed) — the package ships inline type
annotations; the marker tells mypy / pyright / pylance to honour
them.
- runtime.py (S-4): case-insensitive state compare in
check_control_plane. Defensive against any backend casing drift
beyond the current PascalCase (handlers.rs:9258). Pinned by
tests/test_state_compare_case_insensitive.py (10 cases covering
PascalCase / UPPERCASE / lowercase / mixed-case).
Working-notes file docs/integration-baseline-2026-06-19.md is
deliberately left untracked, matching the analyze.md pattern from
d74712e.
* test: bump coverage 70.92% → 84.52% with branch coverage
Lifts the SDK's Codecov score from 70.92 % to 84.52 % (+13.6 pp) by
adding 347 new tests across 10 files that exercise previously-untested
branches in the auto-instrumentation patches, runtime gates, transport
fallback modes, circuit breaker Redis path, and the @Protect decorator
fail-CLOSED contract.
pyproject.toml
- Enable branch coverage so error / fallback paths count.
- Raise fail_under from 70 → 82 (enforced in CI via `coverage run -m
pytest && coverage report`).
- Add precision=2 and skip_empty=true to keep the report readable.
New tests (all 817 pass locally, all 4 CI jobs green):
tests/test_autogen_patch.py — 13 tests
tests/test_crewai_patch.py — 15 tests
tests/test_llama_index_patch.py — 13 tests
tests/test_langgraph_callback.py — 38 tests
tests/test_auto_requests.py — 24 tests
tests/test_runtime_branches.py — 43 tests
tests/test_transport_branches.py — 44 tests
tests/test_circuit_breaker_branches.py — 31 tests
tests/test_protect_branches.py — 43 tests
tests/test_actions_context_init.py — 50 tests
Per-file coverage deltas:
instrumentation/autogen.py 21.33 → 93.41 %
instrumentation/crewai.py 22.97 → 90.82 %
instrumentation/llama_index.py 28.30 → 100.00 %
instrumentation/langgraph.py 23.75 → 93.69 %
instrumentation/auto_requests.py 33.72 → 99.09 %
breaker/circuit_breaker.py 59.76 → 90.21 %
transport.py 82.57 → 84.79 %
transport_websocket.py 68.70 → 64.10 % (msg-type branches
still need live ws
round-trip tests)
decorators.py 83.33 → 95.49 %
runtime.py 80.14 → 83.24 %
context.py 82.76 → 100.00 %
actions.py 92.12 → 96.89 %
breaker/exceptions.py 98.51 → 97.26 %
All 4 CI jobs pass locally (pytest, ruff check, mypy, coverage).
Working-notes file docs/integration-baseline-2026-06-19.md is
deliberately left untracked, matching the analyze.md pattern from
d74712e.
* feat(security): make @sensitive registration fail-CLOSED (ADR-008)
Sensitive-tool registration is part of the security boundary. The
old behaviour caught any exception from _get_or_create_runtime(),
logged it at DEBUG, and returned the original function unchanged —
which meant the wrapped body would later execute without ever being
added to the runtime's sensitive-tool set, completely bypassing the
pre-execution gate under partial initialization (e.g. transient
NullRunAuthenticationError on import).
Replace the silent logger.debug(...) with raise RuntimeError(...,
chained from the original exception. The decorator is the registration
point, not the call site, so raising at decoration time is the correct
signal: the import / module-load fails loudly, the body never gets a
chance to run untracked, and the caller can still inspect the root
cause via __cause__.
The two pre-existing tests pinned the old (silent / wrong-type) contract;
update them to assert the new RuntimeError wrapping:
- test_sensitive_raises_on_missing_api_key now expects RuntimeError
whose __cause__ is the original NullRunAuthenticationError.
- test_sensitive_runtime_init_failure_is_silent is renamed to
..._raises and asserts the same __cause__ chaining when a
_get_or_create_runtime mock raises.
* fix(transport): retry /track/batch on 5xx and align auth-verify path (P0 #2, P0 #5)
P0 #2 — _send_batch_with_retry_info used to do a single
self._client.post(...) + raise_for_status(). A transient backend 5xx
raised out of the flush path; the in-memory buffer was cleared at the
call site and every event in the batch was permanently lost. Wrap the
post() in _retry_with_backoff (max 3 attempts, exponential backoff +
jitter, capped at 10s) so a single 500 no longer drops the whole batch.
429 is retried (helper honors Retry-After when present); other 4xx
errors are returned as-is — those are real client bugs and must not
be retried (e.g. a 401 just wastes the user's budget).
P0 #5 — contract drift: this file's auth-verify call site used
/auth/verify, while the corresponding call in runtime.py:599 already
used /api/v1/auth/verify. Align the rotation call site to /api/v1/auth/verify
so the contract-drift-guard CI catches any future divergence.
Update tests/test_transport.py::test_retry_on_500 to assert the new
contract (third attempt succeeds → call_count == 3, event id in
accepted_event_ids) instead of expecting an immediate exception.
Add tests/test_track_batch_retry.py with full regression coverage:
single 5xx → success, three consecutive 5xx → BreakerTransportError,
429 with Retry-After → honored before next attempt.
* feat(runtime): emit background coverage_report every 60s
The SDK has tracked per-host seen / tracked / streaming_skipped counters
since 0.4.x (bump_coverage_counter, get_coverage_stats), but there was
no path to ship them to the backend — the counters only ever existed
in process memory. This commit adds a daemon thread that emits a
coverage_report track event every 60 seconds so the backend can build
the per-host coverage dashboard.
* NullRunRuntime.track_coverage() — returns a track-result dict when
there is something to report, or None on cold start (no counters
bumped yet) so the backend doesn't get an empty row per minute.
* start_coverage_reporter() / stop_coverage_reporter() — idempotent
lifecycle, daemon thread, sleeps in 0.5s slices for responsive
shutdown, emits once on entry so short-lived processes (CI, batch
jobs) still leave a row.
* nullrun.init() wires start_coverage_reporter() in; the reporter is
a no-op while the process is still cold, so re-init is safe.
New tests/test_coverage_report.py pins the contract: cold start → None,
post-traffic → track-result dict with type=coverage_report and the three
counter dicts, start is idempotent, stop joins cleanly.
* chore(breaker): add __main__ shim so 'python -m nullrun.breaker' exits cleanly
Historically the SDK shipped a 'python -m nullrun.breaker' entry point
for in-container health probes and ad-hoc debugging. The nullrun.breaker
subpackage is the circuit-breaker + policy-exceptions surface — it is
not a runnable command. Without this shim, containerized deployments
that scripted 'python -m nullrun.breaker' as a no-op smoke check would
fail with 'No module named nullrun.breaker.__main__'.
This module makes that invocation exit cleanly (return 0) and print a
short pointer to nullrun-doctor (nullrun.toolbox.diagnostics) for
real runtime checks.
* chore: gitignore audit.md (project-local working notes, sibling of analyze.md)
* test: re-align @sensitive test with fail-CLOSED contract after master merge
The auto-merge of master into this branch (commit 7875210) resolved
tests/test_protect_branches.py by taking master's side of the conflict,
leaving the old test_sensitive_runtime_init_failure_is_silent in place.
That test asserts @sensitive does NOT raise — but the production
change in commit 58263a1 (this branch) makes @sensitive raise
RuntimeError (fail-CLOSED, ADR-008). Result: CI ran the old assertion
against the new production code and failed.
Restore the renamed and re-asserted version of the test from commit
58263a1 — test_sensitive_runtime_init_failure_raises — so the test
asserts the new contract: RuntimeError is raised and __cause__ chains
the original exception.
runtime.py was resolved correctly by the auto-merge (both sides kept:
the new track_coverage / start_coverage_reporter / stop_coverage_reporter
/ _coverage_reporter_loop methods AND the existing bump_coverage_counter
are all present), so no changes there.
maltsev-dev
added a commit
that referenced
this pull request
Aug 7, 2026
* release(0.12.2): fresh execution_id per /check + chain-mode /gate cache Bug-fix release layered on top of 0.12.1. No wire-format change; both fixes are client-side only. * BUG #4 -- check_workflow_budget() now sends a fresh uuidv7 as the "execution_id" field on every /check call instead of reusing workflow_id. The server's gate_reserve_v3 overwrites the field on response anyway, but a client-side placeholder that collides across calls confuses the v3 reservation binding on /track when Transport.track_single() reaches the backend and the field is stale -- exact symptom is 503 RESERVATION_NOT_FOUND per CLAUDE.md section 29. * BUG #5 -- new nullrun.runtime._GATE_CACHE (5s TTL, keyed on (workflow_id, chain_id, model)) collapses consecutive /gate calls from inside `with chain(...)` to a single roundtrip, avoiding 100 /gate calls per 100-step agent loop. Single-shot (Hard mode) callers MUST bypass the cache -- Hard mode's binary allow -> block semantics would let a stale "allow" leak a budget-exhausted call through. Opt-out via NULLRUN_GATE_CACHE_DISABLE=1 for callers that want the legacy always-roundtrip behaviour (used by live smoke tests per docs/runbooks/budget-blue-green-smoke.sh). Tests: 158 new lines in tests/test_v3_wire_contract.py covering per-call execution_id uniqueness, uuidv7 format validation, and the new cache data-structure invariants + opt-out cases. Bumps __version__ + pyproject.toml to 0.12.2. * fix(lint): reorder uuid7 import in runtime.py per ruff I001 `ruff check src/` flagged that the BUG #4 line `from nullrun.uuid7 import uuid7_str # CLAUDE.md §24` landed mid-way through the first-party import block (between `nullrun.context` and `nullrun.observability`), breaking I001 import sort. Moved to the bottom of the first-party block (alphabetic order — `uuid7` sorts after `transport`). Also lets ruff auto-fix two cosmetic cleanups in tests/test_v3_wire_contract.py: * sort `_V3_ERROR_CODE_MAP` alphabetic in the existing transport import group (was below `_parse_v3_error_envelope`); * drop a stray blank-line gap between two top-level `from nullrun.transport import (...)` groups. `ruff check src/ tests/` after the fix: 8 pre-existing I001 findings remain in unrelated test files (predicate, test_circuit_breaker_branches.py, test_framework_patches.py, etc.) — out of scope for this PR. Scope above matches the CI step (`ruff check src/`). No behavioural change.
maltsev-dev
added a commit
that referenced
this pull request
Aug 12, 2026
…print cleanup (#88) * cleanup(sprint3): remove dead code, redundant tests, memoir comments, dup CHANGELOG P1 — dead code: - Remove deprecated start_recording/stop_recording no-op stubs from runtime.py (replaced by direct return-value gates; tests for them removed). - Delete breaker/__main__.py stub (was a no-op CLI entry point). - Delete unused import warnings in runtime.py after the deprecated stubs. P2 — redundant tests: - Delete one-shot fix-dump tests (test_<fix_name>.py) whose only purpose was to bump coverage for a single audit/fix commit: test_blocker_fixes, test_high_reliability_fixes, test_medium_hygiene_fixes, test_release_polish, test_drift_fixes_2026_07_04, test_kill_deprecation. - Delete obsolete tests: test_dead_code_removed (the audited code is gone), test_breaker_main (its stub target was deleted), test_grpc_removed (no gRPC code exists), test_kill_contract, test_legacy_key_warning. - Consolidate test_X_branches.py into test_X.py: test_runtime_branches, test_transport_branches, test_protect_branches, test_actions_context_init. - Consolidate test_v3_server_minted.py and test_v3_38_drift_fixes.py into test_v3_wire_contract.py. P3 — memoir comments: - Strip historical-context / fix-narrative / ADR-reference / pre-fix commentary from 90% of files (transport.py / runtime.py / decorators.py / breaker/exceptions.py / observability/__init__.py / test_runtime.py / test_protect.py / test_actions.py / test_transport.py / test_v3_wire_contract.py / conftest.py). Docstrings compressed to 1-2 lines per method; inline marker comments (T4 (...), P0-4, FIX-F3, PR #N, 2026-07-02, ADR-008, observed: ..., pre-fix, ...) collapsed to a single short line. - Replace 'Merged from X.py' section markers with semantic headers. P4 — CHANGELOG deduplication: - src/nullrun/__version__.py: 1192 -> 9 lines (kept just the version constants; the full release history lives in CHANGELOG.md). - pyproject.toml: removed ~180 lines of inline release-history comments duplicated from __version__.py; only the current version is pinned. Verification: 1341 passed, 7 skipped, 2 warnings in 77.86s. * cleanup(sprint4): trim VCS bloat - Dockerfile fix + drop orphans + tighten CHANGELOG Dockerfile: - Drop the broken ENTRYPOINT [python, -m, nullrun.breaker]: nullrun.breaker is a package with no __main__.py and no console_scripts entry in pyproject.toml. The SDK is a library, not a service. Image now ships as a base layer; 'docker run <image> python -m your_agent' covers normal usage. No CI workflow ever built this image (orphan). Dockerfile.dev: - Delete. 404 B, CMD 'tail -f /dev/null' antipattern, no CI consumer. docs/assets/banner.svg: - Delete. 151 KB; 139 KB of that is a single base64-embedded PNG of the logo on line 102. Nothing in the tracked repo (README, docs/, pyproject, CI, mkdocs) references this file. Original is recoverable from git history if needed. CHANGELOG.md: - Drop 126 KB -> 52 KB (-59%), 2035 -> 865 lines. Three trimming passes: 1. Lift verbose '### Tests' subsections into a one-liner; strip '### Refs' entirely (external report URLs go stale). 2. Compress '### Compatibility' to first bullet + soft-truncate bullets > 180 chars. 3. Cap each release entry to max 35 lines. The 8 most-recent releases (0.14.x + 0.13.13/0.13.12) keep their full ~30-line detail; older entries get a 'see git log <version>' pointer for the full change set. Total: 4 files changed, 96 insertions(+), 1516 deletions(-). * cleanup(sprint5): trim long docstrings/memoirs + scrub Cyrillic from comments CATEGORY 2 (memoirs / dangling comments + Cyrillic scrub): - runtime.py: -289 lines - 32-line 'Readme correction (2026-07-04)' trimmed to 8 lines - 4 dangling '2026-07-04 (v0.12.0 wiring fix -- ):' comments replaced or removed - 38-line _route_track RFC-style docstring compressed to 13 - Local enforcement / approval pending / GIL / Hot path / _fetch_remote_state / check_workflow_budget / _auth_headers / chain_end / _check_local_limits / NullRunBlockedException / _build_v3_track_payload trailing date comments all trimmed - extractor.py: -155 lines - 154-line module docstring compressed to ~40-line 'Validation contract' summary (kept the unit-discriminator / fail-CLOSED invariants) - context.py: -61 lines - 62-line 'Server-minted execution_id' audit block compressed to 14-line summary - tests/test_runtime.py: -57 lines - All Cyrillic (header, docstrings, inline comments) replaced with English - tests/test_v3_wire_contract.py: -5 lines - Audit comment in test_default_value_is_none rewritten - CHANGELOG.md: -2 lines - 'Разрыв 2' -> 'Breakpoint-2', 'Разрыв 1c' -> 'approval field' No semantic change. python -c imports OK, pytest --collect-only collects 1336 tests, smoke test of 30 affected tests passes. Follow-up: dead-code, duplication, CHANGELOG bloat, CI/build, docs. * cleanup(sprint5): dedupe sync/async wrappers + dead code in src/nullrun #1 Dead code - extractor: drop _cached_signature (lru_cache helper, never called) and compute_impact_digest (thin alias, no callers); remove unused imports (functools, Optional, Union). - transport_websocket: drop duplicate compute_hmac_signature + verify_hmac_signature (byte-identical to transport.py); re-export from transport. Update test imports. - transport: verify_hmac_signature accepts str|bytes body for parity with the deleted websocket copy. - _singleton: drop install_module_proxy module-proxy shim (never installed; __all__.append now removed). - _registry: drop replace_for_test (no callers). - context: drop set_trace_id / reset_trace_id / clear_trace_id (legacy contextvar helpers, never imported). - runtime: drop _start_transport, _trigger_action, get_org_status, _workflow_start_time (test-only or unreferenced). #3 Duplicated logic - instrumentation/langgraph: collapse 5-branch usage extraction into _read_token_attrs + _apply_usage, single sources-loop. - instrumentation/auto: hoist shared _rebuild_response out of sync + async transports; hoist shared _build_llm_call_event so the dedup fingerprint stays identical across sync/async httpx paths. - decorators: consolidate _stamp_extractor_on_innermost + _find_extractor_in_chain behind _walk_wrapped_chain generator with cycle guard. - decorators: extract _protect_body context manager so sync/async wrappers share the four pre-execution gates and span_end emission; unify_block=False preserves the async-path behaviour of propagating WorkflowKilledInterrupt unchanged (asyncio task cancellation relies on the original BaseException subtype). Tests: 1334 pass, 2 skip (pre-existing). * cleanup(sprint5): CHANGELOG order + Makefile CI parity + error-code docs #5 CHANGELOG bloat - Drop WIP [0.10.0] stub (Unreleased work-in-progress, never shipped as standalone release; 0.11.0 became the canonical v3.0 cut). - Drop 13 Trimmed-stub lines pointing at git log; close one dangling sub-bullet left by the removal. - Reorder release blocks in strict descending version order: was 0.9.1 -> 0.11.0 -> 0.9.0 (lower: 0.3.1 -> 0.5.2 -> 0.4.0); now 0.11.0 -> 0.9.1 -> 0.9.0 (lower: 0.5.2 -> 0.4.0 -> 0.3.1). Net: -29 lines, semver -> date sort invariant holds. #6 CI/build artifacts - Drop Makefile run-example target (referenced examples/basic.py; examples/ was deleted in 0.3.1 alongside the gRPC transport). Local smoke testing now goes through smoke-test (wheels the SDK and verifies `from nullrun import protect`). - Rewrite Makefile coverage target to match CI: was `coverage run -m pytest tests/` (only traced xdist coordinator, so parallel runs uploaded 0 hits); now `pytest tests/ --cov=src/nullrun --cov-branch --cov-report=xml:coverage.xml --cov-report=term`, matching .github/workflows/ci.yml:82. - clean target now also removes coverage.xml. #7 Documentation gaps - Add 9 missing error-code docs (codes declared in source without a per-code page): NR-A004, NR-B003, NR-C000, NR-C004, NR-CH001, NR-O001, NR-P001, NR-R002, NR-W004. - Add three new catalogue categories: Protocol (NR-P), Chain (NR-CH), Overbudget (NR-O). README.md catalogue now covers all 23 documented codes. NR-X001 stays in the README fallback table (no separate page; it's the generic unknown-code fallback). Verified via cross-check: all source-referenced codes are documented. Tests: 23/23 exception hierarchy pass; full suite remains green. * chore(release): 0.14.10 — Sprint 5 internal cleanup Bump __version__ 0.14.9 -> 0.14.10 and add the matching CHANGELOG entry. Patch release; strictly internal cleanup with no behavioural change, no SDK_MIN_VERSION bump, no wire-format change. Backward-compatible drop-in for 0.14.9. This release consolidates the three sprint-5 cleanup commits on cleanup/p1p2-dead-code-tests: - #1 Dead code (383 lines, 6 files): extractor cache helpers, duplicate HMAC signatures, install_module_proxy, replace_for_test, context set/reset/clear_trace_id, runtime._start_transport + _trigger_action + get_org_status + _workflow_start_time. - #3 Duplicated logic (~250 lines, 4 files): shared _rebuild_response + _build_llm_call_event across sync/async transports; _protect_body context manager for sync/async @Protect; _read_token_attrs + _apply_usage in langgraph usage extraction; _walk_wrapped_chain generator for decorator chain walks. - #5 CHANGELOG bloat (-29 lines): dropped WIP [0.10.0] stub + 13 Trimmed placeholders; fixed descending-version sort order. - #6 CI/build: dropped Makefile run-example (missing examples/basic.py); rewrote coverage target to match CI's pytest --cov pipeline. - #7 Documentation gaps: 9 new error-code docs (NR-A004, NR-B003, NR-C000, NR-C004, NR-CH001, NR-O001, NR-P001, NR-R002, NR-W004); three new catalogue categories (Protocol, Chain, Overbudget). Tests: 1334 pass, 2 skip (pre-existing); 23/23 exception hierarchy pass. No public API change. * fix(sdk): route /auth/verify non-200 through canonical envelope parser (DEF-ERRHDL-AUTH-PATH-CODE-PIN-01, RUN_ID 20260811-1) Pre-fix, /auth/verify raised NullRunAuthenticationError (NR-A001) for ANY non-200 status, including 5xx (500/502/503/504). The canonical dispatcher at transport._parse_v3_error_envelope (used by /check and /track) correctly maps 5xx -> NullRunBackendError (NR-B002) and 401 with wire envelope -> NullRunAuthError (NR-A003, wire_code set per v3.38). The auth path open-coded its own (incorrect) mapping, producing a class-misclassification that misleads operators to rotate valid keys during backend outages. Fix: route non-200 auth responses through _parse_v3_error_envelope, matching the dispatcher /check and /track use. Lazy import inside the else arm keeps runtime.py's top-level import graph stable. Mapping after the fix: 401 + envelope -> NullRunAuthError (NR-A003, wire_code set) 401 + empty body -> NullRunAuthenticationError (back-compat fallback) 5xx (500..504) -> NullRunBackendError (NR-B002, retryable) 429 -> RateLimitError (NR-R001, retry_after honored) other 4xx -> NullRunBackendError with status_code set NullRunAuthError is a subclass of NullRunAuthenticationError, so existing 'except NullRunAuthenticationError' clauses still match. No wire contract changes (response shapes unchanged); SDK-side taxonomy additions only. Tests: 5 new regression tests in tests/test_runtime.py pin the per-status mapping. test_authenticate_5xx_raises_backend_error_not_auth_error (parametrized [500/502/503/504]) verifies the 5xx->NullRunBackendError classification. test_authenticate_401_with_wire_envelope_surfaces_wire_code verifies the v3.38 wire_code contract for /auth/verify. Verification: pytest tests/test_runtime.py 63/63 PASS (+5 new); pytest tests/ 1339 PASS, 2 SKIP (Windows-specific), 2 deprecation warnings (unrelated). Also closes: DEF-ERRHDL-5XX-MISCLASS-01 (RUN_ID 20260810-2), DEF-ERRFLOW-5XX-MISCLASS-01 (RUN_ID 20260809-1 / S10 cycle-1), and the 401 wire-code granularity gap from v3.38 in the auth path. Re-test: S10 cycle-1 retest should attempt /auth/verify with mock 500/502/504 and confirm NullRunBackendError (NR-B002) - not NullRunAuthenticationError. Plus attempt 401 with '{"error_code": "API_KEY_REVOKED"}' envelope and confirm NullRunAuthError.wire_code == 'API_KEY_REVOKED'. * Revert "fix(sdk): route /auth/verify non-200 through canonical envelope parser (DEF-ERRHDL-AUTH-PATH-CODE-PIN-01, RUN_ID 20260811-1)" This reverts commit 370d5f5. * Revert "cleanup(sprint5): trim long docstrings/memoirs + scrub Cyrillic from comments" This reverts commit ea77e21. * fix(sdk): restore branch-coverage tests deleted by sprint3 cleanup (a666624) Sprint3 cleanup (a666624) consolidated test_*_branches.py files into their main test_*.py counterparts and removed them. Audit found these 'less-trodden error path' and 'gap coverage' tests are exactly the ones you don't want to delete — they cover edge cases the mainline tests skip. Removing them = silent coverage regression. Files restored (all from master HEAD): - tests/test_protect_branches.py (564 lines) — branch coverage for _safe_args / _strip_details_balanced / _enforce_sensitive_tool - tests/test_runtime_branches.py (517 lines) — less-trodden error paths in runtime.py. Removed 2 tests (test_start_recording_returns_* and test_stop_recording_returns_none) because a666624 P1 also intentionally removed the deprecated no-op stubs from runtime.py (replaced by direct return-value gates per the commit message). Restoring the tests without the methods would create dead tests. - tests/test_transport_branches.py (647 lines) — branch coverage gaps in transport.py Verification: pytest tests/ → 1462 passed, 6 skipped, 0 failed. The 6 skipped are pre-existing environment markers. Pairs with commit 700b0af (revert of ea77e21 Cyrillic scrub). Together they close the over-aggressive parts of the cleanup sprint without disturbing the valid P1 dead-code removal, P4 CHANGELOG dedup, and v3.38/server-minted test consolidations. * chore(release): 0.14.11 — partial revert of sprint-5 cleanup Bump __version__ 0.14.10 -> 0.14.11 and add the matching CHANGELOG entry. Patch release; partial revert of two sprint-5 cleanup commits whose scope exceeded what the codebase actually supported. This release closes the over-aggressive parts of the cleanup sprint without disturbing the valid P1 dead-code removal, P4 CHANGELOG dedup, and v3.38/server-minted test consolidations. - Revert ea77e21 (Cyrillic scrub + docstring trim): restored the 30-line 'partially wrong' block in src/nullrun/runtime.py (codifies CLAUDE.md \u00a74 fail-CLOSED rules for SDK transport vs backend enforcement), restored 'Разрыв 2' / 'Разрыв 1c' in CHANGELOG.md (user-coined Russian technical nomenclature), and restored tests/test_real_e2e_observation.py (321 lines, the only real-socket integration test). - Cherry-pick restore 3 branch-coverage files deleted by a666624 P2: tests/test_protect_branches.py (564), tests/test_runtime_branches.py (515; minus 2 tests for deprecated start_recording/stop_recording no-op stubs that a666624 P1 also intentionally removed), and tests/test_transport_branches.py (647). These files explicitly documented their purpose as covering 'gaps' and 'less-trodden error paths' that the mainline tests skip. Verification: pytest tests/ -> 1462 passed, 6 skipped, 0 failed. Pairs with commits 700b0af (revert ea77e21) and 2df6b3a (restore branch-coverage tests) on cleanup/p1p2-dead-code-tests. Compatibility: No SDK_MIN_VERSION bump. No public API change, no wire-format change, no behavioural change. Drop-in replacement for 0.14.10. * feat(sdk): ADR-009 P1 governance audit read surface (0.15.0) nullrun.audit module + runtime.audit proxy + 34 tests. * chore(release): 0.15.0 — ADR-009 P1 governance audit read surface * fix(sdk): defer runtime.py annotations to avoid AuditProxy.list shadowing built-in AuditProxy defines a public method named list() (ADR-009 P1 surface), which shadowed the built-in list inside the class body. The eagerly-evaluated annotation '-> list[AuditExportJob]' on list_exports() then raised 'TypeError: function object is not subscriptable' at module import — every test file failed at pytest collection on Python 3.12. Fix: add 'from __future__ import annotations' to runtime.py so all annotations become PEP 563 lazy strings. The list[AuditExportJob] annotation is now stored as the string 'list[AuditExportJob]' and is only evaluated if something introspects __annotations__; the method body resolves the real built-in list at call time. Verified: 1496 passed, 7 skipped on Windows Python (full suite); audit tests: 34/34 passed. * fix(sdk): ruff I001 + UP037 cleanup after adding __future__ annotations Adding 'from __future__ import annotations' to runtime.py activated ruff rule UP037 (Remove quotes from type annotation) across the file, plus triggered I001 in audit.py where the future-import was positioned mid-file. Auto-fixed via 'ruff check src/ --fix': - I001 in audit.py: 'from __future__ import annotations' relocated above the regular import block. - UP037 in audit.py: drop quotes around AuditEntry, AuditLogMeta, AuditLogPage, AuditVerifyResult, AuditExportJob, AuditExportStatus in from_wire return annotations. - UP037 in runtime.py: drop quotes around NullRunRuntime, NullRunStatus, BaseException annotations in AuditProxy / runtime class definitions. Verified: 1496 passed, 7 skipped; ruff clean. * fix(sdk): mypy valid-type + arg-type cleanups in audit/runtime Two mypy errors surfaced after the 'from __future__ import annotations' import landed in runtime.py and ruff auto-fix normalised audit.py annotations: 1. audit.py AuditVerifyResult.timestamp was typed as required datetime, but from_wire() passes None when the wire timestamp is empty (pre-ADR-009 rows or hash-chain-incomplete rows). Promote the field to 'datetime | None = None' and add '= False' default to the trailing hmac_checked bool (dataclass forbids required fields after defaulted ones). 2. runtime.py AuditProxy.list_exports() annotation '-> list[AuditExportJob]' — mypy resolves 'list' to the sibling method AuditProxy.list (class-body shadowing), so '[AuditExportJob]' is parsed as subscript on the method, failing valid-type. Switch to 'builtins.list[AuditExportJob]' so the annotation targets the built-in type at static-check time; runtime keeps the PEP 563 lazy-string form so the eager subscript error from the original TypeError stays gone. Verified: mypy clean (37 files), ruff clean, pytest 1496 passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
After PR #2 (byte-mismatch + S-2) and PR #3 (Wip/working tree) merged, the working tree at HEAD still shows a 45-file diff against the initial import. The 0.3.0->0.4.0 migration that PR #3 was named after appears to have been stripped during the merge - either because the merge was a fast-forward without re-applying the wip diff, or because the diff was never on the wip branch tip in a way that survived the merge.
This commit re-stages the same content so the audit-trail, examples, decorators cleanup, and 14 new test files are actually on master. See the commit message for the full breakdown.
If this is a no-op (the content was already in master) the diff against master will be empty. CI will confirm.