Skip to content

refactor(observability): send each request's spans when its scope closes - #848

Merged
anandpant merged 2 commits into
mainfrom
refactor/phase3-telemetry-exporter
Nov 16, 2026
Merged

anandpant merged 2 commits into
mainfrom
refactor/phase3-telemetry-exporter

Conversation

@anandpant

@anandpant anandpant commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Send each request's spans when its scope closes

The telemetry exporter now uses native Effect scope ownership instead of its own per-request batch, open-span counter and cross-request held queue. A request's root span owns the spans beneath it. withTelemetryExport({ pipeline, scope }) registers one finalizer on the host's request scope; closing the scope sends every row once and waits for the sends. Each host closes its scope under waitUntil after its late work.

sketchi.telemetry.span.v1 rows, full content, chunking, start_seq, span names and correlation are unchanged. rows.ts, json.ts, the schema, the DuckDB SQL and the log-analysis skill are untouched.

Changes

  • Exporter (packages/observability): RequestSpans replaces SpanBatch. A failed binding send is logged once at Error, never retried and never raised, and other batches still go. A span that ends after its request was sent is reported by console.error. Spans outside every request are no longer stored or held for a later request.
  • Studio runtime: each request has its own scope, and its callback FiberSet is now per request instead of host-lifetime. The scope closes under waitUntil once the handler, response stream and callbacks have ended. The stream recorder now marks a cancelled stream ended only after the source's cancel settles (cde7cab4), so the interrupted model work's spans are kept.
  • Studio and Icons MCP: each handler's scope closes on dispose, after native runtime disposal. serveStatelessMcp already runs dispose under waitUntil after the body.
  • Icons HTTP: the scope closes after the handler resolves or rejects.
  • PNG renderer: one scope per call, closed through the Durable Object's waitUntil.
  • Docs: telemetry mechanism row; quota row records Fable's decision that IconDrawBudget/ClientRateLimit stay the 156-line structured Durable Object KV implementation, a named host gap; architecture and the package README.

Why not native OtlpExporter

An offline probe with a Pipeline-binding HttpClient showed two problems. A binding failure reported as a transport error sends the same batch four times (fixed retryTransient), which would store duplicate rows. Any other failure drops the request's later batches for 60 s, with only a Debug log. Neither policy is configurable, so this is a named upstream gap.

Proof

  • Same rows: one representative request (correlated root, content, typed failure, logs, metric, 1.2 MB multipart attribute, late stream span, late callback) through base and candidate gives identical rows once IDs, times and stack frames are normalized. Sends go from 3 to 1; bytes are equal apart from stack paths and batch framing.
  • Studio boundary tests:
    • An SSE response returns with an open stream, a pending callback and a blocked send. Rows are sent only after the stream and callback end, and the request's waitUntil waits for the send. This test fails if the scope closes before late work ends or if the send is not awaited.
    • Client cancel, request abort and a model call missing its deadline inside the stream each keep the failed or interrupted spans. Client cancel lost the model span before cde7cab4.
  • Concurrent requests: the existing Icons HTTP test still keeps two overlapping requests' rows separate.
  • Gate: full pnpm ci:local --no-signoff against origin/main passed on ec900e0d (3m22s) and on cde7cab4 (3m11s).
  • Signoff: pnpm ci:local on the pushed head cde7cab4 passed and posted the local-ci commit status (success).
  • Independent review: Sol (t3rj) accepts cde7cab4. It reproduced the original cancel loss and the fix. Its own probes (typed rejection, defect, abort before headers, hung stream, nested callback) pass, and two mutations fail as expected.

Size (base b8a559f5 → cde7cab4)

effect-telemetry.ts 687 → 628 lines. Source +125/−171 (−46 net, host seams included). Tests +203/−59: the Studio boundary tests are new; the package tests are 17 lines shorter. Docs +18/−12.

Accepted limits

  • A request's rows go out once, when its scope closes. A hung but live stream delays every row of that request, and its held spans and content grow in memory with no cap until it ends or is cancelled. If the isolate dies first, every row of that request is lost; AI Gateway logs still show its model calls.
  • A callback started after the request's callbacks have drained is not guaranteed delivery; it is reported, not stored. No current product caller does this.
  • Streams with no deadline: Studio chat (MAX_AGENT_STEPS, request signal, client cancel and gateway limits only), the runtime SSE pass-through (inherits its source) and MCP response-body reading.
  • Existing behaviour, unchanged: a request-signal abort closes the source normally, so the stream span lacks http.response.body.complete=false.

Live preview proof (t3rw)

Exact-head Studio and Icons previews of cde7cab4, queried through the canonical, pinned pnpm db --preview (main 3ef71560, after the separate DB fix in #849).

  • Preflight, no paid calls: model-free MCP and Icons HTTP checks; 20 complete v1 spans across 9 smoke traces, with no duplicate parts, missing parents or truncation.
  • One paid request: one POST to /api/v1/generate, one model attempt, no retries or repairs, 2,523 tokens. Gateway's logged cost is USD 0.0002819, not checked against an invoice.
  • Join: 33 complete sketchi.telemetry.span.v1 rows, with no duplicate parts, reconstruct exactly to the full span values. The native attempt's gateway_log_id matches the Gateway log, whose request, response and output equal the retained content. The stored artifact artifact_ce26ea634c234e93ad091da9ada476b6, fetched once, equals the accepted inline file.
  • Not exercised live: sink failure, client abort, deadline, error/defect/interruption, split multipart rows, isolate loss and production. Those failure paths rest on the local boundary tests and review above.

Production is not deployed.

Replace the per-request batch, open-span counting and cross-request held
queue with native Scope ownership. A request's root span owns its spans;
withTelemetryExport registers one finalizer on the host's request scope
that sends every row once and waits for the sends. Hosts close that scope
under waitUntil after their late work: Studio after the response stream
and callback fibers (now a per-request FiberSet), both MCP handlers on
dispose, Icons HTTP after the handler, the PNG renderer after each call.

Rows, content, chunking, start_seq and joins are unchanged. A failed send
is logged once at Error and never retried. Native OtlpExporter was probed
and rejected: it re-sends a batch four times on a transport failure and
otherwise drops later batches for 60 s, with no option to change either.
A client that cancels a streamed response interrupts the stream's model
work after the recorder had already marked the stream ended, so the
request's scope could close and send its rows before the interrupted
model span ended, losing it. Mark the stream ended once the source's
cancel settles; Effect's stream cancel resolves after its fiber stops.

Boundary tests cover client cancel, request abort and a model deadline
inside the stream. The README names the remaining limit: an isolate that
dies before the scope closes loses that request's rows.
@github-actions github-actions Bot added the preview Deploy PR previews for the affected apps label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Sketchi Icons Preview

Status: deleted

@anandpant anandpant removed the preview Deploy PR previews for the affected apps label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Sketchi Playground / Studio Preview

Status: deleted

@anandpant
anandpant marked this pull request as ready for review November 16, 2026 23:52
@anandpant
anandpant merged commit f83975c into main Nov 16, 2026
9 checks passed
@anandpant
anandpant deleted the refactor/phase3-telemetry-exporter branch November 16, 2026 23:53
@github-actions

Copy link
Copy Markdown
Contributor

Sketchi Web Preview

Status: deleted

@github-actions

Copy link
Copy Markdown
Contributor

Sketchi Excalidraw Workspace Preview

Status: deleted

This branch was successfully deployed

1 active deployment
staging — cde7cab4 Deployed Nov 16, 2026 by anandpant via cleanup-preview (playground) #1945
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant