Repository navigation
refactor(observability): send each request's spans when its scope closes - #848
Merged
Merged
Conversation
Replace the per-request batch, open-span counting and cross-request held queue with native Scope ownership. A request's root span owns its spans; withTelemetryExport registers one finalizer on the host's request scope that sends every row once and waits for the sends. Hosts close that scope under waitUntil after their late work: Studio after the response stream and callback fibers (now a per-request FiberSet), both MCP handlers on dispose, Icons HTTP after the handler, the PNG renderer after each call. Rows, content, chunking, start_seq and joins are unchanged. A failed send is logged once at Error and never retried. Native OtlpExporter was probed and rejected: it re-sends a batch four times on a transport failure and otherwise drops later batches for 60 s, with no option to change either.
A client that cancels a streamed response interrupts the stream's model work after the recorder had already marked the stream ended, so the request's scope could close and send its rows before the interrupted model span ended, losing it. Mark the stream ended once the source's cancel settles; Effect's stream cancel resolves after its fiber stops. Boundary tests cover client cancel, request abort and a model deadline inside the stream. The README names the remaining limit: an isolate that dies before the scope closes loses that request's rows.
Contributor
Sketchi Icons PreviewStatus:
|
Contributor
Sketchi Playground / Studio PreviewStatus:
|
Contributor
Sketchi Web PreviewStatus:
|
Contributor
Sketchi Excalidraw Workspace PreviewStatus:
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Send each request's spans when its scope closes
The telemetry exporter now uses native Effect scope ownership instead of its own per-request batch, open-span counter and cross-request held queue. A request's root span owns the spans beneath it.
withTelemetryExport({ pipeline, scope })registers one finalizer on the host's request scope; closing the scope sends every row once and waits for the sends. Each host closes its scope underwaitUntilafter its late work.sketchi.telemetry.span.v1rows, full content, chunking,start_seq, span names and correlation are unchanged.rows.ts,json.ts, the schema, the DuckDB SQL and the log-analysis skill are untouched.Changes
packages/observability):RequestSpansreplacesSpanBatch. A failed binding send is logged once at Error, never retried and never raised, and other batches still go. A span that ends after its request was sent is reported byconsole.error. Spans outside every request are no longer stored or held for a later request.FiberSetis now per request instead of host-lifetime. The scope closes underwaitUntilonce the handler, response stream and callbacks have ended. The stream recorder now marks a cancelled stream ended only after the source's cancel settles (cde7cab4), so the interrupted model work's spans are kept.serveStatelessMcpalready runs dispose underwaitUntilafter the body.waitUntil.IconDrawBudget/ClientRateLimitstay the 156-line structured Durable Object KV implementation, a named host gap; architecture and the package README.Why not native
OtlpExporterAn offline probe with a Pipeline-binding
HttpClientshowed two problems. A binding failure reported as a transport error sends the same batch four times (fixedretryTransient), which would store duplicate rows. Any other failure drops the request's later batches for 60 s, with only a Debug log. Neither policy is configurable, so this is a named upstream gap.Proof
waitUntilwaits for the send. This test fails if the scope closes before late work ends or if the send is not awaited.cde7cab4.pnpm ci:local --no-signoffagainst origin/main passed onec900e0d(3m22s) and oncde7cab4(3m11s).pnpm ci:localon the pushed headcde7cab4passed and posted thelocal-cicommit status (success).cde7cab4. It reproduced the original cancel loss and the fix. Its own probes (typed rejection, defect, abort before headers, hung stream, nested callback) pass, and two mutations fail as expected.Size (base
b8a559f5→cde7cab4)effect-telemetry.ts687 → 628 lines. Source +125/−171 (−46 net, host seams included). Tests +203/−59: the Studio boundary tests are new; the package tests are 17 lines shorter. Docs +18/−12.Accepted limits
MAX_AGENT_STEPS, request signal, client cancel and gateway limits only), the runtime SSE pass-through (inherits its source) and MCP response-body reading.http.response.body.complete=false.Live preview proof (t3rw)
Exact-head Studio and Icons previews of
cde7cab4, queried through the canonical, pinnedpnpm db --preview(main3ef71560, after the separate DB fix in #849)./api/v1/generate, one model attempt, no retries or repairs, 2,523 tokens. Gateway's logged cost is USD 0.0002819, not checked against an invoice.sketchi.telemetry.span.v1rows, with no duplicate parts, reconstruct exactly to the full span values. The native attempt'sgateway_log_idmatches the Gateway log, whose request, response and output equal the retained content. The stored artifactartifact_ce26ea634c234e93ad091da9ada476b6, fetched once, equals the accepted inline file.Production is not deployed.