Skip to content

Recover transient span-plugin timeouts and expose active holds - #209

Open
Stephen Belanger (Qard) wants to merge 1 commit into
mainfrom
fix/span-plugin-timeout-recovery
Open

Stephen Belanger (Qard) wants to merge 1 commit into
mainfrom
fix/span-plugin-timeout-recovery

Conversation

@Qard

Copy link
Copy Markdown
Collaborator

A span's conversion to JavaScript can exhaust the old 50 ms deadline before a small redaction filter executes. The resulting Error: interrupted points into the filter, and an unchanged plugin can leave a route held indefinitely after a transient timeout.

Raise the call budget to one second, retaining the sandbox memory/stack limits and runaway interruption. Classify timeouts from the actual deadline handler or elapsed call budget, report the failing phase and conversion/execution timings, and preserve the underlying exception. Check the conversion budget before invoking the filter. Ordinary plugin errors keep their original messages and remain fail closed.

Timeout-held delivery gets two automatic retries with backoff from its durable SpanOp position. After exhaustion it waits for relevant plugin changes or a runtime version change. Persist the retry evidence and attempt count, avoid caching a timeout across submissions, and continue isolating healthy sessions/routes. No operation bypasses its configured transforms.

Doctor now separates historical error occurrences from current held sessions, with queued canonical span-operation counts, retry attempts, and recovery conditions. These operation counts are distinct from unique spans and from native events awaiting translation. New status/recovery fields deserialize older records with defaults.

Validation on macOS:

  • Full all-feature Rust test suite passed; final library run passed all 287 tests.
  • New regressions cover conversion-budget exhaustion before the filter, a large valid redaction input, retained property-error details, runaway execution, bounded persisted timeout retries/runtime upgrades, automatic journal replay without a plugin edit, and human/JSON hold diagnostics.
  • All-target/all-feature Clippy with warnings denied, formatting, and git diff --check passed.

This addresses the shared daemon behavior behind the Pi timeout and follow-up reports. The reports demonstrate a failure mechanism; they do not establish the triggering spans or historical trace completeness for the October 8 incidents. Deployment requires a bt dependency update and release.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-20T18:21:12.405858Z ab63b77 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant