Repository navigation
Recover transient span-plugin timeouts and expose active holds - #209
Open
Stephen Belanger (Qard) wants to merge 1 commit into
Open
Stephen Belanger (Qard) wants to merge 1 commit into
Stephen Belanger (Qard) wants to merge 1 commit into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A span's conversion to JavaScript can exhaust the old 50 ms deadline before a small redaction filter executes. The resulting
Error: interruptedpoints into the filter, and an unchanged plugin can leave a route held indefinitely after a transient timeout.Raise the call budget to one second, retaining the sandbox memory/stack limits and runaway interruption. Classify timeouts from the actual deadline handler or elapsed call budget, report the failing phase and conversion/execution timings, and preserve the underlying exception. Check the conversion budget before invoking the filter. Ordinary plugin errors keep their original messages and remain fail closed.
Timeout-held delivery gets two automatic retries with backoff from its durable SpanOp position. After exhaustion it waits for relevant plugin changes or a runtime version change. Persist the retry evidence and attempt count, avoid caching a timeout across submissions, and continue isolating healthy sessions/routes. No operation bypasses its configured transforms.
Doctor now separates historical error occurrences from current held sessions, with queued canonical span-operation counts, retry attempts, and recovery conditions. These operation counts are distinct from unique spans and from native events awaiting translation. New status/recovery fields deserialize older records with defaults.
Validation on macOS:
git diff --checkpassed.This addresses the shared daemon behavior behind the Pi timeout and follow-up reports. The reports demonstrate a failure mechanism; they do not establish the triggering spans or historical trace completeness for the October 8 incidents. Deployment requires a
btdependency update and release.