Skip to content

fix(runner): split event uploads before control-plane limits - #944

Open
rudycelekli wants to merge 3 commits into
Tencent:mainfrom
rudycelekli:fix/platform-runner-event-batches
Open

rudycelekli wants to merge 3 commits into
Tencent:mainfrom
rudycelekli:fix/platform-runner-event-batches

Conversation

@rudycelekli

@rudycelekli rudycelekli commented Nov 16, 2026 •

Copy link
Copy Markdown

What changed

A burst of runner output or a backlog after a failed upload can exceed the events API's 500-record or 2 MiB limit. The runner requeues rejected uploads, so sending the same oversized batch—or a single oversized event—can block later logs indefinitely.

Split uploads by both shared server limits and actual encoded JSON size. For a single oversized record, preserve one sequence/idempotency key and timestamp while sending an explicit bounded head/truncation-marker/tail representation. Ordinary log records keep their event type. Structured observation records retain that outer type and encode a valid TelemetryEvent warning (process.output_truncated); slicing their inner JSON would make the server silently discard them. The warning includes a bounded excerpt of the original serialized observation rather than claiming to retain its full semantics. Caller events remain unchanged.

Scope and compatibility

Platform runner output. The route, envelope, sequence deduplication, 500-record maximum and 2 MiB server body limit are unchanged: the previous decode() already delegated to decodeLimit(..., 2<<20). This PR names that same limit through the shared constant; it does not tighten old runners' server limit. Updated runners can upload bounded representations of records that older runners repeatedly failed to upload. Oversized observations lose full structured payload details and become explicit warning observations.

Validation

  • Actual client code with temporary localhost HTTP fixtures: the existing 520-record escaping-heavy batch test plus oversized plain, JSON-escape-heavy, UTF-8, and nested TelemetryEvent cases pass twice. All requests are decoded under the real 2 MiB cap. Sequences and neighboring events remain ordered and unchanged. Structured replacement is decoded with the same domain.TelemetryEvent type used by the server consumer.
  • Before the correction, all four oversized cases fail with the permanent upload-size error.
  • Focused go vet, public-content scan, gofmt and git diff --check pass.
  • Full package/platform checks and browser acceptance were not run locally: the initial package check lacked generated web/native-chat, and disk budget precluded frontend/dependency builds. Current upstream was merged and focused tests rerun. Tests explicitly compile the production client.go with the regression test file, without fake frontend assets. The Linux-only secret-scan script could not download its tool in the restricted environment; no successful secret scan is claimed.

No live runner, credentials, paid model or external knowledge service was used. AI-assisted correction with a separate diff review. Hosted platform CI is requested and its result remains separate from these focused local checks.

@Yanami-2K Yanami-2K left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

按服务端的条数和字节上限拆批、并把两个上限提成 domain 里的共享常量,这个方向是对的,测试也断言了 520 条事件按顺序全部送达。但还剩一种情况会重新引发本 PR 要修的永久卡死:

  1. internal/runner/client.go SendEvents:单条事件编码后超过 RunnerTaskEventBodyLimit 时,直接 return fmt.Errorf("runner event %d exceeds the upload size limit")。按描述,调用方遇到错误会把整批事件重新入队,于是这条超大事件每次 flush 都失败,排在后面的日志也永远上传不了,和原来“超限批次反复重试”是同一类问题。而且报错时,当前已经放进 batch 的合法事件也没有发出去。建议把超大事件的 Data 截断(保留头尾加一个截断标记),或按同一个 Seq 拆成多条,让它无论如何都能上传;至少不要返回一个会触发重新入队的错误。请补一个测试:一批里包含一条超过 2 MiB 的事件,断言其余事件按顺序送达,且调用不返回错误。
  2. internal/httpapi/runners.go runnerTaskEventsIngest:decode 改成了 decodeLimit(..., 2<<20)。请在描述里说明原来 decode 的默认上限是多少。如果原来大于 2 MiB,这就是一次服务端收紧,会影响还没升级的旧 runner,需要写入兼容性说明。

本 PR 只跑了开源扫描,合入前请触发 platform CI。

@rudycelekli

Copy link
Copy Markdown
Author

Thanks—you're right about the single oversized record: returning that error left the same permanent retry blockage, including valid records already buffered ahead of it. I've changed that path to a bounded head/marker/tail representation, preserving the sequence key and timestamp. Structured observation data gets a valid TelemetryEvent warning rather than invalid truncated JSON, so the server consumer can decode it.

The regression now includes >2 MiB plain and UTF-8 records, an escaping-heavy record whose encoded size exceeds 2 MiB, and a real serialized TelemetryEvent. All four cases fail on the previous head; the correction plus the existing 520-event batching test passes twice against a size-capped HTTP fixture. Other logs arrive in order, caller inputs remain unchanged, and the nested observation uses the server's actual domain type for decoding.

I also corrected the compatibility section: the original decode() already imposed 2 MiB (decodeLimit(..., 2<<20)), so the explicit shared constant does not tighten that limit for older runners. Full platform CI still needs to pass; focused local checks are documented separately and no full-platform success is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants