Repository navigation
fix(runner): split event uploads before control-plane limits - #944
rudycelekli wants to merge 3 commits into
Conversation
Yanami-2K
left a comment
There was a problem hiding this comment.
按服务端的条数和字节上限拆批、并把两个上限提成 domain 里的共享常量,这个方向是对的,测试也断言了 520 条事件按顺序全部送达。但还剩一种情况会重新引发本 PR 要修的永久卡死:
internal/runner/client.goSendEvents:单条事件编码后超过RunnerTaskEventBodyLimit时,直接return fmt.Errorf("runner event %d exceeds the upload size limit")。按描述,调用方遇到错误会把整批事件重新入队,于是这条超大事件每次 flush 都失败,排在后面的日志也永远上传不了,和原来“超限批次反复重试”是同一类问题。而且报错时,当前已经放进batch的合法事件也没有发出去。建议把超大事件的Data截断(保留头尾加一个截断标记),或按同一个Seq拆成多条,让它无论如何都能上传;至少不要返回一个会触发重新入队的错误。请补一个测试:一批里包含一条超过 2 MiB 的事件,断言其余事件按顺序送达,且调用不返回错误。internal/httpapi/runners.gorunnerTaskEventsIngest:decode改成了decodeLimit(..., 2<<20)。请在描述里说明原来decode的默认上限是多少。如果原来大于 2 MiB,这就是一次服务端收紧,会影响还没升级的旧 runner,需要写入兼容性说明。
本 PR 只跑了开源扫描,合入前请触发 platform CI。
|
Thanks—you're right about the single oversized record: returning that error left the same permanent retry blockage, including valid records already buffered ahead of it. I've changed that path to a bounded head/marker/tail representation, preserving the sequence key and timestamp. Structured The regression now includes >2 MiB plain and UTF-8 records, an escaping-heavy record whose encoded size exceeds 2 MiB, and a real serialized I also corrected the compatibility section: the original |
What changed
A burst of runner output or a backlog after a failed upload can exceed the events API's 500-record or 2 MiB limit. The runner requeues rejected uploads, so sending the same oversized batch—or a single oversized event—can block later logs indefinitely.
Split uploads by both shared server limits and actual encoded JSON size. For a single oversized record, preserve one sequence/idempotency key and timestamp while sending an explicit bounded head/truncation-marker/tail representation. Ordinary log records keep their event type. Structured
observationrecords retain that outer type and encode a validTelemetryEventwarning (process.output_truncated); slicing their inner JSON would make the server silently discard them. The warning includes a bounded excerpt of the original serialized observation rather than claiming to retain its full semantics. Caller events remain unchanged.Scope and compatibility
Platform runner output. The route, envelope, sequence deduplication, 500-record maximum and 2 MiB server body limit are unchanged: the previous
decode()already delegated todecodeLimit(..., 2<<20). This PR names that same limit through the shared constant; it does not tighten old runners' server limit. Updated runners can upload bounded representations of records that older runners repeatedly failed to upload. Oversized observations lose full structured payload details and become explicit warning observations.Validation
TelemetryEventcases pass twice. All requests are decoded under the real 2 MiB cap. Sequences and neighboring events remain ordered and unchanged. Structured replacement is decoded with the samedomain.TelemetryEventtype used by the server consumer.go vet, public-content scan, gofmt andgit diff --checkpass.web/native-chat, and disk budget precluded frontend/dependency builds. Current upstream was merged and focused tests rerun. Tests explicitly compile the productionclient.gowith the regression test file, without fake frontend assets. The Linux-only secret-scan script could not download its tool in the restricted environment; no successful secret scan is claimed.No live runner, credentials, paid model or external knowledge service was used. AI-assisted correction with a separate diff review. Hosted platform CI is requested and its result remains separate from these focused local checks.