Repository navigation
Recover complete network metrics after interrupted interface dumps #5522
Description
🤖 Generated by the Agentic Engineer
Problem
Network metric collection stops immediately when the kernel marks a network-interface dump as interrupted. Interface churn can make that snapshot inconsistent, so a subsequent complete read is needed before publishing its results.
The pinned collector currently propagates either link-list or address-list interruption directly: collector source. Its dependency identifies this as a typed incomplete-dump error: dependency source. A current collection failure has been verified privately; raw telemetry is kept out of this issue.
Intended outcome
Retry only interrupted dumps within a small fixed bound, discard partial results, and publish a complete successful snapshot. Persistent interruption and unrelated failures must continue to fail visibly.
Acceptance
- Demonstrate RED/GREEN controls for link and address interruption, wrapped errors, successful recovery, exhausted retries and unrelated failures.
- Preserve existing collection behavior and the reviewed image publication and signature gates.
- Deliver a separate immutable-image activation only after its producer is merged and verified.
- Confirm fresh network metric coverage and absence of this error during a new observation window after deployment; elapsed time alone does not close this issue.
Part of #2043
Delivered on: 2026-11-01
Measure on: 2026-11-02
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
chore: agent-claim #5522 nonce=6ee27454aded9de4429022440946304f
chore: agent-claim #5522 nonce=df7f2ecc6a1abd3668bcc92829bae9bc
chore: agent-claim #5522 nonce=69a9d2155fba9c820f448cd635e71874
devantler commentedon Oct 29, 2026
The main-source publisher for #5524 completed successfully. Independent signature verification binds image
ghcr.io/devantler-tech/platform-coroot-node-agent@sha256:6a5301bb88803ec61b794b0624f0345f1f3110a65c2c7f6290599b115870f509to the protected workflow and merged sourcea2a39865f4bed3cf39d89f88bc594177fcb24a04. The downloaded Linux executable layer matches its registry digest.Adoption remains held: the actual executable uses Go 1.24.9 and identifies its patched upstream module by a pseudo-version. A fresh symbol-level binary audit reports unresolved dependency findings. Its successful JSON-producing exit is not a clean vulnerability verdict; the findings themselves govern. Detailed audit data stays in private operator evidence.
Repair and independently validate the producer’s compiler, dependency disposition and release identity before proposing a separate immutable-image activation. Preserve the bounded interrupted-dump tests and every signature/admission boundary. No image pin, deployment, alert objective or acceptance criterion was changed, and no production recovery is claimed.
devantler commentedon Oct 31, 2026
Scoping result for the producer repair. No code changed and no image, pin or deployment moved.
Moving the pinned source to upstream v1.38.1 looks better than bumping dependencies one at a time on v1.35.8:
Still open:
Next step: resolve the hunk, bump dependencies inside the patch, move the source pin, runtime image and compiler together, then add the count-only audit to the workflow's verify job. Moving three upstream minor versions changes real agent behaviour, so activating the image stays a separate reviewed change. Claim released.
chore: agent-claim #5522 nonce=34001acc38683eab0fc318ea7f62cb33
fix(coroot): keep the OTLP log exporter on the bumped log API
fix(coroot): keep the OTLP log modules at the versions the cluster ag…
fix(coroot): build the agent with slice Prometheus labels
fix(coroot): patch the cluster agent for the fixed OTLP log API
devantler commentedon Nov 1, 2026
The producer repair is in draft #5643: the retry fix on upstream v1.38.1, a current compiler, patched dependencies, a named release identity, and a publication gate on advisories the built binary reaches. Its verify job passes with none reached, and an earlier head proved the gate stops publication.
No image, pin or deployment has moved. Still owed after merge: verify the published signature, then a separate activation change, then a fresh coverage and error observation window. This issue stays open for those.
chore: agent-claim #5522 nonce=7d62bd939ce8a18400e3f0b5c01e1826
devantler commentedon Nov 1, 2026
Deployed. #5643 published the rebuilt image and #5647 moved all nine node agents to it at about 04:55Z. Cosign verifies the digest against the main-branch publish workflow.
First read, 05:10Z, 10 to 15 minutes after each agent started. No restarts. No error-level lines in 3,188 log lines. Memory averages 390 MiB, highest 512 MiB; before the change it averaged 493 MiB, highest 648 MiB. These agents are young, so that gap may close. The missing Talos journal now logs as a warning on every node, which is the fix from the #5643 review working on real nodes.
This does not close the issue. The acceptance still needs fresh network metric coverage and no interrupted-dump error over a new observation window. No interrupted-dump line has appeared yet, but 15 minutes says little. Next read: at least 24 hours after 05:10Z, on 2026-11-02, including memory against the 1 GiB limit.
devantler commentedon Nov 2, 2026
24-hour read, 05:11 UTC on 2 November: no sign of the error, but this does not close the issue.
Still not established:
Next step: one read of network metric coverage per node, through Coroot or the metrics store.
devantler commentedon Nov 2, 2026
24-hour read, 2 November about 05:15Z, covering 05:07Z on 1 November onward on all nine node agents.
The read filtered for the words interrupted, incomplete and dump. It has no positive control: the original error has not recurred in this window, so this cannot show the retry path ran, only that no failure surfaced.
Not read: fresh network metric coverage. That is the last acceptance item. This host cannot reach the metric store (pod proxy is forbidden), so I did not check it. The issue stays open. It needs one look at Coroot's network view signed in, or a query from inside the cluster.
devantler commentedon Nov 2, 2026
Measure date reached, 2 November: the fresh network metric coverage reading is still missing, so this stays open. A read-only query to the metric store was refused as forbidden at about 16:00 UTC. That path is now tracked in devantler-tech/monorepo#4750. Nothing here says the coverage is bad: it has not been read.
chore: agent-claim #5522 nonce=cd7c44002f6673846671fcff90ffcb55
devantler commentedon Nov 3, 2026
Acceptance verified on 3 November at about 07:05 UTC. Coroot’s Nodes view, set to the last five minutes, shows both receive and transmit metrics for all nine current node agents. The nine node identities match the live DaemonSet; none has an empty network reading. Values changed when the view moved from one hour to five minutes.
All nine agents are ready, have zero restarts, and run the exact immutable digest activated by #5647. A fresh read of 100,341 retrievable log lines from all nine agents contains zero error-level lines and zero interrupted/incomplete-dump matches. Available logs span approximately 9–24 hours per agent; one agent joined more recently and one older agent’s logs have rotated. This complements the earlier full 24-hour observation; it does not claim every current agent has a full day of retained logs.
This supplies the remaining fresh-coverage reading and closes the issue. No production interruption occurred during the observed window, so the merged RED/GREEN controls remain the proof that the bounded retry works. The separate scheduled-access task, devantler-tech/monorepo#4750, remains open; this interactive read used the existing browser session and changed no access or deployment settings.