Repository navigation
fix(otel): name a CronJob pod's service after its CronJob - #414
Conversation
A CronJob pod within its schedule window has a /k8s-cronjob/<ns>/<cronjob>/<container> container id, but ContainerIdToOtelServiceName returned early for anything not starting with /k8s/, so its CronJob pattern never ran. Those pods' logs, traces and profiles carried the raw container id as service.name, and their metrics an empty app_id, apart from every other run of the same CronJob. They now get /k8s/<ns>/<cronjob>, the name a run outside the window already gets from its pod name. k8s.cronjob.name comes from the pod's resolved owner, not the container id, so it was not affected.
There was a problem hiding this comment.
Code Review
This pull request updates the ContainerIdToOtelServiceName function in common/otel.go to correctly map CronJob container IDs (starting with /k8s-cronjob/) to their corresponding service names under /k8s/, and adds corresponding unit tests in common/otel_test.go. The review feedback suggests replacing the regex matching with a simpler, more efficient, and more robust string parsing approach using strings.HasPrefix and strings.SplitN. This change would also resolve a limitation where CronJob names containing dots are not matched by the current regex. Additionally, the reviewer recommends adding a test case to verify the parsing of CronJob names containing dots, using a hyphenated placeholder name to prevent potential TLD-related test failures.
blue4209211
left a comment
There was a problem hiding this comment.
Approve. Small, correct fix at the cause. Nothing blocks merge.
Non-blocking
- Gemini's dotted-name suggestion isn't needed. The id is built from
cronjobPodName, which only allows[a-z0-9-]+(containers/registry.go:42), so a dot can never reach this regex. String parsing would gain nothing. - The real dotted-name edge is in
registry.go, and it was already there.cronjobPodNameisn't anchored, so a podmy.job-28283967-abcdematches asjoband the id becomes/k8s-cronjob/ns/job/c. A CronJob with a dot in its name gets a shortened service name. Rare, so anchoring it with^…$is a small follow-up. - Old rows keep the raw id until they age out. Searches by service name over a window that spans the upgrade will miss them. Worth a release-note line, no code needed.
Checked and fine
From reading the code; CI green.
- Who is affected: the window is ±7 days (
registry.go:43), so nearly all CronJob pods had this bug. - Callers: all four use the new name as intended (logs, traces, profiles,
app_id)..appIdis otherwise only logged, so nothing branches on it being empty. app_id: one value per CronJob, so cardinality doesn't grow. Those containers' metric series do change label on upgrade.k8s.cronjob.name: set from the owner map (logs/otel.go:66), not the id.- Other consumers: nothing in the backend reads
k8s-cronjob, so no parser depends on the old form. - Regex change: the
^anchor and moving the check ahead of the/k8s/return are both safe. The old loop entry could never match. - Tests: the new cases would fail on
main, since the early return kept the raw id.
What this PR does
A CronJob pod within its schedule window has a
/k8s-cronjob/<ns>/<cronjob>/<container>container id (containers/registry.go).ContainerIdToOtelServiceNamereturned early for any id not starting with/k8s/, so its CronJob pattern never ran. Those pods' logs, traces and profiles carried the raw container id asservice.name, apart from every other run of the same CronJob.They now get
/k8s/<ns>/<cronjob>: the name a run outside the window already gets from its pod name (/k8s/default/hello-28283967-khz2f/xz→/k8s/default/hello).For reviewers:
app_idmetric label (containers/container.go:227). For in-window CronJob containers it goes from empty to/k8s/<ns>/<cronjob>, the value out-of-window runs already have. That's one value per CronJob, so cardinality doesn't grow.k8s.cronjob.namecomes from the pod's resolved owner, not the container id, so it was not affected.Linked issue / context
Reported against the ClickHouse logs setup (
logs.collection=node-agent): searching a CronJob's logs by service name found nothing.How was this tested?
common/otel_test.go: an in-window CronJob id (app container and sandbox) maps to the same name as the out-of-window run, and a too-short/k8s-cronjob/id is left as it is. They failed before the fix and pass after it; the rest ofcommonpasses.go vet ./common/and gofmt are clean./k8s-cronjob/but don't start with it, are left unchanged, as before.To check on a cluster: with a CronJob running on a node with this build and a logs endpoint set, its records carry
service.name=/k8s/<ns>/<cronjob>, not/k8s-cronjob/....eBPF changes?
No.