Skip to content

fix(otel): name a CronJob pod's service after its CronJob - #414

Merged
blue4209211 merged 1 commit into
mainfrom
fix/nb-43217-cronjob-service-name
Nov 5, 2026
Merged

blue4209211 merged 1 commit into
mainfrom
fix/nb-43217-cronjob-service-name

Conversation

@reckless-sherixx

Copy link
Copy Markdown
Contributor

What this PR does

A CronJob pod within its schedule window has a /k8s-cronjob/<ns>/<cronjob>/<container> container id (containers/registry.go). ContainerIdToOtelServiceName returned early for any id not starting with /k8s/, so its CronJob pattern never ran. Those pods' logs, traces and profiles carried the raw container id as service.name, apart from every other run of the same CronJob.

They now get /k8s/<ns>/<cronjob>: the name a run outside the window already gets from its pod name (/k8s/default/hello-28283967-khz2f/xz → /k8s/default/hello).

For reviewers:

  • The same function sets the app_id metric label (containers/container.go:227). For in-window CronJob containers it goes from empty to /k8s/<ns>/<cronjob>, the value out-of-window runs already have. That's one value per CronJob, so cardinality doesn't grow.
  • k8s.cronjob.name comes from the pod's resolved owner, not the container id, so it was not affected.

Linked issue / context

Reported against the ClickHouse logs setup (logs.collection=node-agent): searching a CronJob's logs by service name found nothing.

How was this tested?

  • Unit tests in common/otel_test.go: an in-window CronJob id (app container and sandbox) maps to the same name as the out-of-window run, and a too-short /k8s-cronjob/ id is left as it is. They failed before the fix and pass after it; the rest of common passes.
  • go vet ./common/ and gofmt are clean.
  • A separate review ran edge-case ids. Ids with uppercase, dots or empty segments, and ids that contain /k8s-cronjob/ but don't start with it, are left unchanged, as before.
  • Not run on a node.

To check on a cluster: with a CronJob running on a node with this build and a logs endpoint set, its records carry service.name=/k8s/<ns>/<cronjob>, not /k8s-cronjob/....

eBPF changes?

No.

A CronJob pod within its schedule window has a /k8s-cronjob/<ns>/<cronjob>/<container>
container id, but ContainerIdToOtelServiceName returned early for anything not
starting with /k8s/, so its CronJob pattern never ran. Those pods' logs, traces
and profiles carried the raw container id as service.name, and their metrics an
empty app_id, apart from every other run of the same CronJob.

They now get /k8s/<ns>/<cronjob>, the name a run outside the window already gets
from its pod name. k8s.cronjob.name comes from the pod's resolved owner, not the
container id, so it was not affected.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the ContainerIdToOtelServiceName function in common/otel.go to correctly map CronJob container IDs (starting with /k8s-cronjob/) to their corresponding service names under /k8s/, and adds corresponding unit tests in common/otel_test.go. The review feedback suggests replacing the regex matching with a simpler, more efficient, and more robust string parsing approach using strings.HasPrefix and strings.SplitN. This change would also resolve a limitation where CronJob names containing dots are not matched by the current regex. Additionally, the reviewer recommends adding a test case to verify the parsing of CronJob names containing dots, using a hyphenated placeholder name to prevent potential TLD-related test failures.

Comment thread common/otel.go Resolved
Comment thread common/otel_test.go Resolved

@blue4209211 blue4209211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Small, correct fix at the cause. Nothing blocks merge.

Non-blocking

  1. Gemini's dotted-name suggestion isn't needed. The id is built from cronjobPodName, which only allows [a-z0-9-]+ (containers/registry.go:42), so a dot can never reach this regex. String parsing would gain nothing.
  2. The real dotted-name edge is in registry.go, and it was already there. cronjobPodName isn't anchored, so a pod my.job-28283967-abcde matches as job and the id becomes /k8s-cronjob/ns/job/c. A CronJob with a dot in its name gets a shortened service name. Rare, so anchoring it with ^…$ is a small follow-up.
  3. Old rows keep the raw id until they age out. Searches by service name over a window that spans the upgrade will miss them. Worth a release-note line, no code needed.
Checked and fine

From reading the code; CI green.

  • Who is affected: the window is ±7 days (registry.go:43), so nearly all CronJob pods had this bug.
  • Callers: all four use the new name as intended (logs, traces, profiles, app_id). .appId is otherwise only logged, so nothing branches on it being empty.
  • app_id: one value per CronJob, so cardinality doesn't grow. Those containers' metric series do change label on upgrade.
  • k8s.cronjob.name: set from the owner map (logs/otel.go:66), not the id.
  • Other consumers: nothing in the backend reads k8s-cronjob, so no parser depends on the old form.
  • Regex change: the ^ anchor and moving the check ahead of the /k8s/ return are both safe. The old loop entry could never match.
  • Tests: the new cases would fail on main, since the early return kept the raw id.

@blue4209211
blue4209211 merged commit 6719067 into main Nov 5, 2026
7 checks passed
@blue4209211
blue4209211 deleted the fix/nb-43217-cronjob-service-name branch November 5, 2026 07:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants