Skip to content

Keep Hubble relay replicas on separate production nodes #5576

Closed

Description

@devantler

🤖 Generated by the Agentic Engineer

Expected Behavior

The production Hubble relay remains available after losing one node. Its two replicas use separate eligible nodes, and a placement change preserves the Cilium datapath, admission controls and existing disruption budget.

Actual Behavior

The declared relay placement uses ScheduleAnyway, which expresses a preference and permits both replicas to share a node. The soft spread introduced by #2125 therefore does not guarantee the intended availability outcome. This is visible in the public configuration; private runtime evidence is retained separately.

Steps to Replicate

  1. Render the pinned Cilium chart with the production HelmRelease values.
  2. Inspect the Hubble relay Deployment's topology constraint and selector.
  3. Observe that its scheduling rule permits a placement that leaves both replicas dependent on one node.

Acceptance Criteria

  • Render a placement rule that prevents the two production relay replicas sharing one eligible node, with selectors matching the actual chart output.
  • Preserve replica counts, resource requests, disruption protection, security contexts and datapath configuration; compare the complete chart output to detect unrelated changes.
  • Validate local and production configurations and keep single-node development behavior explicit.
  • Deliver through current-head CI, substantive review and the protected merge queue.
  • Verify two ready deployed replicas on separate nodes and observe the corresponding Coroot availability risk clear without dismissal or a changed observation window.

Related to #2125. Part of #2043.

Scheduling semantics: https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/

Activity

  1. self-assigned this
    on Oct 29, 2026
  2. added theissue type on Oct 29, 2026
  3. moved this to 🏃🏻‍♂️ In Progress in 🌊 Project Boardon Oct 29, 2026
  4. updated Effort Low and Priority Medium on Oct 29, 2026
  5. moved this from 🏃🏻‍♂️ In Progress to 🫴 Ready in 🌊 Project Boardon Oct 29, 2026
  6. moved this from 🫴 Ready to 👀 In Review in 🌊 Project Boardon Oct 29, 2026
  7. added 2 commits that reference this issue on Oct 29, 2026
  8. devantler commented on Oct 29, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    #5577 delivered the placement rule through the protected queue as bfeafbc6ed0c80b60ef838037314e75a1e80d149. Its production run passed the existing 17 public checks and active-readiness gate. Validate Main, Release, Code Quality and CodeQL all passed at that merged source. Native convergence confirmed it in production, and v1.283.8 points to that revision.

    Live readback found the current Deployment fully observed, with both desired replicas updated and ready on two distinct nodes and zero restarts. Coroot's unchanged last-hour risk view no longer contains the relay's availability risk; including dismissed results also finds no relay risk. No dismissal or risk-policy change was made. The complete chart comparison and local validation retained the existing datapath, security controls, resources and disruption protection.

    All acceptance criteria for this placement issue are met. Other application and security findings remain separate work.

  9. moved this from 👀 In Review to ✅ Done in 🌊 Project Boardon Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

Medium

Effort

Low

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions