Skip to content

fix: disable metrics for per-WorkspaceType multicluster managers - #34

Open
2mind wants to merge 1 commit into
kcp-dev:mainfrom
2mind:fix/per-workspacetype-manager-metrics
Open

2mind wants to merge 1 commit into
kcp-dev:mainfrom
2mind:fix/per-workspacetype-manager-metrics

Conversation

@2mind

@2mind 2mind commented Sep 28, 2026 •

Copy link
Copy Markdown

Summary

targetcontroller.createMulticlusterManager() creates one multicluster manager per WorkspaceType, but never sets Metrics in the manager options. controller-runtime then falls back to its default bind address (:8080) for every one of them: the first manager claims the port, and each subsequent manager fails to start:

Failed to run multicluster manager  ctrlkey=root:account
error=failed to start metrics server: failed to create listener: listen tcp :8080: bind: address already in use

The WorkspaceType of the failed manager is then never initialized — its InitTargets are ignored until the pod restarts and happens to start with that target already known.

This disables the metrics server for the per-WorkspaceType managers (BindAddress: "0"). Only the main manager is supposed to serve metrics (--metrics-address, 127.0.0.1:8085 by default), and the secondary managers' metrics are not scraped from anywhere.

"Disable" rather than "one port per WorkspaceType" because the number of WorkspaceTypes is unbounded, while only a single metrics endpoint is actually served; if per-WorkspaceType metrics are ever wanted, they should be registered with a WorkspaceType label in the main registry instead.

Verification

  • go build ./..., go vet ./internal/..., CGO_ENABLED=1 go test -tags unit ./... — pass.
  • Reproduced on our kcp installation (kcp v0.32.x, kcp-operator 0.9.0, init-agent v0.4.0 with multicluster-provider v0.8.0): as soon as a second WorkspaceType appeared, its manager died with the error above and that WorkspaceType stayed uninitialized. The init-agent image we run in that installation carries this change (together with fix: wire event recorder into initializingworkspaces cluster multicluster-provider#117) and initializes all WorkspaceTypes in the cluster.

What Type of PR Is This?

/kind bug

Related Issue(s)

Fixes #26

Release Notes

Fix a metrics server port collision that prevented init-agent from initializing more than one WorkspaceType

Every WorkspaceType gets its own multicluster manager inside this process,
but the manager options leave Metrics unset, so controller-runtime falls back
to its default ":8080" for each of them. The first manager claims that port
and every subsequent one fails to start:

    Failed to run multicluster manager  ctrlkey=root:account
    error=failed to start metrics server: failed to create listener: listen tcp :8080: bind: address already in use

The affected WorkspaceType is then never initialized (its InitTargets are
ignored) until the pod happens to restart with that target already known.

Only the main manager serves metrics (--metrics-address, 127.0.0.1:8085 by
default), and metrics of the per-WorkspaceType managers are not exposed
anywhere anyway, so disable the metrics server for them.

Fixes kcp-dev#26

Signed-off-by: Nikita Aboltin <aboltin.nikita@rwb.ru>
@kcp-ci-bot kcp-ci-bot added dco-signoff: yes Indicates the PR's author has signed the DCO. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Sep 28, 2026
@kcp-ci-bot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign xrstf for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kcp-ci-bot kcp-ci-bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Sep 28, 2026
@kcp-ci-bot

Copy link
Copy Markdown
Contributor

Hi @2mind. Thanks for your PR.

I'm waiting for a kcp-dev member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kcp-ci-bot kcp-ci-bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Sep 28, 2026
@2mind

2mind commented Sep 28, 2026

Copy link
Copy Markdown
Author

/kind bug

@kcp-ci-bot kcp-ci-bot added the kind/bug Categorizes issue or PR as related to a bug. label Sep 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. kind/bug Categorizes issue or PR as related to a bug. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

:8080 metrics port collision when starting a new init controller for a different WorkspaceType

2 participants