feat: gate client traffic until replica initialization completes

[GrigoryPervakov]

Why

• Replicas are user-queriable before the operator initializes them
• Replicas may need time to replicate all data before deletion

What

Add a Pod readiness gate based on the operator-managed condition.

Related Issues

Fixes #266

[Copilot]

Pull request overview

This PR introduces a readiness-based traffic gate for ClickHouse replicas so that the public headless Service only publishes replicas after operator-driven initialization is complete, while internal per-replica Services remain reachable to support peer/operator communication during bootstrap. It also adds controlled traffic draining and replication waiting on scale-down, plus updates to schema-sync/default-db handling to avoid data loss.

Changes:

• Gate client traffic via a Pod readiness gate (clickhouse.com/ReplicaInitialized) and adjust Service publishing behavior (public gates on readiness; internal publishes not-ready endpoints).
• Teach the reconciler to mark replicas initialized after schema/default-db preparation, and to drain traffic + wait for replication before deleting scaled-down replicas.
• Add/extend unit + e2e tests, RBAC permissions, and documentation to cover the new behavior.

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/e2e/clickhouse_e2e_test.go	Adds e2e coverage for gated traffic and scale-down drain behavior; includes EndpointSlice helper.
internal/controller/resourcemanager.go	Adds helpers to patch Pod status conditions and query Pod condition truth.
internal/controller/clickhouse/templates.go	Makes public Service gate readiness; adds Pod readiness gate to the pod spec.
internal/controller/clickhouse/templates_test.go	Verifies readiness gating and public Service publish behavior.
internal/controller/clickhouse/sync.go	Implements initialization gating, traffic draining on removal, EndpointSlice inspection, and revised schema-sync flow.
internal/controller/clickhouse/sync_test.go	Adds unit tests for readiness/initialization/drain logic and EndpointSlice publication behavior.
internal/controller/clickhouse/controller.go	Adds a Pod watch mapping ClickHouse Pods back to their owning ClickHouseCluster.
internal/controller/clickhouse/controller_test.go	Tests Pod→cluster enqueue mapping.
internal/controller/clickhouse/commands.go	Refactors default DB engine enforcement into per-replica API that can preserve populated non-Replicated defaults safely.
internal/controller/clickhouse/commands_test.go	Updates integration tests to the new per-replica default DB engine API.
docs/guides/configuration.mdx	Documents internal vs public Services, readiness gate behavior, and scale-down draining/replication guarantees.
dist/chart/templates/rbac/manager-role.yaml	Grants manager RBAC for pods/status and endpointslices needed by readiness gating/draining.
config/rbac/role.yaml	Same RBAC updates for the non-Helm manifests.
api/v1alpha1/types_test.go	Adds tests for the new DatabaseSyncEnabled() helper behavior.
api/v1alpha1/conditions.go	Adds new condition reasons and the Pod readiness gate condition constants.
api/v1alpha1/clickhousecluster_types.go	Adds DatabaseSyncEnabled() helper to ClickHouseSettings.
.golangci.yml	Updates linter configuration to allow clickhouse-go Row interface returns.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

test/e2e/clickhouse_e2e_test.go:1580

• Use the callback's g.Expect for these assertions. Global Expect aborts the spec instead of letting Eventually retry when the API call is transient or an old Pod still exists during scale-down, making this helper flaky precisely while it waits for convergence.

		Expect(k8sClient.List(ctx, &pods, client.InNamespace(cr.Namespace), client.MatchingLabels{
			controllerutil.LabelAppKey: cr.SpecificName(),
		})).To(Succeed())
		Expect(pods.Items).To(HaveLen(count))

test/e2e/clickhouse_e2e_test.go:310

• This assertion should use the callback's g.Expect; otherwise a transient Pod GET error aborts the E2E spec instead of being retried by Eventually.

This issue also appears on line 1577 of the same file.

					Expect(k8sClient.Get(ctx, podKey, &pod)).To(Succeed())