migrate mcm-provider from openstack to stackit - #272
aniruddha2000 wants to merge 29 commits into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
67e72b1 to
dcf82cb
Compare
b696375 to
25fa93d
Compare
|
/retest-required |
|
/retest |
3 similar comments
|
/retest |
|
/retest |
|
/retest |
|
/test pull-gardener-extension-provider-stackit-dev-artifacts |
|
/test all |
81f51f4 to
af6bcea
Compare
|
/test pull-gardener-extension-provider-stackit-integration-tests |
# Conflicts: # pkg/controller/worker/machines.go
b636923 to
a712745
Compare
stackit-ske-bot
left a comment
There was a problem hiding this comment.
SKE Code Review
Architectural Feedback
- Inverted Migration Transaction Flow & Redundant Patch Overhead: In pkg/controller/worker/machines.go#L456-L499, each machine undergoes three separate Kubernetes API
Patchcalls (shouldMigrateMachineAnnotation+migratedMachineAnnotation, thenSpec.ProviderID, then removingshouldMigrateMachineAnnotation). This transaction complexity and ephemeral annotation were introduced becauseSpec.ProviderIDis updated in Kubernetes before confirming that the STACKIT IaaSUpdateServercall succeeded. Reversing the sequence—executingUpdateServer(an idempotent operation) on STACKIT IaaS first, and then executing a single atomic KubernetesPatchupdating bothSpec.ProviderIDandmigratedMachineAnnotation—eliminates the ephemeralshouldMigrateMachineAnnotation, reduces Kubernetes API patch traffic by 66% during node migrations, and avoids leaving machine resources in a state whereSpec.ProviderIDpoints to STACKIT while the remote server labels were never updated. - Unconditional IaaS Client Initialization in
WorkerDelegate: In pkg/controller/worker/actuator.go#L76-L80,stackitClient.IaaS(ctx, d.seedClient, worker.Spec.SecretRef)is called unconditionally on every Worker reconciliation. Becausew.iaaSClientis exclusively used bymigrateMachines(which only executes whenfeature.MigrateStackitMachineControllerManager(cluster)is enabled and the worker has not yet migrated), Shoot clusters that do not use MCM migration or OpenStack clusters whose secret lacks STACKIT service account credentials (saKeyJSON) will fail secret credential parsing instackit.GetCredentialsFromSecretRef, breaking Worker reconciliation. The IaaS client should only be instantiated whenfeature.MigrateStackitMachineControllerManager(cluster) && worker.Annotations[workerMigratedAnnotation] != "true", or lazily on demand insidemigrateMachines.
Findings & Feedback
All findings and concrete recommendations have been provided as inline code suggestions above.
Verdict
Comment
| if err != nil { | ||
| return fmt.Errorf("migrateMachines: %w", err) | ||
| } | ||
| if m.Spec.ProviderID == "" { |
There was a problem hiding this comment.
we cannot just error out here.
we have to answer the following questions:
can the provider id even be empty? if yes, in which cases? what is the state of the server if the id is empty?
what happens when we return an error here if the id is empty, this will result in a "failed migration" right? is this the expected behaviour?
There was a problem hiding this comment.
@aniruddha2000 why have you resolved this comment?
as discussed, we should no error out here, just remove these 3 lines.
if the providerID is empty, the machine will get cleaned up by the mcm anyways after some time
There was a problem hiding this comment.
:) whats wrong with my git, i deleted it and pushed a commit bellow this comment, but somehow it did not! sorrry! :(
commit also says "revert provider id is empty review"
| err = w.migrateMachines(ctx) | ||
| if err != nil { | ||
| return err | ||
| } |
There was a problem hiding this comment.
| err = w.migrateMachines(ctx) | |
| if err != nil { | |
| return err | |
| } | |
| if err := w.migrateMachines(ctx); err != nil { | |
| return err | |
| } |
| if err != nil { | ||
| return fmt.Errorf("migrateMachines: %w", err) | ||
| } | ||
| if m.Spec.ProviderID == "" { |
There was a problem hiding this comment.
@aniruddha2000 why have you resolved this comment?
as discussed, we should no error out here, just remove these 3 lines.
if the providerID is empty, the machine will get cleaned up by the mcm anyways after some time
| return fmt.Errorf("cannot migrate machine %s: providerID is empty (provisioning in progress)", m.Name) | ||
| } | ||
|
|
||
| if m.Spec.ProviderID != "" { |
There was a problem hiding this comment.
add a comment above on why empty providerID is also fine (this is stated in the ADR), so the next person who reads that code knows that we intentionally skip them
| m.Annotations = make(map[string]string) | ||
| } | ||
| m.Annotations[shouldMigrateMachineAnnotation] = "true" | ||
| m.Annotations[migratedMachineAnnotation] = "true" |
There was a problem hiding this comment.
add a comment what this annotation does, and that we need it for cleaning up the nics (stated in the ADR)
| // TODO refine labels | ||
| "mcm.gardener.cloud/machine": m.Name, | ||
| "mcm.gardener.cloud/machineclass": m.Spec.Class.Name, | ||
| "mcm.gardener.cloud/role": "node", |
There was a problem hiding this comment.
| // TODO refine labels | |
| "mcm.gardener.cloud/machine": m.Name, | |
| "mcm.gardener.cloud/machineclass": m.Spec.Class.Name, | |
| "mcm.gardener.cloud/role": "node", | |
| "kubernetes.io/machine": m.Name, | |
| "kubernetes.io/machineclass": m.Spec.Class.Name, | |
| "kubernetes.io/role": "node", |
There was a problem hiding this comment.
AFAIK these are the labels that the stackit-mcm are using.
double check that, when using the stackit mcm, creating a new machine, what labels are present on this machine?
How to categorize this PR?
/kind enhancement
What this PR does / why we need it:
This PR replaces mcm provider from openstack to stackit. Replaces openstack provider ID with stackit provider ID and add
stackit.cloud/migrated-machine: trueannotation to machines and puts annotationstackit.cloud/machine-controller-manager-migrated: trueto workers once the migration is done.Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Breaking changes: