Skip to content

Exercise each clouds serving stack placement in the local e2e - #472

Draft
haarchri wants to merge 4 commits into
modelplaneai:mainfrom
haarchri:feat/serving-stack-e2e-chainsaw
Draft

haarchri wants to merge 4 commits into
modelplaneai:mainfrom
haarchri:feat/serving-stack-e2e-chainsaw

Conversation

@haarchri

Copy link
Copy Markdown
Collaborator

Description of your changes

Stacked on and a follow-up to #456, the first commit here is that PRs fix - will rebase when merged.

Nothing verified that a clouds serving stack actually lands where it should: the local e2e runs everything on one untainted kind node, so a toleration that was too broad (the wildcard tolerations #456 removes) or too narrow (an Equal toleration whose value drifted from the cloud's real taint) passed locally and only failed on a real cluster.

nix run .#e2e -- --cloud eks|aks|gke|nebius|vultr|existing now installs that cloud's serving stack on the local kind workload cluster, with KWOK faking two nodes labelled and tainted exactly like that cloud's GPU pool while the real kind node plays the untainted system pool. A single values-driven Chainsaw test applies the fake nodes and a ServingStack XR directly (source: Existing hard-wires cloud: Existing, so the XR is applied rather than composed through an InferenceCluster), waits for it to become Ready, then checks placement from both sides: the values file's expected DaemonSets must fan out to the fake GPU pool (a toleration too narrow fails here), and an audit of every pod's tolerations rejects wildcards and unexpected GPU tolerations wherever the pod landed (a toleration too broad fails here, approximating client-side what AKS rejects at admission). Everything cloud-specific lives in e2e/clouds/values-<cloud>.yaml.

Validated with a local --cloud aks

PASS
Tests Summary...
- Passed  tests 1
- Failed  tests 0
- Skipped tests 0
Done.

==> Cloud serving stack OK: the aks stack installed and placed correctly 

Fixes #

I have:

  • Read and followed Modelplane's contribution process.
  • Run nix flake check (or ./nix.sh flake check) and made sure it passes.
  • Added or updated tests covering any composition function changes.
  • Signed off every commit with git commit -s.

…olerating every taint

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant