Place cached replicas only on clusters with cache storage - #466
Merged
Merged
Conversation
|
Docs preview: https://modelplane-docs-pr-466.vercel.app (ready once the site's Content workflow finishes) |
negz
force-pushed
the
cache-me-if-you-can
branch
from
September 26, 2026 02:03
7755f60 to
cc5fb63
Compare
A ModelDeployment that references a ModelCache placed its replicas on any cluster the cache's clusterSelector matched, but compose-model-cache only stages the cache onto matching clusters that report cache storage, an RWX StorageClass in status.cache. Vultr clusters, and Existing clusters that name no RWX StorageClass, report none, so a replica could land where its PVC never appears and hang on the volume mount (modelplaneai#186). This makes only clusters with cache storage candidates for a deployment that references a cache. A replica already on one without is re-placed, as a replica is when the cache's selector stops matching its cluster, and a deployment left with no candidate says so in ReplicasScheduled rather than blaming capacity. It also corrects the ModelCache docs, which said narrowing the selector leaves running replicas where they are. They're re-placed, which is right, since compose-model-cache deletes the cache's PVC on a cluster it no longer stages to. Signed-off-by: Nic Cope <nicc@rk0n.org>
negz
force-pushed
the
cache-me-if-you-can
branch
from
September 26, 2026 02:25
cc5fb63 to
02062ba
Compare
negz
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
#176 had compose-model-cache stage only onto clusters with an RWX StorageClass, and #189 then bounded placement by the cache's
clusterSelectoralone, missing that. #370 added a source, Vultr, that never has a class.A ModelDeployment that references a ModelCache placed its replicas on any cluster the cache's selector matched. compose-model-cache stages only onto the matching clusters that report cache storage in
status.cache. Vultr reports none, nor does an Existing cluster with nocache.storageClassName. So a replica could land where its PVC never appears and hang on the mount, the failure #186 describes.Only clusters with cache storage are now candidates for a deployment that references a cache. A replica already on one without is re-placed, as a replica is when the cache's selector stops matching its cluster, which also fixes any replica the bug already stranded. With no candidate left,
ReplicasScheduledreportsNoCacheStoragerather thanInsufficientCapacity.The ModelCache docs said narrowing the selector leaves running replicas where they are. The code re-places them, which is right, since compose-model-cache deletes the cache's PVC on a cluster it no longer stages to, so this corrects the docs.
A cluster that names a missing or non-RWX StorageClass still counts as having cache storage, and its PVC never binds. I haven't reproduced any of this on a real cluster, since that needs a second cluster that can't host the cache.
I have:
nix flake check(or./nix.sh flake check) and made sure it passes.git commit -s.