Improve scheduler actor by TheHeroBrine422 · Pull Request #63 · FyraStack/odorobo

TheHeroBrine422 · 2026-07-10T06:38:40Z

intended changes for this PR

improve actor cache
- currently the actor cache is setup in a way that doesn't really support the scheduler updating it when messages happen. this makes the cache very difficult to use because it can't be relied on for up to date information which means we can barely use it as a cache. a specific issue i hit with this was trying to implement affinity/placement groups. if we rely on the cache data for calculating affinity, we could break the affinity rules due to race conditions.
- i am replacing the generic version I made with one that is just integrated into the scheduler directly, because otherwise I don't really feel comfortable doing direct modifications to the cache in other segments of the scheduler
- on top of this, the vm cache arguably needs slightly different code because we basically need the ability to query by either actor id or vmid, and so it needed some sort of modification to support this either way.
- todo: have messages sent to scheduler directly update the cache
Placement groups #28
improve scheduling algorithm
- i realized that currently the scheduling algorithm doesnt consider the resource requirements of the new vm. so for example if an agent had 15/16 vcpus scheduled, and then a new VM was to be scheduled that needed another 16 vcpus, it could theoretically schedule it to that agent. which would be bad

All that is left is testing and review by another maintainer.

this was done for two reasons: 1) the vm and agent actor caches have slightly different requirements, specifically around keys (actorid vs vmid) 2) the scheduler needs direct control of the cache so the cache can be updated in response to messages it forwards. being possibly up to 1 second out of date in this data could cause problems. In theory the scheduler could update the generic cache, but I don't think using a generic cache if we need to modify it directly anyway really makes sense. Signed-off-by: Caleb Jones <caleb@calebgj.io>

Signed-off-by: Caleb Jones <caleb@calebgj.io>

scheduler affinity Signed-off-by: Caleb Jones <caleb@calebgj.io>

…k/odorobo into caleb/fix-scheduler-actor-cache

Signed-off-by: Caleb Jones <caleb@calebgj.io>

github-advanced-security

clippy found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

this also fixes an issue where if a VM is never linked (ex: it didnt boot) it would never be removed from the caches.

…ctor-cache

halfcyan · 2026-07-16T23:39:26Z

i will look at this when i get back home. have you run it through clippy?

TheHeroBrine422 · 2026-08-17T23:57:56Z

+                data_cache
+                    .entry(vmid)
+                    .and_modify(|entries| {
+                        if let Some(entry) = entries.iter_mut().find(|entry| {


this code is hard to read.

TheHeroBrine422 · 2026-08-18T00:06:57Z

+                    data_cache.alter(&actor_ref.id(), |_, mut v| {
+                        v.data = data;
+
+                        v.extended_vm_set.retain(|vmid| v.data.vms.contains(vmid));


this has two issues.

should a vm stay in the extended vm set if we sent CreateVM, then sent GetAgentStatus, GetAgentStatus got there first and so its not in the agent status BUT it will be in a moment when CreateVM is received.

if we do want to reset it entirely, we should just remake the set from the iter directly instead of deleting and then adding things weirdly.

+            let cache_is_empty = data_cache.get_mut(&vmid).map_or(false, |mut entries| {
+                entries.retain(|entry| {
+                    let expired = entry.actor_ref.is_none()
+                        && now.duration_since(entry.cached_at) >= UNRESOLVED_VM_CACHE_TIMEOUT;
+                    if expired {
+                        removed_agent_ids.push(entry.scheduled_agent_id);
+                    }
+                    !expired
+                });
+                entries.is_empty()
+            });


TheHeroBrine422 added 13 commits May 1, 2026 12:04

improve scheduling algorithm to use estimated future values.

09c41f0

Signed-off-by: Caleb Jones <caleb@calebgj.io>

committing before lunch so i dont have code just sitting on my machine

d0dd375

Signed-off-by: Caleb Jones <caleb@calebgj.io>

continue to work on scheduler actor caches and placement groups

4449502

Signed-off-by: Caleb Jones <caleb@calebgj.io>

continue to work on scheduler actor (still not finished)

7a63518

Signed-off-by: Caleb Jones <caleb@calebgj.io>

more work on scheduler affinity stuff. still not finished.

11186cd

Signed-off-by: Caleb Jones <caleb@calebgj.io>

theoretically write scheduler affinity scoring. need to test this.

2202c17

Signed-off-by: Caleb Jones <caleb@calebgj.io>

remove ipam actor

fedb58b

Signed-off-by: Caleb Jones <caleb@calebgj.io>

Merge branch 'main' into caleb/fix-scheduler-actor-cache

e26b7d7

fix bug with getting all agent and vm actors in actor finders

29c2a84

Signed-off-by: Caleb Jones <caleb@calebgj.io>

Merge branch 'main' into caleb/fix-scheduler-actor-cache

99e313f

commiting stuff before cypress rebase. was in the middle of testing

2fde501

scheduler affinity Signed-off-by: Caleb Jones <caleb@calebgj.io>

Merge branch 'caleb/fix-scheduler-actor-cache' of github.com:FyraStac…

da311a3

…k/odorobo into caleb/fix-scheduler-actor-cache

github-advanced-security AI found potential problems Jul 10, 2026

View reviewed changes

TheHeroBrine422 added 2 commits July 10, 2026 02:14

Merge branch 'main' into caleb/fix-scheduler-actor-cache

5274b03

Signed-off-by: Caleb Jones <caleb@calebgj.io>

cargo fmt

674c5a9

Signed-off-by: Caleb Jones <caleb@calebgj.io>

github-advanced-security AI found potential problems Jul 10, 2026

View reviewed changes

TheHeroBrine422 added 2 commits July 10, 2026 02:22

fix bugs created by merge commit

06c7885

cargo fmt

2f1ad4e

github-advanced-security AI found potential problems Jul 10, 2026

View reviewed changes

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

TheHeroBrine422 added 4 commits July 11, 2026 03:13

random debugging stuff

1561884

Merge branch 'main' into caleb/fix-scheduler-actor-cache

a575032

fix: remove unused AffinityDirection enum

188a2b9

feat: remove VMs from actors if unreachacble by updaters.

43b6f56

this also fixes an issue where if a VM is never linked (ex: it didnt boot) it would never be removed from the caches.

github-advanced-security AI found potential problems Jul 16, 2026

View reviewed changes

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

TheHeroBrine422 added 2 commits July 16, 2026 05:37

chore: cargo fmt

a784bab

Merge remote-tracking branch 'origin/main' into caleb/fix-scheduler-a…

e42ee50

…ctor-cache

TheHeroBrine422 changed the title ~~Caleb/fix scheduler actor cache~~ Improve scheduler actor Jul 16, 2026

github-advanced-security AI found potential problems Jul 16, 2026

View reviewed changes

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

fix: better extended_vm_set removal and setup

81d5b9c

github-advanced-security AI found potential problems Jul 16, 2026

View reviewed changes

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

chore: clippy and fmt

ecbc2e9

TheHeroBrine422 marked this pull request as ready for review July 16, 2026 20:23

TheHeroBrine422 requested review from halfcyan and korewaChino July 16, 2026 20:23

halfcyan previously approved these changes Jul 17, 2026

View reviewed changes

korewaChino previously approved these changes Jul 20, 2026

View reviewed changes

Comment thread odoroboctl/src/cli.rs Outdated

resolve conflicts

2ecbae0

halfcyan dismissed stale reviews from korewaChino and themself via 2ecbae0 July 31, 2026 06:31

fix clippy warnings

7698f8f

halfcyan requested changes Jul 31, 2026

View reviewed changes

halfcyan added 5 commits July 31, 2026 14:44

Merge branch 'main' into caleb/fix-scheduler-actor-cache

f8473e2

Merge branch 'main' into caleb/fix-scheduler-actor-cache

53fcfad

Merge branch 'main' into caleb/fix-scheduler-actor-cache

a46111a

resolve review comments

929c4aa

fix more of the review issues

8fe3e3b

halfcyan previously approved these changes Aug 17, 2026

View reviewed changes

TheHeroBrine422 commented Aug 18, 2026

View reviewed changes

fix review comments and remove unnecessary exception

4a972aa

halfcyan dismissed their stale review via 4a972aa August 18, 2026 00:19

make sure stuff in the cache actually gets run

a415ef2

github-advanced-security AI found potential problems Aug 18, 2026

View reviewed changes

Uh oh!

Conversation

TheHeroBrine422 commented Jul 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-advanced-security AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

halfcyan commented Jul 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TheHeroBrine422 Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

TheHeroBrine422 Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants