Skip to content

fix(executor): re-seat a stale region latch when the pool changes - #245

Open
Gaozx1 wants to merge 2 commits into
caigee-cmd:mainfrom
Gaozx1:fix/reseat-stale-region-latch
Open

Gaozx1 wants to merge 2 commits into
caigee-cmd:mainfrom
Gaozx1:fix/reseat-stale-region-latch

Conversation

@Gaozx1

@Gaozx1 Gaozx1 commented Sep 27, 2026

Copy link
Copy Markdown

Problem

Pool.PickRoute remembers the region a route last served from (lastRegion, keyed by provider|region|model) so a balanced pool does not flip regions as rotation advances. The latch was only ever written, never re-evaluated, so a change in pool composition underneath a route left it pinned to whatever region it happened to pick first.

In production a migration added 78 cn WorkBuddy accounts to a pool that held 2 global ones. The route had latched onto global, so every request kept landing on those two accounts while the 78 new cn accounts received nothing:

pool requests avg TTFB >15s worst
global (latched) 285 48.1s 39% 528s
cn (starved) 32 2.8s 0% —

Restarting the process was the only way to clear it, and the same trap awaits anyone who adds or migrates accounts into a running instance.

Fix

Drop the latch once its region is left behind by a factor of two, so the route re-seats on the largest region:

if subset := inRegion(available, previous); len(subset) > 0 && !regionLatchStale(len(subset), available) {
    available = subset
} else {
    delete(p.lastRegion, key)
}
func regionLatchStale(latchedCount int, available []Item) bool {
    return latchedCount*2 < largestRegionCount(available)
}

The hysteresis keeps a roughly balanced pool on its current region instead of thrashing between two comparable regions, preserving the original intent for the common case.

Tests

TestPickRouteReseatsStaleRegionLatch fails on the previous code with exactly the production symptom:

--- FAIL: TestPickRouteReseatsStaleRegionLatch
    pool_route_test.go:335: stale global latch starved the new cn accounts: map[global:12]

TestPickRouteKeepsRegionLatchWhenBalanced pins the balanced case so the hysteresis cannot be widened into region thrashing.

Validation

go test ./...            # all packages pass (Linux)
go test -race ./...      # clean
go vet ./...             # clean
gofmt -l <files>         # clean
git diff --check         # clean

Only internal/executor/pool.go changes; no migration, no API, no console change. scripts/release-notes.py validate passes with the added bilingual fragment.

Deployed verification

Patched locally against a live instance: 12 fresh requests spread across 23 cn accounts at 1.0–2.4s TTFB, where the same instance previously pinned 100% of traffic to 2 global accounts.

Gaozx1 added 2 commits September 27, 2026 09:00
PickRoute remembers the region a route last served from (`lastRegion`, keyed by
provider|region|model) so a balanced pool does not flip regions as rotation
advances. The latch was only ever written, never re-evaluated, so a change in
pool composition underneath a route left it pinned to whatever region it picked
first.

In production a migration added 78 cn WorkBuddy accounts to a pool that held 2
global ones. The route had latched onto global, so every request kept landing on
those two accounts while the 78 new cn accounts received nothing: 285 requests
averaged 48s TTFB (39% over 15s, worst 528s) against global, while cn sat idle.
Restarting the process was the only way to clear it.

Drop the latch once its region is left behind by a factor of two, so the route
re-seats on the largest region. The hysteresis keeps a roughly balanced pool on
its current region instead of thrashing between two comparable regions.

TestPickRouteReseatsStaleRegionLatch fails on the previous code with exactly the
production symptom (map[global:12]); TestPickRouteKeepsRegionLatchWhenBalanced
pins the balanced case.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant