QVAC-23767 vae: re-point graph params at the fallback backend during CPU fallback

[DmitryMalishev]

Fixes the silent crash in the VAE automatic CPU fallback on 2026-08-11.

Bug. The fallback switches the runner's runtime backend to CPU and recomputes, but prepare_params stages weights to each tensor state's registration-time compute backend — and staging swaps the live tensor's buffer/data pointers to the staged copy. The CPU graph then dereferences device-staged memory. gdb stack at the fault:

#0 ggml_vec_dot_f16 (x=0x804ec440 <- Vulkan-staged weight)
#1 ggml_compute_forward_mul_mat_one_chunk
#2 ggml_compute_forward_mul_mat        (ggml-cpu.c)

How it surfaced. The qvac diffusion 2026-08-11 pair bump (QVAC-23767 / qvac#3978): the Wan 2.1 img2vid integration test — the only CI leg that exercises the fallback (win32, whose Vulkan driver caps logical buffers at 4GB, rerouting the 8.7GB encode graph) — died with a silent access violation on every run (example). The same test on the 2026-07-03 engine passed with the same reroutes (238s, job) — the July implementation computed from CPU-resident params directly; the August rework introduced the staging mismatch.

Fix (inside the fallback branch of the preflight wrapper, src/core/ggml_extend.hpp):

1. Collect the graph's param tensors before switching backends (switch_runtime_backend frees the compute ctx that owns the graph; the param pointers themselves are stable model tensors).
2. Quiesce the runner's prepared params (runner_done()) and re-point them at the fallback backend via the existing assign_compute_backend API. With compute and params on the same backend, stage_tensors_to_compute_backend skips staging by design and the CPU graph reads the params in place — the July behavior, expressed through the new manager.
3. Restore the assignment on every exit path (success and exception) so later graphs stage back to the real runtime backend.

Validation (local Vulkan build, RTX 4050 + Wan 2.1 VACE 1.3B Q8 / wan2.1 VAE / umt5 Q5_K_M). The stock trigger doesn't fire on this driver (it reports a 1TB max buffer), so the budget was forced down with --vae-auto-cpu-fallback-memory-ratio 0.05:

• Before: access violation (0xC0000005) immediately after the first routing only this graph to CPU — the exact CI signature.
• After: the same forced img2vid run completes 25 reroute cycles across encode and decode tiles (each logging VAE CPU fallback complete; restored runtime backend Vulkan1), with GPU sampling in between — proving both the re-point and the restore — and writes a valid video, exit 0.

Together with #32 this closes the last engine defect found by the QVAC-23767 pair migration; final CI proof runs via the overlay in qvac#3978's validation branch, links to follow in qvac-registry-vcpkg#325.

---

Validation update (2026-08-25, final CI proof). The full qvac addon matrix ran with this fix (+ #32) overlaid on branch tip a20856b (validation stack 9c013c8): run 32857044906 — all integration legs green, including the previously-crashing win32 GPU leg: ok 33 - Wan 2.1 I2V — smoke (img2vid) on qvac-win25-x64-gpu (job 97835656323). This exact test crashed the process before the fix (round-4g runs and qvac#3923's branch today: job 97769332077, exit 1 at Wan I2V load). The single red job in the validation run, cpp-lint, is the known qvac-ubuntu2204-x64 runner-pool clang/libc++ mismatch — infra, unrelated.

---

Stack & review aids (2026-08-25, per reviewer request). This PR is the TOP of a two-PR stack: #32 → #35 (this) — its base is #32's branch, so this diff shows only the VAE-fallback change; it retargets to 2026-08-11 automatically when #32 merges. Stack tip commit f1c8b80 is tree-identical to the previously validated 9c013c8. The consuming qvac addon PR #3978 now carries overlay ports on its own branch pinning ggml 7d9ce11 and stable-diffusion-cpp f1c8b80 (this branch): overlay-ports. Fresh full-matrix proof run on exactly these pins: run 32865301949.

Review-round update (2026-08-26). All three review findings confirmed and fixed in 97b9033 (see the inline replies): shared reroute helper now covers retry_stateless_on_cpu (the reactive path was exercised end-to-end in the local forced repro — "alloc compute buffer failed → retrying stateless graph on CPU" → complete, rc=0), assign_compute_backend is two-phase (validate-then-commit, no partial mutation), and a failed restore is fatal to the feature (error log + vae_auto_cpu_fallback_enabled cleared + call fails; exception path included). Fresh proof, all green: engine matrix 32959554240, full qvac matrix on this tip 32959700401 (win32 74/74 incl. Wan img2vid).

[iancris]

PR #35 hardens one of compute()'s two VAE CPU-fallback entry points and leaves the other, retry_stateless_on_cpu, performing the identical un-repointed backend switch.

retry_stateless_on_cpu (src/core/ggml_extend.hpp:3236-3266) carries the same guard conditions as the preflight route (:3238-3242 vs :3268-3271), does the same switch_runtime_backend(vae_fallback_backend) at :3252, and recurses into compute<T> at :3254 — with no collect_used_param_tensors and no assign_compute_backend.

That is what the new comment added here declares fatal: after :3252 the tensor states' compute_backend is still the GPU, so stage_tensors_to_compute_backend (src/model_manager.cpp:430-441) stages weights to the device while the graph runs on CPU — the same SIGSEGV in ggml_vec_dot, through the other door.

Impact: this is the more important of the two paths. It fires at :3406-3411 after execute_graph has already failed, so it is the reactive safety net running on exactly the memory-constrained devices the feature targets. It also means "closes the last engine defect found by the QVAC-23767 pair migration" does not hold yet.

Suggested fix: lift the fallback_params collection and the repoint_params switch/restore pair into a small private member helper, and call it from both paths — the switch at :3252, the restore at :3260, and the catch restore at :3263. One shared helper is also what stops the two entry points drifting apart again.

[DmitryMalishev]

Confirmed — retry_stateless_on_cpu performed the raw backend switch with no param re-pointing; the same stale-staged-weights crash through the reactive door.

Fixed as suggested in 97b9033: the collect/switch/re-point/restore sequence now lives in one shared helper (compute_on_vae_fallback_backend) called from both entry points — the preflight route and the retry. Params are collected before the switch (the switch frees the compute ctx owning gf; execute_graph's failure paths free buffers but not that ctx, so gf is still valid at retry time).

Validation: the local forced repro (Vulkan, --vae-auto-cpu-fallback-memory-ratio 0.05, Wan T2V+VACE img2vid) exercised this exact path — "alloc compute buffer failed → retrying stateless graph on CPU → VAE CPU fallback complete; restored runtime backend Vulkan1" — plus 3 preflight cycles, rc=0, structurally valid AVI. CI: engine matrix green (https://github.com/tetherto/qvac-ext-stable-diffusion.cpp/actions/runs/32959554240); full qvac addon matrix green on this tip incl. the win32 Wan img2vid leg 74/74 (https://github.com/tetherto/qvac/actions/runs/32959700401).

[iancris]

assign_compute_backend mutates tensor state as it walks and returns false mid-loop without undoing it, so a failed re-point leaves the VAE's weights split across two backends.

ModelManager::assign_compute_backend (src/model_manager.cpp:1145-1173) writes state->compute_backend as it iterates (:1169-1172) but bails with return false mid-loop at :1161 (state active or already staged) and :1166 (loaded state whose params backend would move). Everything already visited keeps the new backend — there is no rollback.

This error path then restores only the runtime backend at :3345 and returns std::nullopt, so an arbitrary prefix of the VAE's weights stays pinned to CPU while the runtime is back on the GPU.

Impact: silent, and the mirror image of the bug being fixed. On the next graph those tensors have compute_backend == params_backend == CPU, so staging is skipped (src/model_manager.cpp:430) and tensor->data stays a host pointer handed to a GPU kernel. graph_requires_backend_fallback only inspects op support, never buffer residency, so nothing catches it.

The runner_done() quiesce at :3334 narrows but does not close the window: release_compute_staging_blocks is all-or-nothing per block, so a block holding any state outside this runner's tracked params survives with staged_to_compute_backend == true — the condition at :1158. --params-backend vae=disk reaches :1163 by a second route.

Suggested fix: make assign_compute_backend two-phase — validate every state, then commit — so partial mutation cannot happen. Failing that, snapshot each tensor's prior backend here and roll back before returning nullopt.

[DmitryMalishev]

Confirmed — the single loop wrote state->compute_backend as it walked and could bail mid-loop with no rollback, leaving a prefix of the VAE's weights re-pointed (and the skip-staging condition then hands a host pointer to a GPU kernel, exactly the mirror image of the bug being fixed).

Fixed with the suggested two-phase shape in 97b9033: ModelManager::assign_compute_backend now validates every required state first (active/staged and loaded-params refusals) and only then commits, so a refusal can no longer leave partial mutation. This also makes the caller's error path exact: the fallback helper's comment now documents that a false return means nothing moved, so switching the runtime back restores the precise pre-call state.

CI: engine matrix green (https://github.com/tetherto/qvac-ext-stable-diffusion.cpp/actions/runs/32959554240); full qvac addon matrix green on this tip (https://github.com/tetherto/qvac/actions/runs/32959700401).

[iancris]

A failed restore is treated as non-fatal — warned about on the success path and discarded entirely on the exception path — so the runner keeps generating with the wrong param backend.

The two restore sites disagree with each other. Here at :3357-3361 a failed restore emits LOG_WARN and execution continues, returning the computed output. At :3368 the catch path discards repoint_params' return value with no log at all.

A failed restore means the runner's weight bindings no longer match its runtime backend: params stay on CPU while the runtime is back on the GPU. The next graph then hands a host pointer to a GPU kernel — a wild dereference, or silently wrong output — for the rest of the session. Unlike the partial-mutation case, this needs no unusual configuration; any restore failure reaches it.

Impact: the process continues and keeps producing images, so nothing surfaces to the caller. That makes this the worst-behaved of the three exit paths rather than the mildest.

Suggested fix: treat a failed restore as fatal to the feature rather than recoverable — log at error level, clear vae_auto_cpu_fallback_enabled so later graphs cannot re-enter a path whose restore is known broken, and fail the call instead of returning output computed against inconsistent bindings. At minimum, mirror this LOG_WARN onto the catch path at :3368 so the condition can never be invisible.

[DmitryMalishev]

Confirmed — the two restore sites disagreed (WARN-and-continue vs silent discard), and continuing with mismatched bindings is the worst outcome since nothing surfaces to the caller.

Fixed per your suggestion in 97b9033: a failed restore is now fatal to the feature — logged at error level, vae_auto_cpu_fallback_enabled is cleared so later graphs cannot re-enter a path whose restore is known broken, and the call fails (returns nullopt) instead of returning output computed against inconsistent bindings. The exception path runs the same handler instead of discarding the result, so the condition can never be invisible. Both sites share the one helper now, so they cannot drift apart again.

CI: engine matrix green (https://github.com/tetherto/qvac-ext-stable-diffusion.cpp/actions/runs/32959554240); full qvac addon matrix green on this tip (https://github.com/tetherto/qvac/actions/runs/32959700401).