QVAC-23767 abot: destroy ModelManager before the runners that own its tensors - #32
Merged
Merged
Conversation
The VAE auto-CPU-fallback switches the runner's runtime backend to CPU and recomputes the graph, but prepare_params stages weights to each tensor state's registration-time compute backend - and staging swaps the live tensor's buffer/data pointers to the staged copy. The CPU graph then dereferences device-staged memory: SIGSEGV in ggml_vec_dot_f16 on a Vulkan-staged weight, killing the process right after the "routing only this graph to CPU" log line. Fix inside the fallback branch: collect the graph's param tensors (before switching backends - switch_runtime_backend frees the compute ctx that owns the graph), quiesce the runner's prepared params (runner_done) and re-point them at the fallback backend via assign_compute_backend. With compute and params on the same backend, stage_tensors_to_compute_backend skips staging entirely and the CPU graph reads the params in place. Restore the assignment on every exit path so later graphs stage back to the real runtime backend. Surfaced by the qvac diffusion 2026-08-11 pair bump (QVAC-23767): the Wan 2.1 img2vid integration test - the only leg that exercises the fallback (win32, whose Vulkan driver caps logical buffers at 4GB) - died with a silent access violation on every run, while the same test on the 2026-07-03 engine passed with the same reroutes. Reproduced locally by forcing the budget down (--vae-auto-cpu-fallback-memory- ratio 0.05) on a Vulkan build; root-caused with gdb. After the fix the same forced run completes 25 reroute cycles across encode and decode tiles, with GPU sampling in between, and writes a valid video.
~ModelManager force-frees the param storage blocks and writes through the registered ggml tensors (free_params_storage_block: state->tensor->buffer = nullptr). Those tensors live in the runner contexts, and runners hold only weak_ptr refs to the manager - so the manager must be destroyed first, while the tensors are alive. That is exactly how StableDiffusionGGML orders its members; both ABot entry points had it inverted: - AbotWalkSession declared model_manager before runner/tae, so member destruction killed the runners first and ~ModelManager corrupted freed memory at sd_abot_session_free. - sd_abot_scene_create declared the manager local before the t5/vae runners, with the same inverted teardown at scope exit - crashing (STATUS_HEAP_CORRUPTION / SIGSEGV) after the scene pack had already been written successfully. Fix by declaration order alone, mirroring StableDiffusionGGML: the manager is declared after the runners in both places. Surfaced by the qvac diffusion 2026-08-11 pair bump (QVAC-23767): the addon's abot-world integration test died with exit 139 right after the umT5 load on both linux-x64 GPU legs. Reproduced locally on CPU and root-caused with gdb (free_params_storage_block <- release_all <- ~ModelManager <- scene-create scope exit); with this change scene creation and a full walk block (frames decoded and written) both complete and exit cleanly.
DmitryMalishev
force-pushed
the
qvac-23767-abot-modelmanager-teardown
branch
from
August 25, 2026 15:19
eea9be6 to
81b8ccf
Compare
- Lift the collect/switch/re-point/restore sequence into one shared helper
(compute_on_vae_fallback_backend) and call it from BOTH fallback entry
points: the preflight route and the reactive retry_stateless_on_cpu. The
retry previously performed the raw backend switch with no param
re-pointing -- the same device-staged-weights SIGSEGV through the other
door -- and a shared helper stops the two paths drifting apart again.
- Make ModelManager::assign_compute_backend two-phase (validate every
state, then commit) so a mid-loop refusal can no longer leave a prefix
of the tensors re-pointed at the new backend with no rollback.
- Treat a failed restore as fatal to the feature: log at error level,
disable vae_auto_cpu_fallback_enabled so later graphs cannot re-enter a
path whose restore is known broken, and fail the call instead of
returning output computed against inconsistent bindings; the exception
path now reports it too instead of silently discarding the result.
Validated locally (Vulkan, forced with --vae-auto-cpu-fallback-memory-ratio
0.05): 3 preflight reroute cycles AND one reactive-retry cycle ("alloc
compute buffer failed" -> "retrying stateless graph on CPU") all complete
and restore the runtime backend; rc=0, structurally valid AVI.
iancris
approved these changes
Aug 26, 2026
gianni-cor
approved these changes
Aug 26, 2026
QVAC-23767 vae: re-point graph params at the fallback backend during CPU fallback
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Teardown-ordering fix on
2026-08-11, declaration-order only — no API or behavior change.Bug.
~ModelManagerforce-frees the param storage blocks and writes through the registered ggml tensors (free_params_storage_block:state->tensor->buffer = nullptr). Those tensors live in the runner contexts, and runners hold onlyweak_ptrrefs to the manager — so the manager must be destroyed first, while the tensors are still alive.StableDiffusionGGMLgets this right by declaringmodel_managerafter its runner members. Both ABot entry points on this branch had it inverted:AbotWalkSessiondeclaredmodel_managerbeforerunner/tae→ member destruction killed the runners first →~ModelManagerwrote through dangling pointers atsd_abot_session_free.sd_abot_scene_createdeclared the manager local before the t5/vae runners → same inverted teardown at scope exit — crashing (STATUS_HEAP_CORRUPTION/SIGSEGV) after the scene pack had already been written successfully.How it surfaced. The qvac diffusion
2026-08-11pair bump (QVAC-23767 / qvac#3978): the addon's abot-world integration test died with exit 139 right after the umT5 load on both linux-x64 GPU legs (run). Reproduced deterministically on a CPU-only Windows build; gdb stack:Fix. Mirror
StableDiffusionGGML's ordering: declare the manager after the runners in both places (members inAbotWalkSession, locals insd_abot_scene_create).Validation (local, CPU, umT5-Q8 + Wan2.2-VAE-f16 + DiT-Q8 + taew2_2-f16):
sd_abot_session_free.scene written to … 832x480); full walk block rc=0 (9 frames, 832x480, decoded via TAE, clean exit).Consumed by qvac-registry-vcpkg#325 (the
2026-08-11engine-pair ports) — the port REF bumps to this branch's merge result together with qvac-ext-ggml#64.Validation update (2026-08-25, final CI proof). The full qvac addon matrix ran with this fix (+ #35) overlaid on branch tip
a20856b(validation stack9c013c8): run 32857044906 — all 9 prebuilds, all cpp-tests, and all integration legs green, including the previously-crashing linux GPU legs:ok 3 - ABot-World: full world generation - native scene creation + KV-cache walkonqvac-ubuntu2204-x64-gpuandqvac-ubuntu2404-x64-gpu. (The single red job in that run, cpp-lint, is the known qvac-ubuntu2204-x64 runner-pool clang/libc++ mismatch — infra, unrelated.) Counter-evidence that the bug bites without this fix: qvac#3923's branch (pins the tip without #32) dies with exit 139 right after ABot model load on the same leg: job 97769331999.Stack & review aids (2026-08-25, per reviewer request). This PR is the BASE of a two-PR stack: #32 (this) → #35 (its base is this branch; merge this first, #35 then retargets automatically). The branch was rebased onto the current
2026-08-11tipa20856bfor the stack (commit is now81b8ccf, content unchanged). The consuming qvac addon PR #3978 now carries overlay ports on its own branch pinning ggml7d9ce11(merged 2026-08-11 tip) and stable-diffusion-cppf1c8b80(this stack's tip, tree-identical to the validated9c013c8): overlay-ports. Fresh full-matrix proof run on exactly these pins: run 32865301949.Review-round update (2026-08-26). #35 (the stack top) gained review follow-up hardening
97b9033(shared fallback-reroute helper coveringretry_stateless_on_cpu, two-phaseassign_compute_backend, fatal failed-restore handling) — this PR's diff is unchanged. New stack tip =97b9033; the #3978 overlay is repointed at it. Fresh proof, all green: engine matrix 32959554240, full qvac matrix 32959700401 (win32 74/74 incl. both Wan smokes; ABot green on both linux GPU legs).