Skip to content

QVAC-23767 abot: destroy ModelManager before the runners that own its tensors - #32

Merged
gianni-cor merged 4 commits into
2026-08-11from
qvac-23767-abot-modelmanager-teardown
Aug 26, 2026
Merged

QVAC-23767 abot: destroy ModelManager before the runners that own its tensors#32
gianni-cor merged 4 commits into
2026-08-11from
qvac-23767-abot-modelmanager-teardown

Conversation

@DmitryMalishev

@DmitryMalishev DmitryMalishev commented Aug 24, 2026

Copy link
Copy Markdown

Teardown-ordering fix on 2026-08-11, declaration-order only — no API or behavior change.

Bug. ~ModelManager force-frees the param storage blocks and writes through the registered ggml tensors (free_params_storage_block: state->tensor->buffer = nullptr). Those tensors live in the runner contexts, and runners hold only weak_ptr refs to the manager — so the manager must be destroyed first, while the tensors are still alive. StableDiffusionGGML gets this right by declaring model_manager after its runner members. Both ABot entry points on this branch had it inverted:

  • AbotWalkSession declared model_manager before runner/tae → member destruction killed the runners first → ~ModelManager wrote through dangling pointers at sd_abot_session_free.
  • sd_abot_scene_create declared the manager local before the t5/vae runners → same inverted teardown at scope exit — crashing (STATUS_HEAP_CORRUPTION/SIGSEGV) after the scene pack had already been written successfully.

How it surfaced. The qvac diffusion 2026-08-11 pair bump (QVAC-23767 / qvac#3978): the addon's abot-world integration test died with exit 139 right after the umT5 load on both linux-x64 GPU legs (run). Reproduced deterministically on a CPU-only Windows build; gdb stack:

#0 ModelManager::free_params_storage_block  (model_manager.cpp:1039)  state->tensor->buffer = nullptr;
#1 ModelManager::release_params_storage_blocks(force=true)
#2 ModelManager::release_all
#3 ModelManager::~ModelManager
#9 sd_abot_scene_create  (scope exit)

Fix. Mirror StableDiffusionGGML's ordering: declare the manager after the runners in both places (members in AbotWalkSession, locals in sd_abot_scene_create).

Validation (local, CPU, umT5-Q8 + Wan2.2-VAE-f16 + DiT-Q8 + taew2_2-f16):

  • Before: scene create crashes at teardown every run (pack already written); walk session crashes at sd_abot_session_free.
  • After: scene create rc=0 (scene written to … 832x480); full walk block rc=0 (9 frames, 832x480, decoded via TAE, clean exit).

Consumed by qvac-registry-vcpkg#325 (the 2026-08-11 engine-pair ports) — the port REF bumps to this branch's merge result together with qvac-ext-ggml#64.


Validation update (2026-08-25, final CI proof). The full qvac addon matrix ran with this fix (+ #35) overlaid on branch tip a20856b (validation stack 9c013c8): run 32857044906 — all 9 prebuilds, all cpp-tests, and all integration legs green, including the previously-crashing linux GPU legs: ok 3 - ABot-World: full world generation - native scene creation + KV-cache walk on qvac-ubuntu2204-x64-gpu and qvac-ubuntu2404-x64-gpu. (The single red job in that run, cpp-lint, is the known qvac-ubuntu2204-x64 runner-pool clang/libc++ mismatch — infra, unrelated.) Counter-evidence that the bug bites without this fix: qvac#3923's branch (pins the tip without #32) dies with exit 139 right after ABot model load on the same leg: job 97769331999.


Stack & review aids (2026-08-25, per reviewer request). This PR is the BASE of a two-PR stack: #32 (this) → #35 (its base is this branch; merge this first, #35 then retargets automatically). The branch was rebased onto the current 2026-08-11 tip a20856b for the stack (commit is now 81b8ccf, content unchanged). The consuming qvac addon PR #3978 now carries overlay ports on its own branch pinning ggml 7d9ce11 (merged 2026-08-11 tip) and stable-diffusion-cpp f1c8b80 (this stack's tip, tree-identical to the validated 9c013c8): overlay-ports. Fresh full-matrix proof run on exactly these pins: run 32865301949.

Review-round update (2026-08-26). #35 (the stack top) gained review follow-up hardening 97b9033 (shared fallback-reroute helper covering retry_stateless_on_cpu, two-phase assign_compute_backend, fatal failed-restore handling) — this PR's diff is unchanged. New stack tip = 97b9033; the #3978 overlay is repointed at it. Fresh proof, all green: engine matrix 32959554240, full qvac matrix 32959700401 (win32 74/74 incl. both Wan smokes; ABot green on both linux GPU legs).

The VAE auto-CPU-fallback switches the runner's runtime backend to CPU
and recomputes the graph, but prepare_params stages weights to each
tensor state's registration-time compute backend - and staging swaps
the live tensor's buffer/data pointers to the staged copy. The CPU
graph then dereferences device-staged memory: SIGSEGV in
ggml_vec_dot_f16 on a Vulkan-staged weight, killing the process right
after the "routing only this graph to CPU" log line.

Fix inside the fallback branch: collect the graph's param tensors
(before switching backends - switch_runtime_backend frees the compute
ctx that owns the graph), quiesce the runner's prepared params
(runner_done) and re-point them at the fallback backend via
assign_compute_backend. With compute and params on the same backend,
stage_tensors_to_compute_backend skips staging entirely and the CPU
graph reads the params in place. Restore the assignment on every exit
path so later graphs stage back to the real runtime backend.

Surfaced by the qvac diffusion 2026-08-11 pair bump (QVAC-23767): the
Wan 2.1 img2vid integration test - the only leg that exercises the
fallback (win32, whose Vulkan driver caps logical buffers at 4GB) -
died with a silent access violation on every run, while the same test
on the 2026-07-03 engine passed with the same reroutes. Reproduced
locally by forcing the budget down (--vae-auto-cpu-fallback-memory-
ratio 0.05) on a Vulkan build; root-caused with gdb. After the fix the
same forced run completes 25 reroute cycles across encode and decode
tiles, with GPU sampling in between, and writes a valid video.
~ModelManager force-frees the param storage blocks and writes through
the registered ggml tensors (free_params_storage_block:
state->tensor->buffer = nullptr). Those tensors live in the runner
contexts, and runners hold only weak_ptr refs to the manager - so the
manager must be destroyed first, while the tensors are alive. That is
exactly how StableDiffusionGGML orders its members; both ABot entry
points had it inverted:

- AbotWalkSession declared model_manager before runner/tae, so member
  destruction killed the runners first and ~ModelManager corrupted
  freed memory at sd_abot_session_free.
- sd_abot_scene_create declared the manager local before the t5/vae
  runners, with the same inverted teardown at scope exit - crashing
  (STATUS_HEAP_CORRUPTION / SIGSEGV) after the scene pack had already
  been written successfully.

Fix by declaration order alone, mirroring StableDiffusionGGML: the
manager is declared after the runners in both places.

Surfaced by the qvac diffusion 2026-08-11 pair bump (QVAC-23767): the
addon's abot-world integration test died with exit 139 right after the
umT5 load on both linux-x64 GPU legs. Reproduced locally on CPU and
root-caused with gdb (free_params_storage_block <- release_all <-
~ModelManager <- scene-create scope exit); with this change scene
creation and a full walk block (frames decoded and written) both
complete and exit cleanly.
@DmitryMalishev
DmitryMalishev force-pushed the qvac-23767-abot-modelmanager-teardown branch from eea9be6 to 81b8ccf Compare August 25, 2026 15:19
- Lift the collect/switch/re-point/restore sequence into one shared helper
  (compute_on_vae_fallback_backend) and call it from BOTH fallback entry
  points: the preflight route and the reactive retry_stateless_on_cpu. The
  retry previously performed the raw backend switch with no param
  re-pointing -- the same device-staged-weights SIGSEGV through the other
  door -- and a shared helper stops the two paths drifting apart again.
- Make ModelManager::assign_compute_backend two-phase (validate every
  state, then commit) so a mid-loop refusal can no longer leave a prefix
  of the tensors re-pointed at the new backend with no rollback.
- Treat a failed restore as fatal to the feature: log at error level,
  disable vae_auto_cpu_fallback_enabled so later graphs cannot re-enter a
  path whose restore is known broken, and fail the call instead of
  returning output computed against inconsistent bindings; the exception
  path now reports it too instead of silently discarding the result.

Validated locally (Vulkan, forced with --vae-auto-cpu-fallback-memory-ratio
0.05): 3 preflight reroute cycles AND one reactive-retry cycle ("alloc
compute buffer failed" -> "retrying stateless graph on CPU") all complete
and restore the runtime backend; rc=0, structurally valid AVI.
QVAC-23767 vae: re-point graph params at the fallback backend during CPU fallback
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants