Skip to content

evict layer backdrop textures on resize and destroy to prevent VRAM leak - #87

Open
sophietheopossum wants to merge 2 commits into
bea4dev:mainfrom
sophietheopossum:leak-fix
Open

evict layer backdrop textures on resize and destroy to prevent VRAM leak#87
sophietheopossum wants to merge 2 commits into
bea4dev:mainfrom
sophietheopossum:leak-fix

Conversation

@sophietheopossum

Copy link
Copy Markdown
Contributor

The layer-surface counterpart to 1ac0829 (#59), which fixed this leak class on the window side; both new helpers land beneath its purge_shared_effect_pipeline_caches_for_window and are named to match.

layer_backdrop_cache is insert-only: 42 references across tty.rs, winit.rs and state.rs, not one of them a remove, retain, clear or drain, and because the key ends in the effect rect's size (__layer_background_effect_DP-1_<layer_id>_0_1520x471), a resize mints a fresh full-size texture for every size a layer has ever had. The texture is allocated in physical pixels while the key carries the logical rect, so that key costs 2.7 MiB at scale 1.

Scope: only configs that put a backdrop effect on a layer surface, a per-layer behind: effect, or background_effect and a client declaring a blur region through ext-background-effect-v1. A stock config never writes an entry, and a plain backdrop blur on a Top/Overlay layer takes the framebuffer path instead, so in practice it's behind: effects and Background/Bottom layers.

Nothing reclaims those entries today. layer_destroyed doesn't touch the map; the hot-reload reset at state.rs:2481 clears five sibling effect caches and skips this one; refresh_layer_effects_for_output sweeps configured_layer_effects and layer_effect_cache for dead layer ids and skips this one; and the 128-entry pipeline LRU frees the intermediates while the final texture stays pinned by the clone stored here.

What changed

  • evict_stale_backdrop_sizes(cache, current_key) at the four insert sites (two in tty.rs, two in winit.rs): splits the key at its last _ and keeps only the current size for that layer and variant. The current key is exempt so the insert can still read back its element Id and commit counter. It runs on the insert path only: the cache-hit branch returns before it, though an Always-invalidation effect takes that path every frame; the retain is a prefix compare per entry over a map bounded by outputs × layers-with-backdrop × variants, inside a function already doing a full-size capture and N blur passes.
  • purge_backdrop_cache_for_layer from layer_destroyed, clearing a departed layer across every output. On an abrupt client exit wl_surface().client() is already None, so layer_runtime_id degrades to unknown-client:<id> and the purge matches nothing.
  • retain_backdrop_cache_for_live_layers closes that gap, called from both refresh_layer_effects_for_output paths beside the existing layer_effect_cache sweep: it drops any entry whose layer is not in the live set, so it needs no event to fire and also covers a close the compositor missed. It cannot reuse retain_effect_texture_cache_for_live_ids: that tests {id}@ as a key prefix, while these keys carry the id between underscores after the output name, so handing it this map would match nothing and empty the cache on every refresh.
  • Three of the four sites keyed on a bare wl_surface().id().protocol_id(), unique only within a client, rather than layer_runtime_id() ({client_id}:{protocol_id}); the purge searches for _{layer_runtime_id}_, so it cannot match a key built from a bare protocol id. The correct binding was already in scope at all three: tty.rs's top-layer and winit.rs's lower-layer sites each ignored one earlier in the same function, and winit.rs's top-layer site shadowed one, so that fix is a deleted line, which also re-keys that function's three sub-element rect keys from the protocol id to the runtime id (reads and writes move together, so it's inert, but it's why one deletion touches more than stable_key). On its own the collision reads as cache thrash plus two render elements sharing a damage-tracker Id and commit counter, since the signature check, which hashes the effect rect and capture geometry, gates the actual texture reuse. Found by inspection, not reproduced.

Testing. Both new paths fire in a real session here (Intel xe iGPU):

purged destroyed layer backdrop textures layer_id="...:59" removed=1
evicted resized layer backdrop textures current_key="__layer_background_effect_DP-1_...:36_0_1520x471" removed=1

cargo check -p shoji_wm --bins is clean against main at 0d4eab1 itself: I built this branch in a fresh worktree on that base, zero warnings. cargo test -p shoji_wm --bins I have only run on my own tree: 186 passed, 4 failed, all ssd::evaluator::tests::workspace_* kinetic-scroll assertions that fail the same way with this patch stashed, so they are not caused by the change; my tree diverges from main inside ssd/, so I can't say whether they also fail for you. The session logs above were captured on my tree too, though every region this patch touches is identical in both, and the branch is one commit on 0d4eab1, so it applies as a fast-forward.

Two commits: the eviction and the keying fix, then the liveness sweep on top. The keying fix rides with the first because it is the same expression built wrong, happy to lift it into its own commit if you'd prefer that.

…RAM leak

`layer_backdrop_cache` was insert-only, so every `CachedBackdropTexture` and
the full-size `GlesTexture` it holds survived for the whole session. Worse,
the key ends in the effect rect's `{width}x{height}`, so a layer that resizes
mints a new key and a new texture for every distinct size it has ever had:
a dock whose width tracks window titles grows the cache without bound and is
never destroyed. Add `evict_stale_backdrop_sizes` at each insert site to drop
the entries a layer left behind at its previous sizes, and
`purge_backdrop_cache_for_layer` from `layer_destroyed` to clear a departed
layer across all outputs, mirroring the window-side fix in 1ac0829.

Three sites also built the key from a bare `protocol_id()`, which is unique
only within a client, rather than `layer_runtime_id()`; two clients whose
surfaces shared a protocol id would collide on one key, sharing its texture,
its render-element `Id` and its commit counter. The signature check gates the
actual texture reuse, so the realistic effect is cache thrash rather than a
visibly wrong backdrop; found by inspection, not reproduced. In `configured_background_effect_elements_for_layer` the
correct id was already in scope and had been shadowed.
`purge_backdrop_cache_for_layer` is driven by `layer_destroyed`, which does
not fire for every departure. On an abrupt client exit `wl_surface().client()`
is already `None`, so `layer_runtime_id` degrades to `unknown-client:<id>` and
cannot match the keys written while the client was alive, leaving those
entries orphaned. Sweeping the cache against the live layer set instead needs
no event to fire, so it also covers a close the compositor missed.

this fix is called from both `refresh_layer_effects_for_output` paths, beside the existing
`layer_effect_cache` sweep, so the two caches are pruned by the same rule at
the same point.

It cannot reuse `retain_effect_texture_cache_for_live_ids`: that helper is
generic over the value type and so would compile if handed this map, but it
tests `{id}@` as a key *prefix*, while these keys carry the id between
underscores after the output name. It would match nothing and empty the cache
on every refresh while looking like correct reuse.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant