Vulkan: PTQ1_0 kernel hangs GPU (device lost, xe driver job timeout) on Intel Arc/Battlemage at longer prompts

[flatlinebb]

Summary

The PTQ1_0 Vulkan kernel hangs and takes down the GPU on Intel Arc/Battlemage once the prompt gets large enough (~1900+ tokens). Short prompts (tens of tokens) work fine; the kernel driver logs a compute-job timeout and resets the GPU, which surfaces to llama.cpp as vk::Queue::submit: ErrorDeviceLost.

Given #185 says the PTQ1_0 Vulkan kernel was only ever validated on an AMD RDNA3.5 iGPU (RADV) and "committed as untested" elsewhere, I suspect Intel/ANV was simply never exercised — filing this as the first Intel Arc data point.

Environment

• GPU: Intel Arc B580 (Vulkan reports Intel(R) Graphics (BMG G21), deviceID 0xe20b, Battlemage)
• Driver stack: Mesa mesa-vulkan-drivers 25.0.7-2+deb13u1 (ANV), kernel xe DRM driver
• Host kernel: 7.0.14-16-pve (Debian, Proxmox LXC w/ GPU passthrough via /dev/dri)
• llama.cpp fork commit: 1a07bfa5f4144274c8f1c9963821dd9d9a51854b (branch prism, 2026-09-17)
• Build: cmake -B build-vulkan -DGGML_VULKAN=ON -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
• Model: Ternary-Bonsai-2-27B-Abliterated-PTQ1_0.gguf (BoldingBuilds' abliterated build of PrismML's own PTQ1_0 pack)

Repro

./build-vulkan/bin/llama-server -m Ternary-Bonsai-2-27B-Abliterated-PTQ1_0.gguf \
  --host 127.0.0.1 --port 5800 --jinja --reasoning off -ngl 999 --flash-attn on \
  -ctk q8_0 -ctv q8_0 --parallel 1 --threads 12 --threads-batch 16 --ctx-size 32768

A short chat completion (~20 tokens prompt) succeeds normally. A prompt of ~1900+ tokens reliably fails on the first prompt-processing batch:

1.21.972.052 E ggml_vulkan: device lost on Vulkan0
1.21.972.644 E srv  update_slots: decode() failed: vk::Queue::submit: ErrorDeviceLost

Retrying after the failure sometimes instead hits an assertion during recovery:

/opt/llama-cpp-prismml/tools/server/server-context.cpp:2730: GGML_ASSERT(batch.slot_batched || batch.size() == 0) failed

Host kernel log (dmesg) confirms it's a real GPU hang, not just a driver-side error surface:

xe 0000:05:00.0: [drm] Tile0: GT0: Timedout job: seqno=692, lrc_seqno=692, guc_id=2, flags=0x0 in llama-server [pid]
xe 0000:05:00.0: [drm] Xe device coredump has been created

The devcoredump identifies the failing engine as GT0 (main), GuC firmware xe/bmg_guc_70.bin version 70.72.1 (driver notes it wanted 70.54.0 — flagging in case that mismatch is relevant, though I have no evidence either way).

Notes

• Other (non-ternary) models on the same GPU/driver/kernel, built from a different llama.cpp fork, run correctly with long prompts — this appears isolated to the PTQ1_0 ternary Vulkan kernel path, not a general Arc/Xe/ANV instability on this box.
• Short prompts never reproduce it in my testing; it seems tied to batch/prompt size rather than being immediate.
• Happy to pull more from the devcoredump or test a patch if useful — just say what's needed.

[mamama1]

This is from my personal documentation because I ran into what was the same issue, I think. If that's the case, then it is a xe driver issue and you need to recompile the xe module with increased timeouts because the max timeout the driver allows is not high enough. After I applied that fix, I NEVER hit that issue again:

7. Raise the GPU job timeout (rebuild xe.ko)

Without this, long attention kernels (>10 s at ~250k context) trigger Engine reset: engine_class=ccs → Timedout job → UR_RESULT_ERROR_DEVICE_LOST. job_timeout_max is capped at 10000 ms by CONFIG_DRM_XE_JOB_TIMEOUT_MAX.

7.1 Source tree

sudo dnf install rpm-build rpmdevtools dnf-plugins-core kernel-devel-$(uname -r)
sudo dnf builddep kernel
rpmdev-setuptree                      # as your normal user, not root
dnf download --source kernel
rpm -ivh kernel-$(uname -r | sed 's/\.x86_64//').src.rpm
rpmbuild -bp ~/rpmbuild/SPECS/kernel.spec --target=x86_64

export SRC=$(find ~/rpmbuild/BUILD -maxdepth 4 -name 'linux-*.fc*.x86_64' -type d | head -1)
cd "$SRC"

7.2 Configure

cp /boot/config-$(uname -r) .config
make olddefconfig
./scripts/config --set-val DRM_XE_JOB_TIMEOUT_MAX 60000
grep DRM_XE_JOB_TIMEOUT .config          # must show 60000

Do not run make olddefconfig again after --set-val.

7.3 Fix version strings, then build

cp /usr/src/kernels/$(uname -r)/Module.symvers .     # modules_prepare doesn't make this
echo "-$(uname -r | cut -d- -f2-)" > localversion    # e.g. -200.fc44.x86_64
rm -f include/generated/utsrelease.h
make modules_prepare
make -j$(nproc) M=drivers/gpu/drm/xe modules

modinfo drivers/gpu/drm/xe/xe.ko | grep vermagic
modinfo xe | grep vermagic

Both vermagic strings must match exactly, including the preempt variant. If yours lacks the -200.fc44.x86_64 suffix, localversion was created after utsrelease.h — remove the header and rerun make modules_prepare.

7.4 Install

Fedora modules are xz with a small dictionary. Plain xz -9 uses a 64 MiB dictionary that the in-kernel decompressor rejects — the module silently fails to load with decompression failed with status 6.

sudo cp -a /lib/modules/$(uname -r)/kernel/drivers/gpu/drm/xe/xe.ko.xz \
           /boot/xe.ko.xz.stock-$(uname -r)

cp drivers/gpu/drm/xe/xe.ko /tmp/xe.ko
strip --strip-debug /tmp/xe.ko                       # 164 MB -> ~10 MB
xz -f --check=crc32 --lzma2=dict=1MiB -c /tmp/xe.ko > /tmp/xe.ko.xz

sudo cp /tmp/xe.ko.xz /lib/modules/$(uname -r)/kernel/drivers/gpu/drm/xe/xe.ko.xz
sudo depmod -a
sudo dracut -f                                       # xe lives in the initramfs

dracut -f is mandatory — the initramfs copy loads first and shadows /lib/modules otherwise.

7.5 Verify

sudo modprobe -r xe 2>/dev/null; sudo modprobe xe
sudo dmesg | tail -20                                # no "decompression failed"
sudo /usr/local/sbin/gpu-bar-resize.sh
cat /sys/class/drm/card1/device/tile0/gt0/engines/ccs/job_timeout_max
# 60000

Rollback: sudo cp /boot/xe.ko.xz.stock-* /lib/modules/$(uname -r)/kernel/drivers/gpu/drm/xe/xe.ko.xz && sudo depmod -a && sudo dracut -f

---

8. Apply the timeout at boot

/etc/udev/rules.d/99-xe-job-timeout.rules:

ACTION=="add", SUBSYSTEM=="drm", KERNEL=="card*", DRIVERS=="xe", \
  RUN+="/bin/sh -c 'echo 60000 > /sys$devpath/device/tile0/gt0/engines/ccs/job_timeout_ms'"

sudo udevadm control --reload
sudo reboot

---

9. Final verification

cat /proc/cmdline
lspci -s 25:00.0 -vv | grep -E 'Kernel driver|current size' 
sudo dmesg | grep -i 'small bar'                     # must be empty
cat /sys/class/drm/card1/device/tile0/gt0/engines/ccs/job_timeout_ms   # 60000
journalctl -u gpu-bar-resize -b
sycl-ls

Note: the reported link (LnkSta: 2.5GT/s Width x1) on the GPU function is a normal Arc reporting quirk — the real link is on the upstream switch port (0000:23:00.0), which shows 8.0 GT/s x8.

[flatlinebb]

Tested the job_timeout_max/job_timeout_ms theory from the comment above against my original repro, without doing a full xe.ko rebuild — my host runs a custom Debian/Proxmox kernel (pve-kernel, currently 7.0.14-19-pve), so I wanted to rule this in/out with a cheap, reversible test first.

Turns out the runtime knobs were already there: /sys/class/drm/card1/device/tile0/gt0/engines/ccs/job_timeout_ms was set to 5000ms, below the compiled job_timeout_max of 10000ms on this build. So I could test the whole hypothesis with a plain sysfs write, no kernel rebuild needed.

• job_timeout_ms=5000 (original): repro fails cleanly at ~25s wall time — HTTP 500, decode() failed: vk::Queue::submit: ErrorDeviceLost. dmesg shows the expected Timedout job on ccs.
• job_timeout_ms=10000 (raised to the compiled ceiling): still hits Timedout job in dmesg — same hang, just later, and this time it took the process down harder (an uncaught vk::DeviceLostError C++ exception killed llama-server outright, instead of the handled 500 I got at 5000ms).

Doubling the timeout to the driver's max didn't let the job complete or even meaningfully change the outcome — it just moved the same reset a bit further out and made the failure mode worse. If this were purely "needs more wall-clock at this batch size," I'd expect raising the ceiling to actually help. Since it doesn't, I think this points to a real hang/bug in the PTQ1_0 Vulkan kernel itself on Intel/ANV (consistent with #185 — only ever validated on AMD RDNA3.5/RADV) rather than a timeout-configuration issue, at least on this GPU/driver combo. Not planning to pursue an xe.ko rebuild on this box on the strength of this result — seems like it would just delay the identical crash.

Environment unchanged from the original report (Arc B580 / BMG, Mesa ANV, xe DRM driver, Debian/Proxmox host kernel now at 7.0.14-19-pve, same prism branch commit).

[bri-prism]

Thanks for the detailed repro and for testing the timeout theory. Raising the timeout not helping suggests a single submission genuinely takes too long rather than hitting a marginal limit. The PTQ1_0 Vulkan mat-vec on Xe2 GPUs, including the B580, now uses a much faster integer-dot kernel (#238), which is in the new release prism-b10735-842b188. Could you retry your ~1900-token prompt on that build? If it still hangs, a run with a smaller -ub (for example -ub 128) would tell us whether it's tied to batch size.

[flatlinebb]

Retested on prism-b10735-842b188 (built from the tag, same flags: -DGGML_VULKAN=ON -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release), same Arc B580 / Mesa ANV / xe / 7.0.14-19-pve host, same model and server command as the original report. job_timeout_ms is back at the stock 5000.

It's fixed on my end:

Prompt tokens processed	Result	Prefill	Decode
3,232	200 OK, 39.6s	94.8 tok/s	9.0 tok/s
3,232 (cache_prompt: false, full re-prefill)	200 OK, 39.4s	101.3 tok/s	n/a
8,752 total (~5.5k new, rest from prompt cache)	200 OK, 83.0s	76.2 tok/s	7.9 tok/s

No Timedout job / devcoredump in dmesg for any of these, and the process stayed healthy throughout. I didn't need the -ub 128 run since nothing hung. For the same prompt, the previous build (1a07bfa) failed every time with ErrorDeviceLost.

Thanks for the quick turnaround on #238. Fine to close from my side.