Intel GPU backend, shared graph cache, and checkpoint hardening (0.2.0) - #18
Open
khanhnd61-vr wants to merge 34 commits into
Open
Intel GPU backend, shared graph cache, and checkpoint hardening (0.2.0)#18khanhnd61-vr wants to merge 34 commits into
khanhnd61-vr wants to merge 34 commits into
Conversation
ctest registers six tests but the workflow named only two, so the other four reported Not Run and the job exited 8. Building the default target also keeps the next test that gets added from breaking it again.
A crafted checkpoint could overrun host buffers in five places. The smolvla safetensors reader trusted data_offsets against a shape-derived buffer, bitvla and gr00tn1d6 trusted patch-count keys that the image grid contradicts, vla_adapter trusted head_blocks past the layer count, and the qwen3-vl position resample only clamped the upper interpolation index. Each is now a load-time check or a clamp. While here, smolvla, evo1 and bitvla treat a missing state vector as zeros like the other eight archs instead of dereferencing it. All eleven checkpoints load unchanged and predict_check output is identical.
Both servers drained multipart frames with a blocking recv, so one client that announced a frame and went quiet held the loop forever and shutdown never ran. A 5s receive timeout bounds it and the drain now reports the stall instead of replying into a socket that is still mid-message. Also reject a cache path containing a quote and a repo id starting with a slash: both reach the hf download shell command through VLA_CACHE or HOME.
Backend::is_cuda and is_gpu were copied into every arch and never read back. Anything that needs the device type can ask ggml_backend_get_device.
vla_model_load leaked the whole engine if the handle allocation threw. The sycl guide named GGML_SYCL_DISABLE_DNN, which does not exist; ggml reads GGML_SYCL_ENABLE_DNN. The chat example linked a file that was never written.
Numerics: GR00T N1.5 and N1.6 move by up to 4.6e-4 on actions peaking near 0.87 (mean 6e-5), from an upstream ggml kernel change on the SigLIP tower path the two share. The other nine archs are bit-identical. Also drops GGML_CUDA_GRAPHS from the Dockerfile, which llama.cpp already defaults on.
Only gr00tn1d7 kept its compute graph between predict calls; the rest rebuilt theirs every time and reused just the arena. graph_cache in scratch_ctx.h holds the context, allocator, graph, shape key and input handles, so an arch supplies only its key and a build lambda. gr00tn1d7 moves onto it and evo1, pi0, pi05, gr00tn1d5, gr00tn1d6, vla_adapter, openvla_oft and vla_jepa gain it. A stable graph is also what lets ggml-cuda capture and replay. Dump modes still rebuild, since they add graph outputs. All eleven archs verified bit-identical with vla_predict_check.
sinusoidal_time_emb was copied into pi0, pi05 and smolvla, and the causal mask loop into five archs; both move to dit_common.h. dual_tower derives the patch count from the conv output instead of assuming 256. test_config_guard now calls config_is_sane rather than reimplementing it, so a change to the real guard can fail the test. It links vla_core for that, which takes it out of the standalone sanitizer job.
The quickstart needed raw token ids, which meant setting up the python eval client before the first command would run. --text calls scripts/tokenize_prompt.py with the tokenizer the arch was trained on and feeds the ids straight in; --tokens still works. VLA_PYTHON picks the interpreter and VLA_TOKENIZE_SCRIPT the script, so a release tarball can carry its own. Release also builds linux-aarch64-cpu on a native arm64 runner. Jetson is the deployment target and only x86_64 and macOS had binaries. CUDA on aarch64 still has to be built on the device.
Latency re-measured on the 5090 at the graph-cache code, best of three sweeps. Also surfaces the LIBERO success rates that were sitting in eval/reports, with their hardware and commit stated: latency alone does not say the policy works.
anindex
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an Intel GPU backend, collapses the per-arch backend selection that made
adding one painful, then extends the same treatment to the compute graph. Tagged
as 0.2.0.
Intel GPU via SYCL
src/backend.his the new shared ladder. Every arch used to open-code the sameaccelerator-then-CPU sequence, so a new backend meant editing a dozen files; they
now call
vla::backend_init. Exactly one accelerator is compiled in, picked bythe CMake flag (
GGML_CUDA/GGML_SYCL/GGML_METAL), and configure fails iftwo are on. BitVLA keeps its own CPU init: it pins its ggml graph to the CPU and
offloads through its own ternary CUDA kernels.
VLA_DEVICEselects the ordinal on both multi-device backends. The SYCL pathrange-checks it, because
ggml_backend_sycl_initguards the index with anassertthat a Release build compiles out.Intel Arc A380 (6 GB, Xe-HPG) against the CPU backend on an AMD Ryzen 5 5500,
8 threads:
Two SYCL workarounds.
GGML_SYCL_ENABLE_VMM=0is defaulted at backend init:ggml-sycl's VMM pool hands out pointers oneDNN cannot wrap in a
dnnl::memory,which aborts the GEMM for any non-F32
src0. It is only a default; setting itexplicitly wins.
GGML_SYCL_ENABLE_DNN=0also clears the crash but costs ~8%.Setup and the memory ceiling are in docs/backend/sycl.md.
Shared graph cache
Only GR00T N1.7 kept its compute graph between
predictcalls; the other archsrebuilt theirs every time and reused just the arena.
vla::graph_cacheinsrc/models/scratch_ctx.hholds the context, allocator, graph, shape key andinput handles, so an arch supplies only its key and a build lambda. N1.7 moves
onto it and eight more archs gain it. A stable graph is also what lets ggml-cuda
capture and replay.
Measured on an RTX 5090 by alternating before/after binaries per model, several
rounds: 3-10% on pi0, pi0.5, GR00T N1.5/N1.6, VLA-Adapter and VLA-JEPA, neutral
elsewhere. A single sequential sweep across all 11 models is not usable for this
comparison; the models at the tail drift by 50% or more between runs.
Hardening
Five paths where a crafted checkpoint could overrun a host buffer: the smolvla
safetensors reader trusted
data_offsetsagainst a shape-derived buffer; bitvlaand gr00tn1d6 trusted patch-count keys the image grid contradicts; vla_adapter
trusted
head_blockspast the layer count; the Qwen3-VL position resampleclamped only the upper interpolation index. Each is now a load-time check or a
clamp, verified by loading hand-patched GGUFs.
Two reachable without a crafted file: a peer that announced a multipart frame and
went quiet held either server's loop forever, and the
hf downloadcommand tookits cache path from
VLA_CACHEorHOMEunescaped.Also fixes the red CI.
ctestregisters six tests and the workflow built two, sothe other four reported not run.
Numerics
Every change is bit-identical across all 11 archs, checked with
vla_predict_checkperCONTRIBUTING.md. One exception, stated in the changelog:bumping llama.cpp to b10331 moves GR00T N1.5 and N1.6 by up to 4.6e-4 on actions
peaking near 0.87, from an upstream ggml kernel change on the SigLIP tower the two
share. The other nine are unaffected.
Distribution
vla-cli --text "pick up the bowl"tokenizes with the tokenizer the architecturewas trained on, so the quickstart no longer needs raw token ids. Release builds
linux-aarch64-cpuon a native arm64 runner for Jetson-class boards. The READMEgains the LIBERO success rates that were sitting in
eval/reports, labelled withtheir hardware and commit.