Ai/hardening ii/vllm authority - #20
Open
DarthPJB wants to merge 48 commits into
Open
Conversation
added 30 commits
August 24, 2026 14:21
Phase 1 delivers model package infrastructure only. CPU inference validation belongs to Phase 2 (module enhancement). Removed misplaced criteria from Phase 1 gate.
…Phase 3.2) Qwen/Qwen3-Coder-30B-A3B-Instruct — 30B-parameter MoE code LLM, 3B active. Pinned to commit b2cff646eb4bb1d68355c01b18ae02e7cf42d120 with per-file SRI hashes for all 24 files (config, tokenizer, LICENSE, 16 safetensors shards). Replaces qwen3-coder:30b-a3b-q4_K_M in Ollama. Apache-2.0. Builds to /nix/store/zv4bv993gy5gpkz6p3667ymxf2442sxr-qwen3-coder-30b-a3b-2512; formatting, deadnix, topology-coverage, and golden-validation checks pass.
vLLM-only migration complete. Ollama decommissioned on LINDA, all models now served by vLLM with Nix-managed packages and full Prometheus coverage. Phase 1: Foundation - pkgs/models/ template for HuggingFace model packages - qwen3-8b, qwen3-30b-a3b, qwen3-coder-30b-a3b model packages - Models pinned to commit SHAs with per-file SRI hashes Phase 2: Module Enhancement - vLLM module: device (gpu/cpu), modelPath, cpuKvCacheSpace, cpuOmpThreadsBind - Per-model systemd service generation with MemoryMax for CPU models - CPU model deployment on LINDA (port 8002) Phase 3: Model Migration - LINDA: GPU qwen2.5-vl (:8001), CPU qwen3-30b-a3b (:8002), CPU qwen3-coder (:8003) - LiteLLM routing updated to vLLM backends - Ollama removed from LINDA, archived to services/archive/ Phase 4: Monitoring - Prometheus scrape targets for all vLLM models and LiteLLM - LiteLLM callbacks = ["prometheus"] enabled - AI inference Grafana dashboard (ai-inference.json) Phase 5: Cleanup - CUDA scoped to vLLM via pkgsCuda overlay (no global cudaSupport) - Open-WebUI pkgsNoCuda replaced with overrideScope - All 19 golden tests pass Phase 6: Documentation - ai-stack.md updated for vLLM-only architecture - vllm-architecture.md updated with implementation details - ai-upgrades.md: all P1-P5 priority issues resolved Laguna models flagged for review (custom GGUF, not on HuggingFace). Cluster-box not migrated (external Malayalam flake).
vllm CUDA build was failing on remote builder with 'Can't find CUDA or HIP installation' using nixpkgs_llm (unstable). Switching to nixpkgs_stable which has working CUDA toolchain propagation.
nixpkgs_stable vllm 0.16.0 has starlette version conflict. Reverting to nixpkgs_llm and building locally to investigate CUDA issue.
The pkgsCuda overlay overrode only vllm + torch at the vllm level, but vllm's transitive python deps (torchaudio, torchvision, xformers, triton) stayed bound to the CPU torch from the un-overridden scope. Both torches landed in the build env; cmake's find_package(Torch) picked the CPU TorchConfig (CAFFE2_USE_CUDA=OFF), CUDA_FOUND was never set, and the build died with 'Can't find CUDA or HIP installation'. Fix: overrideScope on python313Packages rebuilds every torch-dependent package against the CUDA torch, plus triton -> triton-cuda (from the outer scope to avoid self-referential recursion) so packages using both torch and triton (xgrammar) don't hit a duplicate-triton closure conflict. Validated: LINDA toplevel builds (exit 0), vllm 0.24.0 links CUDA (libc10_cuda, libcudart, libcuda), cmake reports 'Found CUDA 12.9'.
…agation Root cause: pkgsCuda overlay only overrode vllm and torch, leaving transitive deps (torchaudio, torchvision, xformers, triton) on CPU torch. cmake found CPU torch's TorchConfig.cmake and failed with 'Can't find CUDA'. Fix: overrideScope on python313Packages ensures CUDA torch propagates to all transitive dependencies. triton-cuda referenced from outer scope to avoid recursion. LINDA system build succeeds. Golden regenerated for llama-cpp/vintagestory version bumps from nixpkgs_llm flake input.
- Replace pkgsCuda overlay with separate nixpkgs import (cudaSupport=true) - Revert modifier_imports/cuda.nix to ai/hardening-II (ollama-cuda restored) - Remove Laguna model references (custom GGUF, not on HuggingFace) - cluster-box backend now serves ornith:9b and ornith:35b only - Ornith-1.5-9B available on HuggingFace in safetensors format
cluster-box backend must not be altered — external machine, dlyon-operated. Restored laguna-xs-2.1:q4_K_M to cluster-box models list. vLLM migration changes preserved: linda-vllm, linda-vllm-cpu, linda-vllm-coder backends added, old linda Ollama backend removed, Prometheus callbacks enabled.
vLLM needs to locate CUDA tools during initialization. Added to the systemd service definition.
FlashInfer needs nvcc for JIT compilation of CUDA kernels. Added pkgs.cudaPackages.cudatoolkit to the service PATH.
FlashInfer JIT needs ninja to build CUDA kernels. Added pkgs.ninja to the service PATH alongside which and cudatoolkit.
ProtectSystem='strict' blocks /bin/sh which ninja needs for posix_spawn. Changed to ProtectSystem=true — still protects /usr, /boot, /efi but allows /bin and /sbin for JIT compilers.
FlashInfer JIT needs c++ compiler for building CUDA kernels. Added pkgs.gcc to the service PATH.
FlashInfer JIT compiles CUDA kernels at runtime via ninja/c++/nvcc. posix_spawn fails under any ProtectSystem setting because the spawned compiler subprocess needs full filesystem access. Disabled: NoNewPrivileges, ProtectSystem, ProtectHome. TODO: Pre-compile FlashInfer kernels at build time to re-enable.
ninja spawns /bin/sh -c 'c++ ...' which doesn't inherit systemd PATH. GCC and ninja must be in environment.systemPackages for system-wide availability.
FlashInfer requires runtime JIT compilation (gcc, ninja, nvcc) which breaks under systemd security hardening. Instead of disabling hardening, use pre-compiled FLASH_ATTN backend and triton sampling fallback. - attentionBackend default: FLASH_ATTN (pre-compiled, no JIT) - VLLM_USE_FLASHINFER_SAMPLER=0: triton fallback for sampling - Restored security hardening (ProtectSystem=strict, NoNewPrivileges) - Removed gcc/ninja/cudatoolkit from PATH (no longer needed)
FLASH_ATTN requires GPU access. CPU models automatically use TORCH_SDPA attention backend instead.
CPU models need pkgs_llm.vllm (CPU-only build) because the CUDA build always tries to use the CUDA platform. Added vllmPackageFor function that returns the appropriate package based on device type.
CPU-only vllm package can't infer device type without CUDA driver. Set VLLM_TARGET_DEVICE=cpu environment variable for CPU models.
0.24.0 only trusts importlib.metadata for CPU. Wrap, don't rebuild.
… KV cache - Replace qwen3-30b-a3b (57GB MoE) with qwen3.8-27B (55GB dense VL model) - Add util-linux to vllm-cpu wrapper PATH (lscpu needed by CPU worker) - Reduce cpuKvCacheSpace from 40 to 4 GiB (55GB model on 68GB RAM) - Document systemd socket activation as future work for model lifecycle - Remove --device cpu flag (CpuPlatform auto-detects from +cpu metadata)
- New per-model autoStart option (default true) - qwen3-coder-30b-a3b: autoStart = false (manual: systemctl start vllm-qwen3-coder-30b-a3b) - qwen38-27b starts at boot, coder only when needed
128GB RAM, 102GB usable. One CPU model at a time (~55GB). 40GB KV cache + 55GB weights + 7GB headroom.
torch.compile hangs on CPU for 27B+ models (warmup never completes). enforceEager skips compilation — fast startup, ~20-30% slower per-token. KV cache reduced from 40 to 30GB (55+30+20+8=113GB on 128GB).
Top-level enforceEager exists but wasn't exposed in the per-model submodule. CPU models can now set enforceEager = true independently.
Qwen2.5-VL-7B-AWQ used ~9.2GB (weights+vision encoder), leaving no room for KV cache. 3B variant uses ~5GB, leaving ~5GB for KV cache and headroom. Single safetensors file, same architecture, same API.
added 18 commits
August 27, 2026 11:48
- linda-vllm-cpu: qwen3-30b-a3b → qwen38-27b (vision-language, 27B dense) - prometheus: update model label for vllm-cpu scrape target - litellm.nix: update fallbacks example
LiteLLM sends tool_choice=auto by default. vLLM requires --enable-auto-tool-choice and --tool-call-parser hermes for Qwen models.
First compilation is slow but cached to TORCHINDUCTOR_CACHE_DIR. Subsequent restarts load compiled kernels from disk instead of recompiling. The warmup forward pass still runs but compilation is the expensive part that now persists.
Clients sending max_tokens > model max_model_len causes vLLM rejection. drop_params=true lets LiteLLM silently drop unsupported params.
Client sending max_tokens=32000 to qwen2.5-vl (max 8192) is a client bug, not a server config issue. maxTokens is correctly set per model.
New per-backend option maxTokensParam sets litellm_params.max_tokens, which caps the parameter before forwarding to the backend. Unlike maxTokens (advertised context window), this prevents client overshoot. Set on linda-vllm to 8192 (model max_model_len).
LiteLLM sends tool_choice=auto by default. All Qwen models need --enable-auto-tool-choice and --tool-call-parser hermes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.