FreeToken-Intel: 35B MoE, live tokens off an Arc Pro B70 (SYCL / XPU port)

[aangelinsf]

FreeToken-Intel: 35B MoE, live tokens off an Arc Pro B70

Happy to report that work is underway. 🥳

Repo: Performant-Labs/FreeToken-Intel

Note that I just reached a major milestone: first tokens from the card! Woot! 🎉

Contents

• Glossary
• Summary
• Dig into the Details
• Appendix

Glossary

Term	Meaning
Runner	A machine that picks up GitHub Actions CI jobs. “GitHub-hosted” = GitHub’s cloud VMs (no Intel GPU). “Dedicated / self-hosted” = a worker we run on this workstation.
B70	Intel Arc Pro B70, the 32 GB Xe2 GPU in this box.
XPU	PyTorch’s name for an Intel GPU (torch.xpu, vs torch.cuda).
SYCL / oneAPI / Level Zero	Intel’s compile and runtime stack. Replaces CUDA and nvcc.
MoE / expert	Mixture of Experts: most of the 35B is experts that stay off until a token routes to them.
Offload	Keep experts in host RAM; copy one onto the GPU when needed. How it fits today.
Hybrid / q*	Run some experts on the CPU at the same time as the GPU. Bandwidth-adaptive policy from the FreeToken paper. Not online yet.
Prefill / decode	Prefill = chew the prompt. Decode = emit the next tokens.
Epic / issue / PR	GitHub: epic = parent ticket; issue = a slice of work; PR = the code change.
oc-monitor	A Claude session whose job is to watch Qwen (OpenCode transcript + logs) and unwedge it.
Wedge	The coding model hangs (GPU queue stuck, vLLM idle) until someone restarts or unsticks it.
Dual-venv	Two Python environments: CPU tests must not import torch; the XPU env is where the GPU stack lives.

---

Summary

This is a port, not a from-scratch engine. Upstream (FlashML FreeToken, flashml.ai, paper) already has CUDA, q*, LRU, agents. We had to stand up the Intel harness first. Two tracks.

Who wrote what. Claude Opus wrote all the initial epics (the map of the port). After that, Qwen3.8 took over: it updates those epics, files new issues, and wrote all the PRs. That is still not an automatic port. I watch every agent, reject dead ends, unstick wedges, and decide what is allowed to touch the B70.

Harness (Intel-only — not in the NVIDIA repo)

#	Milestone	Status	When	Notes
H0	Repo scaffold, epic #1, dual-venv, ft CLI	done	8/23	#41 — week starts here, not at first XPU kernel
H0b	Engine loop + loader + OpenAI HTTP	done	8/24–8/25	#14 #17 #25 — ft serve exists; GPU kernels still placeholders
H1	SYCL / Level Zero / torch.xpu device layer	done	8/28	#2 #3 — Intel compiler in place of NVIDIA’s nvcc
H2	Two CI workers on this workstation + our own machine image	done	8/26–8/28	Epic #42. GitHub’s cloud VMs have no Arc Pro, so they cannot run GPU jobs
H3	CPU vs GPU tests + oc-monitor watching Qwen	done	8/28–8/30	Qwen and the 35B share one GPU
H4	🎉 Live tokens on the B70	done	8/30	#93 / cf9e38c — prefill + decode on the card

Upstream design, on Intel (what FlashML calls FreeToken)

#	Milestone	Status	When	Notes
1	Readable chat / agent APIs	done	8/30	#95 — real encode/decode (same day as first tokens). Thinking-mode dialects still #97
2	Correct hero + LRU offload closed	next	—	#7 #12 #13 #24
3	Hybrid (q) online*	open	—	#8 #9 — CPU and GPU overlap on experts. Paper title.
4	Interactive rates	open	—	#30 then #10 #15 — a tokens/sec we will quote
5	Elastic memory	open	—	#16
6	Semantic KV cache	open	—	#32
7	FTW fast load	open	—	#11
8	Ship it	open	—	#27 #28 #31
—	Close epic #1	~34%	—	after 1–4: chat at interactive rates, offload and hybrid
—	Design-complete	later	—	5–7 + other models / multi-GPU

Today (cf9e38c) the 35B Qwen3.5/3.6 MoE prefills and decodes on the card through ft serve (#93). Tokenizer #95 landed the same afternoon.

• 32 GB of GPU memory is not enough: dense layers on the GPU, ~28 GB of experts in host RAM
• The coding model (Qwen3.8) and the 35B we are porting share one B70
• Hybrid is not online yet — milestone 3, and why this exists vs llama.cpp / vLLM on Intel
• No tokens/sec I’m willing to quote yet

CUDA vs this port

Layer	Upstream (CUDA / RTX)	This port (Intel / B70)
Compiler	nvcc	icpx -fsycl (oneAPI DPC++)
Runtime	CUDA	Level Zero
PyTorch device	torch.cuda	torch.xpu
Compiled kernels	cubins (NVIDIA binaries)	SPIR-V / Triton-Intel ahead-of-time cache
Attention / MoE kernels	CUDA + Triton	in-tree SYCL + triton_xpu
GPU	NVIDIA RTX 30/40/50	Arc Pro B70 (Xe2, 32 GB)
Multi-GPU	NCCL	oneCCL (not yet)

How the agents are wired (I sit in the middle on purpose). oc-monitor’s job is to watch Qwen:

flowchart TD
  You[Human — watches all of it]
  JI[Claude Opus: initial epics]
  MON[Claude: oc-monitor]
  FT[Claude: FreeToken-Intel]
  OC[OpenCode: Qwen3.8 — writes issues and PRs]
  VLLM[vLLM serving Qwen3.8 on the B70]
  LOGS[Qwen's transcript + vLLM / watchdog logs]

  You --> JI
  You --> FT
  You --> OC
  You -->|"before next epic: what should the monitor prepare for?"| OC
  JI -->|"Claude cross-session"| MON
  FT -->|"paste / instruct"| OC
  OC -->|"llama-swap"| VLLM
  LOGS -->|"oc-monitor reads these — that is the watch"| MON
  MON -->|"paste-ready unwedge"| You
  MON --> FT

Loading

• Claude sessions talk to each other over Claude’s cross-session channel (oc-monitor was spawned that way).
• OpenCode / Qwen3.8 is not a Claude peer, so the monitor cannot send it a message. It watches by reading the OpenCode transcript and logs. I paste the unwedge into the worker.
• Before Qwen starts the next epic I ask: “Before you start the next epic, what do you want the Claude monitor to prepare for?” so watchers exist before the next hang.

---

Dig into the Details

As of 8/30 (cf9e38c), ft serve loads the 35B Qwen3.5/3.6 MoE on an Intel Arc Pro B70, prefills, and decodes on the card through the OpenAI-compatible API (#93). #95 wired real chat encode/decode the same day.

What that is / isn’t

• Prefill + decode on the Intel GPU, with a real tokenizer (prompt text in, tokens out)
• Thinking-mode dialects still #97
• No tokens/sec I’m willing to quote yet
• SYCL / Level Zero / PyTorch XPU port of FlashML’s FreeToken
• llama.cpp and vLLM already generate on Intel GPUs; this is FreeToken’s expert-offload serving design on that stack

Why it fits a workstation

• The 35B does not fit in 32 GB of GPU memory
• Dense layers stay on the card; ~28 GB of routed experts live in 32 GB of host RAM
• Hybrid is when the CPU and the B70 work those experts at the same time instead of taking turns

Who wrote it. Claude Opus wrote the initial epics. Qwen3.8 (OpenCode, via llama-swap → vLLM on this same GPU) updates them, files new issues, and wrote every PR. I set direction, watch the agents, and decide what may touch the B70. Not autopilot.

Hardware

• GPU: Intel Arc Pro B70, Xe2 Battlemage, 32 GB, 608 GB/s
• RAM: 32 GB DDR5 (expert banks)
• CPU: AMD Ryzen 9 9900X
• OS: Ubuntu 26.04

XPU stack

• oneAPI 2026.1, icpx -fsycl, Level Zero
• torch 2.13.0+xpu
• in-tree SYCL + triton_xpu 3.7.2

Same card, two jobs

Early on, Qwen ran GPU tests on the same card it was inferencing on. A crashing test left the GPU queue stuck; the coding session died. That happened a lot at the beginning (7 hangs in ~2 hours on one stretch).

CI was never the problem. We changed how the agent is allowed to test here: CPU-only by default; GPU tests time-capped and not looped; the GPU suite belongs on nightly CI (or a window when Qwen does not need the card).

oc-monitor — watching Qwen

A parallel Claude session watches Qwen so a hung OpenCode stream does not kill the day. Stock logs were not enough: OpenCode showed “stream then Aborted”; restarting vLLM deleted the container logs for the hang.

We added a log follower that survives container restart, plus wakes when the watchdog fails. After that, the first hang had real engine evidence instead of a guess. The monitor reads Qwen’s transcript plus those logs and hands me a paste-ready “how to proceed.”

CI

This workstation is the CI machine, not just the GPU. Two GitHub Actions runners (workers) live on it and pull jobs from the FreeToken-Intel repo. GitHub’s own cloud VMs have no Arc Pro and no oneAPI, so GPU jobs would never run there — we had to bake a machine image and keep the workers here. A fork’s pull request still runs cheap CPU checks on GitHub’s cloud, so a stranger cannot execute code on this card.

Tests

~120 tests, 23 files. Only Qwen3.5/3.6 is known-good on the B70. Other model names in the tree are placeholders.

Where	What
CPU (GitHub’s cloud + local default)	CLI, engine, loaders, expert cache, OpenAI + Anthropic HTTP
GPU (nightly on this box / careful local windows)	device, SYCL, attention, fused MoE, offload, live ft serve

#93 landed with a test that drives the real engine through ft serve, not a notebook print.

What closed 8/30

Time	What
~12:06	#18 Qwen3.5/3.6 as the B70 model
~13:18	#93 live tokens via ft serve
~15:25	#95 real chat encode/decode

---

Appendix

Two tracks on purpose. FlashML assumes CUDA, an NVIDIA GPU, and a desktop installer. This repo had to build the Intel harness first.

Harness

#	When	What landed
H0	8/23	Epic #1; package, docs, dual-venv, ft CLI (#41). Day zero.
H0b	8/24–8/25	OpenAI ft serve (#25), loader (#17), engine loop (#14). GPU kernels still placeholders.
H1	8/28	Device probe #2, SYCL toolchain #3.
H2	8/26–8/28	Two CI workers on this workstation, custom image (#42). Forks stay on GitHub’s cloud CPUs.
H3	8/28–8/30	GPU nightly; CPU-only local tests for the agent; oc-monitor watching Qwen.
H4	8/30	Live tokens #93 / #18.

Upstream design on Intel

Epic #1 closes when ft serve on a B70 speaks OpenAI chat at interactive rates, with offload and hybrid. That is rows 1–4. Rows 5–7 are also FlashML core features; they are not the hero-serve gate.

#	Status	Feature
1	done 8/30	Readable chat #95. HTTP was already up (#25 #26). Thinking-mode #97 still open.
2	next	Correct hero + close expert-cache #7. KV #12, scheduler #13, remaining layers #24.
3	open	Hybrid (q)* #8 #9. Offload is “it fits.” Hybrid is the paper title.
4	open	Quoted tokens/sec #30 (smaller weights #10, captured GPU graphs #15 if needed).
5	open	Elastic memory #16 (expert cache vs KV, no reload).
6	open	Semantic KV #32 (agent edits without recompute).
7	open	FTW fast load #11.
8	open	Ship: shell/daemon #27, ft launch #28, installable packages #31.

To run this (not pip install on a laptop)

You need	Why
Intel Arc Pro B70-class GPU (Xe2, ~32 GB)	The card this port targets
oneAPI 2026.1 / icpx + Level Zero	Replaces CUDA / nvcc
~32 GB host RAM	Routed experts live here
Ubuntu + torch.xpu	Device and driver stack
Weights on local disk	Tests and serve do not download a 35B MoE by default

Repo: Performant-Labs/FreeToken-Intel