Testing FreeToken : 1.75x Over a Tuned llama.cpp
The last post covered the FreeToken paper, which argues that running a big MoE model locally is a scheduling problem rather than a fitting-in-VRAM problem. Fine as an argument. I wanted to know if it beats the tools I already have installed.
So I ran it on my own desktop. My first answer was 3.81x, and it was wrong.
The Short Version
One RTX 5070 Ti, 16 GB of VRAM, serving gpt-oss-120b. That’s 117 billion parameters, 61 GB of weights on disk.
FreeToken 42.42 tokens/s
llama.cpp (tuned) 24.20 tokens/s 1.75x slower
Ollama (default) 11.14 tokens/s 3.81x slower
Same MXFP4 weights in all three. Same prompt, 256 output tokens, greedy. Everything separating them is a decision about what lives where.
Here is what that gap looks like. Both engines answering the same question, replayed at the speed they actually ran:
1. The Setup
An ordinary gaming desktop:
- GPU — RTX 5070 Ti, 16 GB VRAM, sm_120 (Blackwell)
- CPU — Ryzen 9 9900X, 12 physical cores
- RAM — 128 GB DDR5
- Link — PCIe 5.0 x16
- OS — Windows 11, everything run inside WSL2 (Ubuntu 24.04)
FreeToken’s CLI is Linux-only, so WSL2 wasn’t optional. It quietly costs about 7%, and I’ll come back to that.
The 128 GB of system RAM is what makes this machine interesting. Expert offloading treats host memory as the model’s real home and lets the GPU borrow pieces of it, so RAM capacity sets the ceiling on what you can serve at all.
Picking the model took longer than expected. The README advertises 290B+ models, but I checked the actual checkpoint sizes first and most of them don’t fit in 128 GB. GLM-5.2 wants about 400 GB. DeepSeek-V4-Flash wants 167 GB. gpt-oss-120b at 65 GB was one of the few that did fit, and it had a property I cared about more than size:
Ollama’s
gpt-oss:120band the Hugging Face checkpoint are the same MXFP4 weights.
Local inference comparisons are usually contaminated. One engine runs a 4-bit GGUF, the other runs something else, and part of the speed difference is really a precision difference nobody mentions. Here I could confirm all three engines load identical numbers, so the benchmark measures engines.
2. FreeToken Measures Your Machine First
Before serving anything, FreeToken runs a calibration pass over the two routes an expert can take. Copy it across PCIe to the GPU, or leave it in RAM and compute on the CPU.
ft bench bw --dtype mxfp4,bf16
PCIe came in at 56.6 GB/s. The CPU’s entire memory read bandwidth is 54.2 GB/s. So shipping an expert to the GPU costs roughly what reading it on the CPU costs, before the CPU does any arithmetic at all. PCIe 5.0 is just that fast next to dual-channel DDR5.
FreeToken only splits work across both paths when the CPU is at least twice as fast. It isn’t close here, so it picked pure PCIe streaming:
Auto-selected MoE backend: offload
Which means the paper’s headline mechanism, the bandwidth-adaptive split, does nothing on my hardware. That’s the policy working. Its job is to look at the machine and decide, and on a PCIe-rich desktop the right decision is to leave the CPU alone. It would earn its keep on a laptop with a narrow PCIe link, or a workstation with eight memory channels feeding a slower bus. I don’t have either, so I can only report that it declined to engage and was right.
3. The First Answer, and Why It Was Wrong
I started with Ollama, since that’s what most people have installed. FreeToken hit 42.42 tok/s against Ollama’s 11.14. A 3.81x gap is larger than I predicted, so I went looking for a bug in my harness instead of believing it.
The harness was fine. The baseline wasn’t. Running ollama ps during the benchmark showed what was actually happening:
gpt-oss:120b 66 GB 78%/22% CPU/GPU 4096 ctx
"offloaded 8/37 layers to GPU"
Eight complete layers on the GPU, twenty-nine running entirely on the CPU. When I went looking for a way to improve that, I found Ollama doesn’t really offer one. Its environment variables cover context length, KV cache type, flash attention. Nothing that changes how the model gets split. The one offload knob is num_gpu, which sets how many whole layers go to the GPU, and Ollama had already filled VRAM to 14.5 of 16.3 GB. Maybe one more layer would fit.
“3.81x faster than Ollama” is a true measurement of a real default configuration. It is not a fair statement about the state of the art. Ollama wraps llama.cpp, and llama.cpp has a much better option that Ollama never exposes.
4. The Real Comparison
The option is --n-cpu-moe, and why it matters is the whole point of this post.
A transformer layer holds two things with completely different shapes. Attention is small but does heavy math. The experts are enormous but only a handful get used per token; gpt-oss-120b has 128 per layer and picks 4. When Ollama sends a layer to the CPU it sends both. The small compute-heavy part gets dragged onto the slow device along with the bulky part, for no reason anyone would choose.
--n-cpu-moe N pulls them apart. Every layer’s attention stays on the GPU, and only the expert weights of the first N layers move to the CPU. Heavy math on the fast device, bulk wherever there’s room.
Getting this measured was more annoying than it should have been. CUDA 13.1’s runtime segfaults inside WSL’s driver stub, so I built llama.cpp against 12.8 instead. Then Ollama’s GGUF refused to load, because it declares its architecture as gptoss while upstream llama.cpp expects gpt-oss. Ollama’s model files aren’t readable by the project it’s built on. I ended up downloading llama.cpp’s own MXFP4 build of the same model.
With that sorted, llama.cpp reaches 24.20 tok/s. That is more than double Ollama’s 11.14, on the same engine, same weights, same GPU. One flag.
5. Why llama.cpp Still Hits a Ceiling
The sweep tells you more than the peak number does. Every time I moved two more layers’ worth of experts onto the GPU, throughput rose about 2%. The climb stops at N=28, with 15.7 of 16.3 GB used. There is nowhere left to go.
At that limit, 78% of the expert work is still on the CPU. Moving layers two at a time picks at the edge of the problem without changing it. The CPU carries the bulk either way, and it manages roughly 16 GB/s effective, under a third of what PCIe delivers here.
FreeToken gets 42.42 tok/s using less VRAM, 13.9 GB. It never makes the layer-level decision. All 36 layers compute on the GPU and individual experts get fetched across PCIe as the router asks for them. You’re moving books instead of shelves.
One measurement shows the payoff cleanly. FreeToken keeps recently-used experts cached in VRAM, 702 of the model’s 4,608, about 15%. If the router picked at random, that cache would hit 15% of the time. It hits 31.5%:
15.2% of experts resident -> 31.5% of lookups served
Routers have favourites. Some experts are far more popular than others, and a cache working at expert granularity can exploit that. A layer-granularity cache can’t, because it committed to whole layers before the router said anything.
6. What I Take Away
The paper’s framing held up, but not through the mechanism it spends the most pages on. The bandwidth-adaptive split sat inert on my hardware. It measured the machine, concluded the CPU wasn’t worth using, and got out of the way. The gap came from the plainer decision to offload at expert granularity rather than layer granularity, plus a cache that exploits the fact that routers have favourites.
That’s the part worth carrying elsewhere. A layer is a convenient unit when you’re writing the code, because it’s how the model is built. It’s a bad unit when you’re scheduling, because it forces two unrelated decisions to be made together and one of them is usually wrong.
The other thing I’d keep is about method. My first number was measured correctly, reproduced cleanly, and misled anyway. It only became honest after I went and built the thing I had dismissed in a caveat. When a benchmark writeup says “of course, X wasn’t tuned,” that’s usually the experiment that needed running.