Hey guys,
After my CPU-only to 96GB VRAM test, I tested Qwen3.8-Flash-Next across llama.cpp, SGLang and FreeToken on the same workstation.
This time I wanted to see what changes when you keep the hardware and model family fixed, but change the engine, weight format and memory placement.
I also tested newer builds, PR patches and speculative decoding: llama.cpp's MTP fork, SGLang's Blackwell support patches, n-gram speculation and an experimental PLE read-path build.
Short version:
- At the full 262K window, time to first token was 35.4s in SGLang, 80.4s in FreeToken, 210.2s in llama.cpp + MTP and 258.4s in the llama.cpp baseline.
- That is a 7.3x difference in waiting time between the fastest and slowest tested configurations.
- In the separate context sweep, llama.cpp decode fell from 101.9 to 20.2 tok/s. FreeToken stayed much flatter at 100.1 to 94.8 tok/s.
- On matched coding tests, llama.cpp MTP improved decode by 1.63x at 8K and 1.69x at 32K.
- GSM8K scores were 95.22–95.75%; MATH-500 was 92.20–93.00%. The paired tests did not detect a significant difference.
- Startup went the other way: llama.cpp reached an answer in 16s, SGLang in 108s, FreeToken in 126s.
https://preview.redd.it/ztjuce0wfcoh1.png?width=1725&format=png&auto=…
Setup
- GPU: NVIDIA RTX PRO 6000 Blackwell, 96GB VRAM
- CPU: AMD Ryzen 9 9950X
- System RAM: 96GB DDR5
- OS: Ubuntu, CUDA 13, Docker
- Model: Qwen3.8-Flash-Next
- llama.cpp: UD-IQ4\_XS GGUF; MTP tested on the qwen4exp/mtp fork
- SGLang and FreeToken: the same NVFP4 checkpoint revision
- Client: AIPerf, with thinking off and the same non-thinking sampler
I ran one engine at a time, with fresh starts and GPU cooldowns. The runs saved resolved configurations, outputs, memory use and GPU telemetry.
Important note about the comparison
These are results for the tested stacks on this workstation. Quantization, KV-cache format, memory placement and speculative decoding differ.
SGLang uses its NEXTN draft head. FreeToken has no speculative decoding in the tested setup, I couldn't get it to work. llama.cpp has separate baseline and MTP results.
So the headline does not isolate the engine software alone. The repository includes the configurations so you can see what produced each number.
Newer builds, PRs and speculative decoding tested
- llama.cpp MTP: the
danielhanchen/llama.cpp qwen4exp/mtp fork, pinned to d1a92352, with the roughly 2.6GB draft head. On matched coding tests, decode improved 1.63x at 8K and 1.69x at 32K. Those gains compare the same build with the head off and on. - SGLang on Blackwell: the tested image included PRs \#36567, \#36556, \#36749 and \#36750, plus a local FP8 KV-cache patch. These were part of the working configuration, not individually benchmarked speedups.
- N-gram speculation:
ngram-mod gave +6.8% decode on the tested code workload, but generated zero drafts on the tested prose with the 24-token match setting. - Experimental PLE reads: I built llama.cpp PR #28136, but withdrew the read-mode comparison after discovering that a renamed flag was ignored. The intended direct-read mode was never exercised, so I am not claiming a speedup from that PR.
The report records the pinned builds and withdrawn findings alongside the successful tests.
1. All four configurations fit the full window. The waiting time is very different.
This test uses roughly 261,500 input tokens and a 128-token answer inside the 262,144-token window. The accepted input counts differ by two tokens across configurations.
|Configuration|First token|Decode|
|:-|:-|:-|
|SGLang|35.4s|126.9 tok/s|
|FreeToken|80.4s|87.5 tok/s|
|llama.cpp + MTP|210.2s|52.6 tok/s|
|llama.cpp baseline|258.4s|20.3 tok/s|
Going from over four minutes to about 35 seconds changes how usable a large prompt feels.
The two columns measure different things: first-token time is the initial wait; decode is how quickly the answer arrives after that.
2. A short-prompt test misses the long-context behavior.
The separate prose sweep uses 2,048-token answers and three measured requests per input length.
https://preview.redd.it/x7dnlip1gcoh1.png?width=1575&format=png&auto=…
|Configuration|Decode at 2K input|Decode at 259,584 input|
|:-|:-|:-|
|SGLang|182.7 tok/s|191.5 tok/s|
|FreeToken|100.1 tok/s|94.8 tok/s|
|llama.cpp + MTP|126.8 tok/s|61.4 tok/s|
|llama.cpp baseline|101.9 tok/s|20.2 tok/s|
Prefill also changes the ranking. FreeToken starts behind llama.cpp at 2K input: 1,525 vs 1,869 tok/s. At 128K it reaches 3,231 vs 1,362 tok/s, about 2.4x faster.
3. MTP helps llama.cpp, but it does not remove the long-prompt wait.
On real coding prompts, comparing the same fork build with the draft head off and on:
https://preview.redd.it/cbj78fx5gcoh1.png?width=1425&format=png&auto=…
|Input|MTP off|MTP on|Decode gain|
|:-|:-|:-|:-|
|8,192 tokens|94.9 tok/s|155.1 tok/s|1.63x|
|32,000 tokens|83.1 tok/s|140.4 tok/s|1.69x|
The draft head is roughly 2.6GB.
At the full window, the tested MTP configuration reached 52.6 tok/s, versus 20.3 tok/s for the baseline configuration. That is a 2.59x gap, but the full-window comparison also involves a different build. The matched-build coding tests above isolate the draft-head change more cleanly.
I would not attribute the 258s → 210s first-token improvement to MTP alone.
4. I checked accuracy as well as speed.
https://preview.redd.it/8bb55qjegcoh1.png?width=1725&format=png&auto=…
|Stack|GSM8K|MATH-500|
|:-|:-|:-|
|llama.cpp baseline|95.60%|92.60%|
|SGLang|95.22%|93.00%|
|FreeToken|95.75%|92.20%|
The llama.cpp MTP arm scored 95.75% on GSM8K.
The tests used 1,319 GSM8K problems and 500 MATH-500 problems. The paired comparisons did not detect statistically significant differences.
That does not prove the stacks have identical quality. These are two short math benchmarks, with no full-precision reference on this machine.
5. Starting the model is a separate benchmark.
https://preview.redd.it/hw5nqk4dgcoh1.png?width=1425&format=png&auto=…
Median time from starting the container to receiving the first answer:
- llama.cpp: 16s
- SGLang: 108s
- FreeToken: 126s
FreeToken returned HTTP 200 from /health after about 3.3s, but took about 82s to reach serving readiness, followed by roughly 44s for its first generation.
That first request includes compilation work. Measuring only the health endpoint would give a very misleading startup result.
6. Loading modes barely changed speed with the experts on the GPU.
I compared none, mmap, mlock, mmap+mlock and dio on the same llama.cpp image, with the same tensor placement and real coding prompts.
- At 8K input, prefill ranged from 2,036 to 2,124 tok/s — a 4.3% spread.
- At 32K, it ranged from 1,946 to 1,956 tok/s — about 0.5%.
- No arm ran out of memory or restarted.
My earlier 1.87x RAM-resident loading gain used a different placement, with 23 expert layers computed on the CPU. In this test, all experts stayed on the GPU.
Loading mode can matter when the CPU computes the experts. It made little difference in this configuration.
https://preview.redd.it/sas55spwhcoh1.png?width=1575&format=png&auto=…
7. MTP became slower when experts were offloaded to the CPU.
https://preview.redd.it/ib6wnkk7icoh1.png?width=1650&format=png&auto=…
The MTP gains above do not apply to every memory budget.
I repeated the test with smaller usable VRAM pools on the same RTX PRO 6000, using 2,048-token coding prompts and 256-token answers. Both arms used the same fork build.
|Usable VRAM|Expert layers on CPU|MTP off|MTP on, head on GPU|
|:-|:-|:-|:-|
|16 GiB|45|33.0 tok/s|9.5 tok/s|
|24 GiB|42|34.7 tok/s|10.2 tok/s|
|32 GiB|36|38.1 tok/s|11.9 tok/s|
|48 GiB|23|48.3 tok/s|18.2 tok/s|
|96 GiB|0|99.8 tok/s|160.2 tok/s|
At the full 96 GiB budget, MTP gave 1.61x faster decode. At 24 GiB, it made decode about 3.4x slower.
Moving the draft head to the CPU did not fix the 24 GiB result: 9.6 tok/s, versus 34.7 tok/s with MTP off.
In these tests, MTP helped only when all experts stayed on the GPU. Verifying drafted tokens adds work, and CPU expert execution can outweigh the benefit.
These are VRAM-capacity limits on one Blackwell card, not measurements of actual smaller GPUs. Their bandwidth and compute performance will differ. The lookup table used the build’s default lazy-read mode in both arms.
8. Finishing sooner also reduced estimated GPU energy per request.
For the full-window test with a 128-token answer, I estimated GPU energy as median power while busy × request duration.
https://preview.redd.it/mpefmbggicoh1.png?width=1425&format=png&auto=…
|Configuration|Median GPU power|Approximate GPU energy|
|:-|:-|:-|
|SGLang|358 W|13 kJ|
|FreeToken|404 W|33 kJ|
|llama.cpp + MTP|489 W|104 kJ|
|llama.cpp baseline|440 W|116 kJ|
The fastest configuration used roughly one-ninth the estimated GPU energy of the baseline for this request.
The main difference was how long the GPU remained busy: about 36 seconds for SGLang versus 265 seconds for llama.cpp, including generation.
This is an estimate using median power, not integrated energy or a wall-socket measurement. It excludes CPU, RAM and SSD power. No thermal throttling was recorded in these runs.For the full-window test with a 128-token answer, I estimated GPU energy as median power while busy × request duration.Configuration Median GPU power
The fastest configuration used roughly one-ninth the estimated GPU energy of the baseline for this request.The main difference was how long the GPU remained busy: about 36 seconds for SGLang versus 265 seconds for llama.cpp, including generation.This is an estimate using median power, not integrated energy or a wall-socket measurement. It excludes CPU, RAM and SSD power. No thermal throttling was recorded in these runs.
Resources
I made a full video covering the memory placement, engine setup, flags and these results:
Full video: https://youtu.be/RlsxXB5q-cA**
GitHub — report, scripts, configurations, raw results and charts
The new report is engine\_benchmark\_report.html.
PS: AI was abused while making edits.
Has anybody tested the same model across these engines on a different GPU or memory setup?
I am especially interested in whether FreeToken stays this flat at long context, and how much MTP helps when some experts are offloaded to the CPU maybe on the other models.
And maybe you found more efficient methods to run it too,
