Qwen3.8 Flash Next llama.cpp config tuning
Hola all.
Do you guys mind sharing your LLama.cpp config and system setup details for Qwen3.8 Flash Next?
Model's quite big and tryining many combinations of llama.cpp options takes lots of time, so looking for other people setup details. I've attached my current config at the bottom, so if anyone sees something that could be improved please shout.
My current best result:
\- PP within 130...200 tps (limited by cpu?)
\- TG within 14..22 tps (\~15tps on average)
Hardware:
\- Dual RTX 3090 (48GB VRAM)
\- 128GB DDR4
\- Some old Xeon 40 core
\- Proxmox VM, pcie passthrough, numa binding to a single phys cpu
Llama.cpp config:
llama-server --port ${PORT}
--model /nvme/gguf/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
--mmproj /nvme/gguf/mmproj-Qwen3.8-Flash-Next-F16.gguf
--load-mode none
--lazy-mode off
--parallel 1
--ctx-size 131072
--cache-type-k q8_0
--cache-type-v q8_0
--flash-attn on
--fit off
--temp 1.0
--min-p 0.0
--top-p 0.95
--top-k 20
--presence-penalty 0.0
--repeat-penalty 1.0
--batch-size 2048
--ubatch-size 512
--split-mode layer
-ts 26,10
-ngl 99
-ncmoe 26
--no-mmproj-offload
--override-tensor per_layer_token_embd=CPU
--chat-template-kwargs '{"reasoning_effort":"xhigh"}'
ngl, ncmoe, ts - manually adjusted to fit the model without crashing
\------------------------------------------
For the record, if you have +128GB RAM and dual RTX3090 setup try this:
https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE
\- Prompt processing jumped to anything between 300 to 600 tps (can do more!)
\- Token generation 40..50 tps
\- Stable work for hours with 256k context in agentic coding scenario
\- Feels like a frontier model at home, wow!