← r/LocalLLaMA
▲
12
+1
8👁
r/LocalLLaMA · u/nirurin · 15d ago

Qwen3.8-Flash-Next, on 5090+64gb, with Llama.cpp - Seems to not use ram?

I was actually pretty happy with my Qwen3.8-27b setup, and I'd been tinkering with Ninfer to have a version that was "fast but maybe a bit stupid" and the speed was nice to have as a backup. But I was curious how the Flash-Next version might work, after I learned it didn't need to all fit in VRAM to work. I picked up the Atomic quant (let me know if there is a better one I should use, this one seemed good from what I could find). I used the build setup below. It can still be tweaked some more, as I still am only using about 27gb of my vram. > ./build/bin/llama-server \\ \--model "/mnt/SPCC-2TB/Projects/AI-APPS/LLM-Models/Qwen3.8-Flash-Next-Atomic/Qwen3.8-Flash-Next-AD-4.27bpw-Q4\_K\_M-M64 \-00001-of-00033.gguf" \\ \--no-mmproj \\ \--load-mode mmap \\ \--lazy-mode on \\ \--fit off \\ \--gpu-layers all \\ \--n-cpu-moe 32 \\ \--ctx-size 64768 \\ \--flash-attn on \\ \--jinja \\ \--parallel 1 The odd thing I noticed though - I know that some parts of this are meant to run from the SSD for the sake of saving vram space etc. Fine. But I kinda expected that some of it at least would get buffered into system ram, as running from ram would be a whole lot more efficient than running from my NVME drive. But this run gets me the following results: 40tok/s decode. 50tok/s prompt processing (It's a short prompt so probably not accurate) 27gb of vram used 8gb of system ram used... So... I mean, am I just wrong and this is normal? The speed doesn't seem as bad as I expected (I thought I was going to get more like 10tok/s at best) but it seems like I might be missing a trick somewhere?

13 0 12 10/3 06:32 10/3 15:39 UTC
scorecomments8 sightings
first seen 2026-10-03 06:32 UTClast seen 2026-10-03 15:39 UTCscore then 11score now 12gained +1sightings 8
open on reddit ↗ 💬 31