I turned my gaming PC into a inference machine and got 2x to 9x over default llama.cpp on an 8 GB card
Everyone keeps saying you need expensive dedicated hardware for local agents. I have an RTX 4060 Ti with 8 GB and 64 GB of system RAM, and I wanted to see how far a normal gaming PC gets if you stop running defaults.
So I let Claude (Opus 5.5) go through the whole setup, change one thing at a time and measure. Same card, same models, only the config changed:
|Model|Quant|Context|Download defaults|Tuned (Windows)|Tuned (headless Linux)|
|:-|:-|:-|:-|:-|:-|
|Qwen3.6-35B-A3B|Q4\_K\_XL|131k|\~25 tok/s|39-45 tok/s|52-65 tok/s|
|Qwen3.8-Flash-Next 125B|iQ4\_XS|131k|\~4 tok/s|9-10 tok/s|17-19 tok/s|
|Ternary Bonsai 27B|PTQ1\_0|64k|\~4 tok/s|36 tok/s|36 tok/s|
Bonsai is the odd one out: it fits fully in VRAM, so there's nothing to offload and no defaults to beat. It's just the fast option for small, well scoped tasks.
What actually moved the needle:
- Experts in system RAM, everything else in VRAM. Layer-wise offload is far worse for MoE.
- Dense models are bad, couldn't optimize Qwen-3.8 27B over 6 tok/s, Flash-Next is better anyways.
- Take the display off the GPU. A desktop eats 0.5-1.2 GB of VRAM plus GPU time, and moving it to the iGPU was worth 20-30%.
- Native Linux over Windows (WSL2): another 33-38% on the same hardware.
- llama.cpp pinned per model family. The wrong tree made VRAM thrash.
- KV cache quant and MTP tuned per profile.
None of this needs expensive hardware. A consumer GPU plus a machine that does nothing but inference gets you most of the way, and the models now run comfortably below their listed system requirements. Every non-default setting in the repo is there because something failed on real hardware first.
I also tried an RX 570 8 GB over Vulkan. If you have another 8 GB card, I'd like to see your numbers.
Repo, one install script (Linux or WSL2): https://github.com/voxlo-dev/qwen-agent-8gb