Qwen 27B and Flash Next on 2× RX 7900 XT: am I missing something?
I've tested quite a few settings and collected the results in a spreadsheet. I keep seeing people reporting 100+ tokens/s with 16 GB VRAM, or generally much higher speeds with less VRAM. I'm trying to understand whether my setup is underperforming or I'm comparing completely different things.
My PC:
- Ryzen 7 5800X3D, 128 GB DDR4 at 3600 MT/s
- 2× RX 7900 XT, 20 GB each, XFX and PowerColor
- Gigabyte B550 EAGLE WIFI6
- XFX on PCIe 4.0 x16; PowerColor on a chipset-connected PCIe 3.0 x1 slot
- Windows 11, AMD driver 32.0.31041.1004
The cards have reduced clock settings: XFX 1700 MHz core, PowerColor 1800 MHz, both 2500 MHz memory and −10% power limit.
Here are the main single-response results:
| Setup | Generation TPS | Including prompt processing |
|---|---:|---:|
| Qwen3.8-27B IQ4_XS, direct ROCm + MTP3 | 46.4 | 43.1 |
| Qwen3.8-27B IQ2_XXS, direct ROCm + MTP3 | 66.5 | 59.9 |
| Flash Next UD-IQ4_XS, one GPU, warm ROCm run | 8.6 | 5.7 |
| Flash Next UD-IQ4_XS, two GPUs, Vulkan | 5.5–6.5 | 3.3–5.7 |
The 27B tests used roughly 700 input tokens, 8k context and 1024 output tokens. IQ4 had three runs; IQ2 is the median of nine prompts. Settings were ROCm 2.46.0, Flash Attention, f16 KV, MTP3 and batch/microbatch 2048/512, with thinking off.
Two separate IQ4 copies reached 103.5 TPS combined, but that required 16 concurrent requests. I haven't reached 100 TPS for one response. Splitting one model across both cards was slower. Tensor split initially produced broken text; --no-mmap fixed that.
Flash Next is unsloth UD-IQ4_XS, around 93.7 GB. The single-GPU profile used ROCm 2.49.0, --n-cpu-moe 42, f16 KV and 8k context. The dual-GPU profile used Vulkan 2.51.0, tensor split 1:1, --n-cpu-moe 28, q8 KV and 256k context. Both used eight threads, PLE on CPU and MTP off.
The Flash measurements were individual short runs. The configured 256k window was mostly empty, and the different profiles weren't a controlled single-versus-dual comparison.
What would you check first: CPU/RAM offloading, the x1 connection, or backend settings? If you're getting 100+ TPS on 16 GB or less, could you share your exact model/quant, hardware, backend, MTP settings and actual context length? Also whether that's one response or combined throughput.
Update Oct 6: Strata 0.1.39 works on RDNA3 with Windows/HIP. Same Flash UD-IQ4_XS, one 7900 XT, 8k, int8 KV, prefill512, 8 workers, thinking off/greedy. 24 GiB expert RAM + ~7.7 GiB auto GPU expert cache. Three 128-token text runs per setting:
| Setting | Decode TPS | Including prompt |
|---|---:|---:|
| MTP2 | 11.0 | 8.6 |
| MTP4 (tested at start/end) | 10.6–10.8 | 8.4–8.7 |
| MTP8 | 10.0 | 8.2 |
| MTP4, min-p 0.2 | 9.7 | 7.9 |
| MTP2, 32 GiB expert RAM | 13.6 | 10.5 |
MTP8 helped counting but slowed the text prompt. With 512 output tokens and 24 GiB RAM, MTP2 gave 12.7 t/s vs 11.2 for MTP4 (11.7 vs 10.6 including prompt), three runs each. More expert RAM helped most in the short tests. Sequential runs/cache conditions vary; this isn't a quality comparison or a controlled comparison with the old backend. Some small projections are rounded to BF16 by the pack. Still no 100 t/s for one answer. Staying with IQ4; not testing Q2. Linux/custom gfx1100 builds are still untested here.