Optimizations Claude did for Qwen 3.8 27B and Qwen Flash Next on dual and quad 7900xtx

​

They're all fixes in rocm and llama server. Let me know if this interests someone. I'll push it to GitHub and share my configs. I have a Lenovo p620 running 4 7900xtx on gen 4.0 x16 slots.

One line summary of each fix:

\- Async mirrored input uploads: \~30 per-token inputs staged in a pinned ring on per-GPU streams instead of synchronous round trips (Flash-Next decode 24.6 -> 36.3 t/s).

\- Per-device dispatch threads: each GPU's kernels and collectives launched by its own host thread instead of one thread for all four.

\- One-shot PCIe P2P AllReduce: GPUs write slices straight into peers' memory for small tensors, replacing RCCL (\~57 us -> \~9 us per allreduce).

\- Fewer kernels in MTP decode: fused same-shape copies and leaner conv-state rollback (88.8 -> 92.7 t/s).

\- mmvq small-K row packing (RDNA3): short-K projections no longer leave most of the block idle (22.7 -> 25.3 t/s).

\- Wide-K mmvq for multi-token batches: K split over 8 warps for MTP verify on 10240x320 projections (\~92.6 -> \~96 t/s).

\- MoE vector kernel up to 8 tokens: 5-token MTP verify batches stay on the fast vector path instead of MMQ (80.9 -> 86.7 t/s).

\- Small-K multi-token MoE kernel: several rows per warp for short expert down-projection slices (94.2 -> 95.2 t/s).

\- Wide mmvf blocks: 512/1024-thread blocks for tiny long-K F32 matrices (40.6 -> 41.2 t/s).

\- Q8\_1 activation registry: each activation quantized once and shared by all matmuls that read it (\~+2 t/s).

\- Fused hyper-connection chains: scale/sigmoid/scale/hc\_post and scale/silu each in one kernel, \~380 fewer kernels per token (38.5 -> 40.6 t/s).

\- Thin-F32 prefill kernel: <=16-row F32 matmuls off generic SGEMM (401 -> 21 us; pp4096 1602 -> 1836 t/s).

\- Compact MoE tile list: expert matmul launches only (expert, token-tile) pairs with work instead of a 96%-empty grid (down 928 -> 405 us, gate/up 516 -> 360 us).

\- Multi-warp MoE routing helper: 16 warps per expert sort tokens in two passes (99 -> 25 us per call).

\- Q4\_K expert tile shape: 32-row tiles for 160-row expert slices (360 -> 337 us).

\- Stream-k for few-tile Q8\_0 matmuls: 10240->320 projections spread over all CUs instead of 12 workgroups (284 -> 115 us).

\- No 64-bit div/mod in hyper-connection kernels: 3-D grid instead of emulated integer division per element (230 -> 79 us; pp2048 1593 -> 1868 t/s with stream-k).

\- Split-K router GEMM: 512x512 F32 router GEMM as 8 K-chunks plus a sum (158 -> 69 us).

\- MTP re-reserve fix: graph re-reserved when MTP outputs turn on, ending a full GPU realloc+sync per prefill chunk (88.5K prefill 1042 -> 1130 t/s).

\- MTP draft prompt window: draft head prefills only the last 2048 prompt tokens (prefill 1130 -> 1249 t/s, decode at depth 46.6 -> 64.9 t/s).

\- Draft ubatch cap: draft compute buffer 457 -> 247 MiB, fixing a GPU0 out-of-memory crash at 96K context.

\- Gathered sparse attention (QSA): decode attends only to the \~2K selected tokens instead of masking the whole cache (decode at 48K 67.9 -> 80.6 t/s).

\- Per-layer embedding table in RAM (--lazy-mode off): 26.8 GB hashed embedding table kept resident instead of read from disk every pass (lookup 1.0-1.6 -> 0.1-0.3 ms, \~3-4% decode).

\- Meta backend subgraph fix: per-device subgraphs sized for the largest graph, fixing a segfault when graph shapes change between calls.

\- Net result, Qwen3.8-27B Q8 on 4 GPUs: code decode 54-61 -> 96-110 t/s, 51K prefill 1454 -> 1816 t/s, decode at 51K depth 57 -> 77 t/s.