I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been watching other redditors post their custom configurations, and I wanted to share what I tested to optimize for a single RTX 5090.
I was on llama.cpp (Q5\\\_K\\\_M GGUF, q5\\\_1 KV, 262K context, MTP), but it's limited to parallel=1 and I wanted concurrent serving. So I tested vLLM and NInfer as NVFP4 replacements and did a proper quality evaluation instead of just vibes.
Hardware
\- RTX 5090 32GB (eGPU, OCuLink Gen4 x4) (yes, it's in an eGPU dock 😂 but that only affects model loading)
\- Ryzen 7 7840HS, 32 GB DDR5
\- Ubuntu 26.04, nvidia driver 610.43.02 (open)
## Engine configs
| | **\*\*llama.cpp\*\* | \*\*vLLM\*\* | \*\*NInfer\*\*** |
|---|---|---|---|
| Quant | Q5\\\_K\\\_M GGUF | NVFP4 | NVFP4 |
| KV cache | q8\\\_0 | FP8 | FP8 |
| Context | 196K | 262K | 240K |
| MTP | On (gate failed) | None | MTP3 (76% acceptance) |
| Concurrency | parallel=1 | Continuous batch | x2 lanes |
| VRAM | 31.6 GB | 29.6 GB | 30.5 GB |
How the eval worked
I built a custom harness with 6 tiers, 50 items each, all from my actual production workload (not generic benchmarks):
- **\*\*Relevance classification\*\*** \- is this industry relevant? (binary, 50 labeled deals)
- **\*\*Needle retrieval\*\*** \- planted facts in real industry podcast/video transcripts at 64K/128K/192K/240K context
- **\*\*Multi-transcript QA\*\*** \- questions across 3-4 concatenated diarized transcripts, 120K+ tokens, including unanswerable controls
- **\*\*Reasoning with thinking\*\*** \- numeric/logic problems, thinking mode on, greedy pass@1
- **\*\*Structured extraction\*\*** \- custom extraction prompt, json\\\_mode (skipped on NInfer, it doesn't support json\\\_mode)
- **\*\*Tool replay\*\*** \- replayed recorded agent episodes against each engine (diagnostic only, all engines fail this one)
Everything paired across engines: same prompts, same seeds (42), same gold labels. No cache\\\_prompt. Statistical comparison uses paired cluster-bootstrap CIs with pre-registered non-inferiority margins.
And when I do development with Claude Code, I often leverage multi-model consultations for design, planning, and code review. I did a 4-model review panel (Codex/GPT-5, DeepSeek V4 Pro, Grok 4.5, Kimi K3) audit the methodology mid-campaign. They found 10 issues, including 4 mislabeled gold items where the models were actually right and my labels were wrong. Fixed everything and reran.
Quality results
| **\*\*Tier\*\* | \*\*llama.cpp\*\* | \*\*vLLM\*\* | \*\*NInfer\*\*** |
|---|---|---|---|
| Relevance | 86.0% | 84.0% | 86.0% |
| Needle (conditional) | 100% (29/29) | 100% (41/41) | 100% (41/41) |
| Transcript QA | 82.0% | 78.0% | 88.0% |
| Reasoning | 100% | 100% | 98.0% |
| Extraction | F1 0.300 | F1 0.350 | skipped |
| Tool replay | 0% | all errors | 0% |
Needle counts differ because llama.cpp's 196K context can't fit the 192K items (need room for max\\\_tokens + headroom). NInfer and vLLM both handle 192K fine. All engines score 100% on every needle they can fit.
Statistical comparison (NInfer vs llama.cpp, bootstrap):
\- Needle: delta = -0.29 (NInfer better, p=0.0006) - this is entirely from context capacity, not retrieval quality
\- Transcript QA: delta = -0.03, p=0.69 - no difference
\- Reasoning: delta = +0.02, p=0.72 - no difference
\- Relevance: McNemar p=1.0 - identical
\- Tool replay: delta = 0.0 - both fail equally
**\*\*Takeaway: quality is statistically indistinguishable across all engines.\*\***
Speed results (perf probe, server-side timings)
| **\*\*Metric\*\* | \*\*llama.cpp\*\* | \*\*NInfer\*\* | \*\*Speedup\*\*** |
|---|---|---|---|
| **\*\*Decode 1K\*\*** | 114 tok/s | 158 tok/s | 1.4x |
| **\*\*Decode 32K\*\*** | 109 tok/s | 213 tok/s | 2.0x |
| **\*\*Decode 128K\*\* | 72 tok/s | 202 tok/s | \*\*2.8x\*\*** |
| Prefill 1K | 1,545 tok/s | 7,265 tok/s | **\*\*4.7x\*\*** |
| Prefill 32K | 2,155 tok/s | 6,892 tok/s | 3.2x |
| Prefill 128K | 1,528 tok/s | 3,904 tok/s | 2.6x |
| TTFT 1K | 670 ms | 138 ms | 4.9x |
| TTFT 32K | 15.2 s | 4.8 s | 3.2x |
| TTFT 128K | 85.9 s | 33.6 s | 2.6x |
vLLM speed excluded from the table because the perf probe used wall-clock timing (includes prefill + scheduling + decode) instead of server-side timings, so the numbers aren't comparable. From community reports and my own task-level measurements, vLLM does about 70 tok/s decode at short context, which actually matches NInfer's raw step rate (\~66 tok/s). The speed difference is entirely MTP3 speculative decoding.
What I learned
**\*\*NInfer's speed advantage is all MTP.\*\*** The raw NVFP4 kernel speed is about the same between NInfer and vLLM (\~66-70 tok/s). NInfer's MTP3 speculation with 76% acceptance gets you to 158-213 tok/s. If vLLM or llama.cpp had working MTP on NVFP4, the gap would mostly close.
**\*\*The decode speedup grows with context.\*\*** At 1K context it's 1.4x. At 128K it's 2.8x. MTP acceptance stays high even at long context while llama.cpp's dense decode gets slower as context grows.
**\*\*NInfer's tokenizer endpoint is great.\*\*** It exposes \/v1/messages/count\_tokens\ (Anthropic Messages format) which gives exact token counts. No more \len(text)//3\ heuristics.
**\*\*NInfer does NOT support json\\\_mode (as far as I can tell).\*\*** \response\_format: json\_object\ returns 400. If you need structured JSON output, you'll need to route those calls elsewhere or use prompt-based enforcement.
**\*\*Don't trust vibes for quality.\*\*** I went in expecting NVFP4 might lose a few points vs Q5\\\_K\\\_M. It didn't. Not on any tier. The biggest delta across 250+ items was 2 percentage points, well within noise at n=50 (SE \~5.6pp).
Verdict
NInfer NVFP4 replaces llama.cpp as my production engine. Same quality, 1.4-2.8x faster decode, 2.6-4.7x faster prefill, concurrent serving. The only gap is json\\\_mode.
I put together a detailed poster with all the charts and methodology details: \full results poster\
Setup if you want to try it:
\\\`
\# NInfer (from source)
git clone https://github.com/Neroued/ninfer && cd ninfer
mkdir build && cd build
cmake .. -DCMAKE\_BUILD\_TYPE=Release -GNinja && ninja
\# Model (HuggingFace)
\# https://huggingface.co/neroued/Qwen3.8-27B-nvfp4-NInfer (20 GB)
\# Run
./ninfer-serve /path/to/model.ninfer \\
\--model-id qwen3.8-27b \\
\--host 0.0.0.0 --port 8080 \\
\--max-context 240000 --kv-capacity 240000 \\
\--max-concurrency 2 --kv-dtype fp8 \\
\--spec mtp --draft-tokens 3 \\
\--vision --preserve-thinking
\\\`
