If you benchmark llama.cpp on AMD with the official ROCm builds, check that you're actually on the GPU

I spent the last few weeks comparing llama.cpp output across backends on a small multi-vendor GPU fleet, and the single most useful thing I learned wasn't about inference quality — it was that on four AMD hosts, the official ROCm prebuilt (b11327) never touched the GPU at all. The server started fine, /health was green, and everything looked normal. It was running on the CPU the whole time.

The reason is dull but nasty: the prebuilt's HIP backend wants libamdhip64.so.7 / libhipblas.so.3 / librocblas.so.5, and a stock Ubuntu host with ROCm 6.3.0 has .so.6 / .so.2 / .so.4. The backend library fails to load, and llama-server quietly serves from the CPU. No error on the console. Even -ngl 999 doesn't change anything. Details here: https://github.com/ggml-org/llama.cpp/issues/26964#issuecomment-6024889471

If you benchmark tokens/sec you'll notice eventually. But if you compare output quality — perplexity, eval scores, side-by-side generations — nothing gives it away. On my boxes the "ROCm" perplexity matched the CPU perplexity to the last digit, every time, because it was the CPU doing the work.

The cheap check that catches this: run the same prompt set through the CPU build on the same box and compare per-prompt wall time. Ratio around 1.0 = you're on the CPU. Well under 0.5 = the GPU is actually working. I now run this check before trusting any benchmark number off a new box.

The same sweep also produced a small cross-backend conformance dataset (same GGUF, same prompts, temperature 0, per-token top-5 logprobs on every backend) — the short version: identical stacks are bit-for-bit deterministic across machines, and anything that changes the numerical stack (different backend, different build, even a different host CPU) starts flipping near-tie token choices, with the effect getting much worse the heavier your quantization is (F16 mostly agrees, Q4 mostly doesn't). Happy to share the data if anyone wants it.