Probably I'm doing something wrong using PR#29887 (Add a GPU cache for MoE experts kept in host memory)

I have 8GB VRAM(4060) + 32GB RAM(DDR5 5600). Tried this feature with b11491. Experimented with both cmoe & fit. Not getting expected t/s.

Please fix this for me.

And others, what are you getting for your limited VRAM? Share your t/s stats.

llama-server -m E:\LLM\models\MOE\Qwen3.6-35B-A3B-IQ4_XS-00001-of-00002.gguf -cmoe
3.25.566.603 I slot print_timing: id 3 | task 0 | prompt eval time = 7546.59 ms / 342 tokens ( 22.07 ms per token, 45.32 tokens per second)
3.25.566.621 I slot print_timing: id 3 | task 0 | eval time = 74878.16 ms / 1750 tokens ( 42.81 ms per token, 23.36 tokens per second)
3.25.566.624 I slot print_timing: id 3 | task 0 | total time = 82424.75 ms / 2092 tokens

llama-server -m E:\LLM\models\MOE\Qwen3.6-35B-A3B-IQ4_XS-00001-of-00002.gguf -cmoe --moe-cache-mib 1536
2.07.904.789 I slot print_timing: id 3 | task 0 | prompt eval time = 11915.27 ms / 342 tokens ( 34.84 ms per token, 28.70 tokens per second)
2.07.904.798 I slot print_timing: id 3 | task 0 | eval time = 86812.44 ms / 1305 tokens ( 66.57 ms per token, 15.02 tokens per second)
2.07.904.800 I slot print_timing: id 3 | task 0 | total time = 98727.70 ms / 1647 tokens

llama-server -m E:\LLM\models\MOE\Qwen3.6-35B-A3B-IQ4_XS-00001-of-00002.gguf -cmoe --moe-cache-mib 2048
2.45.980.222 I slot print_timing: id 3 | task 0 | prompt eval time = 15102.25 ms / 342 tokens ( 44.16 ms per token, 22.65 tokens per second)
2.45.980.408 I slot print_timing: id 3 | task 0 | eval time = 124490.76 ms / 1669 tokens ( 74.63 ms per token, 13.40 tokens per second)
2.45.980.412 I slot print_timing: id 3 | task 0 | total time = 139593.01 ms / 2011 tokens

llama-server -m E:\LLM\models\MOE\Qwen3.6-35B-A3B-IQ4_XS-00001-of-00002.gguf -cmoe --moe-cache-mib 4096
1.45.860.593 I slot print_timing: id 3 | task 0 | prompt eval time = 19690.18 ms / 342 tokens ( 57.57 ms per token, 17.37 tokens per second)
1.45.860.818 I slot print_timing: id 3 | task 0 | eval time = 58251.17 ms / 1143 tokens ( 51.01 ms per token, 19.60 tokens per second)
1.45.860.821 I slot print_timing: id 3 | task 0 | total time = 77941.35 ms / 1485 tokens

llama-server -m E:\LLM\models\MOE\Qwen3.6-35B-A3B-IQ4_XS-00001-of-00002.gguf -cmoe --moe-cache-mib 8192
6.12.049.848 I slot print_timing: id 3 | task 0 | prompt eval time = 27804.55 ms / 342 tokens ( 81.30 ms per token, 12.30 tokens per second)
6.12.049.864 I slot print_timing: id 3 | task 0 | eval time = 311652.61 ms / 3753 tokens ( 83.06 ms per token, 12.04 tokens per second)
6.12.049.866 I slot print_timing: id 3 | task 0 | total time = 339457.17 ms / 4095 tokens

llama-server -m E:\LLM\models\MOE\Qwen3.6-35B-A3B-IQ4_XS-00001-of-00002.gguf -c 131072 -cmoe --moe-cache-mib 2048
1.28.103.097 I slot print_timing: id 3 | task 0 | prompt eval time = 13046.26 ms / 341 tokens ( 38.26 ms per token, 26.14 tokens per second)
1.28.103.169 I slot print_timing: id 3 | task 0 | eval time = 52369.09 ms / 802 tokens ( 65.38 ms per token, 15.30 tokens per second)
1.28.103.171 I slot print_timing: id 3 | task 0 | total time = 65415.35 ms / 1143 tokens

Above ones with -cmoe while below ones without -cmoe & fit is on by default

llama-server -m E:\LLM\models\MOE\Qwen3.6-35B-A3B-IQ4_XS-00001-of-00002.gguf --moe-cache-mib 2048
1.11.969.119 I slot print_timing: id 3 | task 0 | prompt eval time = 5196.27 ms / 341 tokens ( 15.24 ms per token, 65.62 tokens per second)
1.11.969.129 I slot print_timing: id 3 | task 0 | eval time = 38764.96 ms / 939 tokens ( 41.33 ms per token, 24.20 tokens per second)
1.11.969.131 I slot print_timing: id 3 | task 0 | total time = 43961.23 ms / 1280 tokens

llama-server -m E:\LLM\models\MOE\Qwen3.6-35B-A3B-IQ4_XS-00001-of-00002.gguf -b 2048 -ub 2048 --moe-cache-mib 2048
1.11.944.661 I slot print_timing: id 3 | task 0 | prompt eval time = 5403.22 ms / 341 tokens ( 15.85 ms per token, 63.11 tokens per second)
1.11.944.672 I slot print_timing: id 3 | task 0 | eval time = 39398.70 ms / 971 tokens ( 40.62 ms per token, 24.62 tokens per second)
1.11.944.674 I slot print_timing: id 3 | task 0 | total time = 44801.92 ms / 1312 tokens

llama-server -m E:\LLM\models\MOE\Qwen3.6-35B-A3B-IQ4_XS-00001-of-00002.gguf --moe-cache-mib 4096
1.05.634.810 I slot print_timing: id 3 | task 0 | prompt eval time = 6284.73 ms / 341 tokens ( 18.43 ms per token, 54.26 tokens per second)
1.05.634.821 I slot print_timing: id 3 | task 0 | eval time = 32617.75 ms / 533 tokens ( 61.31 ms per token, 16.31 tokens per second)
1.05.634.822 I slot print_timing: id 3 | task 0 | total time = 38902.48 ms / 874 tokens

llama-server -m E:\LLM\models\MOE\Qwen3.6-35B-A3B-IQ4_XS-00001-of-00002.gguf -c 131072 --moe-cache-mib 2048
1.18.552.620 I slot print_timing: id 3 | task 0 | prompt eval time = 6213.00 ms / 341 tokens ( 18.22 ms per token, 54.88 tokens per second)
1.18.552.627 I slot print_timing: id 3 | task 0 | eval time = 47349.21 ms / 859 tokens ( 55.19 ms per token, 18.12 tokens per second)
1.18.552.629 I slot print_timing: id 3 | task 0 | total time = 53562.22 ms / 1200 tokens

llama-server -m E:\LLM\models\MOE\Qwen3.6-35B-A3B-IQ4_XS-00001-of-00002.gguf -c 262144 --moe-cache-mib 2048
2.09.088.373 I slot print_timing: id 3 | task 0 | prompt eval time = 14680.06 ms / 341 tokens ( 43.05 ms per token, 23.23 tokens per second)
2.09.088.387 I slot print_timing: id 3 | task 0 | eval time = 88642.28 ms / 997 tokens ( 89.00 ms per token, 11.24 tokens per second)
2.09.088.389 I slot print_timing: id 3 | task 0 | total time = 103322.34 ms / 1338 tokens

Below one is from past without this PR. 20 t/s for 128K context is not bad with 8GB VRAM + RAM.

llama-server -m E:\LLM\models\MOE\Qwen3.6-35B-A3B-IQ4_XS-00001-of-00002.gguf -fa 1 -ctk q8_0 -ctv q8_0 -kvu --cache-ram 24576 --cache-idle-slots -np 1 -cb -fit on -fitt 512 -t 8 --mlock --no-mmap --no-warmup -ctxcp 64 --no-mmproj -c 131072
4.39.110.891 I slot print_timing: id 0 | task 0 | prompt eval time = 1717.32 ms / 35 tokens ( 49.07 ms per token, 20.38 tokens per second)
4.39.110.903 I slot print_timing: id 0 | task 0 | eval time = 178110.45 ms / 3448 tokens ( 51.66 ms per token, 19.36 tokens per second)
4.39.110.905 I slot print_timing: id 0 | task 0 | total time = 179827.77 ms / 3483 tokens

Tried Q2 of Qwen3.8-Flash-Next just for fun.

llama-server -m E:\LLM\models\MOE\Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf -ctk q8_0 -ctv q8_0 --load-mode none
5.19.601.200 I slot print_timing: id 3 | task 0 | prompt eval time = 54472.12 ms / 379 tokens ( 143.73 ms per token, 6.96 tokens per second)
5.19.601.214 I slot print_timing: id 3 | task 0 | eval time = 186902.08 ms / 1500 tokens ( 124.68 ms per token, 8.02 tokens per second)
5.19.601.216 I slot print_timing: id 3 | task 0 | total time = 241374.19 ms / 1879 tokens