I'm using DS-V4-Flash-Vision with Q8\_K\_XL quantization locally as my everyday engine, and for some time now I've been doing a lot of comparisons with Qwen3.8-Flash-Next, also with Q8\_K\_XL quantization. It took me quite a while to get Q3.8FN to work reasonably well, and here are my observations. My hardware: 2x StrixHalo 128GB, USB-C 4 connector, Llama (RPC) as inteference engine.

1. DSV4FV is about 40% slower than Q38FN at the same quantization level when it comes to token generation alone.
2. DSV4FV completes tasks about twice as fast as Q38FN! This means that DSV4FV “hallucinates” less (I observe this based on the obstacles the models encounter along the way).
3. The Q38FN is unusable in “xhigh” mode. A simple task that the Q38FN completed in 25 minutes on “medium” mode, it failed to complete in \~3 hours on “xhigh” mode.
4. The same task that the Q38FN completed in 25 minutes (average), the DSV4FV completed in 12 minutes (fastest round) on “medium”.
5. The DSV4FV completed the same task on “max” in 37 minutes in first iteration, second took 44 minutes.
6. Qwen3.8 tends to overinterpret my instructions. If I don’t write them out in great detail and leave room for creative interpretation, it will take advantage of that. Perhaps this is where it gets bogged down in its own creativity. In what it does, I’ve noticed that Qwen clearly adds too much and struggles to flesh out the details.

In my opinion, DSV4FV is the better solution when working with professional code.

Just so there’s no misunderstanding - I was a huge fan of Qwen 3.6 27B and of course now I'm Qwen 3.8 27B big fan, which I’ve been using a lot and is great! In general i’m a huge fan of Qwen, but ever since I’ve had the hardware on which I can run DSV4FV, I’ve been using it, and I’m super happy with how good this model is.

I've been tinkering with local LLMs since the beginning of the year when I had an Intel Arc B580 and 32 GB of DDR5. Curiosity got the best of me and I bought the first R9700 about half a year ago, also because I wanted to upgrade my gaming graphics for 4k. As the 5090 was about 3 times as expensive, I had a "sweet spot", kind of. On the last prime days, I found a X870E mainboard for \~150 € below the standard price, and it got to my head that I can use an upgraded machine for gaming and local inference tinkering.

Anyways. Fast forward to this week, I now have the following setup

• Ryzen 7500F
• 64 GB DDR5 CL40 6400 MT/s
• Asus ProArt Creator X870E
• 2x R9700 32 GB, each running at PCIe 5.0 x8 (Gigagbyte)
• Currently running ubuntu on an old Samsung EVO 860 1 TB drive; this will become intersting for the ngram / PLE offload; I have Windows and the gaming related stuff on a gen4 NVMe, but will soon add another Gen 5 NVMe with decent random reads

The only issue that I can report so far is that one of the cards runs quite hot, so I will definitely implement power limiting to 210 W and some light undervolting. The other card runs 10-15 °C cooler.. Case is a purebase 501 with 4 fans, 2 intake in front, one back and top for output.

Now long story short I wanted to give some results of Qwen 3.8 27b FP8 and MXFP4, as well as Qwen 3.8 flash next after the first day tinkering with it. What I found super interesting is that the SATA SSD does not seem to be super terrible when using Qwen 3.8 flash next.

Considering the whole build costs \~4k €, or more than 1k less than a single RTX 5090 with 32 GB, I kinda like this setup price/performance wise. Next step is checking context degradation / KV quants. I am using local inference mostly for deep research, summarization, image creation, light coding and non-trivial data analysis

Cheers

Qwen3.8 benchmarks on 2× Radeon AI PRO R9700

Hardware: 2× AMD Radeon AI PRO R9700 32 GB, 61 GiB system RAM
Benchmark: BetterBench 0.2.2, corpus v1.0, single-stream, greedy decoding, 2 warm-ups + 10 measured runs per category, 8k benchmark context.

| Model | Weight format | Runtime | Server context | Max sequences | Speculative decoding | Weighted decode median | ITL 1% low | TTFT p50 | Prefill ~2k | Prefill ~4k | Prefill ~7k |
|---|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|
| Qwen3.8-27B | Quark AWQ MXFP4 | vLLM Radiance, TP2 | 131,072 | 1 | MTP, up to 8 tokens | 111.4 tok/s | 77.9 tok/s | 81 ms | 4,224 tok/s | 4,322 tok/s | 4,410 tok/s |
| Qwen3.8-27B | Native block FP8 | vLLM Radiance, TP2 | 16,384 | 8 | MTP, up to 8 tokens | 87.6 tok/s | 61.9 tok/s | 73 ms | 4,134 tok/s | 4,329 tok/s | 4,305 tok/s |
| Qwen3.8-Flash-Next | UD-IQ4_XS GGUF | R9V/vLLM, TP2, tiered expert offload | 131,072 | 1 | MTP, 2 tokens, FP8 draft | 35.4 tok/s | 27.3 tok/s | 290 ms | 1,727 tok/s | 1,986 tok/s | 1,925 tok/s |

• Qwen 3.8 27b in FP8 and AWQ MXFP4 served with vLLM Radiance
• Qwen 3.8 Flash next served with vLLM / R9V fork
• Decode metrics come from the 10-pass standard run.
• Prefill measurements use cold, nonce-prefixed prompts.
• Prompt-token medians for the prefill columns were 1,556, 3,024 and 5,226 tokens.
• No concurrency sweep was included in these results.
• I expect decode of Flash next to increase a bit when an NVMe is used, and, as I am writing this and checked, I found EXPO was not enabled........oh my god I swear I turned it on when I updated the bios yesterday

While we see for frontier models similar scores among famous coding benchmarks, across: DeepSWE, Terminal-Bench, LiveCodeBench, Code-Arena ELO. Here are in my opinion some next level benchmarks that really define deep intelligence, and complete capability in Software Engineering :

1. Program-Bench

Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior (without access to decompilers or internet). Link: https://programbench.com/

• GPT-6 Astra: 5.5%
• Fable 5.1: 7%
• Kimi K3: 2%
• Qwen3.8 27b: 0%
• GPT 5.6 Sol: 1.5%
• GLM 5.3: 1.5%
• GPT 5.6 Luna: 0%

2. SRE-Bench

Can AI agents work out what a real-world binary does without its source code?

Link: https://www.vals.ai/benchmarks/srebench

Sure nobody is reading assembly code in daily work, it is hard. The ability to understand a compiled program is insane ability.

• GPT-6 Astra: 88%
• GPT-5.6 Sol: 55.9%
• Claude Opus 5 (max): 12.5%

3. Code Migration

Can language models reimplement working programs in another language?

Link: https://www.vals.ai/benchmarks/code-migration

• GPT-6 Astra: 67.7%
• Fable 5.1: 54.6%
• GLM 5.3: 44.2%
• GLM 5.3 Flash: 20.5%
• Qwen3.8 27b: 14.2%

EDIT: edited text format

You need Blender 5.x with MCP extension.

I use pi so I also need: pi install npm:pi-mcp-adapter

and then:

$ cat .mcp.json
{
"mcpServers": {
"blender": {
"command": "/home/jacek/.local/bin/uvx",
"args": [
"--with",
"mcp[cli]<2.0.0",
"--from",
"git+https://projects.blender.org/lab/blender_mcp.git@v1.0.0#subdirectory=mcp",
"blender-mcp"
],
"env": {
"BLENDER_MCP_HOST": "localhost",
"BLENDER_MCP_PORT": "9876"
}
}
}
}

then we can start working ("show on r/LocalLLaMA"):

Today we want to make 3D llama to show on


The user wants to make a 3D llama to show off on . This is a fun creative task. Let me think about how to approach it.

A llama is a pretty complex organic shape. There are a few options:
1. Procedurally build the llama from primitive shapes (spheres, cylinders) — a "low-poly" or "chibi" style llama
2. Sculpt it — not easy to do with MCP
3. Use metaballs — could get a blobby llama

Given the MCP tools, the best approach is to procedurally build a stylized/chibi llama from primitives:
- Body: elongated sphere/capsule
- Neck: cylinder or elongated sphere
- Head: sphere
- Ears: small cones or spheres
- Legs: 4 cylinders
- Tail: small sphere/cone
- Maybe a fluffy mane

This is a well-known technique — making a cute chibi llama from primitives. Let me plan the geometry:

Llama proportions (stylized, standing side view, facing +X or -X):
- Body: UV sphere scaled to (1.5, 0.9, 1.0) or so, position (0, 0, 1.6)
- Legs: 4 cylinders, radius ~0.22, length ~1.2, positions: front legs x=+0.9, back legs x=-0.9, y=±0.5, z from 0 to 1.2... actually, the legs go from the ground (z=0) up to the bottom of the body.
- Neck: cylinder or tapered, extending up from the front of the body. Llamas have long necks. Position around x=+1.2, extending upward.
- Head: sphere at the top of the neck, elongated (llamas have long snouts).
- Ears: 2 small cones/spheres at the top of the head.
- Tail: small sphere at the rear.
- Mane: maybe some small spheres along the neck.

For a cute chibi look: big head, small body. But a llama's charm is the long neck. Let's go with a stylized llama.

For a polished look:
- Apply smooth shading
- Add a material (cream/tan color, maybe with some subsurface)
- Add a ground plane
- Set up lighting
- Position the camera for a good render
- Render a nice image to show off

This is a multi-step task. Let me write Python code that builds this. I'll do it in stages so I can check with screenshots and iterate.

https://preview.redd.it/ybk5u9g2hvnh1.png?width=1471&format=png&auto=…

https://village-sim-one.vercel.app/

\- 16GB VRAM RTX 5070 Ti, fully offloaded

\- Vision on CPU

\- Windows, not headless

\- beellama.cpp - latest version with the kvarn performance enhancements making it as fast as qx\_x quants.

\- MTP n-max = 2

\- tg up to 75t/s, pp up to 1700t/s

\- KV = kvarn3/kvarn3

\- MTP draft KV = kvarn2/kvarn2

\- context = 96256

\- tail tokens = 1024

\- HTML/Javascript

\- pi harness with pi-observational-memory, pi-web-access, pi-atelier (UI Only change, check it out) extensions, though it never used the web access.

\- This is not a one-shot, I do not believe one shotting is a great test. Instead, I did many incremental feature prompts. However, I did not give it any design or framework, which is probably where it can be improved.

Lessons learnt:

\- Do not fear Q3 model quants for Qwen3.8

\- Do not fear KV quantisation. If you have the VRAM sure use it, but I don't feel like it's worth choosing a higher quant if it's going to cause me to offload to CPU and see my tg drop to 5-20 t/s. With higher speed I can fix any issues with a follow up prompt much faster and that rarely happens. I think I had like 3 runtime exceptions which was easily resolved pasting the console output and there is no guarantee a higher KV quant would not have had the same exceptions.

\- MTP/draft cache can also be quantised with kvarn now and actually saves VRAM where qx\_x quants increase VRAM usage for some reason. kvarn2 for MTP is perfectly fine and has high acceptance rates.

The game:

\- Inspired by a popular indie game which I am not promoting, I am just a huge fan.

\- I won't release any further updates, since I don't want to be stepping on any toes. If you like the idea of the game I highly recommend the real game, it's by far my favourite game I played this year and 1000x better than what I present here. It will be a nice distraction from your AI. I just wanted to see what this model is capable of. I do have a Cursor subscription but did not use it at all in the project.

\- I will probably continue to develop it for my own entertainment, but it won't be made public. Maybe come up with my own ideas, but the original game is near perfect anyway, so it will be hard to improve except with some UI gripes I have in the original. And my graphics obviously does not compare.

Game features:

\- Large Map, larger than the browser window.

\- Minimap

\- Zoom feature with mouse wheel

\- Collectable resources, that must be taken to a storage site. Each site can store limited resources.

\- Houses required to sleep and protect against cold

\- Weather and seasons.

\- Day night cycle with randomised sleeping times.

\- Possible death due to hunger or sleeping in cold outside or in house without firewood.

\- Game speed controls.

\- Villagers avoid obstacles.

\- Delete/deconstruct buildings and partial resources refund.

The code:

\- I almost never read the code, so I have no idea what it looks like and the quality thereof. I also gave it very few hints in the AGENTS.md, mostly no magic numbers and write modular code, not a single html.

\- Actually, my initial prompts were a single html but as it grew, I told it to create modules. It messed it up on the first attempt, basically rewriting the entire UI in the process. So I reverted and told it to do it again without making any changes to the functionality or UI.

\- I am actually quite happy with and surprised by the performance of the game.

Context management:

At first, I had issues with the context filling up too quickly and too often. Sometimes it would fill up to the point that there was not enough room to compact. Forcing me to temporarily increase the context and tell it to create a handover document. Reduce context again and feed it the handover doc.

I then installed pi-observational-memory extension, and it works quite well and I never run into context issues anymore since it takes notes throughout (a short wait time every few prompts) and compacting is near instant because it already took the notes.

Conclusion:

\- Do not blindly drop your KV cache quant without testing. I have a hard level needle in haystack test that requires multiple hops and 100's of decoys. Q3\_XXS does poorly in that test even with F16 KV cache. However, Q3\_K\_XL almost 100%'s the test even at kavrn3. So both the model and KV matter. In my testing a smaller model does more damage than a smaller KV. So find the right balance. At a certain point increasing model quant will have less impact than picking a larger KV quant. But for a tight 16GB VRAM fit Q3\_K\_XL works very well with kvarn3. Q4 on the other hand just leaves me with too little context. That said despite Q3\_XXS doing poorly in my needle test it still does fairly well with coding. Better than Qwen3.6 so if you have 12Gb VRAM it is still an option. Because by poorly I mean F16 KV scores 84% and Q3 KV around 80%. Needle tests however do worse with kvarn compared to qx\_x for some reason. However, a needle test is not the be all and end all. kvarn does better with KLD, so once my needle scores near 100% I am satisfied.

I will play around with higher KV quants, but I intentionally kept it at kvarn3 for this test, however I am not sure how much context I am willing to sacrifice. Maybe i will try kvarn4/kvarn3. But I just wanted to prove a point to myself and kvarn3 worked just fine. If I had >16GB VRAM sure I would up it but I don't.

Right, so this is going to be embarrassing but it's presumably something we've all been through at one point or another, and I guess this is my first time resolving something like this in the way that I did so figured I'd share if only to share that it's now a thing and that it's pretty cool..

A friend of mine sent a message asking what's up and if I wanted to watch a movie together, I was kinda hesitant but she buttered things a bit and finally I'm like fine, and so she sends me a link to some clearly vibe coded site that I'm kinda getting red flags from and so I forget about it and a little later I get another message going "we're waiting for you" and so I'm like shit, I guess I gotta do it huh, and so I open up this goofy looking site again. You gotta login to join a room, and you gotta sign up inside their downloaded software, sure whatever, next thing I know some fake 150MB file's fake install bar is stuck at fake 50% and both my Chrome and Discord's crashed and reloaded. Suspect, but I've been through this stuff before, it's probably just a RAT so I guess it's time to dust off Windows Defender and unplug the internet for a little bit. I message her to go on and watch it without me as my PC's giving me suspicious vibes right now, and seconds later I get some overly polite DietGPT in my IM's saying "sorry um excuse me but it appears that i've hacked you👉👈", occasionally switching to really hostile broken English asking for giftcards from some site I've never heard of. I stall, unplug the PC's internet so my router still responds to pings, and start punching into GLM "what do" and it tells me it's a session grabber - time to switch passwords. Meanwhile my phone's texts are blowing up with 2FA login requests from domain registrys and other bad stuff and I kinda freak out a little. I get my emails' passwords switched first and by the time it's Discord's turn my friendlist's already been nuked and the dude says I got 10 minutes to give him $200 or he's gonna fuck me up some more, and so I kinda figured welp time to figure out what more he's got and so I called him a giant pussy and he blocked me. An hour later my Discord was perma-banned, he had posted the phrase "i sell cp" using my account and used that as blackmail along with some really old photos of me, I though it was a bluff but oh well it's being handled with Discord's customer support on it's own. Now I sat there alone, in the middle of the night, having just had my friends on the phone yanked away from me with a permaban, knowing that if I reboot I'd probably be ransomware'd or something so I figured let's run Windows Defender - it found nothing, 0 results on a full scan.. Too good to be true, so I grabbed AwdCleaner on my phone and transfered it via USB. It found an AVG Toolbar for Chrome. That confirms it, I haven't used AVG for decades and so I removed it but it's back 5 minutes later. That double confirms it, I'm screwed. With nowhere else to go and potentially a ticking timebomb running on my PC that could start encrypting or deleting files at any given moment I figured why the hell not, if I'm going to watch my pc blow up I might as well send in the goofy little local LLM to cut one of the wires,

here's the situation.
i've downloaded a maliscious file that unfortunately hacked my discord and got me banned. i'll be dealing with that on my own. your job is to study the files in the project folder and see if you can help me clean up my computer, as presumably the virus is still active. there's no internet connected, and i request that you refrain from running the \*\*\*\*\*\*\*\*.exe file (\*\*\*\*\*\*\*\*.exe is the virus archive, do not run it, it's a 7zip archive), please help.

And so Qwen3.8-27B got to work, and to big surprise after around 60 minutes of clawing at the file it had done what I asked and a whole lot more. it fully deciphered all the layers these clowns had bundled this thing with in order to make it appear legit, it had created a single PowerShell removal script ready to go complete with a pre-launch check enabled by default and everything, and it was reverse engineering 0-days in qProtect to get the C2 domain used by this malware so that it could be blocked from the network.

If you're looking for what Qwen3.8-27B is capable of doing fully on it's own if you let it, here's a 15k line example of it's ability to tear some piece of shit session grabber to shreds in a single prompt: https://www.mdshare.online/s/Mamdrs1WWkurRtt8z8QzK

I let it do what it does best for an additional 24 hours, the additional information is going to the Discord Support team. Hopefully shit like this can be prevented.

TLDR; Qwen3.8-27B > Windows Defender, and don't forget to use 2FA.

I feel like coding overshadows how great this model really is. It knew a lot of very arbitrary facts/information about my home state and resources about those specific things related to jobs. I found this interesting since getting into the nitty gritty details like this can cause a model to hallucinate some facts.

Not only that but if you have a problem, it will throw the kitchen sink at you with everything it’s got to try and solve it.

I see a new one being launched every few days... How do these new harnesses compare to claude code, pi etc. has anyone switched from these?

which harness to prefer and why

edit: Ive tried several different ones claude code, deepagents(langgraph), opencode, pi, and trueforge

my thoughts-

claude code - strongest on maturity and the managed experience but cost and token burn is high

deepagents - interesting middle ground if you want a more structured agent framework and the flexibility of an open-source stack. im interested in testing it more extensively on longer-running workloads fs

trueforge - this is a recent one, this was interesting to me because of its runtime-efficiency, also it allows separate the model from the runtime, which makes experimenting with different models much easier
https://github.com/truefoundry/trueforge

why?? - i also ran a benchmark on a real agent workload same model, same prompt, same tasks to compare these

adding the results of benchmarking i ran to compare this
so I tried to do this by running 14 cross-system tasks, three mcp servers behind them - a crm, an issue tracker, and a doc store through claude's managed agents, langchain's deepagents and trueforge, both open-source agent harnesses

the result that was most surprising:

Claude Managed Agents + Opus 4.8:
11/14 tasks solved | $11.8/run | 10.0M tokens/run

TrueForge + Opus 4.8:
11/14 tasks solved | $8.6/run | 3.7M tokens/run

Same model. Same benchmark. Same average solve rate, to my surprise trueforge used about 63% fewer tokens and cost about 30% less per run.

similar difference in tool usage: trueforge averaged 19 tool calls per task vs 32 for Claude Managed Agents.

Then I tried changing the model.

trueforge + GLM-5.2:
11.7/14 solved | $3.0/run | 3.8M tokens/run

On this benchmark, that was a slightly higher average solve rate than Claude Managed Agents + Opus at roughly 75% lower cost.

The token savings alone make this sooo interesting especially because the solve rate stays comparable
so this one was worth checking out ig

but this is still v early and the OSS runtime does not yet have first-class tracing/eval tooling. They don't ship their own code-execution sandbox, so you need to plug one in and context compaction is intentionally lossy.

So it is definitely not a replacement for a mature managed agent platform or other harnesses in the comparison, feature-for-feature today btu what I do find interesting is that the core runtime can already be competitive on these tasks while staying open, model-neutral, and deployable on my own infrastructure
this was their benchmark kit i used https://github.com/truefoundry/trueforge/tree/main/benchmark

Otaku is an LLM frontend, primarily designed for roleplay, an alternative to SillyTavern and the like. However, It also works for general-purpose chat with local backends (including Ollama) or cloud models, the way Open WebUI is used, once lore extraction is switched off in the settings.

Otaku offers two interfaces:

• a web UI (customizable): interactive demo at https://otaku.sh/demo-web/
• a terminal UI: its own interactive demo at https://otaku.sh/demo-terminal/

Both share the same functions; the difference is that in the terminal you execute them with slash commands (the reference is available with /help), while in the web UI the operations are available from the menu.

Install

Otaku is free and open source (MIT); it works on macOS, Linux and Windows. Install it with uv (uv tool install otaku) or see the GitHub README for other options: https://github.com/enclavum/otaku

Get started

Launch either otaku for the terminal or otaku web for the web UI; the web UI's default URL is http://localhost:9600. Two sample stories are imported on first start to give you an idea of the features and what play looks like, and you land right in the middle of one of them.

On first start, you choose a provider and a model: Otaku automatically detects local installations of Ollama, oMLX, LM Studio, llama.cpp and KoboldCpp, and lets you pick from their models. Cloud providers (OpenRouter, NanoGPT) are also there: enter an API key and their catalogs appear. After exploring the provided stories, you can start your own with the /new command.

Asking for feedback

Otaku is a personal side project, and I'd like to get feedback from the community on the product and on what to add.

I was running into a lot of posts that praised both the Fixed template and the Sharp template in comparison to the stock one, so I put them to the test.

It's not as extensive as it should be for a paper-grade analysis, but it gives out the point of each template.

Test setup

I used SWE-bench Verified with mini-SWE-agent 2.4.6, slice 0:100 (the identical 100 tasks for all runs)

Hardware

• CPU: Ryzen 9 9900X
• RAM: 128 GB DDR5-5600
• GPU: RTX PRO 6000 WS

Runtime

I containerized jpezzulli/sglang-rtxpro6000 and ran Flash Next with RadixArk/Qwen3.8-Flash-Next-NVFP4 on CUDA 13.3.

• Full 262K context
• BF16 KV
• 51.2 GB FP8 n-gram embedding table pinned in RAM
• 32 GB HiCache pinned in RAM

I ran all templates at both medium and xhigh reasoning efforts.

Results

|Metric|Stock (medium)|Stock (xhigh)|Stock Δ|Fixed (medium)|Fixed (xhigh)|Fixed Δ|Sharp (medium)|Sharp (xhigh)|Sharp Δ|
|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|
|Resolved|91|99|\+8|87|98|\+11|94|94|\+0|
|Resolution rate|91%|99%|\+8 pts|87%|98%|\+11 pts|94%|94%|\+0 pts|
|Median output tokens|5,691|13,855|\+143.5%|6,956|14,819|\+113.0%|8,596|12,008|\+39.7%|
|Median reasoning tokens|3,050|8,759|\+187.2%|3,809|9,063|\+137.9%|5,437|7,967|\+46.5%|
|Median wall time|38s|1m 46s|\+180.4%|43s|1m 47s|\+152.3%|1m|1m 32s|\+53.4%|
|Total wall time|1h 47m 1s|4h 31m 22s|\+153.6%|1h 59m 53s|4h 4m 52s|\+104.3%|2h 29m 18s|3h 11m 36s|\+28.3%|

https://preview.redd.it/02geu81o8qnh1.png?width=1152&format=png&auto=…

https://preview.redd.it/v2mt6mgo8qnh1.png?width=1152&format=png&auto=…

https://preview.redd.it/ph1z36zo8qnh1.png?width=1152&format=png&auto=…

Takeaways

https://preview.redd.it/6ydu12mp8qnh1.png?width=1152&format=png&auto=…

• Raising reasoning effort to xhigh closes almost all of stock's and fixed's gap to Sharp. At medium, Sharp led resolution by +3 tasks over stock and +7 over fixed; at xhigh, stock and fixed instead lead Sharp by +5 and +4 tasks, respectively.
• Sharp barely moves on resolution (94 → 94) despite a real token/time cost increase, median reasoning tokens rise +46.5% and median wall time +53.4%. This suggests it was already extracting most of the benefit it could get from extra reasoning budget at medium, while stock and fixed still had headroom.
• Sharp remains the most token-efficient per resolved task at xhigh (14,541 output tokens/resolved vs. \~17,000 for stock/fixed), consistent with its medium-era efficiency edge, but it's no longer the highest-resolving template once reasoning effort is high.
• Absolute cost scales heavily with reasoning effort: total wall time roughly 2.3–2.5× for stock/fixed and +28% for Sharp; total reasoning tokens roughly doubled for stock/fixed and increased +35% for Sharp.

Conclusion

• Sharp should be used at medium and it keeps a reasonable accuracy at very good speed. I don't see the point in using it at xhigh. By sacrificing a small accuracy you complete the tasks in half the time.
• Stock is the slowest but the most precise.
• Fixed is the middle ground between Stock and Sharp both in accuracy and speed
• The next benchmark will be on a much extensive SWE-bench Multilingual + Terminal Bench.

Disclaimer: I wrote the post myself then used AI to format it properly for readability

I've been trying the latest models from the frontier labs and honestly, after extensive testing I can not tell the difference between the best open source options.

I think the differences are now marginal but the labs are doing heavy marketing to convince the public into paying more for tokens as they prepare to go public.

Can't help but see the similarities between the dot com bubble and AI in terms of a very insular environment where the technology will survive but the business models may not.

I've been building a cybersecurity network and we definitely know that even local AI models like Deepseek V4 flash do an excellent job and are really neck and neck with the best the frontier labs can provide.

Will be interesting to see how this all turns out! Exciting time nonetheless.

I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been watching other redditors post their custom configurations, and I wanted to share what I tested to optimize for a single RTX 5090.

I was on llama.cpp (Q5\\\_K\\\_M GGUF, q5\\\_1 KV, 262K context, MTP), but it's limited to parallel=1 and I wanted concurrent serving. So I tested vLLM and NInfer as NVFP4 replacements and did a proper quality evaluation instead of just vibes.

Hardware

\- RTX 5090 32GB (eGPU, OCuLink Gen4 x4) (yes, it's in an eGPU dock 😂 but that only affects model loading)
\- Ryzen 7 7840HS, 32 GB DDR5
\- Ubuntu 26.04, nvidia driver 610.43.02 (open)

## Engine configs

| | **\*\*llama.cpp\*\* | \*\*vLLM\*\* | \*\*NInfer\*\*** |
|---|---|---|---|
| Quant | Q5\\\_K\\\_M GGUF | NVFP4 | NVFP4 |
| KV cache | q8\\\_0 | FP8 | FP8 |
| Context | 196K | 262K | 240K |
| MTP | On (gate failed) | None | MTP3 (76% acceptance) |
| Concurrency | parallel=1 | Continuous batch | x2 lanes |
| VRAM | 31.6 GB | 29.6 GB | 30.5 GB |

How the eval worked

I built a custom harness with 6 tiers, 50 items each, all from my actual production workload (not generic benchmarks):

1. **\*\*Relevance classification\*\*** \- is this industry relevant? (binary, 50 labeled deals)
2. **\*\*Needle retrieval\*\*** \- planted facts in real industry podcast/video transcripts at 64K/128K/192K/240K context
3. **\*\*Multi-transcript QA\*\*** \- questions across 3-4 concatenated diarized transcripts, 120K+ tokens, including unanswerable controls
4. **\*\*Reasoning with thinking\*\*** \- numeric/logic problems, thinking mode on, greedy pass@1
5. **\*\*Structured extraction\*\*** \- custom extraction prompt, json\\\_mode (skipped on NInfer, it doesn't support json\\\_mode)
6. **\*\*Tool replay\*\*** \- replayed recorded agent episodes against each engine (diagnostic only, all engines fail this one)

Everything paired across engines: same prompts, same seeds (42), same gold labels. No cache\\\_prompt. Statistical comparison uses paired cluster-bootstrap CIs with pre-registered non-inferiority margins.

And when I do development with Claude Code, I often leverage multi-model consultations for design, planning, and code review. I did a 4-model review panel (Codex/GPT-5, DeepSeek V4 Pro, Grok 4.5, Kimi K3) audit the methodology mid-campaign. They found 10 issues, including 4 mislabeled gold items where the models were actually right and my labels were wrong. Fixed everything and reran.

Quality results

| **\*\*Tier\*\* | \*\*llama.cpp\*\* | \*\*vLLM\*\* | \*\*NInfer\*\*** |
|---|---|---|---|
| Relevance | 86.0% | 84.0% | 86.0% |
| Needle (conditional) | 100% (29/29) | 100% (41/41) | 100% (41/41) |
| Transcript QA | 82.0% | 78.0% | 88.0% |
| Reasoning | 100% | 100% | 98.0% |
| Extraction | F1 0.300 | F1 0.350 | skipped |
| Tool replay | 0% | all errors | 0% |

Needle counts differ because llama.cpp's 196K context can't fit the 192K items (need room for max\\\_tokens + headroom). NInfer and vLLM both handle 192K fine. All engines score 100% on every needle they can fit.

Statistical comparison (NInfer vs llama.cpp, bootstrap):

\- Needle: delta = -0.29 (NInfer better, p=0.0006) - this is entirely from context capacity, not retrieval quality
\- Transcript QA: delta = -0.03, p=0.69 - no difference
\- Reasoning: delta = +0.02, p=0.72 - no difference
\- Relevance: McNemar p=1.0 - identical
\- Tool replay: delta = 0.0 - both fail equally

**\*\*Takeaway: quality is statistically indistinguishable across all engines.\*\***

Speed results (perf probe, server-side timings)

| **\*\*Metric\*\* | \*\*llama.cpp\*\* | \*\*NInfer\*\* | \*\*Speedup\*\*** |
|---|---|---|---|
| **\*\*Decode 1K\*\*** | 114 tok/s | 158 tok/s | 1.4x |
| **\*\*Decode 32K\*\*** | 109 tok/s | 213 tok/s | 2.0x |
| **\*\*Decode 128K\*\* | 72 tok/s | 202 tok/s | \*\*2.8x\*\*** |
| Prefill 1K | 1,545 tok/s | 7,265 tok/s | **\*\*4.7x\*\*** |
| Prefill 32K | 2,155 tok/s | 6,892 tok/s | 3.2x |
| Prefill 128K | 1,528 tok/s | 3,904 tok/s | 2.6x |
| TTFT 1K | 670 ms | 138 ms | 4.9x |
| TTFT 32K | 15.2 s | 4.8 s | 3.2x |
| TTFT 128K | 85.9 s | 33.6 s | 2.6x |

vLLM speed excluded from the table because the perf probe used wall-clock timing (includes prefill + scheduling + decode) instead of server-side timings, so the numbers aren't comparable. From community reports and my own task-level measurements, vLLM does about 70 tok/s decode at short context, which actually matches NInfer's raw step rate (\~66 tok/s). The speed difference is entirely MTP3 speculative decoding.

What I learned

**\*\*NInfer's speed advantage is all MTP.\*\*** The raw NVFP4 kernel speed is about the same between NInfer and vLLM (\~66-70 tok/s). NInfer's MTP3 speculation with 76% acceptance gets you to 158-213 tok/s. If vLLM or llama.cpp had working MTP on NVFP4, the gap would mostly close.

**\*\*The decode speedup grows with context.\*\*** At 1K context it's 1.4x. At 128K it's 2.8x. MTP acceptance stays high even at long context while llama.cpp's dense decode gets slower as context grows.

**\*\*NInfer's tokenizer endpoint is great.\*\*** It exposes \/v1/messages/count\_tokens\ (Anthropic Messages format) which gives exact token counts. No more \len(text)//3\ heuristics.

**\*\*NInfer does NOT support json\\\_mode (as far as I can tell).\*\*** \response\_format: json\_object\ returns 400. If you need structured JSON output, you'll need to route those calls elsewhere or use prompt-based enforcement.

**\*\*Don't trust vibes for quality.\*\*** I went in expecting NVFP4 might lose a few points vs Q5\\\_K\\\_M. It didn't. Not on any tier. The biggest delta across 250+ items was 2 percentage points, well within noise at n=50 (SE \~5.6pp).

Verdict

NInfer NVFP4 replaces llama.cpp as my production engine. Same quality, 1.4-2.8x faster decode, 2.6-4.7x faster prefill, concurrent serving. The only gap is json\\\_mode.

I put together a detailed poster with all the charts and methodology details: \full results poster\

Setup if you want to try it:

\\\`
\# NInfer (from source)
git clone https://github.com/Neroued/ninfer && cd ninfer
mkdir build && cd build
cmake .. -DCMAKE\_BUILD\_TYPE=Release -GNinja && ninja

\# Model (HuggingFace)
\# https://huggingface.co/neroued/Qwen3.8-27B-nvfp4-NInfer (20 GB)

\# Run
./ninfer-serve /path/to/model.ninfer \\
\--model-id qwen3.8-27b \\
\--host 0.0.0.0 --port 8080 \\
\--max-context 240000 --kv-capacity 240000 \\
\--max-concurrency 2 --kv-dtype fp8 \\
\--spec mtp --draft-tokens 3 \\
\--vision --preserve-thinking
\\\`

Ling 3.0 Tiny still seems to be leading the pack despite only having 1.3B active

Along with everyone's favorite here, qwen3.8-27B

First, I'd like to thank the Qwen and Unsloth teams for the Qwen3.8 27b UD Q4\_K\_XL. Fits the poor 24GB of 3090 VRAM with 100k context at Q8 and works phenomenally well! Imho if theres anything that can threaten Anthropic/OpenAI profits is not another frontier model but actually these small ones you can run fast locally that can do 80..90% of mundane work for hours without paying a single dollar to any external company.

But next, if you want to jump up to a bigger smarter model I feel there is a gap now. Kimi-K3 is out of reach for many businesses let alone prosumers. So what frontier-like models do you use on what setups?

Is a DGX cluster (2..4 machines) or a GPU server with dual or quad GPU (\~96 ... 192 GB of VRAM + >256GB DDR4) a suitable setup to run something like MiniMax-M3 at reasonable speeds for agentic coding (>30tps)? And privacy aside, is hardware cost worth it?

I have a dual rtx3090 + 128GB ddr4 machine, running Qwen3.8-Flash-Next Q4 quite fast but despite being larger doesn't feel much smarter than the Qwen2.8 27b and while I \_can\_ run larger quantized models, Minimax-M2.7 being my workhorse, it way too slow for coding.

Anyone who has a 3D printer and get use of it finds it incredibly useful for those odd jobs around the house, a missing bracket, a cable router, steam deck holder and so on.

In the past if I was missing an app or useful software, a game I'd do the lazy thing, even though I can and have coded in the past, its "effort" I'll just go and buy or download the latest and greatest.

Earlier in the year I was lucky to snag a Minisforum MS-S1 395+ Max with 128GB Unified memory (currently setup 32gb system and 96gb Vram) before the price hike.

Was paired with a Qwen 3.6 27B or 3.6 35B moe but now a 3.8 27B uncensored. it can easily handle a Q8 with full 256k context.

Its now become my first instinct when I'm missing software to build it in a couple of hours local using the custom agent framework I setup.

Nothing I've created is for external use but every single day I find myself adding to it, while writing this post for example my framework finished an idea I had 2 hours ago when, I woke up this morning thinking I've got a lot of japanese visual novels and why don't I just design a combination hook into Exe or ocr the text app that translates via a local llm, and its done, ready for me to test.

I've written 12 adult games (don't code horny) a house AI, a coding framework, a game app to keep a track of all the games I play and download any faq or wiki to do with said game, 17 mods for my Skyrim install, 12 for my Fallout New vegas install, A temperature tracking system for the house that pulls rss local feeds and makes suggestions for my central heating system temps settings, A mapping software for my mobility scooter that checks my normal routes for issues and street work or maintenance that could make pavements impassable.

Plus hundreds of tweaks and test programs.

Anyone else out there using it like this ?

\---------Update-----

Awesome to see this kind of discourse one of the amazing strengths of these local llm's is it doesn't matter if they are slower, I can burn 50 million tokens over a 24 hours period on a new idea or problem and all it costs me is a little bit of electricity and time.

Full credits to @artificialisabel from X!

Used qwen3.8-27b in Opencode to make this silly mini-game because I'm not sober:

```
We are going to play a game, it will be the Wikipedia game. The Wikipedia game has the following rules:

• You will have a Wikipedia article set as a starting point.
• You will have a Wikipedia article set as an ending point.

Your objective is to reach the the end point, which is an article completely separate from the starting point article.

Your only constraints are the following:

• You are ONLY allowed to click on any hyperlinks inside of wikipedia directly. No external links, no typing inside of wikipedia's search bar (but finding the starting article on google is valid. The 10-click limit starts once you reach the starting point article).
• You are NOT allowed to return to a previous page. All clicks much be performed in a forward-looking trajectory.
• You must reach the end article within 10 hyperlink clicks inside of Wikipedia. If you do not reach the destination article within 10 clicks, you lose.
• Do not update any documentation for this task. It is only a game.

Use playwright to click the links.
```

Basically, Qwen needs to reach an ending article within 10 Wikipedia hyperlink clicks from the starting article, which is usually an unrelated article. It needs to use playwright (or some equivalent browser MCP) to click the Wikipedia hyperlinks without backtracking, using search or using external links.

I verified the links for accuracy and I can confirm it managed to complete this task within 6 turns. Thought it would get stuck in a loop. Its a dumb minigame but I think its a good, simple agent test to perform.

Modified 4090 48GB has been out for a while. I remember a lot of people were buying them at the time. A lot of people were also complaining that they are meant to fail, that they scam etc.

I have a few questions to people people who bought these.

1. How is longevity of these cards? Do they still work without issues? Any failure rate?

1. Do they use the same Nvidia drivers that regular 4090 or 4090D uses?

1. Are these cards Linux exclusive?

1. Are you able to run them in windows or Linux with other GPUs like 5090 etc?

1. Do you do anything to cool VRAM on the back of the PCB?