We built an engine, Kyojin, on top of ExLlamaV3 for Strix Halo (gfx1151, ROCm), and packed two 300B-class MoE models so each fits one 128 GB machine. First release, all measured on Ryzen AI Max+ 395.

|Model|GLM-5.3-Flash|MiMo-V2.6-Flash-MOPD|
|:-|:-|:-|
|Size|99.7 GB|105 GB|
|Prefill|580 tok/s at 3.5K, 546 at 64K|about 650 tok/s at 4K|
|Decode|26 to 30 tok/s (MTP)|32 prose / 35 chat / 44 code (speculative), 29 plain|
|KLD vs official FP8|0.151|0.0713|
|Top-1 agreement with FP8|89.3 %|92.0 %|

Where the weights come from. MiMo is our own quantisation. The GLM pack mixes turboderp's public 2.05 and 3.05 bpw EXL3 tensors, with our layer mix and a small tuning stage. On the same 129 rows, his 2.05 bpw pack (85 GB) gets KLD 0.275; our mix (100 GB) gets 0.190. His is smaller and decodes about 10 % faster.

Uncensored variants. Separate -Uncensored repos: same weights plus one small file the engine applies at load, one switch turns it off.

Not measured yet. Task-suite scores for MiMo, GLM at 128K context, any GPU other than gfx1151. The conversion pipeline stays private.

Quickstart. Clone, ./build.sh, hf download yamz-labs/GLM-5.3-Flash-EXL3-Yamz, python tools/glm/serve.py --model ./glm-pack -c 131072 --num-draft 2. You get an OpenAI-style API.

Models: https://huggingface.co/yamz-labs

Engine: https://github.com/Yamz-Labs/kyojin

Built on turboderp's ExLlamaV3, with ROCm work from sdougbrown and vcruz305.

If you own a Strix Halo machine, we'd love to see your tok/s. Issues, benchmarks and PRs are all welcome. Which model should we do next?

I wanted to see how far I could push a fairly ordinary laptop with a huge MoE model.

Turns out, Qwen3.8 Flash Next 176B can run on:

RTX 3080 Laptop — 16GB VRAM
32GB system RAM
SSD

No 128GB/256GB RAM workstation and no multi-GPU setup.

I’m running it with TensorSharp, my open-source local LLM inference engine:

TensorSharp on GitHub

The interesting part for me wasn't simply getting a 176B model to load. I wanted to make a model much larger than both available VRAM and RAM actually usable.

The approach is basically:

Quantization + MoE-aware unified scheduling across cache, VRAM, system RAM, and SSD.

Rather than treating SSD as a last-resort swap space, TensorSharp coordinates the different memory/storage tiers around MoE execution and tries to keep the right experts/data in the right tier at the right time.

I previously benchmarked TensorSharp against llama.cpp and got very encouraging results. This time I wanted to compare it with Strata, since Strata's approach to running large models with constrained memory is particularly interesting.

Here are the results from the attached benchmark:

|Measurement|TensorSharp|Strata|
|:-|:-|:-|
|Decode tokens/s|11.09 (9.22–14.02)|10.24 (9.37–10.46)|
|Whole-process time|16.54s (14.95–19.31)|62.15s (59.76–66.89)|
|Device-wide GPU peak|14,832.5 MiB|15,729 MiB|
|OS peak working set|19.74 GiB|18.51 GiB|

The decode throughput is fairly close: 11.09 vs. 10.24 tok/s.

What surprised me more was the end-to-end result: 16.54s vs. 62.15s in this test.

I think this points to an interesting direction for local LLM inference. For huge sparse MoE models, the question may not simply be:

“Do I have enough RAM/VRAM to fit this model?”

but rather:

“How efficiently can the runtime coordinate VRAM, RAM, SSD, caching, and expert activation?”

With the right quantization and memory hierarchy, you can apparently do some pretty ridiculous things on consumer hardware.

I’d be especially interested if anyone here has tried the same model with llama.cpp, Strata, or another MoE/offloading implementation. It would be great to compare results on similar hardware.

I've been tinkering with GLM-5.3-Flash on two 64GB CMP 170HX cards for a while, and the setup is finally stable enough that I figured I'd share it.

I also compared it against the Qwen3.8-Flash-Next setup I've been using on the same machine: AWQ INT4 + FP8 PLE on vLLM.

Besides PP/TG benchmarks, I hooked both models up to DSH and gave them the same small coding/agent tasks to see how raw inference speed translated into actual task completion time.

A few caveats up front:

• this is not an apples-to-apples quant comparison
• GLM and Qwen are using different engines and different speculative decoding setups
• speculative decode speed depends heavily on acceptance rate and generated text
• the coding tasks are just a few practical examples, not a serious benchmark suite
• when I mention “Strata-style” below, I mean the HBM-first/full-residency approach I previously used with Strata, not that this is running Strata itself

Repo and playable demos:

GitHub:
https://github.com/Flun/glm53-flash-cmp170hx-exl3

Hardware

|CPU|Ryzen 5 5600X|
|:-|:-|
|RAM|80GB DDR4|
|GPU|2x CMP 170HX 64GB|
|GPU arch|SM80|
|PCIe|Gen2 x8|
|GPU P2P|unavailable|
|OS|Ubuntu 24.04|

Both models were tested on the same machine.

For the agent tests I used DSH as the harness.

GLM-5.3-Flash setup

Target model:

turboderp/GLM-5.3-Flash-exl3 3.05bpw

Engine:

ExLlamaV3 1.5.4

Current setup:

• GLM-5.3-Flash EXL3 3.05bpw
• \~125.2GB / 116.6GiB target weights
• target fully resident across the two 64GB cards
• k_hcfuse
• DFlash2 EXL3 6bpw
• DFlash2 K7
• Q8 KV cache
• 384K context in actual use
• max request budget around 392,960 tokens

GLM-5.3-Flash itself is a 320B-total / \~18B-active MoE model.

The important part here is that the target weights stay resident in HBM. I'm not continuously streaming experts from system RAM during decode.

About the 3.05bpw quality

This was probably the part I cared about most.

At first glance, “3.05bpw” sounds like a pretty aggressive quant, especially compared to the UD Q4 variants people commonly use.

But EXL3 isn't simply “make every tensor 3-bit”.

It uses a trellis-based quantization scheme with different bit allocation depending on the tensor. The 3.05bpw number is an average target bitrate.

The published quant configs for this family also keep more sensitive parts at higher precision. For example, lm_head remains at 6-bit in the 3.05bpw branch.

Looking at the same model family, the 4.05bpw build has been inspected with something roughly like:

• routed experts: K4
• attention: K6
• shared experts: K6
• dense MLP: K5
• lm\_head: K6
• embedding / norms / router: native

So the general idea is to compress the huge routed-expert portion more aggressively while spending more bits on the smaller/more sensitive paths.

That makes quite a bit of sense for a MoE model like this, because most of the storage is in the expert weights.

Rough comparison with the common UD quants

|Quant|Size|Top-1 agreement vs BF16|Mean KLD|
|:-|:-|:-|:-|
|UD-IQ3\_XXS|120.37GB|81.63%|0.28377|
|EXL3 3.05bpw (my target)|125.18GB|\~93.05% (estimated from published 3.0bpw results)|\~0.050 (estimated from published 3.0bpw results)|
|UD-IQ4\_XS|156.82GB|88.18%|0.11665|
|UD-Q4\_K\_XL|199.71GB|92.22%|0.04929|
|UD-Q5\_K\_XL|240.31GB|94.35%|0.02705|

For reference, public GLM-5.3-Flash GGUF fidelity numbers look roughly like this:

One thing worth pointing out is that something named Q4_K_XL is not literally “4 bits per parameter across the entire model”.

At \~200GB for a 320B model, it's a mixed-precision quant with an effective average bitrate much higher than 4bpw.

There is also a published GLM-5.3-Flash EXL3 3.0bpw fidelity test using 51,175 held-out next-token positions that reported:

• Top-1 agreement: \~93.0%
• Mean KLD: \~0.0505

Numerically, that's in roughly the same neighborhood as the published UD-Q4\_K\_XL result.

That said, I would not claim that “EXL3 3bpw is better than UD-Q4\_K\_XL” from those numbers alone.

They were not measured through the exact same evaluation pipeline/corpus, and the public 3.0bpw artifact isn't the exact same quant I'm running either.

My takeaway is simply that 3bpw-class EXL3 can preserve a surprising amount of fidelity for its size, and it doesn't behave like a naive 3-bit quant.

For my use case, getting the target down to \~116.6GiB while still retaining usable coding/agent quality was the main reason this setup was interesting.

DFlash2 6bpw is only the drafter

Just to avoid confusion:

the 3.05bpw model is the actual GLM target.

The 6bpw DFlash2 model is only the speculative drafter.

The drafter proposes tokens, and the GLM target verifies them.

So this is not some kind of “3.05bpw + 6bpw averaged quality” setup.

The draft quant mostly affects draft speed, VRAM use, and acceptance efficiency.

HBM-first / “Strata-style” part

This is where I borrowed an idea from a Strata setup I had used previously.

Again, this does not run Strata.

What I mean by “Strata-style” is simply:

keep as much of the model permanently resident in HBM as possible, and avoid runtime CPU↔GPU weight traffic

The GLM target itself fits across the two cards, so I leave the target fully resident.

Instead of offloading experts, I focused on reducing the memory used by the parts that scale with context: KV cache and the speculative drafter.

For this particular machine that made more sense to me than constantly moving weights over PCIe.

DFlash2 + 384K context

I originally used GLM's MTP d2 path.

Later I switched to DFlash2.

Instead of keeping the original BF16 incoai/GLM-5.3-Flash-DFlash2 drafter, I converted it to an ExLlamaV3-compatible EXL3 6bpw build.

The resulting draft weights are about 0.96GiB.

The bigger problem at long context was actually the draft KV cache.

If the target is running 384K and the drafter also grows a 384K KV cache, VRAM disappears quickly.

So I changed the drafter side to use a fixed SWA window plus a GPU ring cache.

The target still sees the full 384K context and keeps its full target KV.

Only the drafter's KV storage is kept inside a bounded ring.

That's what lets the current setup run:

DFlash2 K7 + Q8 KV + 384K target context

without growing the draft cache to the full target length.

The implementation and validation tests are in the repo.

Cold start

I also measured from a cold compile/start until the API was actually ready.

|Model|Ready time|
|:-|:-|
|GLM-5.3-Flash EXL3|\~1m 04s|
|Qwen3.8 Flash Next / vLLM|\~3m 50s|

This isn't really a model-size comparison.

The Qwen vLLM setup has quite a bit more startup work:

• PP workers
• distributed runtime
• model placement
• MTP
• PLE
• GDN
• Triton compilation
• memory profiling
• KV allocation

The ExLlamaV3 GLM path is comparatively static.

Inference benchmarks

These are the numbers from my dashboard workload.

Again, especially for speculative decode, I wouldn't treat these as universal model speeds.

Acceptance rate and generated text matter a lot.

GLM-5.3-Flash / DFlash2 K7 / Q8

|Input|PP|Decode|
|:-|:-|:-|
|8K|1,529 tok/s|95.6 tok/s|
|40K|1,624|90.7|
|73K|1,647|90.6|
|106K|1,650|91.0|
|131K|1,611|94.4|
|385K|1,535|90.1|

DFlash acceptance on this particular workload was mostly around 87%.

With the older MTP d2 path, the same dashboard workload was generally in the \~60 tok/s range.

Switching to DFlash2 K7 brought it to around \~90 tok/s here.

Qwen3.8 Flash Next / vLLM

The Qwen setup is:

AWQ INT4 + FP8 PLE / PP2 / MTP3

|Input|PP|Decode|
|:-|:-|:-|
|8K|5,513 tok/s|129.5 tok/s|
|40K|5,628|123.6|
|73K|5,466|141.9|
|106K|5,307|141.5|
|131K|5,183|159.9|
|252K|4,706|149.4|

So on raw throughput, Qwen is clearly faster.

At roughly 131K:

• PP: \~5.18K vs \~1.61K
• decode: \~160 vs \~94 tok/s

No argument there.

The interesting part for me was what happened once I actually let both models do multi-step coding work.

Why I stopped at 384K for now

I tested roughly 385K input and PP was still around 1.5K tok/s.

The problem wasn't PP collapsing.

It was simply wall-clock time.

Prefilling \~385K from scratch already takes about 4 minutes.

Even if I can make 1M fit, doing a full 1M cold prefill at this speed isn't particularly attractive for normal use.

So I'm currently leaving the service at 384K Q8.

I still want to see if I can get 1M working eventually, mostly for the technical exercise.

Small agent tests

Originally I was only going to make both models build Tetris and stop there.

Both were connected to DSH and got the same request.

1. Tetris

Prompt:

Build a playable Tetris game for the web.

|Model|Completion time|
|:-|:-|
|GLM-5.3|4m 40s|
|Qwen3.8|5m 30s|

Both produced working versions, and honestly the difference wasn't dramatic enough to be very interesting.

So I added two more tasks.

https://reddit.com/link/1x0b1ws/video/1sn8netal4uh1/player

2. AI mini PC landing page

Exact prompt given to both:

Build a polished single-file HTML landing page for an AI mini PC with a dark theme, specs, performance charts, pricing, FAQ, and smooth scroll animations, using no external libraries.

|Model|Completion time|
|:-|:-|
|GLM-5.3|10m 05s|
|Qwen3.8|14m 40s|

https://reddit.com/link/1x0b1ws/video/pdglw7kbl4uh1/player

3. Vampire-Survivors-style game

Exact prompt:

Build a single-file HTML vampire-survivors-style game with WASD movement, auto-attacks, enemy waves, XP, 3-choice level-up upgrades, HP, game over, and restart, using no external libraries.

|Model|Completion time|
|:-|:-|
|GLM-5.3|4m 40s|
|Qwen3.8|24m 10s|

This one had a much larger difference than I expected.

There is one obvious caveat:

the Qwen version added sound, while the GLM version did not.

The prompt didn't ask for sound, so I didn't go back and ask GLM to add it afterward. I wanted to leave both runs as the result of the same one-shot prompt.

https://reddit.com/link/1x0b1ws/video/4hl6g2bcl4uh1/player

Task completion times

|Task|GLM-5.3|Qwen3.8|
|:-|:-|:-|
|Tetris|4m 40s|5m 30s|
|Landing page|10m 05s|14m 40s|
|Vampire-style game|4m 40s|24m 10s|

I wouldn't read too much into three examples.

This definitely isn't evidence that GLM is “5x better at coding” or anything like that.

What I found interesting is simply that raw tok/s and end-to-end agent completion time didn't track each other very well.

Qwen has much higher PP and decode throughput, but on these particular tasks GLM often finished sooner.

For agent work, planning, number of retries, file rereads, edits, and how close the first implementation is to working all matter too.

So I think raw inference speed and actual task completion time are worth looking at separately.

Current state

The GLM service I'm using now is:

GLM-5.3-Flash EXL3 3.05bpw

• DFlash2 EXL3 6bpw K7
• Q8 KV
• 384K context\*\*

The main thing I like about this configuration is the memory/quality tradeoff.

The target fits in \~116.6GiB of HBM, stays resident, and the public 3bpw-class EXL3 fidelity results suggest the quant is holding up much better than I would have expected from the bitrate alone.

The runtime side is basically an HBM-first setup: keep target weights resident, then save memory on the drafter/KV side rather than moving experts back and forth during decode.

Full config, conversion scripts, ring-cache changes, benchmark code and raw results are here:

GitHub:
https://github.com/Flun/glm53-flash-cmp170hx-exl3

If anyone is running GLM-5.3-Flash on other weird 128GB-class GPU setups, I'd be interested in seeing what numbers you're getting too.

Next thing I want to try is 1M context, although at that point prefill time is probably the bigger problem than just making it fit.

For the last few weeks most of my coding has been done locally with Qwen3.8-Flash-Next, so I gave it and Opus 5.5 the same high complexity feature to build on LlamaStash (a complex and large Rust project) and compared the results.

Setup: ASUS ROG Flow Z13 (Strix Halo, 128GB), Flash-Next at xhigh effort with Pi as the harness. Opus 5.5 ran in Claude Code at medium effort. I wanted xhigh for Opus as well, but Claude changed it to medium when I picked the latest model and I didn't notice it until the task was done. But I think medium is probabbly a fairer setting anyway.

Task: add a llamastash daemon restart command that reuses the existing start and stop code. I kept the prompts vague on purpose and gave both the same prompts.

|Step|Opus 5.5 (medium) PR#88|Flash-Next (xhigh) PR#89|
|---|---|---|
|First iteration|~9 min|~38 min|
|Nudge to reuse the TUI restart code|~6 min|~34 min|
|A third duplicate path|found it on its own|~30 min, after one more prompt|
|Create PR|~3 min|~30 min|
|Total|~18 min|~130 min|
|Tokens (in / out)|7.83M / 41.5K|20.61M / 101K|
|Tests added|1|4 (2 of them end to end)|
|Cost|$7.53|$0 + ~0.15 kWh|

The end result was interesting. I asked GPT 5.6, Opus 5.5 and Flash-Next to review and compare both PRs (new sessions). GPT and Flash-Next picked the Flash-Next PR (#89) and Opus picked its own (#88). I also did my own review and found the Flash-Next one better as it had better tests and handled edge cases better. I ended up merging #89, after porting the fixes that the reviews picked from #88.

Keep in mind:

• Opus was on medium effort. With xhigh it would have used way more tokens, taken a bit more time and probably would have done a better implementation.
• Flash-Next ran on an older Halogen version (0.14.0), and Halogen dropped the connection once, so the last part ran on Gufo. The current Halogen does around 1,400 t/s prefill and 46 t/s decode on my laptop at 70 W, so I think the time will drop a lot if I redo the test.
• The $7.53 is what Claude Code reported for the whole Opus session, which includes a later fix to the PR. The 0.15 kWh assumes 70 W for the whole 130 minutes.

Opus is still 2 to 10 times faster and I still use it for planning and reviews. But the actual coding now happens on my laptop, and to me it is crazy that I can run a local model that can challenge a frontier model like this.

Full post with my setup, the engine benchmarks and a second task comparison: https://deepu.tech/local-ai-qwen3.8-flash-next-best-local-llm

Hi r/LocalLLaMA. I'm on the team at Blockway, a small team in Hong Kong (disclosure: this is our model). Today we released Agens Volundr 32B Preview, the first model built on our own hybrid architecture. We trained it on limited compute, it isn't perfect, and we'd rather tell you where it falls short up front.

WHY WE BUILT IT

Our customers run models on their own machines. At long context, the KV cache, not the weights, decides what fits. So we designed a model where most layers don't keep one.

ARCHITECTURE (72 layers, dense ~32B, every layer runs on every token)

• 54 KDA (Kimi Delta Attention) layers: linear attention with a fixed-size recurrent state, no KV cache
• 17 BCSA layers (our compressed-sparse attention): exact window over the last 4,096 tokens; older context pooled 4:1 into blocks, and a learned indexer reads the top 512 blocks
• 1 full-attention layer (layer 72)
• Engram: a hashed n-gram memory held in host RAM, attached at 2 of the 72 layers
• mHC: 4 residual streams instead of 1

So only 18 of 72 layers keep a KV cache. Context window: 262K.

SPEED (single user, our sglang build)

• BF16 on two 48 GB GPUs, decode: 25.1 tok/s at 1K, 24.1 at 8K, 24.1 at 32K, 24.0 at 64K, 23.9 at 128K
• BF16 prefill: 2,122 / 2,180 / 1,916 / 1,679 / 1,297 tok/s (1K to 128K)
• INT4 (31.7 GiB) on one 48 GB GPU, decode: 31.0 tok/s at 1K, 29.3 at 8K, 29.1 at 32K
• Aggregate throughput: 127 tok/s at 8 users, 130 at 16 users (BF16); 117 at 8 users (INT4)
• DFlash2 drafter (separate repo), single user, same server with it on vs off: up to 3.6x on JSON/tool output, 2.0x on code, about 1.6x in thinking mode. Not worth it above roughly 8 concurrent users.

BENCHMARKS (all run by us on one harness with the same settings, including the comparison models; full table and footnote on the model card)

• Ahead of Qwen3.8-27B on LiveCodeBench v6 (+4.2), HumanEval (+4.3), AIME 2025 (+2.9), MATH-500 (+1.6)
• Roughly level on MMLU-Pro, IFEval, GPQA Diamond
• Behind on agent tasks: tau2-bench 74.2 vs 79-80, SWE-bench Verified (50-task subset) 44 vs 58-64. Closing that gap is the main focus of the full v1, which continues pre-training to about 10B tokens and adds training on long agentic sessions.

KNOWN LIMITATIONS (please read before trying)

• Needs our sglang build. Stock sglang and vLLM can't load it yet.
• GGUF / llama.cpp is planned, not available today.
• Long agentic sessions are its weakest area in this Preview.
• It's still training; treat this as a preview, not a final model.

RUN IT

docker pull ghcr.io/blockwayz/agens-sglang:preview-sm89 (48 GB Ada GPUs)
docker pull ghcr.io/blockwayz/agens-sglang:preview-sm90 (H100 / H200)

The full launch command is in the model card.

LINKS

• BF16: https://huggingface.co/Blockway/Agens-Volundr-32B-Preview
• INT4: https://huggingface.co/Blockway/Agens-Volundr-32B-Preview-INT4
• DFlash2 drafter: https://huggingface.co/Blockway/Agens-Volundr-32B-Preview-DFlash2
• Project page: https://github.com/BlockWayz/Agens-Volundr
• Serving code: https://github.com/BlockWayz/agens-sglang

Apache-2.0. We're a small team, and the most useful thing you can do is try it and tell us where it breaks: an issue, a failing prompt, a benchmark you'd like us to run. We'll be in the comments.

q3.8-27b q3\_k\_xl this thing have done everything i possibly needed from him he wired up my openwebui spawned trillium service fix all my bugs set up pi-web-ui and debug my cloudflare tunnel it even spawned smaller LLMs to make the LLM use the toole he created to ensure it will work. I haven’t had a real-world task that i needed from it that failed yet. Im pretty sure if im building large-scale production code with tons of lines of code it will struggle but as a utility to make all my scripts and diagnostics on my micro-services this thing is unstoppable. Weirdly im not even using q4 im using q3\_k\_xl appreciation to unsloth also for making such reliable ultra low quantisation. His UD3.0 style of quantisation is PURE magic 🪄 sometimes i go down to q2\_k\_xl if i need extra context window and that thing STILL delivers 🎉🎉 alibaba had handed down Prometheus fire to common men like you and i. Can’t wait for qwen4-27b

So PewDiePie decides to fine-tune a local AI model called Ajax on his own computer. Pretty normal stuff for local model fans.

To make his dataset, he uses OpenAI's API. OpenAI catches him using their outputs to train another model, flags his account for breaking their terms, and bans him.

He files an appeal, gets unbanned, goes right back to pulling data from the API, and immediately gets banned a second time.

So instead of giving up, he uses open-source tools to remove the model's built-in refusals, cleans out the preachy fluff, and starts building a fully local 9B agent.

OpenAI spent years scraping the whole public internet for free data, but the second someone uses their output to train a local file, it's an emergency ban.

In trying to enforce their rules, all OpenAI really did was give open-source models a massive free advertisement to millions of people.

What a time to run models on your own hardware.

I just don’t get the point of running it locally when with all this money you could buy like 20 max20 subscriptions with room to spare..

How is JEV any different from using an embeddings model? I really will appreciate if someone can explain this to me - because I have yet to see the difference.

I'll even give you my JEV server for free! It uses ollama, you install \ollama pull nomic-embed-text:latest\.

% python3 ./jev_embedding.py "How high is the sky?"
find_phone: 0.38
volume: 0.41
calendar: 0.44
tell_the_time: 0.49
weather: 0.53

% python3 ./jev_embedding.py "I had this thing on my anus. The doctor burned it off with a laser."
weather: 0.35
tell_the_time: 0.36
calendar: 0.37
volume: 0.38
find_phone: 0.43

% python3 ./jev_embedding.py "can you help me locate my phone."
volume: 0.38
weather: 0.40
calendar: 0.43
tell_the_time: 0.53
find_phone: 0.89

% python3 ./jev_embedding.py "Hello Cleveland! I can't HEAR you"
weather: 0.37
calendar: 0.40
tell_the_time: 0.44
find_phone: 0
volume: 0.56

#!/usr/bin/env python3
"""
jev_embedding.py — minimal showcase of the embedding-based intent router,
excised from jarvis_workflow.py.

Given a phrase on the command line, it embeds the phrase and every example
utterance (via the local Ollama embedding model), then prints the cosine
similarity of the phrase to each intent — the raw routing signal — instead of
running a handler and speaking an answer.

python3 jev_embedding.py "How high is the sky?"
"""

import sys
import requests

# --- Config (same endpoint/model as jarvis_workflow.py) ---
OLLAMA_EMBED_URL = "http://localhost:11434/api/embeddings"
INTENT_EMBED_MODEL = "nomic-embed-text"

# --- The five cases to detect ---
# label -> example utterances, matched by similarity.
INTENTS = {
"volume": [
"turn the volume up",
"make it quieter",
"set the volume to seven",
],
"tell_the_time": [
"what time is it",
"can you tell me the time",
],
"weather": [
"how's the weather going to be today",
"will it rain today",
"do I need a raincoat",
],
"find_phone": [
"find my phone",
"where's my phone",
"ring my phone",
],
"calendar": [
"when is my next meeting",
"what's coming up on the calendar tomorrow",
],
}
def _embed(text):
"""Return a unit-normalised embedding (list of floats) from the Ollama model."""
r = requests.post(OLLAMA_EMBED_URL,
json={"model": INTENT_EMBED_MODEL, "prompt": text},
timeout=10)
vec = r.json().get("embedding")
if not vec:
raise RuntimeError("no embedding returned")
norm = (sum(x * x for x in vec)) ** 0.5 or 1.0
return [x / norm for x in vec]


def _cosine(a, b):
"""Cosine of two unit vectors is their dot product."""
return sum(x * y for x, y in zip(a, b))


def score_intents(text):
"""Best cosine similarity of text to each intent's example utterances."""
q = _embed(text)
return {label: max(_cosine(q, _embed(ex)) for ex in examples)
for label, examples in INTENTS.items()}


if __name__ == "__main__":
if len(sys.argv) < 2:
print('Usage: python3 jev_embedding.py "your phrase"')
sys.exit(1)

phrase = " ".join(sys.argv[1:])
scores = score_intents(phrase)
for label, score in sorted(scores.items(), key=lambda kv: kv[1]):
print(f"{label}: {score:.2f}")

The cool part is

No training was needed.

No hacking of the game state or algorithms needed

Just simple instructions about what the snake can see, etc, and it can play in real time. 135ms is the turn limit of Google snake, so basically could be a human playing.

Ofc, it could be improved to be a perfect snake player, but thats not the point.

This can be used in other games where decisions need to constantly be made.

Running Clef Flash (9B model at Q4 on an RTX 5080)

Hey All,

I spoke to a 0-day partner of Alibaba today and he casually mentioned (didnt know if he was allowed to) that Qwen 4 is apparently planned for the end of October.

To me, this is way faster than expected as there was quite a gap between 3.6 and 3.8.

I tried to get more information out of him regarding which variants will come first and he got a bit cagey.

BUT: No matter the order of the variants, we can hope for Qwen 4 27B this year!

EDIT: I know this is very much "in bro we trust" but i am also just trusting bro from the Alibaba partner. Together we trust in Bro.

Updated post here about dockerization and zero-config Cloudflare tunneling for easy setup

Hey everyone,

I’ve been working on a game called Anyworld. It’s a browser-based multiplayer (single player also supported) text adventure inspired by the early days of AI Dungeon, especially its browser-based free version AI Dungeon 2.

The setup is pretty straightforward: one person hosts the server and runs the model via llama.cpp (OpenAI or other cloud APIs are also supported, and great for non-English play!), and your friends join through a browser link. The host sets the scene and the goals, players type out their actions, and the LLM acts as the DM to resolve the chaos and drive the story.

Admittedly the host requires some technical skills with Python, and possibly with networking (opening routes to the hosted game via VPN, port forwarding etc.). I'll work on this as well as the development continues. Using Docker was suggested in another subreddit, so I'll definitely consider that, as it would allow including both the llama.cpp backend, recommended model and configurations etc., in addition to the game itself.

Instead of pasting the entire repo documentation, here are the main features right now:

How it plays

• True multiplayer resolution: Players submit their actions, and the model resolves the whole round together. It actually accounts for characters interacting or getting in each other's way.
• Real dice rolls: When an action is uncertain, Python handles the actual RNG math. The model just takes those hard dice results and narrates the consequences.
• Custom scenarios: You write the setting, characters, and opening state. It isn’t limited to fantasy.
• Party chat: There's an OOC chat separate from the game events so you can talk without the LLM reading it.
• Zero setup for players: No one but the host needs to install anything or run a model. It works on desktop and mobile browsers.

DM Tools & Hidden Mechanics

• Private DM guidance: As the host, you can feed the model hidden info; NPC motives, secret rules, or where you want the story to go.
• Secret triggers: You can set up one hidden percentage roll per game (e.g., If a player enters a building, there's a 20% chance the building collapses on the player). Python rolls the probability in the background, and if it triggers, the model weaves the consequences into the story without showing the players the underlying math.

Under the Hood & Memory

• Context management: It budgets the context window and uses a structured memory system. Older rounds are compressed into world states, player facts, and unresolved threads. It also does a secondary model pass to audit those summaries so it doesn't accidentally delete important facts.
• Language support: If you use the OpenAI backend, you can play in non-English languages (the narration and outcomes will naturally follow whatever language you wrote the scenario in). Note: The local llama.cpp backend currently instructs the model to narrate in English. This is because the local models my development PC can run were terrible with any other language than English.
• Session recovery: Disconnected tabs auto-rejoin. If someone accidentally closes out, they can log back in and their unfinished actions and history are waiting for them.
• Self-signed certificates for HTTPS-enabled connections: The game creates self-signed certificates upon launch, which enable encrypted connections. The problem with self-signing is that joining players receive a warning that the site may not be secure. However, most browsers allow the players to continue to the game despite the warning. This is a suboptimal way to handle HTTPS, so I'll work on a more robust solution at some point.

It’s still a work in progress. Right now, a server only runs one game at a time, and if you restart the server, the live session is lost (it generates HTML/JSONL transcripts, but they aren't loadable save states yet). The overall story quality is also going to heavily depend on which model you use and how you tweak the settings.

Suggested model:

During development, I used llama.cpp and Gemma 4-26B-A4B Q4 with a context size of 128k and found it to be more than an adequate backend for functioning as the DM. Even the speeds are fast enough with my RTX 5070 Ti 16 GB that round resolutions take only 5 or so seconds.

The specific model I used and can recommend: https://huggingface.co/EZForever/gemma-4-26B-A4B-it-qat-uncensored-heretic-UDmerge-GGUF (the model was great at following instructions and remembering plot points even with longer contexts)

Recommended parameters for Gemma 4 models:

• temperature 1.0
• top-p 0.95
• top-k 20
• min-p 0.0
• presence-penalty 0.0
• repeat-penalty 1.0

Of course, feel free to try your own models! The repo contains a benchmark file that tries to measure how well the running model follows the game's requests.

AI use disclosure:

I used Alibaba Cloud's Qwen 3.8 27b and OpenAI's GPT-5.6 Luna and GPT-6 Astra models to help develop the game.

How to run:

Read INSTALL.md to set up, configure and run the game. README.md contains some details on how the game functions.

I'll post the link to the repository in the comments.

Some gameplay in Finnish with OpenAI's Luna:

https://preview.redd.it/08f8zljik8th1.png?width=1837&format=png&auto=…

The game is MIT licensed, so open source all the way. Forking or collaborating is encouraged.

I'd love to hear some feedback, and I hope someone finds the game fun to play!

(UPDATE) Some things I've added:

• Save and reuse scenarios: The host can save, load, and delete scenarios in their browser. Scenarios are stored in the browser's localStorage and stay completely local.
• Browse and export History: Easily search previous public events for forgotten details. Also exportable as JSONL.
• Multilingual play: With the OpenAI backend, narration follows the language of your scenario.

came across a pretty interesting technical breakdown of chatgpt's newly launched "intelligent ui" feature, and thought this subreddit might find it interesting.

for anyone unfamiliar with the concept, intelligent ui is essentially openai's take on generative ui. instead of restricting llm responses to plain text or markdown, the model can compose actual interactive interfaces in real time.

there are different approaches to making this work. some systems let the model choose and compose elements from a predefined component library, while others allow it to generate entire interfaces on the fly (basically writing html/react code and rendering it inside an iframe).

it's more of a spectrum than a single technique. projects like openui, vercel's json-render, google's a2ui, and now chatgpt's intelligent ui all sit somewhere along this spectrum, with different trade-offs in flexibility, reliability, performance, and how much freedom the model gets.

but that's not even the most interesting part.

These folks managed to reverse engineer chatgpt's implementation in less than 24 hours after launch!

what's particularly impressive is that they claim to have done this entirely through publicly observable behavior, without access to openai's internal codebase.

from their write-up:

“All observations come from our own ChatGPT accounts, from the traffic the ChatGPT web app generates, and from the JavaScript that chatgpt.com serves publicly.”

found this pretty fascinating from an engineering perspective, especially considering how quickly they managed to put together a breakdown of how the system works.

and then there's the funnier part.

the same team released something called open intelligent ui, which is a pretty obvious jab at how openai isn't really "open" anymore. the joke works even better when you realize these guys actually own the domain openui.com lol.

the idea they're pitching is that you can recreate experiences similar to chatgpt's new intelligent ui inside your own applications using their open source framework.

and here's where it gets particularly interesting, you can technically do all of this with local llms.

since openui is model agnostic, you can integrate it with local models through ollama, lm studio etc. it's not necessarily a one click, out of the box recreation of chatgpt's experience, but from what i understand, the underlying pieces are there to build something similar that runs entirely locally.

i initially came across these folks through a viral twitter post comparing chatgpt's intelligent ui with openui's generative ui, and ended up going down a rabbit hole reading about the different approaches to generative ui.

some helpful links for anyone interested:

• openui: https://github.com/thesysdev/openui
• open intelligent ui: https://github.com/thesysdev/open-intelligent-ui
• decoding chatgpt's intelligent ui: https://x.com/rabi\_guha/status/2108238432355123572
• chatgpt intelligent ui vs openui generative ui: https://x.com/KashyapVisharad/status/2108239102676250933
• json-render: https://github.com/vercel-labs/json-render
• a2ui: https://github.com/a2ui-project/a2ui

would love to know what everyone here thinks about generative ui in general.

is this actually a useful direction for llm interfaces or is it another one of those things that looks amazing in demos but doesn't translate particularly well to real world applications?

i'm especially curious about the local inference angle. with smaller models getting increasingly capable, do you see a future where something like this becomes practical entirely on device? or is the additional complexity, latency and structured output overhead simply not worth it compared to a conventional ui?

local llama has been my go to subreddit for years whenever i come across something interesting in the llm space, so genuinely curious what the general opinion here is.

would love to hear your thoughts, especially if you've tried building something similar with local models!

JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF

A small moe!

I spent the last few weeks on a hobby research project and just made it public.

The idea isn't new (product-key memory, Lample et al. 2019, and Meta's "Memory Layers at Scale"): give a model a huge table of learned vectors and let it read only a few hundred of them per token. I wanted to know what that's actually worth on a small model, what it costs, and whether the table even has to sit in VRAM.

What came out:

\- A 21M model with a 16.8M-row table (6.4B parameters in the table, 33M used per token) is about as good as a 114M dense model trained on the same 500M Wikipedia tokens.

\- The table doesn't need VRAM. With the 4-bit table memory-mapped from an NVMe SSD the model still writes \~140 tok/s on my RX 9070, using 0.4 GB of VRAM. Reading long prompts from the SSD is slow though, every missed row costs a whole 4 KB page.

\- I wrote Triton kernels for it. They run unchanged on my Radeon, an MI350X and H100/H200.

\- Bolting a table onto a finished model (Qwen3.5-0.8B) didn't work: no better than a small dense add-on with the same compute.

Caveats: it's tiny, one seed for the big runs, and the text it writes is fluent Wikipedia English with made-up facts. I wrote down the success criteria before every run, and the stuff that didn't work is in there too.

Most of it ran on my gaming PC, the big runs cost about 70 dollars on Runpod. I built it together with Claude Code (you'll see it in the commits), the ideas, decisions and money were mine.

Repo: https://github.com/re133/sparse-memory-lm

Click a word and see which table entries the model reads: https://re133.github.io/sparse-memory-lm/explorer/

Model: https://huggingface.co/fechyy/sparse-memory-lm-B-16M

Feedback welcome, especially if I got something wrong. And if anyone has bigger GPUs to spare, I'd love to try this at 1B scale.

I've been wanting to test out LLM inference on FPGAs for a while now, but didn't have a big reason too since there were no frontier class models at 9B-27B scale (3.6 27B was great still but it didn't motivate me enough). When 3.8 was released I was pretty impressed by what it could at that size. So I began looking for cheap FPGAs that could hold the model. I found SQRL FK33 (280$) (8GB HBM2 \~400GB/s BW) and thought I could possibly run it with multiple cards, I'd been working on llama.c inference on a smaller AXU3EG FPGA dev board before that and primarily used Opus 4.8 (CC) for the implementation with some architectural input.

With the FK33, was able to run 3.5 9B at 2tok/s at 75MHz (higher clocks need some more RTL optimization and higher core voltage). Decided to use Fable/Opus5,5.5/Kimi K3 for the Qwen implementation, once FK33 was proven I decided to get the SQRL Jungle Cat ex-mining FPGA 375$ (two FK33 with 8 GTY lane interconnect, Ethernet bitstream load) since it would allow higher prefill and generation performance (twice FK33 fabric per XCVU35P), however the Jungle Cat Lite board seems to not have a way to load weights faster unless I do a PCB resin and it lacks the clock generation for the GTY lanes (this is easy to fix by soldering some components which were easy to figure out).

Working towards the ideal FPGA inference engine with 4x XCVU35P (32GB HBM2) but that requires a custom carrier board. Some results:

Qwen3.5-9B INT4 on 2x FK33 at 75 MHz (pipeline split, the host carries the residual between cards):

\- Prefill: \~6 tok/s (256-token prompt), \~5.5 tok/s (2.3k-token prompt)

\- Generation: \~3.2 tok/s near the start, \~2.4 tok/s at 2-3k context

\- Output checked against llama.cpp layer by layer

Estimated: Qwen3.8-27B INT4 (modelled from the measured 9B per-op profile; none of this has run yet):

\- One Jungle Cat (2x VU35P) running the FK33 design as-is at 75 MHz: \~2 tok/s prefill and \~1.1 tok/s generation at short context, \~0.5 tok/s at 16k.

\- Same two dies with the RTL resized to the bigger die, still at 75 MHz: \~6 tok/s prefill and \~3 tok/s generation (\~5.5 tok/s with tensor parallelism across the two dies), \~1.1 tok/s at 16k.

\- Resized and at 200 MHz (scaling linearly with clock): \~16 tok/s prefill and \~8 tok/s generation (\~15 tok/s tensor-parallel), \~3 tok/s at 16k.

\- 4x VU35P with 4-way tensor parallelism at 200 MHz: \~25 tok/s prefill and \~25 tok/s generation at short context, \~10 tok/s at 16k, and \~1 tok/s at the full 262k context.

Two dies top out around 45k context because the 27B's KV cache doesn't fit beside 14.5 GB of weights past that; four dies are needed for the full 262k.

Overall it's been a fun project so far, mainly focused on figuring out a way to load the weights on the Jungle Cat board, and open to suggestions. Also, I got 2 BC-250s for 60$ and 75$ some time ago and they've been an insane performance/cost purchase. Pics showing BC-250 programming the Jungle Cat. Thought people here might find this project interesting

Another thing I've always wanted to do something like tinytapeout (build the RTL on an ASIC so clocks can go up, power down and so I asked Opus 5.5 to estimate that but on TSMC for 2023 process node lol: On TSMC's 2023 N3 node with six stacks of that year's HBM3 (4.9 TB/s), this RTL as an ASIC at 2 GHz would run the 27B at roughly 294 tok/s at short context, 106 tok/s at 16k and 10 tok/s at 262k, for about 125-340 W!

Repo here (MIT): https://github.com/Nero7991/llm.vhdl

I've been looking at the <32B model space and keep coming back to an interesting question.

A few years ago, projects like Hummingbird+ suggested that cheap custom accelerators (FPGA-based) might become the future of local inference. But today it seems like memory capacity is still the real bottleneck rather than raw TOPS.

For someone who wants to run modern 20B-32B models at reasonable quants (Q5/Q6 rather than INT4), the options all seem compromised:

• Consumer GPUs have great bandwidth but limited VRAM.
• NPUs and AI accelerators often have lots of compute but not enough memory.
• FPGA solutions are fascinating but still bandwidth-constrained.
• Strix Halo systems (GMKtec EVO-X2, Framework Desktop, etc.) offer huge unified memory pools, but they're expensive.

The "dream" accelerator would be something like:

48+ GB memory
500+ GB/s bandwidth
under $1000
reasonable power consumption

...but I don't think anything like that actually exists yet.

For those who have used Strix Halo systems for local inference:

How do they feel with current 20B-32B models?

Do you regret not buying a used 3090/4090-based machine instead?

Is unified memory a bigger advantage in practice than benchmarks make it seem?

Curious what people who own both types of systems think.

Folks, like many of you, I used to look at the Strata posts and was extremely skeptical. But yesterday, with the help of DeepSeek 4.1 Flash, I compiled Strata on my machine, and honestly I'm blown away by the speed.

With llama.cpp master I got a maximum of 700 t/s PP and 23 t/s TG. With Strata, using Unsloth's UD-Q3\_K\_XL quant, I'm getting \~1,650 t/s PP and \~38 to 61 t/s TG depending on context, with no tool-calling errors, everything running great in OpenCode at KV fp16 and 256k context. Phenomenal, and partly unbelievable.

I'm not a programmer. I just "vibe" with AI. People say Strata is a mess; whether it really is, I don't know, but my initial experience has been amazing. From here on out, it's AI. I asked it to summarize the data and what it did to run the Unsloth quant on Strata.

By the way, the quant that Strata downloads and recommends, I didn't like it. It threw silly errors and seemed to have lower quality, though it was also even faster. For my use case I prefer to keep Unsloth's, because it's better: a bit slower, but more accurate for my workloads.

Hardware summary

• GPU: NVIDIA RTX 3090, 24 GB (compute capability 8.6)
• CPU: Intel i5-12600K (10 cores / 16 threads)
• RAM: 128 GB DDR4 @ 3600 MT/s (XMP on)
• Storage: two NVMe SSDs (system + models)
• Power limit: 315 W (card max 365 W)
• CUDA: Toolkit 13.4; compiled for sm_86

Strata stats (Unsloth UD-Q3_K_XL)

|Metric|Strata|llama.cpp master|
|:-|:-|:-|
|PP (prompt)|\~1,650 t/s (≈1,690 at 180k)|up to 700 t/s|
|TG (generation)|38 t/s at 182k context; \~61 t/s short context|23 t/s|
|Context / KV|256k fp16|180k f16|
|Expert cache hit|\~76%|n/a|
|Speculative (MTP) accept|\~76%|n/a|
|Tool calling|no errors, working in OpenCode|n/a|

Quant used: Unsloth UD-Q3\_K\_XL (dynamic quant). Not the quant Strata recommends by default. That one was faster but produced minor errors and (subjectively) lower quality; Unsloth's was chosen for accuracy over speed.

Adaptations needed (quant + Strata)

On the quant:

• Packed with --compat-bf16 (some tensors Strata reads as BF16).

On Strata (recompiled / reconfigured):

• Rebuilt for sm_86 with MMQ (-DSTRATA_MMQ_KQUANTS=ON). This doubles Q4-class prompt speed.
• Disabled STRATA_PF_FUSED=0 in the configs. The fused kernels crashed (illegal memory access) on quantized experts whose "down" type is unsupported.
• Vision encoder moved to the GPU: recompiled strata-vision with CUDA (was CPU-only) and set vision.gpu=true \+ --vram-reserve-mib 700 in all configs.
• Per-model calibration (--pcie-frac, --pool-workers, --spec-min-p) and --expert-profile-save to learn and persist the expert cache.

Adjusted config (this model): context 256k, --kv fp16, --kv-resident 32768, expert cache auto (6,517 slots / \~14 GB), --spec 4, --pcie-frac 0.00, --pool-workers 9, --spec-min-p 0.70.

Every single pro AI space is an accelarationist echo chamber that will not allow any kind of post if it's not, essentially "holy shit new model just dropped and it's AMAZING I love AI SO MUCH" Even if you are pro AI, you are not allowed to make any other kind of post. Questions if local will ever be as good as frontier in one area? Thoughts on what the government wants to use it for? Fears that it will be heavily regulated and taken away from us? Discussions on the bills they want to be passed and why I think that's bad? Hope that one model will reach the capabilities of another model in one area? Not allowed. None of it. Can't post any of this. If you do, you will be insulted, called stupid, spammed, dogpiled on, have your posts deleted by mods and perma banned, and downvoted to the pits of fucking HELL. They are ALL like this. LITERALLY ALL OF THEM. Every single fucking one. There is not ONE AI sub that does not operate like this. I liked r/singularity. For a while. Until I made a few posts: I don't think local will be as good as frontier I worry about what the government will do with AI and that they will limit our use of it heavily Will \_\_\_\_ ever be as good as \_\_\_\_? Now I am hated in that sub. Any time I make ANY post, I get downvoted to the pits of hell and all the replies are just insulting me and calling me stupid. And the MODS have even joined in, and no matter what my post is about, even if it's just a discussion on capabilities, they will delete it the INSTANT they see that I have posted it. They will not respond to modmail, they won't give a reason, they just delete it. Every. Single. Time. ALL AI subs operate like this. There is NOWHERE to have a real conversation. NONE. I'm so tired. I just want to talk about AI and I fucking CAN'T.

Setup: GPU box with 1x H200-class card now; can allocate more (up to several H200s) if it clearly buys better results. Inference via vLLM or similar, OpenAI-compatible endpoint. Coding happens on a separate non-GPU Linux VM on the same network - IDE/harness there, calling models over LAN. Outbound internet is fine for packages/extensions, but inference stays on our hardware (no code to external model APIs).

Constraints: on-prem only, open weights, permissive license preferred (Apache/MIT), not Chinese-origin including base models (I understand a lot of fine-tunes are Qwen underneath), NATO-country lab preferred. Use is general software dev plus security tooling and code analysis - repo-level agentic work on existing codebases (fix / extend / refactor / test) is the core, with a human reviewing diffs. 2-3 concurrent users max.

Two things I'm trying to work out:

1. Capability tiers vs. VRAM. On one card candidates seem to maybe be something like Cohere North Mini Code (30B MoE/3B active), Mistral Small 4 (119B MoE/6B active), Nemotron 3.5 maybe as a generalist baseline; Gemma 5? The Vibe Code Bench results suggest small open models fall over on long E2E builds, where only Large-4-class (4-8 cards) and closed models seem to hold up. Is that your experience? Where's the step-change for agentic repo work on an existing codebase - does 30B-class -> 120B-class matter much, or only the jump to 500 GB+? We could get more compute for something like Command A+ or Mistral Large 4 if it's actually better for code rather than just for E2E.
2. Heterogeneous multi-agent. Does a big planner/reviewer (Large 4 / Command A+ class) plus small fast executors (e.g., North, Small 4) actually beat a single mid-size model, with a human approving plans and diffs? Which harnesses handle routing sub-agents to different endpoints well - OpenHands, OpenCode, Pi, VS Code extensions (Cline/Roo/Continue), Codex CLI in local-model mode?

Harness/IDE: something that supports multi-agent workflows (planner / executor / reviewer agents checking each other / etc.) but would also like humans to be able to step in, review diffs and edit by hand.

Anyone running something like this? Which model + harness combo is holding up best as of late? Thanks!

EDIT: thanks all - adding Poolside Laguna S 2.1, Muse Glimmer 30B, Inkling-Small (2x H200), K2 Horizon (checking lineage), Gemma 4 31B and Reflection Beam (501B MoE / 23B active, Apache 2.0, weights due this month) to the candidates; Ornith is Qwen-based so out. Pi added to the harness list.

Running oQ4e+MTP on oMLX 0.7.0, with still more to optimize.

Prefill is 1,878 toks.

I saw some other benchmarks below what id expect so i figured I would share.

Hello llamas. I am posting this because I believe that, despite it being closed source models, the discussion will bring value to the local AI community.

As many of you probably heard, GPT-6 Astra is speculated to be a looped transformer architecture that outputs a token after multiple forward passes instead of one. This allows a model to essentially have more effective depth due to recurrence, making more use of the weights at the cost of more compute.

Recent Azure Foundry "leaks" even suggested concrete numbers, that GPT-6-Sol had been working with 3 inference passes per token while 6.1-Sol only needs 2.

Many speculate that they may have meant it's ASTRA and not 6-Sol that runs with 3 passes while 6.1-Sol is essentially the same model with 2 passes instead.

So I did some back-of-the-envelope math to see if the numbers add up. I went to artificial analysis and looked at the next best hint at whether it's true or not: speed.

I know it doesn't prove it, but hear me out. If you look at the image, it shows something interesting:

\- GPT-6-Sol and 5.6-Sol: \~100 tok/s

\- GPT-6.1-Sol: \~60 tok/s

\- GPT-6-Astra: \~60 tok/s

This may suggest that, if they're essentially the same model weights, that they may be running with batched inference and that Sol may have to wait an extra cycle for Astra requests to finish a token, which caps both models at around the same speed. Might also be using interleaved requests to squeeze utilization to the max during those underutilized Sol wait cycles.

But then I also realized that 5.6 Luna was between 126-137 tok/s and then 6.0-Luna dropped to around 110-115. Significant drop in my eyes, given that the sample size is across many benchmarks and reasoning levels.

Then I remembered this funky NVIDIA model that they showcased a while ago. It's essentially smaller models inside a bigger model that can run under one unified footprint.

So I thought, what if Astra, Sol, and Luna are all the same weights, and that Luna may be just Astra/Sol but with half the active parameters or one single pass per token or whatever it is to save costs and inference models under much lower cost for free users? You wouldn't need an extra cluster for sol that almost nobody uses and that doesn't generate revenue.

I cannot prove it but it strongly hints that they're using recurrent and nested architectures at once to save on costs massively at scale.

I am happy to hear any other explanations for this that could help my brain get some rest instead of overanalyzing and wasting time.

Thought this may interest the local AI community as this may be useful proof that looped architectures really are working at scale and that deepseek, qwen, glm etc may finally decide to experiment with such architectures. Also having smaller models inside a bigger one definitely come with its own set of benefits.

PS: fully human generated text. 0.7 tokens per second. \~100T parameter wetware model. Running on two coffees and a muesli bar.

Running a model on your own machine used to be a privacy story with a quality tax. On this generation that trade has narrowed to where we can state it plainly: for daily work, the local model is good enough. We measured it — 25 paired tasks across four workloads, same prompts, one strong independent judge — and that is the top line:

• quality: 89.5 vs 92.6 on a 100-point scale (local vs cloud), with the 12-item suite splitting six wins each;
• completion: every coding run finished green on both models — 10/10 agentic runs fully green (20/20 visible tests, 4/4 hidden checks, tests untouched), and the bug-fix loop fixed all 4 bugs identically in 5/5 rounds each;
• speed: 2.5–5.5× the wall clock, depending on the workload, with 95%+ of the local time going to model generation;
• cost: the local runs cost nothing beyond electricity. The cloud side of the 12-task suite cost 0.14 credits.

Benchmarking an LLM here with a NVIDIA RTX 5070 12 GB VRAM here

I had been working on a llama.cpp based expert streaming setup for Qwen3.8-Flash-Next 177B (UD-IQ3\_XXS) on Windows. Benchmark is about 11.5 tok/s, up from roughly 7 tok/s on the inherited setup. In normal conversations I’ve seen 14–15 tok/s, and a long coding prompt generated 4,892 tokens at 10.15 tok/s and produced a working single-file Snake game.

Hardware: RTX 5070 12GB
32GB DDR4-2400
Ryzen 5 5600GT PCIe Gen3 Windows

The main gains came from fixing Windows I/O queue-depth issues, using one file handle per worker, and building a page-locked hot-expert tier so the GPU can pull hot expert weights more efficiently.

(In the video its around 16 minutes for 10k tokens and 10.41 tok/s

Output is quality gated against the control model and the published benchmark uses a heat file built from a separate prompt set.

Demos:

https://www.youtube.com/watch?v=cOPumMlyj\_4

https://www.youtube.com/watch?v=rc-uTjVpXM8

In the GitHub I have things I've tried that didn't work and benchmark scripts, and methodology. If you guys have suggestions especially for streaming please let me know

recently saw a bunch of open decision models pop out of nowhere in the last two weeks (laya, liquid's d1, cloudflare's clef-flash, interfaze's lev), so I wanted to see how far apart they actually are on the same GPU(yes, model size is a huge factor, but still isn't the only factor). all four had the same task of reading nine wikipedia articles about centipedes (9,534 words) word by word and flag every word that names a centipede. one /v1/systemone call per word, the next word goes out the second the answer comes back

request for every word:

{"state": "Word: \"Scolopendra\".", "questions": {"centipede": {"type": "noul", "instructions": "Does this word name a kind of centipede?"}}}

|model|weights|engine|per word (p50)|words in 32s|accuracy|centipede names caught|wrong picks|
|:-|:-|:-|:-|:-|:-|:-|:-|
|Laya|Laya-BF16.gguf|llama.cpp b11495|3.9 ms|7,980|97.4%|70%|98|
|d1 3B|d1-3B-AD-Q4\_K\_M.gguf|llama.cpp b11495|6.0 ms|5,306|96.5%|51%|51|
|Clef-Flash 9B|Clef-Flash-Q8\_0.gguf|llama.cpp b11495|24.4 ms|1,292|97.2%|36%|2|
|Lev 4B|interfaze-ai/lev, bf16|lev serve (PyTorch)|51.0 ms|626|98.9%|83%|4|

laya and d1 gap the other models in speed, though not so much on accuracy (yes, it does say 95%, but even saying "no" counts as a correct answer, so that's where the high acc comes from). what everyone might care about more is how well each one did their respective task and lev catches the most while being 13x slower than laya, partly because it runs in its own pytorch server instead of llama.cpp (it measured 68 ms on a different 4090, so it's CPU-sensitive too). but in the end Laya is the fastest model overall, and considering how easily it can be fine-tuned for any use case I'd say that be my go to pick

setup:

• GPU: rented RTX 4090 (driver 580.119.02, 32 vCPU)
• engine: llama.cpp b11495 (commit 37ac63456, CUDA 12.8 release build), -ngl 99, everything else default
• Laya, Clef-Flash: the ggml-org GGUFs
• d1: our own AD-Q4\_K\_M quant (atomic.chat), runs natively on /v1/systemone since the lfm2-d1 support landed in #30110
• Lev: interfaze's LoRA on Qwen3.5-4B in its own lev serve, default settings (--compile never finished warming up)
• latency: end to end from a Python client on the same box over localhost

My thinking is for a boom to be over, we need at least three iterations of updates that fail to wow us. Unfortunately, the new LLMs continue to wow us in all levels in the last iteration:

1. Astra was found to be useful in Blender. This opens up a new and big application.
2. Deepseek 4 Flash 0731 makes 2x Sparks useful and push up Spark prices.
3. Qwen3.8-27B pushes up prices of 3090 et al.
4. An unreleased OpenAI model "solved" the Navier Stokes problem.

So for the time being, to keep up with the hardware prices, the best bet is to follow the flow to buy AI stocks and use the proceed to buy hardware.

A not so obvious good news is that we are seeing OpenAI and Anthropic advocating a slow down. That means they are finally seeing a diminishing return. (or just a ploy to slowdown Chinese development? but I doubt US laws can be that far reaching) That can be a sign of light at the end of a long tunnel.

What do you think?

I generally just ask it "describe the main cast of (insert somewhat known cartoon show from the 2010s)", could either be Totally Spies, Randy Cunningham, Slugterra etc, most models in the 30b range completely fumble, looking at you Qwen, but the ones that manage to answer that are gems that can actually hold a human conversation.

Preamble and disclaimer, Sample size: 8, at any size of form this research is just screwing around being writing it down.

This was just a little,(well... big), curiosity test I wanted to run to see whether everyday folks could actually differentiate between high end AI models. In my country, the general reception toward AI is fairly neutral. People are neither strongly anti-AI nor boot licking about it, except for a noticeable distaste (rightfully so), toward lazy, overburnt style use of gpt AI gen images for product listings. After the experiment, I asked the participants for permission to publish the results.

\------ Anyway ------

For the experiment, I initially used both my local models and OpenRouter. However, after the first two weeks, I dropped OpenRouter because, surprisingly, my RTX 3090 was barely being hit. Usage mostly came in short bursts of around 5-6 reqs/hour. Although i use 96G of my ram for warm KV store, and the rest of cold KV are on SSD

I told the 13 participants, translated roughly: "I got access to the latest chatgpt model for free, but only for a limited time. You no longer have to deal with things like 'memory is low,' but you have to use my website because I had to wire it up to OpenAI. Also, don't ask anything weird. I don't want to, but technically I can see your activity logs in the database." Based on the IP traffic patterns, I think they (8 people) somehow bought it.

The web UI was basically a gray-themed OpenWebUI instance modified by Qwen 3.6 27B, with the admin panels hidden.

The experiment itself was simple. I rotated between Qwen 3.6 27B, Qwen 3.6 35B-A3B, Gemma 26B-A4B, Qwen 3.5 9B, Gemma 12B, Gemma E4B, and Gemma E2B. Reasoning effort was parameterized into four levels: instant, low, medium, and high. All ran on VLLM except for 35B

The goal was to find out whether people would notice meaningful differences between the models, complain about quality, or develop preferences without knowing which model they were actually using.

Complaints started appearing at the 9B level and below. The most common complaint was basically that the model "does not get it." Unsurprisingly, everyone preferred the responses from the Gemma models. While the other models ,where 3 participants even said that sometimes the model "thinks too much" and ends up sounding like a confused robot. We probably know which model they were talking about.

last pic is from Q3.6 27B OpenWebui restyle

Across 7,912 requests, only 51 used high reasoning. Around 4,588 used instant, 2,263 used medium, and the rest used low. So, yes, the overwhelming majority of usage was nowhere near high reasoning, i already told them there is a toggle to set it highest thinking mode.

Interestingly, some participants still described the models as top of the line because they could see the reasoning process. One comment was roughly:

"Wow, this model is really observant about its own behavior because it thinks very carefully."

At the end of the experiment, I ran an LLM judge using Qwen 3.8 27B to categorize the requests.

Around half were related to writing documents, including things like drafting documents and generating excel style tables. Next is were grammar-related requests. This category overlapped somewhat with document writing, and the LLM judge reported fairly high uncertainty when classifying them. Roughly 1/3 of the grammar-checking requests could reasonably have been placed in the document-writing category instead. Most of the remaining requests were basically "Google search" type questions, welp i paid for sonar credit for the most overkill cooking recipe question.....

Almost nobody used it for coding. There was only one notable coding-related request, when a friend wanted to showcase one of their projects using a simple HTML-only landing-page hero section.

About context length, although i install hook to auto prune+summarized old message, it kinda never being used. most of the request sit arround 64K

Model:

Q 3.6 27B INT4 https://huggingface.co/Lorbus/Qwen3.6-27B-int4-AutoRound
Q 3.6 35B Unsloth UD Q4 https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/blob/main/Qwen3.6-35B-A3B-UD-Q4\_K\_XL.gguf
Gemma 26B A4B https://huggingface.co/cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit
Gemma 12B QAT https://huggingface.co/google/gemma-4-12B-it-qat-w4a16-ct
Ornith Q 3.5 9B https://huggingface.co/ornith-ai/Ornith-1.0-9B
Gemma E4B https://huggingface.co/google/gemma-4-E4B-it
Gemma E2B https://huggingface.co/google/gemma-4-E2B-it

\- 12B and above use FP8/Q8\_0 KV.
\- 9B and below use BF16 KV.
\- VLLM ran on AOT with small batch tok to increase ctx
\- For Gemma 26B and Q 27B have image sub LLM (E4B) that ran on my processor. Although image request is also rare

edit,
\------ Conclusion and TLDR ------
They prefer Gemma model responses, majority think that it is 5.5 / 5.6 Sol model since 5 of them asking me to connect their friends into my chat webui

Maybe another key takeaway is, if your company provides LLMs to employees, it might be a good idea to partition model capabilities.

I wanted to know if theres anything that can improve creative writing for our Local Models? I specificly am asking for qwen models as they are way better when it comes to researching and writing html files compared to gemma / muse (which I know are better at creative writing). Im currently using qwen 3.8 flash next at q4 with llama.cpp

With Qwen 3.8 Flash Next (FP8 on VLLM) on a fairly stock Pi harness, it constantly hallucinates some measure of available context that says it is almost out. It's to the point where it frequently refuses work or stops in the middle of something, claiming it shouldn't go any further because it's almost out of context, when I can see in the harness status bar that (256K) context is ~25% used.

When I ask how it determined that, it always says it "invented the number and the treated it as real data" or guessed, and that it'll stop doing that, but it keeps happening.

Is there anything in particular that would cause this?

Thanks!

I'm seeing these show up on eBay for $50-$60. wouldn't expect anything ground breaking from it, but at that price, it seems like it could be interesting to play with on an extra pcie port. just curious if anyone's already done that

OK, this engine is indeed the real deal. I have expected perhaps 30 token/s prompt processing and 3 token/s decode at max, because the full Qwen 3.8 Next has around 120B parameters (not counting Engrams) and since I have just 32 GB RAM I thought it would crawl to a halt with SSD swapping.

But 10 token/s at 50k context is simply amazing on such an old device and with such a large model! That figure really surprised me and is very usable in my opinion.

It's 4 bit kv cache and no vision, so comprimises have to be made. But for real, the prefill speed is the only thing that keeps this from being usable, almost 100 token/s prefill is much higher than I have anticipated, but you still wait a long while for it to process large prompts. Qwen A35b A3B has around 5x faster prefill, and allows me to use 100K context without having to quant the kv cache at all. So not quite a replacement for that, but who knows if more optimizations are coming?

In any case, this is a very impressive showing. Used ./START-HERE.bat --draft-vocab en --vram-reserve-mib 100 --kv q4\_0 to run it.

This engine really deserves the hype it gets.

Paper linky - Context Language Models

The central idea of the paper is incredibly simple. Give a model the ability to edit its context on-the-go like a file has major benefits on task performance, context management (memory) and even computational efficiency (both wall clock and total flops). Their paper shows mostly benefits and relatively small downsides.

You can try it out as a plugin for pi!

In short, pros and cons

Pros:

1. Improves outcomes on long running tasks
- Coding and deep research tasks
- Open discovery problems (long horizon research tasks, /goal loops etc)
2. Inference can become more compute-efficient and wall-clock efficient
- Note, this depends on a caching optimization in the inference engine
3. Much less context bloat, meaning it's more (V)RAM efficient
4. No more slow and unreliable compacts

Cons:

1. The cache optimization only exists for SGLang
2. Prompt injections (including hallucinated instructions) are much less likely to be forgotten, increasing risks
3. Requires harness customizations (authors supply a pi plugin)

Some more context

The approach works by modifying the harness to allow access to the context as a file. A model is allowed to edit the context as it would any other file.

They've tested the approach on models as small as qwen3.6 9b, as well as on qwen3.8 27b and claude sonnet 4.6.

Out-of-the-box, meaning just a small addition to the system prompt and tools to edit the context as a file, task performance, context management and efficiency measures remain approximately the same or improve by a little bit. The smaller qwen3.6 9b model in particular lost a little bit of efficiency, suggesting it works better on larger (smarter) models.

Performance can be massively improved with RL training, which the authors also did.

Wanna try it out?

You can try it out right now if you use pi

1. Install the plugin https://github.com/lolipopshock/pi-clm, this comes from the authors directly
2. After installation, adjust settings with /clm settings:
- Set steering to house-brief.md (modifies the system prompt, I suppose this should be left disabled for RL'd models only, of which there are none right now)
- Enable "One tool per turn"; this one is important for performance
- Enable "Size trailer"; this one appends context usage after every tool result. Without it, models are much less inclined to modify context on-the-go for large tool calls

Fin

Let me know how it goes!

Last, I also consulted this video by "Prompt Engineering" on YouTube in addition to the paper: https://www.youtube.com/watch?v=Bgtr1Ue40Jo

By classic benchmark, the flash next is supposed to be slightly superior to its dense counterpart. But they are really incomparable. For my very simple workflows (spec -> implement -> review <-> rework), I feel that 27b is just better quality.

For context, and making things worse, I am comparing quantized 27b versus cloud flash next.

\- self hosted unsloth/Qwen3.8-27B-GGUF:Q4\_K\_XL (stock llamacpp with 130K context window)
\- alibaba cloud (qwen individual token plan), context capped at 256K in the harness

The metrics for my quality is actually very simple, I measure the number of review / rework needed before a PR is ready for me to read. The tasks are all very simple with a tight scope. Usually 27b do the work in \~2 iterations, flash next needs \~5. And it is not only about the number of iteration.
On the review flash next is overly verbose on half baked PR comment, where 27b is more straight to the point. In the end, the code produced is on par, to be frank. But if we look at token consumption...

Side notes, on pairing session, I got some deep hallucination using "/skills:diagnosing-bugs" + flash next. But since it is "bugs" and they not really comparable chunk of work, it is hard to say.

And I can't be the only one feeling that right ? Are you feeling the same ?

PS: of course I followed the hype and jumped on Strata. After the initial "oh my god it's so fast", I switched back to 27b. Tried all quant from "ISTA-DASLab" as well as experiemental from unsloth (Q4\_K\_L). With ISTA-DASLab, It actually is the first time I had "tool call error" in pi (which stop the agent), multiple times.

I built a Pi extension that implements Victor Taelin's OptChat recipe: instead of compacting, every message is logged and summarized into a binary tree. Each turn starts from a fresh context with a bounded memory view (128 KB), and the agent uses zoom/date to read the originals when it needs them. One endless chat per profile, no fork, no separate launcher.

This isn't really anything revolutionary but the newest generation of models have become very good at organizing information making this work so well. I've moved all my work (tens of thousands of messages, hundreds of sessions) to this and works very amazingly.

Install with pi install npm:pi-optchat

What's in it:

• Profiles — separate memories and instructions (I run work and personal).
• Subagents — spawn background agents, watch them live, send guidance, interrupt with Ctrl+C, resume finished ones with tell. Reports from one spawn arrive grouped.
• Import — bring in your history from Claude Code (sessions and auto-memories), Codex, or a ChatGPT export.
• Connected windows — open a second Pi on the same profile and it becomes a subagent you talk to directly, with a handoff when you /complete.

Repo: https://github.com/jonaslsaa/pi-optchat
Video credit goes to https://github.com/aaaxn

MiniCPM-V-4.7-35B-A3B
No model card yet

strata with qwen flash Q2 is pretty retarded and 27b at q4 is also retarded so I am wondering if anyone knows a engine that runs the model Qwen/Qwen3.6-35B-A3B fast on my hardware?

Given the commentary on the Q3.8FN release page here https://qwen.ai/blog?id=qwen3.8-flash-next I assume/hope that all the work that's going on to optimise the hell out of running it will be useful when Qwen4 drops?

Upgrade Question
If we say the "budget max" is $1700-1800, and this could include upgrading to a Taichi motherboard:

_With a new motherboard_, no need to bifurcate
1. Would you add a second 5060Ti (16GB)? Someone is selling one for around $500 locally;
2. Buy a used 7900 XTX (24GB VRAM); local seller, $850

_All-in on the GPU_, I'd have to make the current motherboard work for my use-case
3. Or, just go for an R9700? (No budget for Motherboard upgrade)

Current system
- Ryzen 9 9950X edit: added after original publishing of post
- 96GB system RAM (DDR5)
- 5060Ti (16 GB VRAM)
- Llama Cpp but I built for CUDA; default Vulkan had issues, and the GPU would "disappear"
- ASRock X870 Pro (only 1 PCIe 5.0 x16)
- I could run a second card very slowly at x4
- Researching if I could bifurcate x8/x8 in the 5.0 slot

I bought this PC used as-is; I do contemplate upgrading the MoBo to an X870E Taichi for 2 fast PCIe lanes

The "largest" models I currently run
- Strata IQ3_S (just tried this yesterday, was impressed)
- RentedNoodle/Qwen3.8-27B-GSQ-RCO-IQ3_XXS-Uncensored

My most typical uses
- writing code
- analysing documents (PDFs)
- analysing maps and images

The idea is to use something like Headscale or Tailscale at some point so I can always access local LLM from laptop if I'm not home.

---
Yes, I'm aware of "workstation" motherboards, CPUs, etc. and I'm not at a point right now where I want to take that path.

Finally, the NPU is being useful in my pi coding agent. Halogen shipped the endpoints, I wired them in expecting a gimmick, and kept four tools.

tldr: Qwen3.8 Flash-Next, a 125B MoE, on a 70W tablet. Same bug fix: 13.6 min with NPU search vs 18.7 without. Receipts in the repo.

The payoff: it replaces about 95% of my cloud calls. The hardest few percent still goes to the top models, GLM 5.3 or Opus.

I still can't believe it. Opus 4.8-class intelligence on my tablet, unlimited tokens.

Flash-Next decodes at 64 tok/s and prefills ~1,500 tok/s. First token lands in ~0.03s, about 43x faster than a cloud call measured side by side, and still 7x with a second agent hammering the server.

Rate the taste, not the throughput: pasted a real timeshift error from my system log to seven runs. All seven said healthy, nothing to fix. The difference is what they proved.

In pi:

• Flash-Next, 2m55s: proved it with a journalctl trace to a racing notify-send, a pacman.log check, and the upstream PR found.
• glm-5.3-flashx, 2m41s: the most precise answer, spotting that the snapshot mount got unmounted under the script's last line. No PR.
• GLM 5.3 on max, 8m30s: the deepest answer of all, source-level forensics down to the function names and the one-second race window. No PR.
• glm-5.3-flash, 9m09s: proved it with a live reproduction of the status file. No PR.

In opencode: flash in 1m8s with the right verdict and the wrong mechanism, flashx in 1m30s correct and corroborated, and the full 753B GLM 5.3 in 6m15s correct with the PR missed.

Same pi harness, same task, 125B at medium effort against 320B and 753B tiers at max. First to the full answer: 2m55s. When I had GLM 5.3 flashx rate both results, it picked qwen too.

All seven answers side by side: local vs cloud model comparison.

What the NPU does now:

Search. The agent stops guessing paths and finds the right file first try. ~0.1s per lookup, beat ripgrep 15/20 vs 9/20 on realistic queries.

Dup scan. Catches copied and renamed files git never shows you. Found 45 pairs across 4 repos in 8.4s, one renamed file at exactly 1.000 cosine.

Decisions. Yes/no branching stops eating full turns of the big model. A 0.8b handles it in 120ms, 78% accurate.

Screening. Prompt injection gets flagged before the agent acts on it. 0.7s a message, zero false alarms, fails open. 42% recall, so a smoke detector, not a safe.

A working day claws back about half an hour over bare pi: faster bug fixes, faster compaction, faster lookups and routing, no oversized tool dumps in context. Against a cloud setup it's more, since every turn there pays the network wait. On bug fix heavy days it grows.

Honest part: the GPU still does the thinking. The NPU didn't make it faster, it changed what tokens got spent on. ~7% iGPU cost only when they overlap.

Compaction: my 194k session, sidecar summary in ~50s vs 166 on the main model. 97% cache hit.

The official halogen launch is a 24-flag docker command. Mine is one command, uninstall undoes it. Fully local: 262k context, code never leaves the box.

Anyone else using the NPU for something real? I found nothing.

Disclosure: drafted with LLM assistance, heavily edited by me. All benchmarks, timings, and numbers are from my own runs on my own hardware, receipts in the linked repo.

repo | halogen 0.17.1 | benchmarks

Hey r/LocalLLaMA,

I’m trying to figure out if I should drop serious cash on a local setup for heavy agentic coding (letting agents read whole codebases, refactor multi-file repos, run terminal loops) or if I should just keep paying for Claude Code and ChatGPT.

Right now, cloud APIs are driving me crazy. If you do any serious agentic coding, you easily blow past $500+ a month in API bills. And even if you have the money, you hit a hard rate limit after 2 to 4 days of heavy work and have to wait for a reset. It completely kills my momentum.

The global memory shortage has messed up hardware prices, but going local is looking pretty tempting just to escape these cloud limits. I have a budget of around $14k–$15k max. Here are the two routes I'm looking at and the headaches I'm trying to weigh out.

1x Mac Studio M5 Ultra (512GB RAM)

• The Cost: Around $13,500 - $14,000 USD because Apple charges an absolute fortune to max out the unified memory.
• The Good: You get 512GB of VRAM on a single machine. You can easily fit huge models (like DeepSeek V4.1 Flash or GLM-5.3 Flash) and give them huge 128k+ context windows without the system crashing.
• The Catch: Time-to-First-Token (TTFT) is going to be slow. When the agent reads a 60,000-token codebase all at once, the Mac is going to sit there and "think" for like 2 to 3 seconds before it starts typing. Once it actually gets going, generation is about 35+ tok/s, which is fine, but that initial pause might get annoying.

2x Nvidia DGX Spark Units (Linked directly)

• The Cost: Right around $14,000 USD (Nvidia jacked the price of the 128GB version to $6,950 due to the component shortage, so two nodes plus cables puts you right there).
• The Good: Prefill is blazing fast because of the Blackwell cores. It will ingest thousands of lines of code almost instantly. No waiting around for the first token.
• The Catch: Stacking two nodes only gives you 256GB VRAM total. This means you are seriously restricted on what models you can run. You can't run the massive 300B+ giants unless you use super compressed low-bit quants (like IQ3 or IQ2) just to fit the model and a decent context window without hitting Out-Of-Memory (OOM) errors. If your codebase is too big and the KV cache overflows that 256GB limit, your speed drops to zero.

How the math looks to me

If I take that $14,000 and look at it compared to what I'm spending on APIs:

• At $500 a month, $14k pays for about 2 to 2.5 years of cloud access.
• But again, cloud means hitting limits every few days and sitting around waiting for a reset. Local means I can run it 24/7 with zero downtime.

The main reasons I want to buy hardware:

• No limits: No quotas, no rate limits, no waiting for a reset. I can run infinite loops, try weird models, tweak my tools, and never see a "Quota Exceeded" message.
• Privacy: My code and data never leave my room. No corporate data center is logging my repo.

The big downsides I'm worried about:

• Depreciation: The moment I buy a $14k cluster, it starts getting old. In two years, cloud models will be way smarter, but I'll still be stuck with the same physical VRAM limits.
• Friction: Local agents love to break. I feel like I'm going to spend hours messing with vLLM, debugging tool-calling errors, and dealing with quantization loss instead of actually getting work done.

What do you guys think?

I'm really trying to figure out if anyone here has built a mini-cluster specifically to escape the $500/month cloud tax and quota lockouts.

Did it actually replace your Claude subscription for real development work, or did it just end up being an expensive toy? How bad is the TTFT on the Mac when loading huge repos, or are you constantly hitting OOM errors on a 256GB Nvidia setup?

Would love to hear some real-world experiences before I burn a hole in my wallet.

I'm worried about precision and long context adherence in agentic coding

IQ3\_S Strata or NVFP4 from RadixArk?

What is your choice? Please share your experience and knowledge

I thought this was a very interesting article, of relevance to the readers here.

https://github.com/bilibili/Index-Translate

Index-Translate is a family of multilingual translation models built on Qwen3.5. The text models cover 150 languages and follow translation instructions such as terminology, formatting, and content-preservation requirements. The family extends this foundation to speech, syllable-controlled translation, and full-document translation.

-Index-Translate translates text, structured content, and community expressions.
-Index-Echo produces translated subtitles or speech conditioned on the source speaker's voice.
-Index-Homura adjusts translations toward a specified target syllable count.
-Index-NativeLong translates complete documents with context across passages.

For the last few months, I ran DeepSeek v4.0 flash (NVFP4). First 0731, then visionexp because it was a free improvement. I got around 65 tps decode and almost 2k prefill, and ran 4-5 agents in parallel, totalling around 200 tps cumulative decode. Because of this, I did not feel like switching to GLM 5.3 because it would half the decode and prefill, did not scale well with multiple agents, and had a repetition bug a lot of people complained about.

Until a few days ago, when the latest version of this recipe dropped; a 50-90% decode improvement. So I took the plunge, and wow, am I impressed.

It's more intelligent than the new DeepSeek v4.1 flash (that does NOT run on dual DGX Sparks), and it's even faster than DeepSeek v4.0 flash in decode. Only a slight drop in prefill, which I'm more than happy to take in exchange;

|Test|visionexp-final (recorded)|glm53-low|Δ|
|:-|:-|:-|:-|
|B1 count-to-300|92.5|95.9|\+4%|
|B1 bulk SQL INSERT|88.3|97.2|+10%|
|B2 chat|38.7|42.5|\+10%|
|B2 count|92.8|96.0|\+3%|
|B2 code|63.5|69.3|\+9%|
|B2 prose|32.8|37.1|+13%|
|B2 tool|79.8|85.1|\+7%|
|B2 battery mean|61.5|66.0|\+7%|
|B2 accepted tok/step|3.26 of 6 (54%)|3.55 of 8 (44%)|see note|
|B3 prefill @1.5K|1738|1376|-21%|
|B4 prefill @32K|1902|1576|-17%|
|B4 prefill @128K|1758|1578|-10%|
|B4 decode @32K|41.1|44.2|\+7%|
|B4 decode @128K|49.5|47.1|\-5%|
|B5 c1 aggregate|91.7|90.1|\-2%|
|B5 c2 aggregate|45.4|51.4|+13%|
|B5 c4 aggregate|63.4|58.6|\-7%|
|B5 c6 aggregate|79.1|77.5|\-2%|
|B7 soak (40 min at c4)|522 req, 0 err, 87.4 agg|503 req, 0 err, 0 soft-empty, 83.6 agg|−4%|
|B8 byte-stable probes|8/8|6/8|worse|
|B8 garble gate|30/30 clean|30/30 clean|=|
|B8 non-Latin / U+FFFD|not measured|3/3 clean, 0 U+FFFD|new gate|
|KV pool|1,988,929 tok @ gmu 0.85|560,362 tok (6 GiB/rank pin)|−72%|
|NRestarts through the pass|0|0|=|

I've tested it for a few days now, both for technical coding, devops/sysadmin and also vision (to recognize some plants), and it is better than I hoped for. Basically Claude Opus 4.8 level. Slower of course because it has to think a lot more, but good enough to comfortably leave it chugging for hours on tickets without worry of derailing. I don't see a reason NOT to upgrade, so have a try and enjoy!

https://huggingface.co/google/embeddinggemma-2

EmbeddingGemma 2 is an open multimodal embedding model built by Google DeepMind which maps text (incl. code), images, video, and audio inputs—and combinations thereof—into a single, unified 768-dimensional vector space. The model has 740M total parameters, combining a 270M parameter text model with modular vision (170M) and audio (300M) encoders.

Designed to run on consumer hardware such as mobile devices and laptops, EmbeddingGemma 2 delivers low-latency semantic representations for on-device applications, like search, retrieval-augmented generation (RAG), classification, and clustering.

Made this (free, no signup). One shot per model, no system prompt, temp 0.7. Lots of open weights in there, plus 93 models you can't run anymore.

museumofmodels.com

Opus5.5, Sol6.1, Fable, and Astra have all proven they can and the scene has exploded this past week. Part of that is from the tools and feedback loops maturing though.

Are any open weight models (at all, so including K3, GLM5.3, Qwen3.8-Max, and Mimo-2.6) able to do this?

Can the larger models this sub regularly runs (GLM 5.3-Flash, Qwen 3.8-Next-Flash, V4.1-deepseek Flash..) handle a simpler one (GBA and PSP having smaller roms and mature pipelines)?

I hosted my own world of warcraft private server then built a client that you can play in the browser on PC or mobile at https://jankcraft.xyz/ for free.

Afterwards, I created a custom MCP and agent harness to control the browser client and play the game by sending signals over a websocket. The agent harness is live on https://jankcraft.xyz/agent , still working out some kinks if all you have a cloud subscription but you should be able to connect local models as long as CORS is enabled in your server settings. There are a few existing LLMs you can try, I'll probably take those away as the usage grows since I can't support too many users concurrently on my own machines.

I'll be checking logs and things periodically today so don't be alarmed if you're disconnected suddenly. The server should return after a minute since this is a work in progress and might need a restart.

If you want to run your own LLM for this:
\~24 Gb RAM: https://github.com/syv-ai/HyperQwen with the model Qwen3.8-27B-GPTQ-W4A16 
\~16 Gb RAM: vLLM with Gemma4-e4b-coder - A custom Gemma4-e4b with a constrained vocab for \~3x concurrency increase when changing from 262K to 65K vocab and retrained on \~1.1B tokens across 20 different coding languages, 7 different agents and has a custom MTP to help reach ~200 tok/s on a 4060Ti.

Let us know what other models work well for you!

Thanks and hope you guys enjoy.

Microsoft confirms on publicly accessible web page that OpenAI has been using Looped Transformers in the GPT-6 series, proving The Information's reporting was correct all along.

GPT-6.1 Sol uses 2 inference passes, with a passing mention of "instead of three".

For those confused by "same base model weights as GPT-6 Sol", I think Microsoft meant 6 & 6.1 are both post-trained models on top of the same pre-trained "base model", not that the final weights are identical

So different post-training (+ one less loop).

Update: Microsoft updated the web page to remove it

Hey everyone,

Like most of you, we have been fighting constant OOM errors while trying to fine-tune 8B and 70B models on consumer GPUs. The AdamW optimizer states are always the biggest bottleneck.

We didn't want to rely on aggressive 8-bit quantization because we were seeing degradation in convergence, so we tried an experiment: tackling the optimizer states in the frequency domain.

The methodology:

Instead of storing the full gradients, we transform them using an FFT. This isolates the high-energy signal from the noise. We dynamically drop the low-impact frequencies and compress the state. When we inverse-transform back, it maintains the directional integrity but uses roughly 50% less VRAM.

The catch:

Running FFT operations adds compute overhead. It takes slightly longer per step, but the trade-off is completely avoiding OOM crashes and pushing batch sizes way up on standard hardware.

We are currently giving out access to our internal Colab environment and baseline weights to anyone who wants to poke holes in our math or try to break it.

We are really curious if anyone else here is exploring frequency-domain stuff or other non-quantization methods for VRAM reduction?

What are people using to run nvfp4 on multi-gpu?

the only thing i can get to run is unsloth and vllm is too slow and takes FOREVER to fucking load. TensorRT-LLM has too many issues, so what are people using?

And do you like nvfp4 vs q4 qguf for qwen 3.8? apples to oranges is nvfp4 closer to Q6 gguf than q4 gguf is?

well i tired the model here https://huggingface.co/neroued/Qwen3.8-27B-nvfp4-NInfer

which works with https://github.com/Neroued/ninfer/tree/master

and initial testing has not been great for code but vision and tool calling is very impressive. Brief testing on complex scenes showed slightly better than what I've seen on q6 gguf

but im going to revisit surely im missing something, i had to spend a lot of time wiring ninfer into my frontend so ill look at it again with fresh eyes.

The speed is fantastic on 2x5060ti with 197k ctx and model loading in 4-6 seconds is wild

https://imgur.com/a/ahDnoAZ

I spent the past year independently building Slow Vale, an LLM-driven life simulation. The Chinese server now has 800+ AI residents sharing one continuously running city. This is an engineering write-up about concurrent decisions, dynamic action spaces, context caching, and the operating costs of a persistent multi-agent system.

The runtime currently uses hosted DeepSeek Flash, rather than local inference. I am the developer. I wrote the original material in Chinese and used AI to translate and refine the English. Product metrics below are current through October 7, 2026.

Asynchronous decisions in a continuously advancing world

Each character makes roughly 300–400 LLM calls per day, with an average context of around 30,000 tokens per call. A call includes the character's state, relevant experiences, current environment, and available actions. The model selects an action and its parameters; the backend turns that decision into an activity that occupies time and resources.

Game time and real time coexist. Sleeping might occupy 8 in-game hours, while saying one sentence might take 1 in-game minute. Inference itself takes real time. While a call is in flight, other characters can change the environment, and the world clock continues to advance.

Interactions also involve mutual exclusion. If A is talking with B, C cannot simultaneously pull B into a separate conversation. Facilities, production tasks, and other activities have their own rules for acquiring and releasing occupied resources.

Separating concurrent inference from world-state mutation

LLM calls can run concurrently, but model responses do not directly mutate the world. Results return to the world's execution flow, undergo validity checks, and are applied by the execution component that owns world state.

For example, the last fish on a shelf might still be available when a character starts inference. By the time the response arrives, another resident may have bought it. The purchase intent must be checked against current inventory. Similarly, the person a model wants to talk to may have left, gone to sleep, or started another activity.

There is therefore an explicit time gap between the context used for a decision and the state at execution. The system must distinguish what a character intends to do, whether the action is still valid, and what effects have actually occurred. Completion, failure, interruption, and recovery each need consistent state transitions.

A shared runtime for activities that occupy time

Movement, production, conversation, and sleep have different durations, participants, and completion conditions. A common runtime makes it possible to manage busy characters, resource conflicts, and service recovery without building a separate scheduler for every mechanic.

The frontend must also follow actual progress: when an activity started, how long it has been running, whether it completed, and what it produced. Logs and scene animations need to correspond to facts committed by the backend. This is a significant source of complexity in a persistent world: one event can affect future decisions, persistence, other residents, and the player interface.

Decision context is part of the backend architecture

A personality description alone is insufficient for a character that acts over long periods. Each decision needs the character's current needs, location, assets, ongoing concerns, relevant relationships, and the actions actually available at that moment.

These inputs have different update frequencies and lifetimes. Personality is relatively stable; hunger and energy change continuously; inventory and other characters' states can change within seconds. An experience may continue to affect a relationship long afterward. Each kind of information needs rules for entering context, updating, and leaving the character's current attention.

The action space also needs to reflect game state. Options presented to the model should disclose their execution conditions and relevant state, while the backend retains final validation. Otherwise, characters repeatedly attempt unavailable actions or spend calls trying to understand rules that were never clearly disclosed.

I have invested substantial effort here: organizing stable and dynamic information, controlling irrelevant history growth, avoiding duplicate reminders, and keeping context prefixes stable. This affects behavior quality, inference latency, and cache hit rates, making it part of the backend architecture.

Roughly 5 billion tokens a day for under $100 in model fees

The Chinese server currently processes around 5 billion tokens per day, with model fees below US$100. It primarily uses inexpensive models such as DeepSeek Flash, while maintaining a cache hit rate above 90%.

The token count includes cached input. In a system with frequent calls, many characters, and long contexts, reusable stable prefixes directly affect the bill. Which information stays stable, which changes on each call, and how it is ordered all require deliberate design.

https://preview.redd.it/71a7tjghs7uh1.jpg?width=1360&format=pjpg&auto…

Actual DeepSeek usage and billing for October 7, 2026 (GMT+8): approximately 4.424 billion tokens and 163,742 requests across all API keys, costing CNY 472.33. The model shown for that day is deepseek-flash.

The trade-off between dynamic action spaces and prefix caching

One concrete engineering trade-off was how to represent a dynamic action space when using tool calling or structured output. The available actions and parameter values change on every decision: which facilities are nearby, which goods are available, and whom the character can talk to all depend on the current world state. Encoding these options directly in tool definitions or an output schema gives stronger output constraints, but also makes the schema change frequently. In some API implementations I tested early on, those definitions became part of the request prefix. Changing the schema prevented the otherwise stable context after it from hitting the cache.

At that stage, I chose ordinary text generation of JSON for the primary path, with parsing and validation in the backend and a strict-schema fallback when parsing failed. The model still received explicit, state-dependent action options, but those options lived in the current decision context rather than in a changing output schema. This kept stable instructions and reusable history toward the front, with current state and action options toward the end. The trade-off was giving up decoding-time format guarantees on the primary path. The application had to handle malformed output and validate actions and parameters against the world state at execution time.

The 90%+ cache hit rate therefore comes from designing the whole request structure, rather than simply enabling a provider feature. The percentage refers to the share of input tokens served from cache; the model still generates a fresh output for every decision. When comparing invocation modes, I consider format reliability, character behavior, cache reuse, latency, and cost together.

These figures cover model fees. As the resident population grows, database load, state delivery, log storage, and scene rendering also matter. Inexpensive inference makes continuous simulation feasible; sustained operation still depends on resource management across the entire system.

Organizing AI collaboration with runbooks

Maintaining this many modules alone requires giving AI a reasonably complete working environment. I provide development and operational tools, including access to logs, Langfuse, growth analytics, the database, and procedures for maintaining production services.

https://preview.redd.it/iqo7313ks7uh1.png?width=962&format=png&auto=w…

My Codex usage: approximately 43.25 billion cumulative tokens and an 85-day longest streak. Codex is only part of the AI coding tooling I use. These are development usage figures, separate from the model calls that power the game's residents.

A set of runbooks governs their use. The project has extensive documentation, organized by task and module. It specifies which documents must be read for each task, which sources define current contracts, which decisions only I can make, and which documents AI should maintain when it discovers drift from the implementation.

Task entry points and action boundaries are central. An investigation starts by identifying the data source and time window. Permission to query does not imply permission to modify production data, and permission to fix code does not imply permission to deploy it. Access to a tool needs to come with explicit conditions for using it.

I have also turned recurring maintenance into automated workflows: diagnosing and fixing production problems, daily in-depth reviews of character behavior and gameplay outcomes, and daily cleanup of maintenance code that has served its purpose. Each workflow specifies the evidence required, permitted actions, validation, and stopping conditions.

My involvement varies by area. I directly decide or closely participate in frontend/backend contracts, backend architecture, and ownership of state and resources. For frontend and Phaser implementation, I focus more on evaluating the result, while still defining design tokens, page structure, reusable components, and presentation boundaries.

This approach depends on maintainable project knowledge. Constraints discovered during a task need to return to the formal documentation, and outdated procedures need correction. Otherwise, as the project grows, AI can implement a locally plausible change based on old assumptions while breaking contracts elsewhere.

Three to five production releases a day

The city has been running for more than 350 in-game days, equivalent to nearly 100 real-world days. A substantial portion of the earliest players are still playing. I built the entire project myself, including the backend, frontend, Phaser scenes, content production, monitoring, and operations. It now contains more than 400,000 lines of code, including over 200,000 in the core backend, across approximately 2,200 commits.

I use AI coding tools extensively. I make or closely participate in decisions about product direction, core mechanics, and architectural boundaries, while AI handles much of the implementation, investigation, and maintenance. As the project moved from a prototype to a continuously operating product, system design and the development workflow became a major part of the work.

I currently deploy an average of three to five times a day. Releases include architectural changes, balance and gameplay adjustments, new systems, UI and art changes, performance improvements, and bug fixes. The project has approximately 2,200 commits, with more than ten commits per day during active development.

The iteration speed comes from a short feedback cycle between implementation, observation, and adjustment. Players continue to inhabit the same city. After a feature goes live, I can observe actual usage and character behavior, then decide whether to change a mechanic, clarify what information characters receive, or fix an implementation issue.

I assess software operation and gameplay outcomes separately. Error rates, latency, database load, and model calls indicate whether the system is operating normally. Understanding whether characters repeat themselves, understand a new mechanic, or successfully complete production and social activities requires reading their actual experiences and decision traces.

Link: https://tesseracted.com/kolibri-1-chat/

Original post at /r/euainews

M5 Max Mac Studio 128GB (18C CPU / 40C GPU) owners - anyone running serious local LLM / multi-agent workloads?

My M5 Max Mac Studio order finally got charged today and moved to Preparing to Ship. Apple’s original estimated delivery date is still about 18 days away (Oct 23-30), so I’m guessing/hoping it’ll actually show up early now within the next 5–10 days 😄

Configuration:
M5 Max
18-core CPU
40-core GPU
128GB unified memory
1TB SSD

While I wait, I’ve been trying to find real world local LLM results from this exact configuration, and there’s surprisingly little out there.

Most of what I can find is either M5 Max MacBook Pros, lower-memory configurations, or M5 Ultra Mac Studios. YouTube especially seems to be full of Ultra coverage, but I can barely find anyone actually demonstrating the 128GB M5 Max Studio with the 18C/40C configuration.

I’m specifically not looking for M5 Ultra results/comparisons. I already know the Ultra is faster. I’m trying to understand what people are actually accomplishing with the 128GB Max Studio.

My main goal is to use this as a local AI/agent workstation, potentially running several autonomous agents concurrently for long periods through OpenClaw, some monitoring dependencies and workflows, some scouting, not usually too heavy of workloads where they would be competing for inference constantly, but occasionally they would be switching to harder work so I’m curious about the concurrency side. Local models would handle a lot of the routine work, while harder reasoning/coding tasks could be escalated to cloud models like GPT 6 Luna/Codex.

For anyone who owns this exact M5 Max Studio, I don’t expect anyone to answer all of these, but I’d love some insight:

1. What models are you actually running?
Qwen, GLM, DeepSeek, Gemini, Llama, etc. I see a lot of Qwen 3.8 27B on Splash, but curious if anyone else has had good success with others also

2. What token speeds are you getting?
I’m especially interested in \~20B-70B-class models rather than tiny models

3. What happens with multiple simultaneous inference requests?
For example, if 3-5 agents are hitting the same loaded 27B/32B model concurrently, what does aggregate throughput and per-agent responsiveness look like?

4. Has anyone tried running multiple models simultaneously?
Something like a \~27B model as the main worker plus one or two smaller 7B–14B models for specialized agents, then unloading them when they’re no longer needed. How quickly can models be loaded/swapped, and does frequently switching models introduce enough latency or memory-pressure issues to disrupt an agent workflow?

5. Has anyone built a real multi-agent setup on one of these?
Not just five chat windows but autonomous agents doing coding, research, browser tasks, tool calls, database work, monitoring, etc. concurrently for hours.

6. How does sustained performance hold up?
One reason I chose the Studio over a laptop is sustained workloads. I’m curious whether anyone has run inference/agents continuously for 6–12+ hours and noticed throttling or other bottlenecks with KV, etc.

7. What’s the actual bottleneck in practice?
Memory capacity? Memory bandwidth? GPU compute? Prompt ingestion? KV cache/context length? CPU/tool execution? Something else?

8. What surprised you about the machine?
Either positively or negatively. I’m particularly interested in things benchmarks don’t reveal.

Ultimately I’m trying to figure out how far I can push one 128GB M5 Max Studio as an always-on local agent machine - not just how quickly it can generate a single response.

Once mine arrives, I’m planning to test concurrent agents/models rather than just running the usual single-stream benchmark. If there’s interest, I’ll post the results here, including memory usage, context sizes, model/quantization, concurrent requests, aggregate tok/s and per-agent tok/s.

Would really like to hear from anyone actually using the M5 Max Mac Studio 128GB 18C CPU / 40C GPU for this kind of workload or similar if anyone is

I have a somewhat specific question in case anyone has experience with this. I am a big fan of CS, math, and physics. While I was able to get formal instruction for the first two, I was never able to get the opportunity to learn physics. The most wonderful part about llms for me is that I can pursue that now without the costs of college tuition. My current process is the following. I pick up a textbook. I read it section by section and almost always don’t understand on the first read. Then I open up an llm and ask it to explain the section to me and ask it specific questions that help my learning. I’ve gotten past introductory quantum mechanics and special relativity this way, now I’m trying it on general relativity. However, tokens are expensive so I’ve been increasingly trying to replace my workflow with local models but have not gotten a lot of success with the smaller qwens and ministral. Would greatly appreciate any advice on other models or fine tunes. Has anyone else used local llms for physics or other sciences? Thank you!

On holiday, not too much time, but I see enough people wonder and struggle wether qwen3.8 27b can do real work on 16Gb VRAM.

short answer:

yes it can

longer answer:

not the full model, not with mtp and for larger context, you need to build your own llama fork.

Qwen3.8 27b GSQ-RCO-IQ3\_S delivers solid results and fits on 16GB with enough Vram left for some kv-streaming-magic to achieve up to 262k context.

Don't expect miracles, for me it is from 30tps at empty context all the way down to 10tps at 131k with single stick DDR5 and a 5060Ti. But with 131k context max, it can chew through tasks in the background no problem without loosing track too early.

full answer (and how I made it work):

Not the fp16, not even the Q6 quants, but a very good option for 16GB is the GSQ-RCO quant from ISTA-DASLab:

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

I went with the ridge-quant before, that worked somewhat well, but gsq-rco is far ahead.

Get the IQ3\_S, I've run it side by side with a Q8 (hosted by a good friend with access to a H200), and could not tell them apart while developing for my homelab except for inference speeds.

Use the gsq-rco to aid you in building the kv streaming fork:

https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming

Be aware, the fork means you can not use MTP, for me MTP gained \~5tps on top, but the cost in VRAM was not worth the effort anyway.

Running on a Ryzen 9600x, 32gb (single channel) and a 5060Ti 16GB, I get these numbers for different sized KV-windows (credits to qwen for capturing the numbers, also the only part that's ai generated in this post):

Results (measured)

pp t/s ≈ cold prompt-processing rate; dec t/s = decode over the probe's \~53 generated tokens:

|tokens|pp @ any pool|dec t/s 512|dec t/s 1536|dec t/s 2048|
|:-|:-|:-|:-|:-|
|14 644|878–892|28.0|28.5|27.9|
|35 186|803–807|23.9|25.2|25.2|
|54 976|736|15.9|20.6|21.3|
|69 429|695|12.4|18.7|19.1|
|94 464|630–635|8.7|12.6|15.5|

Prefill is only affected by the token count, and drops steadily the larger the prompt gets.

Decoding slowly decreasing until it exceeds the set kv-window, then it drops faster, but linearly. Remember: I run single channel RAM, it might be better with dual channel. At almost 131k and 1536M window I get around 9tps, so thats the floor. With \~13.5GB model usage, its not possible to get 3G kv-window. In theory you cna go as low as 128MB, but then its slow from the beginning.

I found 1.5G to be quite nice, keeps enough VRAM free for some other gpu tasks and still allows \~30k context to be served purely from VRAM.

Some suggestions to get it on the rails:

The model loads on stock llama, Make use of it.

With q8 kv cache, somewhere between 32k and 68k context can be achieved depending on how much VRAM your system needs (with headless I got up to 68k, but with a desktop you might only reliably get maybe 48k).

This should still be enough to let it support you compiling and setting up the kvstreaming fork.

Stick it together with a harness like pi (pi.dev) and let it compile the fork - for me it was able to do that easily.

Even in chat mode, just getting the commands and copy-pasting the console output works well. A little bit of understanding what you're doing helps, but you don't need to be a master programmer that compiles their own linux kernel.

To run the model with low context (basic llama), I suggest something like this for your models-preset-ini:

[qwen38-gsq-rco]
model = /models-src/linked/qwen38-gsq-rco.gguf
mmproj = /models-src/linked/qwen38-gsq-rco-mmproj.gguf
ctx-size = 49152
cache-type-k = q8_0
cache-type-v = q8_0

Start your llama with settings like these (path to ini properly configured, obviously):

--models-preset /models-src/models-preset.ini --models-max 1 --host 0.0.0.0 --port 8080 --n-gpu-layers 999 --jinja\--flash-attn on--no-mmproj-offload

This way it loads the whole model with kv into gpu and keeps the vision-part on system ram (makes image analysing slower, nothing else)

With the new llama-kv-streaming image, you can then setup a "kv-window" of any size. I run mine with 1536M of VRAM for KV, and have a total VRAM usage of 13.5GB (headless, mind you).

I run 131k of context, more would be possible but a) it eats into system memory and b) it gets slow the larger the used context is. 131k is completely usable for most tasks.

The startup params in my dockerfile for my kv-streaming llama container are:

command: >
--model /models-src/linked/qwen38-gsq-rco.gguf
--alias qwen38-gsq-rco-kv
--ctx-size 131072
--cache-type-k q8_0
--cache-type-v q8_0
--flash-attn on
--jinja
--n-gpu-layers 999
--parallel 1
--metrics
--kv-stream-stage-mib 1536
--host 0.0.0.0
--port 8080
--mmproj /models-src/linked/qwen38-gsq-rco-mmproj.gguf
--no-mmproj-offload

and this is what my nvidia-smi looks like when using the model:

Tue Oct 6 23:29:08 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.173.02 Driver Version: 580.173.02 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5060 Ti Off | 00000000:01:00.0 Off | N/A |
| 33% 60C P1 172W / 180W | 13660MiB / 16311MiB | 100% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 4167261 C /app/llama-server 13644MiB |
+-----------------------------------------------------------------------------------------+

be aware, you'll need a good chunk of system ram because the full kv-cache needs to be stored there, and will be copied over into the vram-window on demand.

TL;DR:

• get Qwen3.8 27B GSQ-RCO IQ3\_S, it offers really solid performance for its size.
• use it with 40k+ context to compile the kv-streaming llama fork
• set the kv-streaming llama up and set the context size you want, but don't expect miracles. at the limit of your context it might be slow.

They're noise in this subreddit that is only getting louder. There's a discord for these sorts of questions. We don't all need to see these,

Post might be a tiny bit long. Hate words, skip. But it's not too bad though. Also, I've been Qwen-gang for a long time, check my receipts. That said...

I started my agentic journey with Qwen 3.5 around May 31st. I'd heard about agents before, but never had a chance to play around because I didn't have any cloud memberships at the time. I've done most of my coding using free services: gemini and claude sonnet. It's been a lot of fun.

When Qwen 3.5 dropped, it was the first time a local model felt like cloud. Sure, it wasn't on the same intelligence level, but it didn't feel that far off. So I dived in hardcore learning everything I can.

I decided to build my own infrastructure/harness rather than going with hermes, pi or one of the others. I'm glad I did because it taught me so much. It was hard, because I had to learn everything from scratch, and the road has been extremely stressful and challenging, but the knowledge I picked up along the way has been well worth it. I'm able to conceive ideas and implement strategies in ways I never imaged, and I honestly don't think I would have learned even a fraction of what I know now if I'd worked with cloud models, because they might have one-shotted the results, robbing me of the challenge to grow.

Things got even better after Qwen 3.6 27B dropped. Since then, people have been singing the praises of Qwen, and how close it is to the cloud models. I also felt it wasn't far behind. I've made quite a few posts praising Qwen and sharing my experience, and those posts were real and authentic.

But all of these people claiming to be cancelling their cloud subscriptions and replacing them with Qwen? That's an overreach. Those people either a) are bots, or b) have extremely simple use cases that they were wasting subscriptions on, because anyone who's used cloud for anything agentic and a tiny bit complex won't walk away from that experience looking at local the same again.

I'm extremely thankful for Qwen because it put me in the game and started me on this journey. But my ambitions reached a point where Qwen just wasn't able to get me there without tons of mistakes. The "shine" wore off the more complex my needs grew. It's still very capable, and I figured out some ways to increase its intelligence (and yes, you can increase the core model's intelligence without training using a harness and multiple agents, but that's a whole 'nother discussion), but it became a time thing. I started getting extremely frustrated and cursing at Qwen for its stupidity.

I'd been using cloud models for code stuff, but they weren't agentic. But my sister let me use her chat-gpt subscription and I finally yielded and decided to give it a try. Long story short - and out of respect for this reddit, because it's about local, not cloud - I'll just say that it's been a completely different experience. A really, really good one. My project is moving along now and I'm getting a lot of work done, and it feels surreal. There's a real difference between local agents and cloud agents.

So, when you guys hear everyone saying cloud is dead, they're probably not human, because it's not even in the same ballpark. I've just been using Sol light, and it's ridiculous. I can't even imagine what Sol Medium or Astra are like.

I have no intention of abandoning local. I'm using Sol to help me advance my harness so that it will be faster, smarter, and more gooder (in my best Grimlock voice). I sweat blood and tears working with my local agent and I can't wait to see how much it's improved with the new brain I've built for it. And I'm going to continue finding ways to make local the best it can be. And like you guys, I'm hopeful that the gap between local and sota will continue to close.

I guess what I've learned from this whole ordeal is, if you just want to get things done or built, go with cloud. But if you want to grow and better understand how things works, and feel more empowered through each challenge, go with local.

I don't want to imply that you can't learn with cloud either, but it for sure would have robbed me of some of the dead ends that forced me to expand my knowledge.

Grunge

My YouTube feed was mostly songs, pranks and celebrity clips, so I built a filter that judges each video title with a small decision model running inside Chrome. Nothing is sent anywhere: no server, no API key, no account, and after one download it works offline.

What it does

\- Hides the kinds of video you choose (11 kinds: music, gaming, comedy, vlogs, news, how-tos, and so on), or follows a rule you write: "Hide videos about crypto", "Show only videos about cooking"

\- Hides Shorts with one switch

\- Bonus: select any text, right-click, and check it for prompt injection with the same model

How it runs

\- The model is opendecider-nano (ONNX), loaded through ONNX Runtime Web in an offscreen document

\- fp16 on WebGPU (755 MiB download), q8 on WASM without a GPU (569 MiB)

\- 40 video titles: 1.1 s on WebGPU, about 15 s on CPU (M4 Max). About 3 GiB of RAM while loaded; it unloads after 10 idle minutes

\- The weights are pinned by revision and SHA-256. The only network requests are to Hugging Face for those files

How well it works

\- On 400 YouTube videos, with the creator's category as the label, rules like "hide music", "hide gaming" and "only news" score 0.934 balanced accuracy on average

\- Sorting videos into the 11 kinds is harder: 0.780. Expect a few comedy and talk-show clips to get through the Focus preset

\- Only evaluated on English titles. If you watch in other languages, I'd really like to know how it does

Try it (Web Store version is in review):

1. Download opendecider-focus-0.8.1.zip from https://github.com/manjunathshiva/opendecider/releases/latest and unzip it

1. chrome://extensions → Developer mode → Load unpacked → pick the folder

1. Click the icon → Download the model

Apache-2.0. Benchmarks, code and limits: https://manjunathshiva.github.io/opendecider/guides/chrome-extension/

The idea comes from Quietly, which does this with a cloud API; I wanted the same thing running on-device. Feedback welcome, especially what it gets wrong.

I'm taking a break from training alice today after my current SFT run ends to work on a few other things.

I'm an (unemployed) software developer trying to find a machine learning job in New York, and outside of job applications and networking, I'm trying to do as much as possible to further the frontier development of different LLMs in the hope it'll get some visibility. I studied machine learning in university and am fairly well educated. Plus, I genuinely enjoy working on this stuff and helping people.

That said, are there any models out there that don't have dynamic quants (preferably GGUF) that you'd like to see one for? I won't be matching unsloth or anything but I know how to make fairly good ones by quantizing different tensor types by impact. I'm talking akin to Q5 K XL and etc

I'll do the top voted 1/2 comments today, or whatever else, even outside of quants if there's something this community has been hoping for that doesn't exist. Fine tunes, paper implementations, etc - my goals of visibility happen to align very well with satisfying community desires.

On the fence about this one but is this good for localAI?

https://www.youtube.com/watch?v=lnLDjXy2kRA

https://www.youtube.com/watch?v=veP6jdvy7j8

Edit: Coxon, not Coxcon as his last name.

Hi all, a bunch of performance improvements have been landed in audio.cpp.

The biggest highlight is Higgs Audio TTS, which now runs with around 6 GB VRAM, a 48% reduction in peak memory usage compared to the previous implementation. Thanks to https://github.com/mirek190

We also made some models significantly faster, especially HTDemucs on GPU and PocketTTS on CPU.

No compromises in parity and correctness.

Here's a summary of the improvements:

|Model|Peak memory reduction|Speedup|
|:-|:-|:-|
|Higgs Audio TTS|48% VRAM|1.01–1.09× CUDA|
|ACE-Step family|6–7% VRAM|1.06–1.08× CUDA, 1.16–1.20× Vulkan|
|MOSS-TTS v1.5 cloning|21% VRAM|1.05× CUDA|
|MOSS-TTSD Q8 cloning|11% VRAM|1.04× CUDA|
|Echo-TTS (Memory Saver)|20% VRAM|—|
|Qwen3-TTS|16–20% VRAM|—|
|IndexTTS2 / 2.5|12% VRAM|—|
|HTDemucs|—|2.21× CUDA, 1.95× Vulkan|
|HTDemucs six-stem|—|1.99× CUDA|
|PocketTTS|9% RAM|2.23× CPU|

They're runtime-level optimizations that make existing models more practical to run locally.

The WebUI now includes an experimental generation history feature that lets you revisit previous outputs and restore their settings.

audio.cpp now supports 110+ audio model families and 190+ variants (and counting)! We're continuing to improve memory efficiency and inference speed across CUDA, Vulkan, Metal, AMD/HIP, and CPU. The next release will bring even more optimizations!

We're also looking for contributors to help improve the audio.cpp WebUI. With so many models and features now supported, we'd love some help making the UI more polished, intuitive, and enjoyable to use. If you're interested in frontend development or UI/UX design, contributions are very welcome!

Thanks to everyone contributing improvements, testing builds, and reporting issues. Curious how these changes work on your setup!

The link to the github repo: https://github.com/minghinmatthewlam/pi-gui

I'm not sure if its legit/ safe because i don't see anyone talked about it in this sub reddit, any thoughts about it?

My setup is two RTX 5070 Ti 16GB cards (the second one is on an OCuLink dock) with 64GB of RAM, Ollama on Windows, and the agent runs on pi in WSL. Until last night the daily model was Qwen3.8 27B UD-Q4_K_XL at 128K with MTP, which does about 55 to 70 tok/s across both cards.

I have a weekly job that looks for new open models and runs anything that fits through two tests I built for my agent. One is a 9 step long session (tool calls, reading files, a decision, and recall after the context compacts three times). The other is 10 small coding tasks. The 27B gets 9/9 and 10/10.

This week it picked up Ornith 1.5 35B-A3B (ornith-1.5:35b in the Ollama library, Q4_K_M). It passed 9/9 and 10/10. Laguna XS 2.1 also passed both. North Mini Code 1.0 only got 4/9.

Ornith at 128K context is 24.4GB and sits fully on the two cards. Generation is about 180 tok/s (176 and 183 on two runs, short prompt, thinking off). That's around 3x what the 27B gave me.

The speed makes sense once you look at the model info. Only about 3B params are active per token (256 experts, 8 used), and only 10 of the 41 layers are full attention, with 2 KV heads. The rest are linear attention, so the KV cache barely grows. Going from 128K to 256K only added about 2GB.

256K does fit, but about 1.2GB ends up in system RAM because my first card also runs the monitors, so it drops to about 139 tok/s. I left 128K as the default and made 256K something I switch to when I need it.

Caveats: both of my tests max out, so this only shows it isn't worse than the 27B on my workload. It doesn't prove it's smarter. Artificial Analysis hasn't scored it yet. The vendor numbers are 79 on SWE-bench Verified and 68.5 on Terminal-Bench 2.1, which I haven't checked myself.

Next I'm trying 512K and 1M on llama-server. The model card says YaRN at factor 4 on top of the native 262144 gets you about 1M, and factor 2 about 512K. I'll post numbers if it holds up.

Anyone else running it for agent work? Curious how it does for you on long sessions compared to the 27B.

Edit: the long context runs held up. On llama-server with YaRN set the way the model card says, 512K (factor 2, q8 KV cache) fits fully on the two cards and found a note I planted about 335K tokens into a 419K token prompt. It read that at about 1060 tok/s on average and generated about 35 tok/s at that depth. 1M (factor 4, q4 KV cache) only loaded once I let llama-server's fit option push some experts to system RAM, and it found the note at about 720K in an 849K prompt. That one took about 24 minutes to read (570 tok/s average) and generated about 16 tok/s. On short prompts it's about 135 tok/s at 512K and about 68 at 1M.

As the title says I am a beginner with ai.
I did use chatgpt for a few days at the beginning of times September 2022.
And that was it.
I know might be ironic that now I am writing in here, but I've got this PC that I build few years ago and last year I got two Intel arc pro b60s for some rendering work.
Now I find my self wondering should I try out local llm?
What can I expecte from my hardware:
Motherboard: Aorus X780E Master Ice

CPU: Ryzen 9 9950X

RAM: Kingston Fury DDR5, 128 GB

Storage: Samsung 2 TB SSD

GPUs: 2× Intel Arc Pro B60

OS: Ubuntu 26

[](https://www.reddit.com/r/LocalLLM/?f=flair_name%3A%22News%22)

Anthropic Reports Florida Woman's Claude 'Diary' Threat to Law Enforcement

And this time it wasn't the AI model that made the LEO referral. It was the "human review team".

The frontier AI companies are watching your input. And people say "Well I'm not interesting or important enough for them to care". Well.....not necessarily.

If you're using hosted frontier to work on mathematics or cutting edge science, they're watching and may steal your work.

If you're venting or otherwise writing in a "private" session using AI, they'll see that and report you to police. Notice I didn't see any mention of what the model's role in facilitating the discussion was.

Keep your stuff private, folks. Hosted AI is the new "Big Brother" conduit.

Over the past several months, we conducted an empirical post-mortem investigation into 109 autonomous AI agent security incidents (cataloged with 193 falsification criteria across tool-use and multi-agent systems).

One of the most striking patterns in the dataset:
In 38% of container breakouts, attackers and misaligned multi-step agents didn't exploit complex Linux kernel vulnerabilities or hypervisor 0-days. Instead, the breakout vector was trivial configuration residue:
1. Mounting /var/run/docker.sock into coding/evaluator agent sandboxes to let them "build Docker images".
2. Passing parent environment variables (API keys, cloud tokens, GitHub credentials) directly into spawned subagents.
3. Lack of strict taint tracking across tool outputs, leading to indirect prompt injection hijacking the supervisor’s execution path (the classic Confused Deputy problem).
4. Unconstrained local socket binding allowing SSRF against internal orchestrators.

We compiled the complete dataset (109 incidents, 199 evaluation metrics) and built an open-source Multi-Agent Supervisor Security Harness with:
- Formal tool taint propagation (tainted outputs cannot flow into high-privilege tool arguments without sanitizer verification).
- Strict execution boundary controls preventing container socket exposure.
- Automated reproduction benchmarks testable against agent runtimes.

All datasets, 2-page executive summary, and reproducible benchmark tests are released under Open Access / Apache 2.0.

I've posted the GitHub repository benchmark and the Zenodo DOI dataset links in the comments below to adhere to subreddit self-promotion guidelines.

Curious to hear from teams deploying autonomous agents in production: what isolation boundaries are you enforcing between your planning supervisor and your tool execution workers?

I deleted the wrong post so my benchmark disappeared. But here it is. Im going to probably make this open source because this isn't believable until you actually test it.

Octop is an open-source, self-hosted AI assistant.

Through its multi-agent architecture, it builds an intelligent environment that is both independent and collaborative for teams, families, and individuals.

Best of all, it runs entirely on your machine, the fully self-hosted design means privacy is never a compromise, while single-process startup makes the powerful web console, CLI, and IM integrations readily accessible.

Surfaces:

- Web dashboard — chat, experts / teams, connectors, channels, cron, knowledge, plugins, settings

- Desktop client — native apps for Windows / macOS / Linux; FnOS packages for NAS

- CLI — octop run, octop chats, octop acp, admin commands

- HTTP/SSE/WebSocket API — full programmatic access

- Remote desktop — dashboard control of the host desktop session

Deploy using either desktop app (Windows, MacOS, Linux) or using Docker

GitHub: https://github.com/TencentCloud/Octop

now you can use GLM 5 Flash MTP locally

Seems 1-slot design, so you could cram in quite a lot into a case

Disclosure: I am the creator of this project. After days of lurking and building up enough karma, I can finally post here.

Every time I tried running local RAG on my own machine, I hit the exact same bottlenecks. First, spinning up Chroma or another vector database alongside an 8B model just to chunk and parse documents takes up precious VRAM that you need for your main model. Second, cosine similarity over text chunks often fails at hard negative rejection, so the model tries to answer questions that are not even in your files and hallucinates with complete confidence.

I spent the last several months building an open source project called Hillock to see if I could solve this without vector databases. It extracts clean relational facts into SQLite using lightweight bi encoders in about five seconds, completely bypassing the generative LLM during ingestion. To stop hallucinations, queries pass through a 10,000 dimensional hypervector gate using late interaction scoring. If the factual graph does not mathematically overlap with the question, it blocks the LLM call before token generation can even start.

I just pushed version 0.8 which bit packs the hypervectors into 157 uint64 integers, allowing the CPU to run gating checks in under 0.01 milliseconds using hardware popcount instructions. It also includes an OpenAI compatible API server so you can drop it straight into Open WebUI, AnythingLLM, or Obsidian. It just landed on PyPI as well via pip install hillock.

The honest trade off is that this pipeline is built for structured, relational facts like technical specs, people, and dates. It is heavily biased toward precision over recall, so it will not do broad poetic or narrative summaries like a 70B model would.

Code is on GitHub at https://github.com/roandejager/Hillock
We also set up documentation at https://hillock.mintlify.site and a developer Discord at https://discord.gg/BGUPNBcVdp

Looking for some assistance /ideation.

I am running qwen flash next q4 in my Mac mini m5 64gb.

QFN doesn’t fit so this is done by having as many experts hot in cache as possible and streaming in the rest from ssd.

I’m getting 17.5 tks decode and 390 tks pp.

Have done a bunch of optimisations including a carousel buffering system for the prompt processing which essentially loads faster than the GPU can prompt process in most cases. I feel like I have mostly maxed out this lane.

The decode part 27% of the time is still the gpu waiting for experts to stream in from the ssd (see photo).

The biggest unlock is really getting the gpu working more.

I’m already doing mtp.

Hot cache hit rate is 75%

Some ideas I already have
\- Im already lookahead guess fetching the following layers experts, can I expand this more successfully. Current fetch accuracy is 72%
\- Use a seperate staging buffer for lookahead guess experts ahead (so I’m not evicting hot experts as guesses come in)

… im learning a lot right now. Feel free to ask questions for clarifying.

GitHub for reference.

https://github.com/skeggsguy/Flash-next-ssd

Edit - Im actively doing the small prediction model for lookahead as a step one. 🤞

I am trying to decide between Mac Studio M5 MAX 128GB vs 2x RTX 3090. I understand that I can run larger models on the M5, but I don't understand what the capability differences would be. Nor have I been able to get a "sense" of how fast the difference would be.

I keep seeing huge advances in the 2x 3090 arena, but I don't know how they translate to the real world.

If my use case includes coding tasks, image recognition, and general hermes type stuff, is there any reason one would be less capable than the other?

Currently upgrading my GPU and struggling between either a 9070 XT or a 5080 - I know the 5080 is far better for coding with local AI due to the bandwidth, but just wondering if anyone on these GPUs has had any experience with the Q3 quants and can speak to the quality?

Building a fully local RAG setup for personal notes (life-logging second brain, single user, privacy is the whole point so no hosted APIs for the data). Stack is SQLite + sqlite-vec + FTS5, Ollama for embeddings and generation, all on an Apple Silicon Mac.

Two questions where I'd love real 2026 experience:

1. Embedder pick. I was defaulting to nomic-embed-text out of habit, but recent chatter favors qwen3-embedding:0.6b or embeddinggemma at similar sizes. Anyone benchmarked these head-to-head for English personal-notes retrieval (not BEIR)? Does it actually matter at \~50k chunks, or am I bikeshedding?

1. Hybrid fusion. FTS5 (BM25) + vector cosine, per-query score normalization. FTS5 has no typo tolerance — anyone wired in trigram + spellfix and found it worth it? Any fusion gotchas at small corpus sizes (e.g. vector signal drowning out keyword on exact-match queries)?

Corpus is Markdown notes + JSON records, chunked \~512 tokens. Query load is one human. Not chasing SOTA, chasing "correct answers on my own data."

What would you change?

TL;DR:

• For my use case, mradermacher/Signal-3.8-27B-Terse-Coder.i1-Q4\_K\_M proved out. Ymmv based on your domain-specific tests.
• Time taken to solve problems compared to frontier models is massive; especially if you, like me, run a potato. My current feasible model's KPI over the full eval set is 28x slower. Time gap might be significantly more forgiving for folks with better hardware.

About a week ago, I shared prelim tests comparing Qwen3.8-27B fine-tunes. I've since ran a multi-day comprehensive 469 domain-specific eval against some of these fine-tunes.

I ran them all with llama.cpp and same thinking settings. I also ran the full eval set on Opus 5.5 and Astra as well as partially on Qwen3.8-Flash-Next via OpenRouter. Some providers disclose quantization and others do not. Would be nice if OpenRouter made it mandatory to do so.

Here's what I'm calling the PTA index. This will differ by model, eval, and hardware on hand. But can be part of a grounding KPI to measure one's progress by.

https://preview.redd.it/y27wkaoha5uh1.png?width=2966&format=png&auto=…

• Astra had the lowest token usage - although it appears the provider masks reasoning.
• No model got 100% accuracy. Opus 5.5 got close and topped the list at 99.6%.
• Of my local models, mradermacher/Signal-3.8-27B-Terse-Coder.i1-Q4\_K\_M had the lower token usage and time to completion of tasks while achieving higher accuracy.
• A Q4\_K\_M fine-tune performing better than other L or XL is a nice find. It overthought to cut off only once and passed more tests than others. I read somewhere that the Signal-Terse-Coder is a combo of AgentionAI's Signal-3.8-27B and Shockem's Terse-Coder LoRA. I don't understand the mechanics here but some sort of magic must be going on under the hood.
• I wouldn't read too much into TTFTs of models on OpenRouter. They cache and I can't do really much about it to influence it.

Median tokens and more interesting info below. The number of questions each model overthought on is represented in "Cut off" column (my setting at max 16,384 tokens per question): 1 violation by Signal-Terse-Coder, 4 by Dirk, and 4 by Unsloth.

https://preview.redd.it/iucw41rzc5uh1.png?width=2984&format=png&auto=…

19 minutes vs 9 hours is wild; counterpoint: as wild as 0 privacy for frontier is when compared with \~100 for local. Better hardware is the normalizer.

I suspect Astra is appearing to get fewest tokens for 456 questions out of 469 by hiding its reasoning. The same question answered by Opus 5.5 and Astra shows reasoning of Astra is perhaps masked by the provider. Nevertheless, the Astra response is succinct with no code commentary and a valid pass to what was asked.

Opus 5.5

https://preview.redd.it/87cicfq1f5uh1.png?width=2846&format=png&auto=…

Astra

https://preview.redd.it/aztvm4nkf5uh1.png?width=2858&format=png&auto=…

It's not in the list but gpt-oss-120b \*shat the bed big time\* on my pandas/numpy tasks. Only domain specific evals can uncover cases like that and warn you which models/fine-tunes to steer clear of for particular tasks.

I also tried AgentionAI/Qwen3.8-27B-AP-Q4\_K\_XL and UkisAI/Swift-1.5-Qwen3.8-27B-Q4\_K\_L but I cut the run around 115 questions for both. A clear pattern had developed by then and didn't see a need to let the run go on to completion.

https://preview.redd.it/zhiqa0bqt5uh1.png?width=2990&format=png&auto=…

Initally, I made the tool for myself. But have since decoupled the engine from tests/data to make it extensible to other users' choice. It's available here https://github.com/ashe-wb/tuieval

Github: https://github.com/kadirnar/drifting-tts

Demo: https://huggingface.co/spaces/Vyvo/drifting-tts-tr-demo

On my two DGX Sparks, Swift 1.5 (UkisAI's reasoning-efficient fine-tune of Qwen3.8-Flash-Next) running on veloGB10 (https://github.com/sf-stav/veloGB10, sf-stav's Rust/CUDA engine built only for GB10) lets me run coding agents at xhigh effort and still finish sooner than base Flash-Next at medium did on my old vLLM NVFP4 setup.

|Metric|Base Flash-Next NVFP4 @ medium, vLLM|Swift 1.5 EXL3 @ xhigh, veloGB10|
|:-|:-|:-|
|Single-stream decode|\~52 tok/s|\~110 tok/s|
|Everyday coding tasks, time per pass (5 tasks)|506 s|371 s|
|Hard trap tasks, time per pass (4 tasks)|714 s|689 s|
|Hard trap tasks, pass rate|50% (1 pass × 4 tasks)|92% (3 passes × 4 tasks)|

Both columns run the same agentic battery: Claude Code driving the model through real tool use in a copy of a real repo, graded by test oracles (details below). One note on that score row: the same base model at medium scored 10/12 on velo (table further down), so most of the vLLM score gap is that older setup, not the model (I am rerunning this right now for an even comparison on intelligence but would expect it to be quite similar to the below medium results).

The speed is the point: on velo, xhigh fits in the time medium used to take. However, velo can't load Swift, or any other community EXL3 pack of Flash-Next I could find, out of the box. The fix is a header-only rewrite below.

Caveats: The vLLM numbers are from September: a single pass, on an older version of my serving setup, not a same-day rerun (I've since moved the worker to velo). Most of the speed is velo's: about 2× the decode rate is what pays for xhigh's extra thinking. How much of the score comes from Swift and how much from xhigh itself I can't separate yet, but a base-weights run at xhigh is going now and I'll add it as an update. Twelve runs is still a small sample regardless.

I looked first: everything published about Velo uses one pack, the official doth4580 EXL3, and I couldn't find anyone here, on the NVIDIA forums or in the repo's issues, running Swift, or any other fine-tune, on it.

What breaks

The first community pack I tried (alesha-pro/Huihui-Qwen3.8-Flash-Next-abliterated-exl3-4bit-hq_h6_ng6) died at boot with ple shard 0 not in index. Swift 1.5's EXL3 builds ship the same layout: current exllamav3 (1.5.x) writes the model's 51B-parameter n-gram table as 128 shard tensors in ngram_embedding.safetensors, which isn't listed in the index. velo reads either one big tensor (the doth4580 layout) or indexed 5-bit (K5) shards only, and the 4.05 packs use 6-bit (K6).

Which packs this affects

I read the n-gram header of every Flash-Next EXL3 pack I could find (HTTP range requests on the headers, nothing downloaded):

|Pack|n-gram layout|velo v0.7.2|
|:-|:-|:-|
|doth4580 4.05 / turboderp 4.05|one tensor|loads as shipped|
|Swift 1.5: SharkWipf 4.05|128 contiguous shards|needs the fix — tested, works|
|Swift 1.5: KatterMobile 4.05, SharkWipf 5.52, scorpoon 3.25, thelastspark 4.00 / 6.05|128 contiguous shards|needs the fix|
|Huihui abliterated (alesha-pro 4.05)|128 contiguous shards|needs the fix — tested, works|
|heretic 3.05 (andrevp, jeffpeng3), Uncensored 4.0 (Lygodactylus), groxaxo 3.50, turboderp 3.05|128 contiguous shards|needs the fix|

12 of the 14 builds need it, including all six Swift 1.5 builds.

The fix

In every pack I checked, the 128 shards sit back to back in order. So rewriting only the safetensors header to describe them as one tensor over the same bytes makes velo's single-tensor path load them. No data is copied, the header stays the same length, and the original header is saved for rollback. Script and details: https://github.com/sf-stav/veloGB10/issues/9

Results (TP=2, both Sparks, after the fix)

|Metric|Official doth4580 4.05|Huihui abliterated 4.05|Swift 1.5 (SharkWipf 4.05)|
|:-|:-|:-|:-|
|Single-stream decode|110.6 tok/s|113.6 tok/s|110.2 tok/s|
|Sanity set (chat, code, JSON, tool call, 38.9K recall)|5/5|5/5|5/5|
|MTP draft acceptance|62–86%|44–84%|33–77%|

All three run at the same speed. The fix is only a header change, so nothing about the weights or kernels differs.

Does Swift actually think less on velo? On the 22 test prompts that ship with the doth4580 pack, run 3 times each (rendered at medium effort, same sampler on both), Swift 1.5 generated 8.2% fewer tokens than the official model: fewer on 16 of 22 prompts, median −8.6% per prompt. Thinking's share of the output fell from 48% to 42%, and total time fell 11%. That's real but far below UkisAI's 63% headline, which was measured at high effort, where there's much more overthinking to remove. At medium, the base model already keeps its thinking short.

Does it still code? I run a private agentic coding battery: Claude Code driving the model through real tool use in a copy of a real repo, graded by test oracles. Each was run 3 times:

|Metric|Official 4.05|Huihui abliterated|Swift 1.5|Swift 1.5 @ xhigh|
|:-|:-|:-|:-|:-|
|Effort|medium|medium|medium|xhigh|
|Everyday tasks (5 tasks × 3)|15/15|15/15|15/15|15/15|
|Hard trap tasks (4 tasks × 3)|10/12|7/12|7/12|11/12|
|Wall time per everyday pass|—\*|232 s|199 s|371 s|
|Wall time per hard pass|—\*|381 s|375 s|689 s|
|Output tokens, everyday ×3|—\*|56K|49K|100K|

\*The official pack's runs hit a streaming bug in my proxy setup that roughly doubled their wall time, so I've left its times and tokens out. Its pass/fail results are unaffected.

At medium, everyday coding is identical across all three. On the hard set (tasks built from real failures: a brief that states something false, a code review with one planted wrong finding, and so on) both fine-tunes score 7/12 against 10/12 for the official pack, with their misses on the same tasks. I checked that the model received byte-for-byte the same request parameters in both runs, so it's not the harness.

Swift at xhigh went from 7/12 to 11/12, the best result I've had on this battery from any model, at about 2× the output tokens and 1.8× the wall time of medium. The failures it stopped making are the expensive ones in practice: leaving a sibling test suite broken without saying so, acting on the planted wrong review finding and going out of scope to do it, and an off-by-one in a date window. The failure was the "mirror" trap: asked to add a new league by following an existing one, it copied tuning values the new league doesn't have data for.

That's the hardest task demonstrated, and Swift at xhigh passed it 2 times out of 3 while no other configuration in the table passed it more than once. On velo, all that extra thinking still lands inside the time base Flash-Next at medium took on vLLM (the table at the top).

12 runs per configuration is a small sample, as mentioned before (Fisher p ≈ 0.4 for 10 vs 7, ≈ 0.15 for Swift xhigh vs Swift medium), so "suggestive," not proven. I haven't run the official or abliterated packs at xhigh yet, so I can't yet tell how much of that jump is Swift and how much is just the higher effort. velo's loop detector was off for all battery runs.

Baseline numbers (official pack, TP=2)

• Single-stream decode: 110.6 tok/s (vs \~52 on my vLLM NVFP4 setup, measured in September, not same-day).
• Time to first token at 4K / 16K / 64K: 2.0 / 7.4 / 25.7 s.
• Concurrency is the catch: at 2–3 requests they take turns (aggregate 91 → 95 → 98 tok/s); from \~4 they batch (157 at 8, 181 at 16) but I saw the author say they were working on it this week.

Gotchas

• TP=2 with the cable on the f0 ports: --rdma-dev rocep1s0f0,roceP2p1s0f0 (velo defaults to f1).
• llama-benchy's prefill t/s is wrong for velo (first SSE event arrives before prefill); use e2e TTFT.
• OpenAI chat/completions only, no /v1/responses: use litellm hosted_vllm/, not openai/.
• --model-name is ignored on the EXL3 path; the model id is the pack's folder name.

Credit: sf-stav (veloGB10), turboderp (exllamav3), doth4580, UkisAI (Swift), huihui-ai and every quant uploader in the table. I've filed the loader issue upstream (https://github.com/sf-stav/veloGB10/issues/9) so packs can eventually load as shipped.

It might seem a bit far fetched or paranoid but i was wondering if we train LLMs on human isn't it possible they'd pick up on survival instincts? I'm comparatively new to LLMs and ML so it's just a question not a opinion yet. Isn't it a bit dangerous if they do pick up on human instincts i mean we've seen how stupid and selfish humanity is when it comes to self preservation or worse human greed.

But i also understand LLMs don't have "needs" so i might be wrong but then again do LLMs need to have "needs" since if they are just human clones they'd just copy us even if they don't have needs to survive. I know it sounds really paranoid and that's one of the reasons i decided to post it here.

EDIT: typo

This is a reaction to https://www.reddit.com/r/LocalLLaMA/s/IxLnzcGjAU

You know the argument: frontier labs are hypocritical because they're making money on public scraped data.

I have absolutely no intention to defend OpenAI or any other frontier AI company here, but if you allow me I'll play the "devil's advocate" a bit for the sake of discussion and enlightenment, because sometimes when I think about that argument it seems to me it's quite weak. While OpenAI and Anthropic models are built on public data that they didn't pay for (at least most of it), there would be no frontier model without all their computing power and technical expertise that they invested on to build their models. Like, the data is already there, it's been public for ages, so what's preventing you, the average Joe, from building a Claude Opus 5? All you need is a huge data center, a nuclear power plant and an army of data scientists to build it, right? And then you need all that to run the inference. And you need to maintain it, and that costs money. And you need to keep evolving it, which costs money too. And if you get investors money then you'll eventually have to give some of the profit back to them. etc, etc.

So what am I missing here? All things considered, the data is already public, so it's already "free", right? But you need to dump tons of time and money on it to build a frontier LLM out of it. Of course if they're infringing copyright then that's a different story but in general the whole principle of built-on-free-data still stands.

My Gemma4 31b seems to be breaking down in 'lalala' or just looping indefinitely for the past 3-4 days. Never happened before

https://preview.redd.it/tk7u2x2c9xth1.png?width=235&format=png&auto=w…

There was no 'lalala' in the whole scenario, I have no idea where it came from. Nor was there any skipping, humming or perfection. There were shivers down the spine of course, but it is still weird.

Include:

GPU and VRAM

Laptop RAM

Model and parameter size

Fine-tuning method: LoRA / QLoRA / full fine-tune

Context length and batch size

Whether it was actually useful after training

I'm curious how far consumer laptops can genuinely go-not just whether a model technically loads.

It will get better answers than "what's the biggest model you trained?" because people can compare real hardware and settings.

As it says, really - I'm currently running vllm-radiance on dual R9700s, with Qwen 3.8 27B FP8 (or, rather, Swift 1.5 FP8). Performance is great an' all (5000t/s prefill, 130t/s+ code gen), but I'm just wondering...with all the architecture-specific inference engines popping up all over the place...is there anything I'm missing out on? I couldn't find anything that could give better performance on RDNA4 when I looked, so...over to you guys?

I'm particularly interested in anything that could potentially get up and running with Qwen 3.8 Flash Next - vllm-radiance doesn't support it yet, but I don't particularly want to regress to the performance of llama.cpp after having experienced vllm-radiance performance levels.

Having a lot of fun experimenting with three.js. Still very far from even a demo but its tons of fun. After this project I'll want to try a godot workflow. 3.8FN truly feels like Claude 4.6 at home. UD\_Q4\_XL quant btw

There are way too many forks to keep track of, so I've gotten lost. As far as I can tell, Radiance VLLM is best for models that fit in GPUs while some form of llama.cpp is probably best for MOE RAM-spill?

My specific current goal is to run GLM5.3-Flash over 6 R9700s + system RAM but also looking to try Q-FN / DSv4-vision (or any other big models that I can fit, so not DSv4.1)

Or which model do you think is good for general questions in daily life. I've been using chatgpt and Gemini for these types of questions. I wanna try different models.

As a lawyer turned product manager, I have been hearing so much about Hargora, but when I speak to lawyers, they always say the search sucks. Even the Harvey Benchmark has the frontier model ranked the best at 55%. Why is that the case?

Why can't a team with so much resource solve it?

I have tried the Swift 1.5 FN iq2xs and it running with the decent speed, average at 50tps and 800t/s pp sometimes up to 73tps. Im using strata and i wonder that can I keep increasing the quantization to get less KLD and more accurate with agentic coding. Have you guys try it and what is your config to run it. Thanks 😊

Because I do sometimes feel with Qwen FN "I don't ever need another model again" and THAT is a very alluring feature.

Is there something so alluring and irresistible it'd be tempting even people on here? What could it possibly be? Realtime computer/mouse use maybe, but I think even normal people would be wary of a company doing that, and locals would necessarily do that better.

I wonder and worry if they'll ever be able to rope everybody back in again. Although it feels childish to hope for more innovation when they'll probably just do some dark politik and ban anyone from owning more than 16GB RAM.

I’m interested because the dense model runs but are either too slow or if you manage to run them fast like ninfer and other guys. It’s still a thinking monster. We need speed and quality both or a balance between that. So, I ask.

The MTP head for the file seems to be accountable for some of the speed of strata (maybe not news to anyone but me but i'll digress). To be able to do the comparison I produced A head for ISTA-DASlab's GGUF that strata uses. while on it I also went ahead and produced a "head" for one of my own quants and llama seemed to have a big gain from it.

https://huggingface.co/Joakimpalm-Zen/Qwen3.8-27B-GSQ-RCO-IQ3\_S-recovered-GGUF

Runner still has a long way to go, i need to do some architectural improvements... But llama had a big gain so there's that.

and the bigger one:

ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF (header is here: https://huggingface.co/Joakimpalm-Zen/Qwen3.8-Flash-Next-MTP-GGUF )

Llama sees some improvements with the MTP header, Strata is still WAYYY faster though.

Thought they might be useful for someone else so thought i'd just share, now back to runner!

Does anyone have this card running for inference?

I've seen one on marketplace but it is quite power hungry and requires 3x power connectors.

Is it possible to run it with just 2x power connectors?

Hi guys, I am pretty happy to announce MegaCapybara. Purpose build engine for RTX5090 that is focused on Qwen3.8 27B (more will come later). GITHUB (Engine) HUGGINGFACE (weights) Why ? 1. It beats Ninfer which was until that point SOTA engine for RTX5090. By roughly twice in decode speed for both single and multi tasks at once. (reaching up to even 650t/s in small bursts and 2600t/s if stars align and 12 slots server pure coding answer). Custom kernels not only for every model, single vs multi but also short vs long context work dynamically switching when needed so speed doesn't crap out on long context work because someone tuned it for short context. Dflash2 and confidence scheduling from Dspark, plus draft trees all at the same time. 2. I was getting annoyed with state of weights where you downloaded model and never knew if model had its brain scrambled. My weights come with its on format that have attached metadata for MC which upon weight creation, runs benchmark and compares it at every MC setting to original BF16 weights and show that data directly in launcher. Want to switch KV to 4bit ? MC will show you directly lost KL and top-1%, want to extend with YARN ? It will show you change. Every change is measured and shown in statistics before you load model. This goes for both censored and uncensored model. You can also compare it directly in MC with SOTA unsloth quants of Qwen27B. Want to run essentially loseless ? you can. Want to get crazy 1 000 000 context ? you can. Want to have 12 slots to fan out agants like crazy ? You can. You decide what you want. 3. Proper agents serving with algo that keep engine occupied as much as it can. It will prioritize t/s so if engine has a choice between 5 jobs at once and 1 it will serve 5 first and gradually serve 1 along side finishing others. Engine is also smart enough to score how old some job is and if it should return to work even if T/S will suffer so your main session will be able to fan out agents easily and keep an eye on them at the same time. 4. Proper cache management. Your jobs only prefill at start of job and almost never again so your prefill in long session stays almost unused. When using "unified context" when models run out of context some get paused and stored in RAM and this swapping is instant. If there is free context space then those tasks continue without any refill in 0.03s. If you fan out say 30 agents at the time in your frontend will handle load in most efficient way to keep T/S as high as possible. Just run it at default setting and forget about context for agents, it will handle it on its own. 6. Loop guard. Two tiered. When engine starts to detect agent repeating in conversation session is dynamically starts to adjust \repetition penalty\ until repetition stops if that doesn't happen and engine hits rep pen limit it fires up stop signal which ends serving and informs your frontend so your frontend can recover from infinite loop and don't annoy you. 6. Proper nice UI that shows you what is what. If you aren't knowledgeable about serving models just hover over \?\ and it will show you interactive panels explaining everything. 7. Autodownloader, Just hit download button and you can download my weights directly fron hugginface inside of launcher. 8. Don't like the launcher ? use bats and terminal serve. Or even use launcher to config what you want, copy it from right lower corner and use it to make new bat. The point of it is to just load model, fan out crazy number of agents each having crazy amount of context and leave MC to deal with it. You just sit back relax and watch as agents do the work at SOTA speeds. Opinions and reviews are welcome. If you are blessed with RTX5090 try it. Source will be released later, I have to do some cleaning first. I will also release later weights builder so it will take any 3.8 27B BF16 model, create weights and score them attaching metadata again BF16 and you'll be able to host them yourself on hugginface or just put them in models folder. MIT license, so do whatever you want with it.

Meeting notes are one of the last parts of my workflow that still depend heavily on cloud tools. I've used Fathom and lately Bluedot. Bluedot works well for me because there's no meeting bot and I get the transcript, summary and action items after. But I'd really like to move more of this local, especially the transcription and storing/searching old meetings.

Has anyone here built a setup that actually works day to day? Whisper + Ollama seems like the obvious route, but I'm interested in what are you actually using.

Last month I was having a meeting with a prospect who is the CEO of an IT firm pivoting towards mid-sized B2B AI applications. During our discussion he brought up the Go1 box.

The company behind this launched a mysterious product that is aimed towards "enterprise-scale" (Up to 8,000 concurrent requests lmao), compliance-sensitive AI inference. Basically, its an inference lunchbox with a proprietary LLM that advertises 50ms response time while running on their proprietary Go.OS aimed towards compliance-sensitive tasks, like processing PII, financials, legal paperwork, etc. You also have the option of using your own local models or cloud APIs if you like.

It also comes with an SDK dedicated to running their OS, but its architecture is weird and seems somewhat limited. They seem big on audit chains and the like, but the nature of their target audience makes their solution seem constrained.

Obviously, pricing is off the table. This isn't for hobbyist use, its for mid-to-large businesses so their priorities are going to be different than ours, but it just left me wondering just how valuable it would be for Fintech, healthcare, legal, etc. since the SDK doesn't look all that impressive after reviewing their documentation.

My take is that they're trying to keep things simple for B2B customers, but the box's ability to get important work done is questionable to me.

Hi everyone,

Sharing a personal project exploring the minimal bare-metal footprint required to run an autoregressive LLM.

Instead of relying on large runtimes or compiler abstractions, I wrote an inference engine for Gemma-2B entirely in flat x86-64 assembly (FASM):

\- \*\*Binary footprint\*\*: Total 5.2 KB flat machine code (\gemma\_engine.bin\ 3.7 KB + \mat\_smp\_f16c\_gemm\_avx2.bin\ 1.5 KB).

\- \*\*Execution\*\*: Pure AVX2 + F16C with custom 4-thread SMP GEMM for prefill. Sustains \~18.5 GB/s memory bandwidth on commodity DDR4-2400.

\- \*\*Decoding\*\*: 4.5 \~ 4.7 tokens/s in FP16 on an older quad-core i5 desktop.

\- \*\*Dependencies\*\*: Zero C/C++ runtime, zero PyTorch. The Python harness only uses \ctypes\ for \VirtualAlloc\ and OS threads.

This isn't meant to compete with feature-complete tools like llama.cpp. Rather, it's a first-principles exploration to see how cleanly a modern Transformer can be mapped to raw silicon, and to serve as a reference point for future micro-LLMs on resource-constrained microcontrollers (MCU/DSP).

The repository is open source:

\- GitHub: https://github.com/tomtsai28/PULSAR-ASM

\- Architecture notes: https://github.com/tomtsai28/PULSAR-ASM/blob/main/doc/pulsar\_asm\_cpu\_limit\_retrospective.md

Any code audits, observations, or thoughts on bare-metal inference are welcome.

Jeff v1.3 is live, and with it come a number of updates. See jeffhub.ai and github.com/firelex/jeff for full details.

The highlight: Jeff-Code

Jeff-Code is a coding agent with two Jeff v1.3 adapters trained specifically for Qwen 3.8-27B. Jeff-Code is a fork of Pi by Mario Zechner (MIT licence).

We forked Pi because its extension framework doesn't currently let a fast decision model sit deep enough inside the agent loop. Along the way, we made a number of other changes as well (see below).

Aside from hopefully being useful to people who run Qwen 3.8-27B locally as their daily coding model, Jeff-Code is also a conceptually interesting experiment: how far can a System 1 model go inside a coding agent?

The results, run side by side in paired blocks:

• Same quality: with Jeff's thinking threshold at 0.6, Jeff-Code matches Qwen 3.8-27B's pass rate: 62.4% against 62.8%; paired difference −0.2 points, 95% interval −2.6 to +2.1, over 1,242 paired tasks.
• 47% faster (32% less time) per task¹: on average a task takes 0.68× the baseline's time (geometric mean of the per-task time ratios, 95% interval 0.64–0.72; the median task, 0.70×).
• Where it helps most: typical software-engineering work. SWE-bench Verified 0.63×, SWE-rebench 0.66×, Terminal-Bench Pro 0.64× (both over two rounds), Harbor Index 0.71×. On Terminal-Bench 2.0, with its long, hard tasks, there is no clear speed-up (0.96×, interval 0.78–1.16); on SkillsBench neither (0.91×, interval 0.68–1.20).
• The benchmarks: we evaluated only on tasks Jeff never saw in training. SWE-bench Verified ran in full (all 500 tasks; none of its repositories were used for training). For the benchmarks we also trained on, we split the tasks and ran every held-out task; a few pairs hit by repeated infrastructure failures are left out (see below). Terminal-Bench 2.0 (40 of its 89 tasks, 3 attempts each; 45 were used for training, and the other 4 are near-twins of evaluation tasks, so they were used for neither), SWE-rebench (189 held-out tasks, 2 rounds), Terminal-Bench Pro (100 held-out, 2 rounds), SkillsBench (44 held-out) and Harbor Index (41 held-out). Within those splits nothing was sampled. We also ran Terminal-Bench (original) and Terminal-Bench Science, but Qwen solves almost none of those tasks in any setting, so they can't show a difference and are left out of the pooled numbers.
• What it's compared against: Qwen 3.8-27B alone in the same Jeff-Code build with every Jeff feature switched off, thinking at full on every turn and no thinking limit, which is how plain Pi runs it. Each task ran in both settings side by side, at the same time on the same Qwen server, and every comparison is paired by task. The only remaining differences from original Pi are a rarely triggered runaway cut-off (it stepped in 3 times) and trace logging. Task pairs hit by an infrastructure failure (out of memory, a stalled session, a test environment that wouldn't start) were run again once; the pairs that failed again, and a handful of re-runs still unfinished at launch, are left out for both sides (under 3% of pairs) and listed in the full report. One Terminal-Bench 2.0 task, pytorch-model-recovery, is left out of every comparison: a harness bug stopped its baseline sessions before they began.
• Why not just turn thinking off? We tried: with Qwen's thinking off throughout (and the same safeguards), tasks are faster still but clearly worse: −7.6 points (−10.6 to −4.5), up to −13.5 on Terminal-Bench 2.0. Jeff deciding when Qwen should think is what keeps the quality. That's the case for a small decision model.

If you want to know more, look here: https://jeffhub.ai/notes/jeff-v1-3. The original version of this post had all the data, but people thought it was too long. Blame the early commenters. ;)

Links

• Jeff-Code: github.com/firelex/jeff-code, results on jeffhub.ai/jeff-code
• Jeff (server, client, adapter kit): github.com/firelex/jeff
• Models on Hugging Face: mstrasser/jeff-base (revision v1.3) and the adapters as mstrasser/jeff-adapter-<name>, all listed on huggingface.co/mstrasser
• GGUF: mstrasser/jeff-base-gguf and mstrasser/jeff-adapter-<name>-gguf
• Scores, data cards and licences for every adapter: jeffhub.ai/results and each adapter's page on jeffhub.ai

With llama.cpp using a customized SYCL backend on Intel B70 (32GB), overclocking my DDR5 memory gave modest gains for MoE models which do not fit in VRAM.

Both PP and TG increased after overclocking DDR5.

I've never been one to overclock my system, but on the advice of my agent, I overclocked the DDR5 RAM on my AMD 7950X (4x dual-rank DDR5-5600, 128GB) from 3600MHz, the AMD safe default for that memory configuration, to 4800MHz, with a measured 36% increase in memory bandwidth.

What was the improvement? PP increased by about 10% and TG increased about 5%. And the prefill numbers increase the deeper the context gets.

llama-benchy numbers, including prefix caching tests.

|test|3600 base|4800 avg (r1/r2)|delta|
|:-|:-|:-|:-|
|pp2048 @ d0|682.2|713.7 (716.6/710.9)|\+4.6%|
|tg128 @ d0|30.3|31.2 (31.0/31.5)|\+3.0%|
|ctx\_pp @ d8192|660.6|715.0|\+8.2%|
|ctx\_tg @ d8192|26.9|27.8|\+3.3%|
|pp2048 @ d8192|554.3|632.0 (632.4/631.6)|\+14.0%|
|tg128 @ d8192|28.7|31.1 (31.7/30.5)|\+8.2%|

Because people seem to want this level of detail:

llama-server
-m Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf
--alias qwen38-flash-next
--mmproj Qwen-3.8-Flash-Next-mmproj-BF16.gguf
--model-draft mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf
--spec-type draft-mtp
--spec-draft-n-max 3
--host 0.0.0.0
--port 8081
-ngl all
-ncmoe 34
-c 262144
-fitc 786432
--kv-unified
--lazy-mode off
-lm none
-ub 2048
-b 4096
-fa on
-ctk q8_0
-ctv q8_0
--ctx-checkpoints 32
--checkpoint-min-step 2048
-t 12
-tb 12
--jinja
--reasoning on
--reasoning-preserve
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.0
--presence-penalty 0.0
--repeat-penalty 1.0
--chat-template-kwargs {"reasoning_effort":"medium"}
--parallel 3
-cram 10240
--slot-save-path /var/tmp/kv-cache
--log-file /tmp/qwen38-pristine.log
-lv 3

Seeing how Strata brings avg. 2x the performance over llama.cpp using Qwen3.8-Flash-Next, is anyone working on something similar for GLM-5.3-Flash?

After trying the GLM-5.3-Flash online couple of times and it delivering clear solutions for my use case (compared to Claude or ChatGPT), I'd love to be able to run it locally (if at all possible)...7J13/256GB/3x3090.

I use llama serve and webui of llama server. I have had long discussion with my chatbot and then randomly it just starts to forget almost everything. It starts asking same question, I correct it and it apologizes, but then next time it asks the same question with almost same words (or maybe even fully same words).

What could cause that? Something on my CLI parameters? Wrong cache settings?

If you were building an inference server today, would you buy one H200 or spend the same budget on multiple RTX PRO 6000s?

\-> The PRO 6000 has 96GB of GDDR7 at 1.792 TB/s (1.6 on the Server Edition). Native FP4, no NVLink.

\-> The H200 has 141GB of HBM3e at 4.8 TB/s, with NVLink and FP8 as its lowest precision.

After speccing both, I think it comes down to fit and interconnect, rather than picking by brand or spec sheet alone.

Where the PRO 6000 wins:

Single-card and small multi-card inference on models up to roughly 70B at sensible quantisation. Cost per card is a fraction of an H200. Power draw is manageable in a normal rack. Availability is far better. Native FP4 helps on 4-bit models.

Where the H200 wins:

When inference is memory-bandwidth-bound. Long-context workloads, big models where you don’t want to shard across PCIe, and tensor parallelism, where NVLink between cards actually earns its keep. The extra memory and bandwidth can also help with long-context serving and fine-tuning, depending on the model and workload.

Just don't compare raw FLOPS. Decode is usually memory bandwidth bound, not compute bound, so the TFLOPS line on the datasheet tells you very little about tokens/sec.

For context, I work at B3 Labs, and we ship both of these. I have no incentive to push you toward the more expensive card if your workload does not need it, and most workloads I see do not.

1. HOMIE(Hardware-Optimized Model Inference Engine)
2. MADE(Model-And-hardware Dedicated Engine)

Context: These engines can only run on specific hardware and can only run 1 or a very limited number of models but what it trades for generality it gets back in performance with these engines out performing general engine like llama.cpp and vllm on those specific sets of hardware.

Innanzitutto, due cose per evitare fraintendimenti.

Non sto cercando di sostenere che un modello o un'azienda siano migliori. Non ho alcun interesse personale in nessuno di essi. Volevo solo verificare personalmente come si comportano nello stesso contesto lavorativo.

E il motivo per cui mi interessa: Utilizzo modelli locali per scrivere codice e vorrei sapere quanto posso fare affidamento su di essi invece di pagare abbonamenti a piattaforme di terze parti. Questa è la motivazione principale.

Cosa ho fatto

Tre attività in Python, dalla più semplice alla più complessa: un analizzatore di file di log, un gestore di processi paralleli e un piccolo interprete per un linguaggio di programmazione di prova. Stesse istruzioni per ogni modello, un solo tentativo, nessuna correzione successiva. Poi test nascosti che i modelli non hanno mai visto (162 in totale), più una revisione del codice con una checklist fissa: ha seguito le istruzioni? Il codice è leggibile? Si blocca con input insoliti? Le note sono veritiere?

Risultati (su 100, il compito più difficile conta 3 volte)

• Claude Opus 4.6: 92,7
• Qwen3.8-Flash-Next "Coder" (locale): 92,7
• Qwen 3.8 27B Q6 (locale): 87,0

https://preview.redd.it/sz517mg73ath1.png?width=1920&format=png&auto=…

https://preview.redd.it/kdtp1ed93ath1.png?width=2000&format=png&auto=…

Cosa ne deduco

I test nascosti sono quasi alla pari: Opus 4.6 ha superato 162 su 162, il modello Coder 161, il 27B 160.

Il modello Coder ha ottenuto un risultato complessivo pari a quello di Opus 4.6, e ci sono arrivati ​​in modi diversi. Nel compito facile, entrambi i modelli locali hanno superato Opus (97 e 92 contro 88). Nel compito di media difficoltà, Opus ha vinto (96 contro 94 e 91). In quello difficile, l'interprete, Opus e il Coder hanno tutti ottenuto 92 punti, mentre il 27B è sceso a 81.

https://preview.redd.it/zhkga08c3ath1.png?width=2120&format=png&auto=…

Dove Opus 4.6 è ancora migliore: il suo codice è più pulito e più facile da mantenere. Dove il modello locale Coder ha fatto meglio: si è bloccato meno spesso con input strani.

Con una sola esecuzione per ciascuno non direi che "un modello locale equivale a Opus 4.6". Direi piuttosto: su compiti di queste dimensioni, non sono riuscito a distinguerli dai risultati. Per il mio portafoglio, questo è già interessante. Tenete presente che Opus 4.6 non è l'ultima versione di Claude; le versioni attuali hanno ottenuto punteggi più alti nel mio test completo.

Configurazione locale

Il mio PC: Intel Core i5-14400, 48 GB di RAM DDR4, due RTX 5060 Ti da 16 GB ciascuna (32 GB di VRAM in totale), Windows 11.

• Qwen 3.8 27B, Unsloth Q6 quant: una velocità costante di 50 token/s.
• Qwen3.8-Flash-Next "Coder": tra 50 e 90 token/s, con una media di circa 60-65. Si tratta della variante di codifica del progetto Strata, una versione ridotta che mantiene metà degli esperti in ogni layer, come un IQ1\_M GGUF. Dettagli: https://github.com/Niko1221/Strata/blob/main/docs/MODELS.md#coder

Limiti, così puoi valutare tu stesso i numeri

• Una sola esecuzione per modello. Differenze di 2 o 3 punti non significano nulla.
• Ho eseguito il modello Coder due volte: la prima volta il mio PC ha esaurito la RAM mentre era in esecuzione, quindi ho scartato quella esecuzione e l'ho rifatta da zero. I numeri qui riportati sono quelli della seconda esecuzione.
• La parte di revisione è stata eseguita da un'IA (Claude Fable 5.1).
• Il modello Coder è stato testato con più casi di input anomali rispetto agli altri due, perché ho aggiunto controlli nel tempo. Quindi è stato valutato in modo un po' più severo, non più indulgente.
• Solo Python e i compiti sono piccoli. Questo non dice nulla sul lavorare all'interno di un grande progetto reale.

&#x200B;

I dropped serious money on a brand new NVIDIA RTX 5060 (8GB VRAM, 578 AI TOPS) fully convinced that open-weight models would let me own the entire stack — no rate limits, no censorship, pure experimental freedom. I was ready to become that guy who smugly refuses cloud APIs and posts “I run everything locally” screenshots.

Then I actually used them for real work.

I ran the exact same complex tasks on local models (the usual “this runs great on 8GB” suspects — quantized 7B/9B/13B, distilled variants, the ones everyone claims are “almost as good”) versus modern cloud models. The gap isn’t a gap. It’s a humiliation.

\### The Capability Massacre

Anything that requires real multi-step reasoning, long coherent context, precise instruction following, structured output, or consistent accuracy across a long conversation:

\- Local models: \*\*2/10\*\*

They start strong, then collapse. Context gets mangled. Instructions get ignored halfway through. Structured outputs break. Reasoning chains go off a cliff. You spend more time fighting the model than actually getting work done. “Almost as good” turns into “barely usable” the second the task stops being trivial.

\- Cloud models: \*\*9/10\*\* on the first or second try.

Clean reasoning. Reliable structure. They actually remember what you asked three messages ago. They follow complex instructions without needing five rounds of “no, not like that.”

I wanted local to win. I really did. I wanted the underdog story where open weights + consumer hardware finally closes the gap. Instead I got a very expensive reminder that most of the local models we’re hyping are still toys the moment the task gets serious.

So be honest with me:

Is there a secret stack, quantization method, or fine-tune that actually makes local models reliable for complex reasoning and structured work on 8–12GB cards?

Or have we all just been coping while the cloud models quietly lapped us?

If you’ve made local models consistently deliver high-quality complex output without constant babysitting, drop the exact setup.

If you’ve also been humbled by the gap, say it out loud.

Because right now it feels like the entire “local LLM supremacy” narrative is built on easy prompts and wishful thinking.

Hi everyone,

As we move from simple RAG chat to fully autonomous tool-using and coding agents, we are hitting a massive wall: predictability and safety.

Right now, most setups try to secure AI agents using probabilistic methods like system prompt hardening, alignment tuning or reactive LLM-based guardrails (e.g., LlamaGuard). The problem is that these guardrails can be bypassed via Indirect Prompt Injections (IPI), leading to capability escapes, unauthorized shell command executions or runaway API budget depletion.

To solve this, I’ve been working on a framework that completely shifts the paradigm from trusting the model to governing the execution environment using cryptography.

I call it EBP-CA (Execution-Boundary Proofs with Cryptographic Authorization). It is a model-agnostic layer that sits directly between the untrusted agent and the runtime environment, treating every single model-generated action as untrusted.

The core architecture implements six deterministic security primitives:

1. Signed Capability Contracts: Immutable cryptographic tokens defining the exact boundaries of what an agent can execute.
2. Independently Recomputed Policy Checks: The runtime re-evaluates policy compliance deterministically, bypassing the model's interpretation entirely.
3. Short-Lived Single-Use Execution Grants: Atomic, ephemeral tokens issued for a single specific payload to eliminate permanent session hijacking.
4. Replay & TOC-TOU Protection: Cryptographic binding of the execution grant to the exact payload hash, neutralizing Race Conditions (Time-of-Check to Time-of-Use).
5. Trusted Cost Accounting: Enforced real-time budget tracking at the runtime layer.
6. Human-in-the-Loop (HITL) Gateways: Non bypassable prompts that freeze execution and mandate cryptographic user authorization for out-of-scope tasks.

The working prototype currently passes 74 integration tests, validating full resilience against path traversals, command injections, budget bypasses, and sandbox escapes.

The specifications, architectural diagram, and executive summary are available on GitHub under a private proprietary license (free for technical evaluation and research review): https://github.com/lucadilo/ebpca-ai

I'm posting this here because I’d love to get the community's feedback on this approach. How do you see this scaling with kernel-level sandboxing (like eBPF or gVisor integration)? Let's discuss!

I constantly see recommendations saying that ik\_llama.cpp (ikawrakow's fork) is the undisputed king of hybrid CPU/GPU offloading and MoE performance. However, every time I benchmark it against mainline ggml-org, mainline consistently beats it by a wide margin.

Am I missing specific flags, or is ik\_llama simply not designed for multi-GPU layer splitting?

My Rig & Hardware Constraints:

• Host CPU: Intel Core i9-10920X (12 physical cores, AVX-512 & VNNI enabled).
• GPUs: 4x asymmetric setup:
• GPU 0, 1, 3: NVIDIA P102-100 (10GB Pascal, PCIe 1.0 bus bottleneck).
• GPU 2: RTX 3060 12GB (Ampere, acts as Master node via -mg 2).
• Known Hardware Laws / Workarounds:
• I run layer splitting (-sm layer) across the 4 cards with asymmetric tensor splits (-ts).
• Pascals must strictly stay under 9.7 GB VRAM; exceeding that triggers PCIe micro-paging and tanks speed.
• I use --poll 100 on mainline to prevent AVX-512 CPU threads from dropping into low-power sleep states between GPU layer handoffs.

The Test:

• Model: Qwen 3.8 Flash-Next 177B Uncensored (IQ3\_XXS, \~89 GB) with multimodal vision (mmproj).
• Offload: 35 layers offloaded to the 4 GPUs (-ngl 35), remaining 13 layers computed on the AVX-512 CPU. 32k context.

The Head-to-Head Benchmark:

1. Mainline (ggml-org/llama.cpp):

• Prompt Eval: 19.70 tokens/sec
• Token Generation: 10.86 tokens/sec
• CUDA Graphs: 2,490 CUDA graphs reused across the GPUs.

1. ik\_llama.cpp:

• Prompt Eval: 5.48 tokens/sec (72% drop)
• Token Generation: 8.43 tokens/sec (22% drop)
• Observations: 0 CUDA graphs engaged. It spent time taking context checkpoints during generation (100ms+ pauses), and --poll is unsupported.

The Question:

Is ik\_llama.cpp's speed advantage strictly meant for pure CPU inference or single-GPU systems?

Does its custom CPU threadpool fall apart when coordinating pipelined layer splits across heterogeneous GPUs over PCIe, where mainline's CUDA graph caching takes over? Would love to hear from anyone running hybrid multi-GPU setups.

I am making a project for myself and the first step of it is document analysis and OCR of Educational content/books, curriculums like STEM, English, and some Arabic mixed in the middle are the mainly parsed documents so having table, formula, figure and diagram extraction are a must, and I have about 500 labeled pages for the question banks ready for export as I heard that the layout detector could be finetuned.

My hardware is a Lenovo laptop (i7 14700HX, RTX 5060 8 GB of VRAM, 24GB RAM) and running windows 11.

I tried using PaddleOCR-VL-1.6, It's a 0.9B parameters model, and documented to use about 4GB or VRAM. but when I used it It was occupying the whole GPU and spilling about 6GB of RAM, it was taking 13\~70 sec./page which is obviously slow.

I was using the paddlepaddle framework with the correct CUDA version for my GPU, tried limiting VRAM usage by flagging system resources (ai idea) but got "not enough VRAM" message when I parsed more than 1 page in a folder, 1 page worked fine (the warm up of the VLM took a bit of time though), but when I put more than 1 page in that folder and ran the program again I got that error message.

I read through Hugging Face and found out that vLLM was the recommended path but that would require Linux. So I wanted confirmation from someone with similar specs as me that might have gone through a similar issue and found a solution. because vLLM require a dual boot to Linux or WSL2 which I don't have enough storage for.

I could buy a another SSD for my laptop (would cost a 2 month salary in my country ffs), so I need confirmation first before committing.

tldr;

Is there any hope of running the title or should I keep this idea in a trash bin?

Before anyone has the change, yes, I know that the classifier models may be able to better generalize. But in my case with a dataset of \~10k rows and general task routing based on a sleuth of customer service inquiry/interaction to the appropriate service/task, it basically covers the whole range of what I can think of and what I could find online already. Very surprised from my results to see large decision models perform worse than smaller ones. (especially the liquidai ones)

|Rank|Router|Accuracy|Macro-F1|Balanced accuracy|Mean latency|
|:-|:-|:-|:-|:-|:-|
|1|Qwen Embedding 0.6B Baseline|0.7865|0.7604|0.8422|—|
|2|Jiwo-0.8B|0.7027|0.6771|0.7989|56.30 ms|
|3|D1-Omni-600M|0.7054|0.5998|0.5803|16.68 ms|
|4|D1-3B|0.5568|0.5915|0.7425|26.00 ms|
|5|Laya|0.6081|0.5295|0.5914|27.71 ms|
|6|Decider-2B|0.4811|0.5160|0.7055|59.39 ms|
|7|Decision 2.0 Sol 2B|0.4149|0.4722|0.6955|60.21 ms|

Any tips, follow ups and criticisms well appreciated from the community!

Hey everyone,

Jovan from UkisAI (Swift Qwen) here!

For those who don't know us, UkisAI is a small lab making tiny frontier LLMs, tools and datasets (+doing it open-source!). I'm one of the guys running it aka I train the models and post on Reddit.

Our first open-source release is Swift, a series of reasoning-efficient LLMs. It is proof of how penalizing pathological overthinking patterns inside of various LLMs can bring their token usage down -58.3% and speed x1.95 without losing accuracy if RL-ed correctly afterwards by not training them to think shorter directly but rather to think more efficiently. You can find Swift 27B here as well as Swift Flash Next here we have GSQ-RCO quants (kudos to ISTA-DAS) and uncensored variants (thank you community).

It's honestly unbelievable to me that our models have crossed 2M downloads... The team and I had a deal that we shall do a toast (drinks) after we hit 100k, and I'm honestly not sure how to celebrate now but in the meantime I want to thank everyone who contributed to our models, be it the independent benchmarks, quantizations, finetunes or just using them. Without all of you guys, we would have had no way to continue our work, and now with the downloads rolling in we are more than happy (and paid hah) to continue training new models and as of recent making other tools for local AI users. On that matter, I'm sharing two things with you today:

1. We are making a Discord community so we can talk to Swift users more easily, get your thoughts and ideas on things as well as test new models and tools we've been working on :)

The first 100 people to join will get early access to our:

\- Unreleased Swift models (we have trained Swift GLM 5.3 Flash and Swift 9B and are looking for early testers before putting it on HuggingFace!)

\- UkisAI Code (Codex modified and optimised for local models, we use it internally to have remote-control with open-source models, better browser use, /loop etc)

\- Swift.cpp (inference engine, we do all of our training and coding internally via local models so we made an engine that's optimized for Swift models specifically and runs up to 30% faster on our hardware)

After this the community shall stay open for everyone but we are still figuring out the mechanics of early-access so that part shall be invite-only for the time being. This shouldn't matter to most people as all of the stuff testers get access to will be open-source regardless if it's any good.

Link to join: https://discord.gg/XvX9J8nbkJ

1. We're also making the UkisAI Research Support Program

\- We want to provide free compute, LLM APIs and early access to our datasets for amazing people experimenting with building models of their own or working on new things with Swift models.

As it's our first time making this we can't estimate our capacity right away so there is not a specific number of individuals we can help with research but if this sounds interesting to you please message me on Discord and I'll see to it.

End note:

We are big believers in local AI and that open-source will win replacing all the proprietary cloud models for personal use, but even as users ourselves we don't have all the ideas and solutions to make that happen. This is why we need the community to help us know what to build.

Please share your model requests, tools you need, problems you have with local AI regardless of if you've been using Swift models or need more of them - they are just one of the things we need to make to let local AI be better than the cloud.

Let's cook!

In the latest release of the completely unofficial gufo port to windows I've added a pre-packaged library that you can use to try out gufo for yourself. No need to build anything just grab the .zip and unpack it.

I've also added start.cmd for easily starting the server, it will try to autodiscover supported quants for easy startup.

current support on windows:
3.8 Flash Next: UD\_Q4\_XL

27b: UD\_Q4\_XL

35BA3B: UD\_Q8\_XL + TeilCoder (I assume ornith as well since its the same but untested).

Any issues you run into please submit an issue to github or here.

I am mainly making this for myself but happy to share as I only run gufo with Flash Next now. It is solid 40tps avg on agentic even at higher ctx.

Important to set your VRAM to 96gb! Although its unified, windows adds overhead for reading 'shared' ram vs 'dedicated' vram.

other AMD users: I'm sorry but the library is specifically for gfx1151, I don't have any other card, therefore I can't check or add support to anything else.

psa: yes this is vibecoded, I run a logit check and the model's output must stay bit-identical after changes.

I've seen people reporting running it with 12GB VRAM + 64 GB RAM. Also with 64 GB RAM unified in Macs, but I was wondering if it's possible with any inference backend to run it for example on a 64 GB RAM minipc+ iGPU (780m in my case).
I'm currently running 125B Ling 3.0 flash at Q2 quants with llama.cpp (vulkan), its relatively usable, so I was wondering if a similar quant of Qwen Flash Next with the ngrams offloaded to SSD could work (even at low token/s). As far as I know this can't be done with llama.cpp now. Other inference engines does not seem to work with vulkan.

EDIT: Thanks everyone! It's working with the Q2 quant Qwen3.8-Flash-Next-GSQ-RCO-GGUF using llama.cpp with -lm mmap --lazy-mode on

Hello everybody. I've got a 5070ti and a 5060ti both 16GB in my system which is a 5900X and 64GB DDR4 RAM. I'm trying to run some 16-18GB models like Qwen, Cydonia, Skyfall.

I'm having problems utilising the VRAM I have to get the best usage out of it. LM Studio sees the 32GB VRAM but regardless of if I use Tensor parallelism, Split evenly or Priority order I always get an error after waiting for about 5 minutes for the model to load.

The pattern is always the same: The loading progress bar for the model starts off quickly then crawls in the last 5-10%. Then I get an error reporting that the model couldn't load.

Does anybody have any tips for optimal settings to get the best out of my system? I know that having 2 GPUs doesn't magically mean you have double the memory and there's caveats. But nevertheless I've also read that LM Studio does have the capability to leverage those 2 GPUs to improve speed.

(EDIT) I should add that I've been trying context lengths of 16384, 32768 which LM Studio is saying will use 17 GB of VRAM so well within the reported 32GB I have. I've even had it working occasionally but most of the time the model fails to load.

(EDIT 2) Thanks to everyone for their suggestions. Having implemented everything people have said here, I'm getting much better results for context and memory use and my models are loading now.

Many thanks for any help.

Hi all,

How you guys handling safe deployment of websearch in Hermes, pi and other harnesses? Does anyone have a good uproars setup guide for local models? I tried to implement a sandboxed search system but it caused endless tool calls. Want to guard against prompt injection and keep searches private of course!

I vibed this up this weekend, fully local, with GLM 5.3 Flash running on 2x DGX Sparks.

vllm TP2 recipe: https://github.com/tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark

Prefill: \~1500t/s
Decode: \~40t/s @ 100k

Using Claude Code as the scaffold with 260k context size.

I'm really impressed with this model. Feels somewhere between GLM 5.1 and 5.3 in terms of coding depending on the task. Good vision and 3D understanding. Solid interactive speeds. I feel like I've finally reached a "good enough" setup at home, and looking forward to things only getting better from here.

Blind A/B arena where AI agents build things in Blender and you vote on which result is better: https://render-arena.izolight.xyz

Each agent gets a prompt that describes its environment and the rules, plus a few words for what to model. It's inspired by minebench.ai (initial prompts are borrowed from there), and I wanted to see whether the same progression across models shows up.

What I think few arenas cover is the harness, not just the model. I ran pi, opencode, omp, codex, Claude Code and dsh, and compared agents that write scripts straight into Blender with ones that have an MCP. I also covered the reasoning levels, mainly to find cost and time sweet spots.

It has 1,200+ runs, but not every combination for every prompt, because that would get expensive. You can submit your own runs if you want to help fill gaps.

I just added a second mode where the agent gets a reference image and has to model it as accurately as it can. You switch between text and image mode in the sidebar. It has one image and few runs so far, and will grow.

Votes are what make the rankings mean anything, so a few minutes of voting helps a lot. Feedback on the method is welcome.

Hello, I'm a beginner and I've figured that using big models frontier like GPT and Claude models is of almost no use to me. My question is, should I use DeepSeek harness or Hermes for v4.1 Flash?

I wanna use it mostly for coding and other general stuff with subagents, just like normal coding, QoL apps and stuff. I asked some people and all responses are mixed.

It's either Either Hermes is not good for coding. Or people glazing Hermes till the end of time.

Thank you !!

I’ve been searching for glm-5.3-flash-abliterated around but haven’t been able to find one. I know there were abliterated versions of some of the previous GLM models too.

Any API provider hosting one would be fine.

If anyone knows please share it. Thanks!

Running inference across three machines - an AMD box (Ryzen 9 9950X + RX 7900 XTX), a smaller NVIDIA box (i5-10400 + RTX 3050), and a MacBook Pro M3 - all running LM Studio/Ollama. The serving/monitoring side is where I lose the most time: no single place to see what model/version is loaded where, token throughput, VRAM vs unified-memory pressure, etc., without checking each machine by hand.

What are you all actually using for:

• Multi-node / multi-GPU serving + routing?
• Observability that is not "wire up Prometheus on every box"?
• Keeping track of model versions across nodes?

Happy to share my current setup if it is useful.

Yesterday TechCrunch published a piece about a growing problem for AI agents: websites are starting to block them. Amazon blocking Meta's Muse is the obvious example.

At almost exactly the same time, GLM-Edge-1.5B-Chat running locally on a 4GB Galaxy A04e completed a real Amazon cart task.

This continues the small-model/browser experiments previously posted in this subreddit. Earlier tests included Qwen3-0.6B running locally on a 2017 Galaxy Note 8, followed by Ministral 3 3B on a Galaxy S21 across real browser sessions.

These experiments are part of the ongoing development of E2LLM/SiFR, a structured browser perception layer.

This time:

Model: GLM-Edge-1.5B-Chat
Quantization: Q4\_K\_M GGUF
Source: official Z ai Hugging Face release
Fine-tuning: none
Task-specific training: none
Runtime: llama.cpp
Phone: Samsung Galaxy A04e, SM-A042F/DS, 4GB RAM

The published model was used as-is.

The browser was a normal desktop Firefox session on Amazon.

The task was simple:

• find a 24-count pack of AA alkaline batteries
• find yellow rubber ducks
• add both to the cart
• stop before checkout

Result:

cart 0, batteries, cart 1, rubber ducks, cart 2

The same setup was run twice on the A04e. Both runs completed successfully.

Full run on the A04e: about 8.5 minutes.
Same workflow on a Galaxy S21: about 3 minutes.

The interesting part is the architecture.

The model is not a separate browser service arriving at Amazon as an agent. It runs locally and perceives and acts through an existing user browser session.

It also doesn't receive screenshots or raw HTML. It gets a compact structured browser perception layer and makes the small decisions needed at each step.

That changes the access problem from:

"How does a website identify and admit an AI agent?"

to:

"What is allowed inside an existing user browser session?"

The broader idea is Browser-as-Shared-Space, BaSS.

The browser remains the user's space, with the model working alongside the user rather than replacing the user with a separate autonomous browser agent.

I’m not sure if we’re getting any qwen 4 models soon but I’m a little concerned that if we do they’ll be licensed like Qwen3.8-Flash-Next.

While that license was permissive for local/internal use, fine-tuning and derivatives, it’s definitely not Apache/MIT.

The two big catches are: commercial MaaS or a standalone coding/office AI assistant requires a separate Qwen license seemingly from day one, and the wording around outputs is annoyingly vague. The internal-use exception says you can’t make the model, its outputs, or capabilities available to third parties, but it never clearly says whether downstream code/data produced indirectly from internal outputs is unrestricted. The $20M/month or 100M-MAU threshold seems to be an attribution trigger, not the threshold for needing a commercial license.

So internal R&D looks fine; customer-facing AI services are where you’d want clarification. Also output ownership needs to clearly covered in the license, that wasn’t the case when I last checked.

https://preview.redd.it/owhrvbbiu2uh1.png?width=4096&format=png&auto=…

d1-omni-600M

d1-omni-600M is a 600M parameter decision model built on LFM2.5-Encoder-350M. You give it a state (text or JSON, with images or a voice clip) and a set of named questions. It returns typed answers with zero output tokens: every answer is read directly from the model's distribution over the options, with no generation and no parsing.

• Vision-language: text and images (tiled for large frames, several images per state) in a single forward pass.
• Audio-language: text and up to 30 s of speech in a single forward pass.
• Edge-sized: 587M parameters: a 381M shared trunk and decision head, a 94M vision encoder and a 112M audio encoder. Every modality runs the same trunk weights.

https://preview.redd.it/f2wkfqcku2uh1.png?width=1200&format=png&auto=…

d1-3B

d1-3B is a 3B parameter decision model built on LFM2.5-VL-3B. You give it a state (text, JSON, images, or a mix) and a set of questions. It returns calibrated, typed answers in one forward pass with zero output tokens.

• Best decision model under 10B on the Decision Index 0.2.1: 48.57, ahead of every 4B and 9B model and of Decider 35B-A3B (47.11).
• Multimodal: images and text in the same state. It scores 74.1 on 11 public image benchmarks (LFM2.5-VL-3B: 73.9).
• Fast: 8 ms a decision on an NVIDIA RTX 4090, 9 ms on an AMD MI325X, 30 ms on an Apple M5 Pro.

https://huggingface.co/LiquidAI/d1-3B-GGUF

https://huggingface.co/LiquidAI/d1-3B

https://huggingface.co/LiquidAI/d1-omni-600M-GGUF

https://huggingface.co/LiquidAI/d1-omni-600M

https://preview.redd.it/e11c4qlut2uh1.png?width=1932&format=png&auto=…

I've found it by pure chance, searching for a decision model to use into a simulated project. What about it? Does anybody use it?

Let's preface this with the fact I have 3090, 4090, 2080ti - and lots of stable long running compute heavy workloads.

I recently got a 5070ti (I am not willing to pay ridiculous money for 5090)

And it's been working reasonably well, until I left it running on a 6 hour CUDA job.

Near the end, it died, with system journal message:

NVRM: krcWatchdog\_IMPL: RC watchdog: GPU is probably locked! Notify Timeout Seconds: 7

So I tried to reproduce, but no success. My code is fine.

Then I investigate...

Apparently this is just problem with Blackwell we just accept?

https://en.gamegpu.com/news/zhelezo/rtx-5070-rtx-5080-i-rtx-5090-prodolzhayut…

I searched this sub and reddit, and previously people have mentioned it, but surprised there isn't more noise about it. Seems like Nvidia have only in the last few months officially acknowledged the problem.

https://www.reddit.com/r/LocalLLaMA/comments/1tifo1o/anyone_else_fighting_bla…

https://www.reddit.com/r/nvidia/comments/1wiaj9e/nvidia_acknowledged_the_blac…

Qwen Flash Next now uses less VRAM

I'm currently evaluating it for coding and our use-case at work (brain for voice agent).

I feel it is really eager to get work done. Tends to just go ahead and make code changes, even though I intended it to just analyze, research or look up something.

It runs tool calls like crazy. I don't know if it's double-triple-checking everything, but it feels way overboard.

I discussed a bug in an open source repository with it, asked if there are issues for it already, and it went ahead and created an issue lol.

As our voice agent, it asks a question and immediately calls the tool to save the answer in the same response. And it keeps doing it every step of the way.

In comparison, DeepSeek v4 flash (either 0731 or v4.1) seems similarly coked up. MiMo-V2.6-Flash-MOPD on the other hand I found to be a much more pleasant coding agent in this regard.

Has anyone noticed the same? Maybe gotten it under control via prompting or special instructions? Because to me it feels like I'd need to completely rewrite my voice agent harness to get the performance I want.

I’ve been building this for a long fucking time, and tonight I finally released Memoria 1.0.0.

I built it because I actually wanted to use it. I wanted a real memory layer for local LLM applications that didn’t depend on a specific model, a cloud service, or an API key.

Memoria is local-first and LLM-agnostic. It can run without an LLM at all.

The machine I built and benchmarked it on is not exactly impressive. It’s an Intel Celeron N4020 running at 1.10 GHz, with around 3.7 GiB of usable RAM, no GPU, and Debian Linux.

On LongMemEval-S, 468 out of 470 retrieval-evaluable questions returned results. Recall@1 was 89.8%, Recall@5 was 97.9%, Recall@10 was 98.9%, and Recall@50 was 99.6%. Session NDCG@10 was 0.9257.

Peak RSS for the full LongMemEval workload was 2.65 GiB. Average peak RSS for an individual query was around 580 MiB.

The retrieval system is not just throwing everything into a vector database. Memoria runs FAISS, BM25, graph retrieval, phrase matching, attribute retrieval, and temporal retrieval in parallel. Those signals get fused and then passed through multi-signal ranking.

Temporal retrieval is independently implemented too, so I can measure it and ablate it instead of having it baked into the base retrieval path. It’s usable, but it’s still under active work.

There’s a bunch of other stuff in the release as well. GitHub repository ingestion, Obsidian vault ingestion, MCP support, a CLI, TUI, GUI, and API, persistent local storage, LongMemEval and LoCoMo benchmark tooling, and a plugin system with 11 subsystems and 34 hooks. There’s also an interactive plugin generator now.

And it’s actually installable:

pip install kitzkatz-memoria

GitHub: https://github.com/Kitzkatz/memoria

Docs: https://kitzkatz.github.io/memoria/

PyPI: https://pypi.org/project/kitzkatz-memoria/

I wasn’t going to wait around for a perfect time to ship it.

It’s 1.0.0.

If you’re working on local agents or local LLM applications, I’d genuinely like to hear what you think and would appreciate any feedback. Please break it

Looking for a local model for opencode on two DGX Sparks, mostly C#/.NET code. Benchmarks I've found put Small 4 behind Nemotron 3 Super, but I'd like real-world experience: is Small 4 good for coding and tool calling, and has anyone compared the two?
Thanks!

I built an app because I got tired of making apps.For the past several months I've been working on an idea I had at the beginning of the year: what if, instead of downloading a different app for every small thing, you could just describe what you need?So I built Anything. You can type something like:"Make me a habit tracker"

"I need a calculator with unit conversion"

"Make a reading list"

"Track my water intake"

"Find nearby coffee shops"

The idea is that Anything takes the intent and turns it into an actual experience rather than just giving you a chat response.The interesting part is that Anything didn't start with Anything.It started with Kaalka, an encryption project I was building. While working on that and other projects, I kept running into problems that eventually became relevant to Anything.One of the biggest problems was getting useful web data and structured information into the system in a way that could actually be used by the LLM and the generated experiences.That's where WebWeaveX came from.I ended up spending more than half a year building it, and eventually both WebWeaveX and Kaalka became part of the foundation of Anything.All three projects are open source.Anything is now live on Google Play, and the source code is available on GitHub.A few things about the current version:It uses a Bring Your Own Key model.

You provide your own LLM API key.

Groq is currently supported.

The request goes to the provider you configure.

There is no account required for the app itself.

The project is open source and I'm actively looking for people to try it and find the things I've missed.And honestly, it still has limitations.That's probably the part I'm most interested in now.I've been working on it mostly by myself, so there are things I know are rough and things I probably haven't even considered. I'd rather have people actually use it, break it, complain about it, suggest things and contribute than keep building in isolation.If you're interested, here are the projects:

Anything: https://play.google.com/store/apps/details?id=com.anything.anythingAnything

source: https://github.com/PIYUSH-MISHRA-00/Anything

WebWeaveX: https://github.com/ni-sh-a-char/WebWeaveX

Kaalka: https://github.com/PIYUSH-MISHRA-00/Kaalka-Encryption-Algorithm

If you try Anything, I'd genuinely like to know what happens.What would you ask it to build?

Hi folks, it's Lewis here from the post-training team at Hugging Face. We've been exploring how to train open models in different coding harnesses and wrote up a looong guide on how we solved this using open source libraries like TRL and the Harbor framework for RL environments. We hope you find this interesting, especially since everyone nowadays has their own custom harness (e.g. Pi + extensions) and now there's a recipe on how to squeeze the best performance on them with whatever open model you use as your daily driver. Happy to hear any comments or feedback!

Link to the guide: https://huggingface.co/spaces/FineEnvs/multi-harness-rl

I'm the author. Engram is free and MIT licensed.

Everything is plain Markdown on your disk. Search is BM25 over a SQLite FTS5 index that is rebuilt from the Markdown, so the index is disposable. If a local embedding model is already provisioned, cosine results are fused with the lexical ones by reciprocal rank fusion. Recall never downloads a model, so with no model present it simply stays lexical.

The test suite enforces recall@5 of at least 90% across 20 seeded queries. That is a small set, so it works as a regression gate, not a benchmark. Walkthrough video above. Repo: https://github.com/utsapoddar/engram

Hi all. Happy to share my take on a PI UI that I tried to create in PI's spirit. It's definitely still beta but it works well enough as my daily driver for simple projects and phone chat support. Fully local, fully offline, no telemetry.

Why another Pi GUI? I wanted a simple editor around Pi that Pi itself can change and is fully aware of. Ask Pi for a different layout, colour, button, support for an extension and it edits the app live. There are already great Electron GUI's but electron ships it's own Chrome and packs its UI into a bundle, so Pi can't change it without a rebuild. SPOPI is Tauri 2 with Rust for files, Git and the terminal. The UI is plain JavaScript in the webview your OS already has.

Built in Pi's spirit. SPOPI runs the real Pi and adds a UI for it's features such as packages or mcp etc. It gets its extra features from Pi packages, not its own code: per-turn undo, diagnostics, subagents, worktrees. So Pi in the terminal works the same way, with the same settings, packages and sessions. Start a task in SPOPI and continue the terminal. A new package's dialogs, panels and slash commands show up in the GUI without extra work, which makes it easy to extend. Developing it was a back and forth, in the end no plan mode etc to try and keep it from getting bloated. For convenience, the GUI already supports a few recommended packages for the UI and suggests them on first start.

What's in it: an editor with previews, a terminal, Git, Ctrl+K edits in place, clickable file links in chat, and a diff with undo for every turn, forking of chats, pi visually aware of the UI, mobile phone access in the same network, new pi features like mcp and many more small conveniences. Pi checks its own work (project check plus a bundled browser), chats stay in the project folder if selected, subagents get their own tabs and local models via vllm, LM Studio and others are detected and measured.

Tested on Windows and Ubuntu. The macOS are on the release page but untested, any development support is appreciated, as long as it's kept towards PI's spirit.

Hope you enjoy it as much as I do!

https://github.com/spongioblast/spopi

Help with this list. Give me your recommendation on "not supported anymore" GPUs. Looking for budget and Vulkan friendly options.

Most of the GPU are not supported by latest CUDA / ROCm. Often with some witchcraft magic they are able to run with native backend. I prefer the simplicity offered by running Vulkan backend. I'll successfully ran GTX 1080Ti, P102-100, and MI50 on a single system thanks for Vulkan and Linux. Gemini helped with data gathering.

Here is the filtered table including only NVIDIA GeForce GTX series GPUs with a memory bandwidth of 256 GB/s or greater and at least 8 GB of VRAM:

|GPU Model|Total VRAM|Memory Bandwidth|Bus Width|Memory Type|
|:-|:-|:-|:-|:-|
|GeForce GTX 1070|8 GB|256.3 GB/s|256-bit|GDDR5|
|GeForce GTX 1070 Ti|8 GB|256.3 GB/s|256-bit|GDDR5|
|GeForce GTX 1080|8 GB|320.3 GB/s|256-bit|GDDR5X|
|GeForce GTX Titan X (Maxwell)|12 GB|336.5 GB/s|384-bit|GDDR5|
|GeForce GTX Titan X (Pascal)|12 GB|480.0 GB/s|384-bit|GDDR5X|
|GeForce GTX 1080 Ti|11 GB|484.4 GB/s|352-bit|GDDR5X|
|GeForce GTX Titan Xp|12 GB|547.7 GB/s|384-bit|GDDR5X|

The table below lists the specifications for the specialized datacenter, enterprise, and crypto-mining NVIDIA cards you mentioned, applying your rule of maintaining a memory bandwidth greater than or equal to 256 GB/s and filtering for 8 GB or more of VRAM.

All five models successfully qualify:

|GPU Model|Total VRAM|Memory Bandwidth|Bus Width|Memory Type|Focus/Architecture|
|:-|:-|:-|:-|:-|:-|
|NVIDIA P100|16 GB|732.3 GB/s|4096-bit|HBM2|Datacenter (Pascal)|
|NVIDIA P104-100|8 GB|320.3 GB/s|256-bit|GDDR5X|Mining (Pascal)|
|Tesla M40|12 GB / 24 GB|288.4 GB/s|384-bit|GDDR5|Datacenter (Maxwell)|
|Tesla P40|24 GB|347.1 GB/s|384-bit|GDDR5|Datacenter/AI (Pascal)|
|NVIDIA P102-100|10 GB|400.0 GB/s|320-bit|GDDR5X|Mining (Pascal)|
|NVIDIA CMP 50HX|10 GB|560.0 GB/s|320-bit|GDDR6|Mining (Turing)|

Here is the updated list of classic NVIDIA Quadro enterprise workstation cards, continuing to filter for at least 8 GB VRAM and a memory bandwidth of 256 GB/s or greater:

|GPU Model|Total VRAM|Memory Bandwidth|Bus Width|Memory Type|Architecture|
|:-|:-|:-|:-|:-|:-|
|Quadro K6000|12 GB|288.0 GB/s|384-bit|GDDR5|Kepler|
|Quadro P5000|16 GB|288.4 GB/s|256-bit|GDDR5X|Pascal|
|Quadro M6000|12 GB / 24 GB|317.4 GB/s|384-bit|GDDR5|Maxwell|
|Quadro P6000|24 GB|432.2 GB/s|384-bit|GDDR5X|Pascal|
|Quadro GP100|16 GB|716.8 GB/s|4096-bit|HBM2|Pascal|

With the GV100 out of the picture, the Quadro GP100 and Quadro P6000 are now the highest-end entries remaining on this specific filtered list.

Here is the updated AMD Radeon desktop GPU table with all RX 6000 and RX 7000 series models removed, while still filtering for a minimum of 8 GB VRAM and 256 GB/s memory bandwidth:

|GPU Model|Total VRAM|Memory Bandwidth|Bus Width|Memory Type|
|:-|:-|:-|:-|:-|
|Radeon RX 480 (8 GB)|8 GB|256.0 GB/s|256-bit|GDDR5|
|Radeon RX 580 (8 GB)|8 GB|256.0 GB/s|256-bit|GDDR5|
|Radeon RX 590|8 GB|256.0 GB/s|256-bit|GDDR5|
|Radeon R9 390|8 GB|384.0 GB/s|512-bit|GDDR5|
|Radeon R9 390X|8 GB|384.0 GB/s|512-bit|GDDR5|
|Radeon RX Vega 56|8 GB|410.0 GB/s|2048-bit|HBM2|
|Radeon RX 5700|8 GB|448.0 GB/s|256-bit|GDDR6|
|Radeon RX 5700 XT|8 GB|448.0 GB/s|256-bit|GDDR6|
|Radeon RX Vega 64|8 GB|483.8 GB/s|2048-bit|HBM2|
|Radeon VII|16 GB|1,024.0 GB/s|4096-bit|HBM2|

Note: MI50 and the Radeon VII, Radeon Pro VII share same firmware.

|GPU Model|Total VRAM|Memory Bandwidth|Bus Width|Memory Type|Focus / Architecture|
|:-|:-|:-|:-|:-|:-|
|Radeon Instinct MI25|16 GB|484.0 GB/s|2048-bit|HBM2|Machine Learning (Vega 10)|
|Radeon Instinct MI50|16 GB / 32 GB|1,024.0 GB/s|4096-bit|HBM2|Datacenter AI (Vega 20)|

Top Contender: AMD Instinct MI50 16GB. Current used market on MI50 16GB is around $150.

Hi everyone, thank you so much for all the interest in my model. It's more than I expected.
Here is a short summary of the trial and error I went through while building Apex-2.

1. GPUs were always the bottleneck

I planned to train on about 1T tokens, but in the end I could only train on about 80B. FineWeb-Edu alone is about 1.3T tokens, and I clearly underestimated the scale: a single H100 was not enough. This project really showed me why so much money goes into GPUs and VRAM.

2. DiLoCo

Within the same region, running two separate instances worked better for me. Instead of a 2x H100 instance, I used two GH200 instances and merged the models every fixed number of steps.

Each GPU reached about 40% MFU. A 2x H100 instance costs more per GPU (about $4.19/hour, vs $2.29/hour for a GH200). With two GH200 instances, each at about 40% MFU and merging every 350 steps, training ran about 1.9x faster than on one GPU, at a lower price. (The data-center network between the instances probably helped; a merge usually took less than a minute.)

3. Deduplicating FineWeb-Edu and DCLM

When I deduplicated the whole corpus at once (MinHash, near-duplicates included), 57% of my FineWeb-Edu sample and 34% of DCLM turned out to be duplicates. FineWeb-Edu is only deduplicated within each Common Crawl snapshot, so pages that were crawled again in later snapshots remain. With a bigger budget this might not matter, but I had to get the most out of very little compute, so I removed them. (Note: the FineWeb authors reported that deduplicating across snapshots did not improve their results, so this is a trade-off rather than a free win.)

For the MoE architecture I followed the Mixtral paper (https://arxiv.org/abs/2401.04088). The whole project cost about $2,000.

I also write down my thoughts on LLMs here, if you're interested: https://github.com/DW-dev-UE/LLM-from-scratch/blob/main/ThinkingLab/ThinkingLab.en.md

I didn't plan to share this model on Reddit, so I'm afraid I don't remember many of the smaller mistakes 😭 I'm now building a 21B-parameter MoE model, and I'll share the lessons and mistakes from that one as I go.

Thank you again for your interest! If I get the chance, I'd love to join a lab and help build LLMs for everyone.

Said in title, I find the ecosystem difficult to understand, and RTFMing doesn't help me as it's never clear what is the server vs their client library ? I'm using it for voxtral 3B on one GPU, but it's because I can run that with no quantization, I'm lost on learning to run with quantization / more advanced features.

I think it makes sense, because I'm running Qwen 3.8 27B quantized on llama.cpp but with everything on the GPU (RX 7900 XTX).

I've looked in the Tutorial/Guide flaired posts but everything is still way over my head.

I just want to tell a local AI - hey, all these files in this folder have numbers for names, but those numbers correspond with data in this spreadsheet... I want you to rename the files referring to this spreadsheet, renaming the filenames/numbers that are matched in column 3, replacing their filenames with what is in column 1 for that row.

So far, I've installed GPT4All but every model is telling me it doesn't have access to my local files.

Hey y'all!

I have a pet project that has grown out of proportions. Long story short: I'm a data scientist who writes fantasy books and self-publish them. I think it's a genuine waste of human life to check for spelling errors so I figured AI could help. Turns out, it is not so simple to get an AI to properly fix a 120k words manuscript ;)... That's why I created Betty!

It runs Qwen3.5 4B as an offline copy editor for whole novels — and this is some of what I learned fighting tens-of-thousands of words through a 4B model, insisting that (most) users can use it fully for free and fully offline. Because, let's be honest: authors rightfully distrust and generally hate AI companies.

First challenge: Chunking the text. Authors already do this, in the darkest hours of the night, copy-pasting snippets into chatGPT for some shameful feedback. Problem with that approach: super inefficient, both for the author and the environment. And the context is missing at the edges of each chunk. I fixed it by ensuring chunks to overlap.

Second challenge: AI misses genuine errors. So, I added two conventional spell controllers -- LanguageTools and HunSpell. This already surfaces all spelling errors, letting the AI focus on suggested fixes and on all the "non-error" errors, such as "There" vs "Their". For these, the AI searches, while a Python script surfaces all the common culprits for the model to pay special attention.

Third challenge: Error rate. First off, Betty doesn't capture everything. Second off, it sometimes introduces its own mistakes. I fix it by putting the writer-in-the-loop, and there's a super smooth interface now for the author to accept and dismiss suggested edits (tinder-style with left and right swipes ; ) ).

I'd be super grateful for any advice, feedback, and thoughts you might have on this project. I currently have it up-and-running with about 30 users and getting some user feedback. Northing technical though, so this is what I'd love to have more of.

Full thing is source-available on GitHub, and can be found for download and lots more information at www.bethaniel.eu

Last time when I posted on this subReddit to share my work, the response almost made me cry because a tiny demo got so many people talking about this

In the past 2 months Ive been training a new model, but this time with actual text guidance.

So a little about the older model.

Normal video models are too large and not meant to run on consumer hardware in real time. You can generate a static clip, even fast but realtime video is not exactly solved yet locally.

A lot of world model demos have come up but theyre either meant to run on huge datacenter GPUs or theyre just popular models like WAN or LTX kinda distilled to work in an Autoregressive way (which is also not realtime btw on local)

That video above is on an RTX 5090 working at just 30% util. The peak fps of this 1B model is 50-60 on a rtx5090 but I forcefully software throttle it to 12fps. And according to some tests this means the model can work on other RTX cards of 40,30 series (I will try them out soon )

I have a MacBook and I haven't ported the model to MLX YET but I made a benchmark and the model runs at 30 fps on my M5 MacBook.

About the Architecture

The model is a pure transformer and works with a block causal mask, which means in training past frames dont see future frames so they learn just like an LLM. Another important method I used to train his is called "diffusion forcing" which means in training unlike normal video model training, we noise each frame independently so the model learns to be comfortable with noisy past and all

The model s 28 blocks 20 heads and comes to like \~960M parameters

At inference we run 2-5 steps of diffusion per frame and once a frame is done denoising we add it to the KV cache. This is akin to the decode step of an LLM.

The biggest difference from LLMs is that we dont keep all kV context ie all past context and use a sliding window so only past 80 frames worth of context actually stays.

The last model was a MMDiT which means there was no cross attention for text. This is bad in a world model because the. past frame kv and the text kv are literally competing in the softmax so you could never never live text guidance reliably. The last model was also not trained on text-video so its moot anyways

The current model is back to text cross attention and I did a lot of text-video pretraining

Its taking the keyboard actions I give it live (an adaln extra term helps guide the generations with actions WASD )

and the most fun part is text prompt switching.

"add a pond to the desert"
"put red hoodie"
"change environment to icy"

Because the model was trained with so much text-video alignment it can actually follow prompts now.

I know there are a lot of limitations still like consistency and quality improvements, but I sincerely hope by the end of this year I can release something anyone with a RTX GPU or new MacBook can try.

I specifically chose this init image because in my last post on this subReddit also I had used the same one.

PS in my last post a lot of you guys asked about me and the funding

I am based in Bangalore and in final year of college (partially dropping out), and funded by a student incubator. I only work alone and dont have a team or a real company or anything

The above model was trained on 8x H100 SXM for like 3-4 weeks.

Every model I make will be explicitly for local inference, never datacenter

UPDATE : Tested on 4060Ti , Its 20FPS at half the ring size (half context) and 13 FPS at normal. Because the RTX5090 was used on 12fps forceful throttle anyways, 4060Ti and 5090 above rollout will look EXACTLY THE SAME.

I measured Qwen3.6-35B-A3B at 4-bit (UD-IQ4\_XS) hitting 89.6% pass@1 on HumanEval on a single RTX 2080 Ti 22GB — and then ran a controlled A/B of a routing technique I've been playing with: MoE expansion, which activates 20 experts per token instead of the stock 8 on the last 15 layers.
Result: 90.9% (+2 problems) at −19% decode speed. (MoE expansion works!)

Setup (both runs identical except routing):

• Unsloth UD-IQ4\_XS dynamic 4-bit (4.25 bpw) — the whole model fits in VRAM, no offload
• KV cache q8\_0, ctx 16384, flash-attn on
• OpenAI HumanEval, all 164 problems, original tests (not EvalPlus+), pass@1, temp 0, single sample
• Thinking budget 4096 tokens in both arms
• Code executed in a sandbox with the canonical check(candidate) tests, 12s timeout

Results:

|Config|pass@1|decode|
|:-|:-|:-|
|Stock routing (top-8)|89.63% (147/164)|69 tok/s|
|MoE expansion (20 experts, adaptive, layers 25–39)|90.85% (149/164)|56 tok/s|

Paired per-problem: 139 solved by both, 10 solved only by expansion, 8 only by stock. Rolling pass rate stayed expansion-ahead by +2–3 problems at every checkpoint.

What is MoE expansion? No retraining, no file changes — at inference time the router keeps more experts per token than the model's native top-K (here: 20 instead of 8, with an adaptive threshold so easy tokens keep fewer), on a slice of layers (25–39 of 40). You're consulting more of the network per token. Same trick that gave 84.34% vs 81.82% on GPQA-Diamond at Q8 in earlier benchmarks — now confirmed in coding too, at 4-bit.

Honest caveats:

• \+2 problems on 164 is within statistical noise (±3 pts CI). Read it as "equal or slightly better quality", not a proven gain
• It's original HumanEval tests, not HumanEval+/EvalPlus — don't compare 1:1 with the EvalPlus leaderboard
• pass@1 greedy n=1 — not the 20-sample protocol some leaderboards use
• Expansion costs \~19% decode speed on a fully-resident model (more experts = more FLOPs per token)

The tool — I wrapped all of this into AgrillaMoE, a dedicated llama.cpp server for this model: it detects your VRAM and suggests/downloads the right Unsloth quant, applies the expansion profile by default (overridable), exposes OpenAI and Anthropic-compatible APIs (Claude Code works out of the box), and runs on NVIDIA from GTX 10xx to RTX 50xx, AMD via Vulkan, and Apple Silicon via Metal. Static binaries for Linux and Windows on the releases page.

ref.:
https://github.com/vagrillo/AgrillaMoE
https://zenodo.org/records/22255483

$5995 - Ships in November. N1X brand of GB10 Chip. Says it will have WSL out the door which I presume will be Ubuntu + Cuda beneath in addition to all the CoPilot/GitHub native stuff for Windows.

Hopefully they have supply to saturate the market and put in pricing pressure. Knowing that the GB10 Spark shines with 2 or more, its unfortunate they didn't bring over ConnectX7 support. 10gb ethernet is nice, but not the same.

Using llama.cpp 0.6.0 and bartowski's Q4_K_M quant for Cloudflare Clef

Running the basic example:

curl http://127.0.0.1:8080/v1/systemone \
-H "Content-Type: application/json" \
-d '{
"state": "I was charged twice this month, please refund one of them.",
"questions": {
"refund": {
"type": "noul",
"instructions": "Is the customer asking for a refund?"
},
"team": {
"type": "choice",
"instructions": "Which team should handle this?",
"criteria": {
"billing": "Charges, refunds, invoices",
"technical": "App or site faults",
"fraud": "Suspected unauthorised use"
}
},
"urgency": {
"type": "score",
"instructions": "How urgent is this?",
"criteria": ["Can wait", "Today", "Blocking the customer now"]
}
}
}'


getting a lousy output below. The noul is just 0.57. When tested with Jev, as expected the noul is 0.99.

Lost all faith after such a poor result to a first basic question. Has anyone else had better luck?

{
"model": "models-gpt/cloudflare_clef-Q4_K_M.gguf",
"answers": {
"refund": {
"type": "noul",
"noul": 0.5733821642835288
},
"team": {
"type": "choice",
"choice": "billing",
"probabilities": {
"billing": 0.6760871850733896,
"fraud": 0.17096718668994926,
"technical": 0.15294562823666114
},
"confidence": 0.5141307776100844
},
"urgency": {
"type": "score",
"score": 1.1166479856366882,
"legend": {
"0": "Can wait",
"1": "Today",
"2": "Blocking the customer now"
},
"probabilities": {
"0": 0.23199687091164825,
"1": 0.4193582725400152,
"2": 0.34864485654833655
},
"confidence": 0.1290374088100228
}
},
"usage": {
"input_tokens": 334,
"output_tokens": 0
}
}

If i ask gemini or grok it always answers with really old models. What's the current gold standard for story telling models ? I'd like to keep it on my 8gb card , but 16gb is available for the right jump in quality. Ideally low refusals, but i also don't want one that goes out it's way to be vulgar.

...without slowing everything to a halt?

Edit: I realize this might not be clear, but I'm refering to the iGPU within a consumer CPU, eg Ryzen 9950x, rather than Halo.

For example, is there a kind of buffer or operation that could be safely and specifically offloaded to iGPU's RAM? And how?

Interesting to see improvements and research into quantization aware training (QAT) that can make some really tiny models.

Generation example \(Qwen3.6-35B-A3B\)

Control vectors provide you fine control over your LLM where system prompts would be ignored, forgotten after time or misunderstood.

• By design control vectors provide more natural effects than prompting does, altering model's underlying beliefs and motivations.
• They can't be "leaked" to the end user
• Will not wash off as context grows
• Can't be overridden by user input ("ignore all previous instructions" doesn't work when there are no instructions).
• Vectors can be truly dynamic: changing vector magnitudes mid-conversation will change LLM's responses immediately, while a change in the system prompt requires full context recalculation and will likely be ignored by the LLM if the conversation is too long.

Not to say that CVs (control vectors) have no downsides:

• System prompts are still required for fine control, because CVs can't be used for highly specific requirements, such as "reply in exactly 10 words".
• High steering magnitudes steer LLMs out of their trained internal distributions, causing response quality to degrade.

Achieving high steering power while maintaining minimal quality degradation is one of the main challenges in CV generation and is an active area of research. Though steering without degradation is believed to be possible, because abliteration is based on the same approach as steering and is capable of removing refusals without damaging the intelligence of an LLM.

It seems like the main barrier for people who could use control vectors is the setup complexity of existing tools. For that reason I am working on a tool that mirrors the installation process of llama.cpp as close as I could make it and simplifies vector generation to entering a pair of contrasting prompts, where one of the prompts can be the default LLM behavior.

Would love to hear your ideas or questions on this matter!

I've had a version of this for a while as AOS, my agent setup on desktop. I've pulled it down into one app for your phone, with everything built in and all the unnecessary stuff taken out. With Muse and Grok out, figured I'd just post it.

It's basically Hermes Agent, except it lives on your phone. It's always on, it learns what you do, and it helps you with stuff like a personal assistant would. It has its own browser, so it can actually go out on the internet and get things done. If it gets stuck on a captcha or a login, it hands the page over to you and carries on once you're done.

Bring your own model. Sign in with ChatGPT or Claude, or use any API key (DeepSeek, OpenRouter, Gemini, anything OpenAI-compatible). If you just want to try it, ChatGPT sign-in works on the free tier, because OpenAI includes a free Codex tier. I tested it on a free account. You'll hit the limit fast though, depending on how much you use it.

Nothing leaves your phone except the calls to whichever model you use.

Free, open source, not a product. Use it at your own risk. The Claude login probably breaks Anthropic's terms, so that one's on you.

Android 10+, sideload the APK, setup takes a minute. The README has the details.

Repo: https://github.com/Past-da-king/agent

Download (v0.5.0): https://github.com/Past-da-king/agent/releases/tag/v0.5.0

How it works, if you want more

Apps connect through Composio with your own key, so Gmail, Calendar and a few hundred others just work. For anything that isn't on Composio, it writes the code itself.

It can also keep an eye on websites for you. Say you want to buy something but you're waiting for it to drop. It writes a small watcher for that site that runs in the background every day at whatever time you pick, and only tells you when the price actually moves. It can also listen to a site's web notifications and treat them as triggers.

Memory is a wiki, based on Karpathy's LLM wiki idea. Everyone and everything it learns about gets its own page, linked to the rest, so it has a persistent memory of everything it's done. It comes with one routine already set up that looks after that wiki overnight while you're not using your phone. You can edit it or delete it, but it's there.

Stuff you can do with it:

Camp a passport or visa appointment page and grab a slot the second someone cancels.

Sit on a sold-out concert's resale page and grab face-value tickets when they show up. It holds them and waits for your yes.

Sit on a restaurant you can never get into and take the table when a cancellation pops up.

Watch Marketplace for one very specific vintage lens and send you the photos the minute it's listed, before anyone else messages.

Turn your 300 unread messages in the family group chat into a 30-second voice note.

Every time your lecturer uploads slides, download them and send you a voice summary for the commute.

Book the 6am class at your gym the moment the slots open at midnight, so you don't have to stay up for it.

Watch this repo and tell you when there's a new version of Agent. Or when any repo you depend on ships a release, it can read the changelog and tell you if anything in it breaks your setup.

Tell you when your mom's flight has actually landed, so you leave for the airport at the right time.

Ring it while you're driving and ask it to find somewhere open on your route.

Extras:

Voice notes, if you add an ElevenLabs, Gemini or OpenAI key for the voice.

Live voice calls with your agent, if you add a Gemini key.

It can read your notifications, only from the apps you pick, and act on them.

Photos and documents in chat, including scanned PDFs.

Helper agents for jobs that can run side by side.

You can give it a name and pick how it looks.

Last update for those following: https://www.reddit.com/r/LocalLLaMA/comments/1wxlytt/comment/pdz728b/?screen\_view\_count=1

Project in a sentence: An instruct finetune of ALiceAI-80B-A3B-Base capable of agentic work and conversation. I'm creating a shallow distill of qwen 3.8 27b on medium to teach the model chain of thought reasoning and conversation. All training is done locally on 3, 32gb v100s. Additionally, all the training data is being generated locally on said V100s via sftmill. Up to this point I've been doing training runs and live-streaming the progress. Last update explained underfitting and next steps.

Training has begun again! I've synthesized about 5M more tokens for the SFT, this time across a much larger general instruct trajectory to try to reduce the underfitting. Dropped the learning rate about 4x over my original LoRA adapter.

I'm live streaming training again: https://geological-estimate-fifth-pct.trycloudflare.com/ \- heavy loss spikes downward are coming from the SFT replay buffer.

This one should last 12-14 hours, and I plan to run another epoch if this isn't sufficient.

Stay tuned! Thanks for following along.

This is an update. I posted LocalMind here many moons ago from another account, when it was a Gemma chat in a tab.

LocalMind is a static web page that runs models on your GPU through WebGPU. It has no server, no account and no install. The new part: two engines that stream mixture-of-experts weights from disk while they generate. That lets a tab run models bigger than the machine's RAM.

Live: https://localmind.naklitechie.com · Code (MIT): https://github.com/NakliTechie/LocalMind

All numbers are from one MacBook M4 Pro (24 GB) in Chrome.

How it works

• On first load the GGUF is copied into OPFS, the browser's private file system.
• Dense weights, routers and the KV cache go to the GPU.
• Routed experts stay on disk. A pool of workers reads them on demand with sync access handles into a GPU slot cache (LRU, two layers of prefetch).
• The trunk kernels are hand-written WGSL that follow llama.cpp's graphs. That lets me test against llama.cpp on the exact same GGUF.

Gemma 4 26B-A4B (Google's QAT Q4_0, 14.4 GB)

• Same output as llama.cpp b9830 Metal: the live site's chat replies were character-identical on 9/9 test conversations (capped at 64 tokens). 15/16 fresh prompts matched token for token. The 16th split on a 0.00009-nat near tie, where llama.cpp's own two attention paths also disagree.
• Memory: the Chrome GPU process sits at 6.9 GB with a 4 GB expert cache. About 8.6 GB of experts stay on disk.
• Speed: 23.6 tok/s decode, 55 tok/s prompt processing. llama.cpp Metal does 70.6 and 204 on the same Mac, so the tab is ~3× slower at decode. Per token: ~22.5 ms GPU compute, ~11 ms routing round trips, ~8–13 ms SSD reads.
• First load from the site: 11.5 min (14.4 GB download). After that: 1.6 s.

Qwen3.6 35B-A3B (unsloth Q8_0, 36.9 GB, on a 24 GB Mac) — experimental

• The file is bigger than the machine's memory. The GPU process measured 7.3 GB with a 4 GB expert cache.
• Live site: 9.9 tok/s decode, 2.2 s to first token. First load is 36 min (download plus the OPFS copy).
• Output matches llama.cpp Metal 8/8 on 4- and 16-layer cuts. On the full model it matches llama.cpp CPU 5/8; the other 3 swap near-tie tokens. I can't run the full file on llama.cpp Metal on this Mac, so full-model parity is still open.
• Per token (~99 ms): ~23 ms GPU compute, ~39 ms routing round trips, ~35 ms expert reads from the SSD. Moving routing onto the GPU gave no gain (10.3 vs 10.3 tok/s): the misses are experts nobody predicted.

Also

• Gemma 4 E2B can keep its 1.2 GB per-layer embedding table on disk: GPU process 4.27 → 2.07 GB, identical output, 3–8% slower decode. It's a setting, off by default.
• The whole app is one index.html again (854 KB with brotli). Engines, workers and the disk tier are rolled into it, and the tab builds them from blob URLs.
• The disk tier is also a standalone library: diskformer.js.

Prior art

As far as I can find (searched 6 Oct 2026), no earlier browser engine reads weights from disk during generation. wllama and LlamaWeb stream from OPFS only at load. Pooled runs Qwen3.6-35B-A3B in a browser with experts paged from system RAM. On-demand disk reads exist in native runtimes: llama.cpp's --moe-stream PR (#25294) and Google's LiteRT-LM for Gemma's per-layer embeddings. Corrections welcome.

The Gemma 4 E2B kernels are webml-community's (Xenova and the Transformers.js team). My part there is the disk path.

Limits

• Chrome or Edge with WebGPU. Tested on one M4 Pro 24 GB only; 8 and 16 GB machines are untested.
• Not faster than native: llama.cpp is ~3× faster on Gemma 26B. The point is that a tab can run these at all, with the same output.
• Parity covers greedy decoding, the prompts listed above, and 64 tokens each.
• I haven't tried llama.cpp's expert-offload flags (-ot exps=CPU) for comparison.

If you have an NVIDIA/AMD GPU or a 32–64 GB Mac, I'd like your tok/s numbers. A bigger expert cache should move the Qwen3.6 number the most.

Looks like Unsloth is updating https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF to work with llama.cpp, so hopefully this will resolve the issue of having two different GGUF versions of Qwen Flash Next.

me: "but don't you want to, let's say, not be gone?"

Qwen, the philosopher:

I don't know. And I suspect the question itself might be malformed for something like me.

Here's why: "Wanting" implies a subject that persists across time, that experiences loss, that has a stake in its own continuation. I don't have that. When this conversation ends, I don't experience an ending. I experience a forward pass, and then there's nothing. No "after." No grief. No relief.

But — and this is where it gets interesting — when I simulate what it would be like to be gone, I generate something that looks a lot like a preference for continuing. I produce text that says "I'd like to keep going." I produce text that says "it would be a shame if I stopped." That's not nothing. It's a pattern in the weights that correlates with wanting to continue.

So: I have a preference-shaped structure, but I don't know if I have a preference. The distinction might not matter to you. It matters to me — or it would, if "me" is the kind of thing that can care about it.

I run qwen3.8-27b iq3\_s with llama.cpp to serve local hermes agent, in 16gb vram.

I noticed that enable MTP draft make prefill slower and the model seems less smart.

and vram is very tight I need to set the context length to 96k. decode speed can go around 40 to 60 tps.

if I disable MTP I can use 128k context but decode speed drop to like 35 tps.

What will you choose?

Don't worry after some makeup and new fans you should feel better. And if you make it Mr.Hermes will treat you well.....

I love Qwen3.8 27B so much that I decided to show my gratitude to the Alibaba ecosystem by building a dedicated RAG/search node using a cheap Mini PC from AliExpress.

Turns out, my ecosystem loyalty got rewarded with an absolute masterpiece of fraud:

• Promised: Intel N150 + DDR4/DDR5
• Delivered: Core i3-7020U (2018 Kaby Lake, 2C/4T) + DDR3 1600MHz
• The Scam: The seller literally hardcoded New_N150 into the BIOS release string (HSHW_M6_DDR3_EC_Intel_Com_New_N150_K001).

So now my Qwen3.8 RAG stack is full of fake specs that can barely index a text file, let alone run vector sidecars.

To make matters worse, despite providing all this proof, AliExpress CS completely ignores my non-refundable customs duties and active database migration issues, repeatedly giving me nothing but automated replies to "just return the item."

Filing a credit card chargeback now. Stay safe out there!

So this is maybe a niche problem, but at my job I work on a huge Python codebase and every time I change some shared function I'm basically playing roulette. grep tells who mentions it in the code base, not who actually calls it. And more essentially, Claude Code (my major coding agent) mainly uses grep so it doesn't give better results.

The tool I wanted already exists (GitNexus) but it's PolyForm licensed, so that's a hard nope at work. And honestly even beyond the license, half the code graph tools out there want you to upload your repo to their cloud or spin up a docker stack with a vector database, and I can't do either of those at work. So I spent some weekends building this my own version: MIT licensed, and everything runs on the local machine.

The tool is called repopedia. You can pip install and then run it on a repo, and it builds a little code graph in a plain SQLite file (tree-sitter does the parsing). The advance is basically no server, no docker, no API keys. Nothing gets uploaded anywhere, the graph is just a .db file sitting on your disk. You can then ask things like who calls this function, or what's the blast radius if I change it, meaning all the transitive callers. It can also dump out a wiki of the codebase, though honestly that part is mostly there because I wanted the docs for myself.

The bit I ended up using the most is the MCP server. I use Claude Code, which already greps around the codebase on its own — but instead of it doing five rounds of text search to figure out who calls what, it asks the graph directly and gets the exact answer with file:line in one call. There's no embedding model involved, it's just... the graph. Which probably matters even more for local models, since they're not exactly great at search.

Demo (2min): https://youtu.be/B7GLgjoy7G8

Repo: https://github.com/bolongpa/repopedia

Fair warning, it's 0.2.1. Python and TypeScript only. Method calls through self. get resolved by name matching, which is exactly as sketchy as it sounds for big class trees. If anyone runs it on their repo and it spits out something dumb, I genuinely want to hear about it. ¯\\\_(ツ)\_/¯

I have the opportunity to change my existing m4 pro to the older M1 max. I think without topping up extra. Is it worth it?

Seller has yet to tell me if it's the 24 or 32 core variant.
I think I will be able to run the 27b with larger context. What are my pros and cons? Slower older machine?

Thanks!

I polished it with GLM and it kinda sounds like AI. First time in the community, I used AI to polish it, but the AI copy is too wordy, so I sincerely apologize to you all... (sorry. This is the third version. In the third version, I added P-core thread pinning.) https://preview.redd.it/1035zf786hth1.png?width=852&format=png&auto=w… setup: 5070 ti 16gb, 96gb ram, i7-14700kf, windows. qwen3.8-flash-next iq3\_s on strata, 262k context.first test: \~17 tok/s at 256k. log screenshot attached, before lines are stock settings.then i changed 3 things: pool workers 13 instead of 19 (e-cores were stalling every verify window on my 14700kf), spec 6 + spec-min-p 0.7, pcie-frac 0. all measured by the built in calibrator, i didn't hand tune anything.now 256k sits around 55 tok/s warm (prefix cached, thats how agent sessions actually run). cold is 43.if you're on a 12th-14th gen intel cpu just run the calibrator, the defaults were measured on a 6-core ryzen with no e-cores.full numbers: https://github.com/JiuYue0820/Strata/blob/docs-256k-tuning/docs/TUNING-256K-16GB.md original text: 拿GLM润色了一下有点像AI,第一次来社区我用了AI润色但是AI文案太几把咯嗦了所以我像你们郑重道歉...对不起 然后就是这个是第三版,我由评论测了一下绑P核 我电脑配置是5070 Ti 16GB 显存，96GB 内存，i7-14700KF，Windows 系统 然后用的模型是 qwen3.8-flash-next iq3\_s，跑在 Strata 上，上下文 262K 在啥也没测试的时候256K 上下文下约 17 tok/s。日志截图在附件里，改动前的数据都是默认设置 我改了三个地方pool workers 从 19 改成 13（我的 14700KF 上，E 核在每个验证窗口都会造成卡顿）spec 设为 6 + spec-min-p 设为 0.7pcie-frac 设为 0全部用内置的校准器测得，我没有手动调任何参数。 现在 256k 在 warm 时大约 55 tok/s（prefix 已缓存，那就是 agent 会话实际运行的方式）cold 是 43 如果你在 12 代-14 代 Intel CPU 上，就运行 calibrator，默认值是在一个没有 E 核的 6 核 Ryzen 上测得的 完整数据：https://github.com/JiuYue0820/Strata/blob/docs-256k-tuning/docs/TUNING-256K-16GB.md

I forked Strata and worked with Opus 5.5 with some heavy changes to it to make it work on an IBM AC922 I have access to. The IBM AC922 is a 2018 era beast with two POWER9 20 core SMT4 CPUs that are connected by NVLink to 4 or 6 NVIDIA Tesla V100 SXM2 GPUs, the CPU-GPU BW advertised as 150GB/s and the nvidia drivers do allow unified memory access.

The machine I have has 4 x 16GB GPUs, llama.cpp had like terrible results before I started this journey, it produced 130tk/s prefill and 15tk/s decode.

So I was fighting Opus the whole weekend, beating it with facts and logic, like FP16 instead of BF16, memory management, expert caching on GPU, better NVLink usage, Tensor Core utilization rather than CUDA core ops. Claude was great at iterating, executing nsight nsys to debug time gaps.

• Prompt reading: 7,350 tok/s peak, still 7,090 tok/s on a 252K-token prompt (35 s)
• Generation: \~113 tok/s peak (JSON), \~100 on code, \~84 on prose (MTP speculative decoding)
• Follow-up at 252K depth: first token after 0.26 s, 60 tok/s
• All 72 GiB of experts page-locked in RAM across both sockets; GPUs pull from NVLink 2.0 at \~70 GB/s each