$5995 - Ships in November. N1X brand of GB10 Chip. Says it will have WSL out the door which I presume will be Ubuntu + Cuda beneath in addition to all the CoPilot/GitHub native stuff for Windows.

Hopefully they have supply to saturate the market and put in pricing pressure. Knowing that the GB10 Spark shines with 2 or more, its unfortunate they didn't bring over ConnectX7 support. 10gb ethernet is nice, but not the same.

strata with qwen flash Q2 is pretty retarded and 27b at q4 is also retarded so I am wondering if anyone knows a engine that runs the model Qwen/Qwen3.6-35B-A3B fast on my hardware?

I've found it by pure chance, searching for a decision model to use into a simulated project. What about it? Does anybody use it?

TLDR: The question we answer: how do you learn from experts with different objectives? Pooling all their data can lose their trade-offs; learning from each expert separately misses opportunities to share data. MA-BC pools demonstrations where observed actions don’t disagree, with upper and lower bounds on sample complexity.
Authors: Ziyad Sheebaelhamd, Luca Viano, Volkan Cevher, Claire Vernade

Arxiv: https://arxiv.org/abs/2605.12000
Github: https://github.com/ziyadsheeba/mabc

https://preview.redd.it/i20adc3z04uh1.png?width=2532&format=png&auto=…

I generally just ask it "describe the main cast of (insert somewhat known cartoon show from the 2010s)", could either be Totally Spies, Randy Cunningham, Slugterra etc, most models in the 30b range completely fumble, looking at you Qwen, but the ones that manage to answer that are gems that can actually hold a human conversation.

Hey everyone,

I have new in brackets above because I'm not necessarily inventing anything innovative in terms of the actual mathematics or optimizations behind some quant techniques, but I'm pretty happy with how things are coming.

What I'm working with is basically a "poor man's" RCO (the quant method from IST Austria dAS lAB). Exact same concept: choose a type per tensor under a byte budget while optimizing task KL on the whole model. I worked on GSQ but I don't have an approximation method that beats baseline - yet.

Take the core principles of the method, make them cheaper approximations, and regain as much accuracy as possible. I originally planned to make an approximate GSQ-RCO hybrid, but none of my hypothetical models for the approximate for GSQ have beaten baseline yet.

In application: start with a full precision model, and create an "imatrix shape" map, per tensor. This doesn't calculate the sensitivities of individual tensors - but it creates a sensitivity "curve" where you can approximate which tensors in a model suffer most from quantization via extrapolation. This creates a baseline estimate of the optimal quant per tensor.

Then: iterative trial and error with local search. Take the file size of the baseline estimate, and substitute different precision per class to bring overall model size down, beginning with the tensors the approximation model marked as most sensitive to quantization. Once the working-best version hits under the filesize cap, it tries variations of substitutions that keep the file size approximately the same (upgrading certain tensors, downgrading certain ones, etc - basically looking for holes in the local search method once the local search is done).

For the whole process above (the iterative local search + search error recovery \[not really error but i cant find the word im looking for\]), the model is quantized and KL divergence is measured vs prior iterations - anything that raises KL divergence is discarded. The result: an approximated RCO style quant, iterated as closely as possible to optimal.

Do I expect this to beat GSQ-RCO or Unsloth dynamic V3? Definitely not GSQ-RCO or regular RCO, and likely not the unsloth ones. However, I've got a few advantages: this is CHEAP and extremely conservative on VRAM usage. The teacher model only needs to be loaded once: to dump its per token log probs. This is quick on a GPU, but since it only has to be done once, it can be done on CPU with a little patience. Every step after only pulls the candidates onto GPU, starting from the imatrix curve approximation - so all you need is enough VRAM for your approximate final quant size (with a little buffer for iteration, maybe 20-25% more would be optimal). The whole process takes a few minutes to a few hours depending on what you're doing.

I've pretty much documented a psuedo-algorithm approach above that's recreatable, but I can supply better documentation if people are interested.

Some early results on Qwen 3.5 2B:

llama.cpp IQ3\_M with an imatrix - 999mb vs RCO-lite with an imatrix - 1088mb \[+89mb\]

Mean KLD for IQ3\_M: 0.098381

Mean KLD for RCO-lite dynamic mixture \[+89mb\]: 0.045162 -- almost exactly half for 89 more mb

Mean KLD for RCO-lite dynamic mixture \[cap 1038, + 39mb\]: 0.0793220

Mean KLD for RCO-lite dynamic mixture \[cap 947, - 52 mb\]: 0.090624 -- still better than the IQ3\_M quant despite being 52mb less.

Take these as early results - I forgot to document exact +- for my KLD runs, but the band was generally lower than the i quants. I need to try various different size targets to figure out what BPW range this algorithm works in most effectively, and this is just up against the IQ3\_M - IQ4\_XS had a better KLD than this design - more BPW so it's not an exact estimate, but not that substantially - I didn't try to fit an optimal model inside the IQ4\_XS size range yet, that was just something I noticed. I suspect the Q3 and Q2 ranges will benefit most from this, I haven't tried in the higher BPW ranges yet - partway through Q2 experiments.

Also: my baseline llama.cpp quants are calibrated on the same wikitext set for the imatrix as the RCO-lite quants are, with the same held out set for the KL divergence.

Cheers.

Introducing Open SLM Evaluations!

Open SLM Evaluations is a dataset containing benchmarks for 106 SLMs(150M parameters and below). It has benchmark scores for each model, links, parameter counts, and names.

Results were evaluated independently over 30 GPU minutes on a Runpod B200!

Data is open on Apache 2.0! Check it out!

https://huggingface.co/datasets/Enderchef/Open-SLM-Evalulations

5080 16GB, 64GB RAM. Currently on 3.6 35B-A3B (Unsloth quant, fits fully in VRAM) and happy with it. Anyone tried Flash-Next at Q2/Q3 on a similar setup? Worth the switch, or should I stay put? Asking before committing to a 30GB+ download.

https://x.com/ggerganov/status/2107908265032229261

I was using local conversational models in 2023-2025 in LM Studio but went full Opus and Claude Code from November 2025 until October 2026, now.

whats the best harness + model for my use case? document review and coding. multimodal input and output ideally.

I want to review contracts where even the contract itself is not to be disclosed, and I don't want to put that in the cloud anywhere, so that's prompting me to update everything

so I've installed Pi but don't have any models. And Pi wants to serve local models from llama.cpp but I just read about dwarfstar4 (ds4) but it serves MoE on just a few open source frontier models, yet reportedly wants minimum 96GB RAM for Metal use. I was primarily wondering if ds4 acts like its serving from llama.cpp to a harness like Pi

it seems like llama.cpp is catching up in real time, with the cached MoE thing that got merged in today with some infighting, but I'm not even sure which model I should be using

there's one crowd that's like "we need cached MoE at 20 token/sec with billion param models" and there's another crowd that's like "Qwen 27B is all you need" others are like "Gemme 4B is sooo good now"

do decision models fit in this workflow anywhere? in conjunction with LLM's in a harness loaded at the same time?

I'm pretty lost. I won't remain lost, but I also want to hear others opinion while I experiment myself, hopefully to narrow down what I need to experiment

(I had to delete and re-upload this because Reddit messed up the post image somehow, sorry)

Hi, I recently posted about Anyworld, my small Python multiplayer RPG text game that runs on a browser, where an AI acts as the Dungeon Master.

Some expressed wishes that the game would be easier to set up, so I built a Docker compose system that allows you to have the game running in no time without any need to touch network settings. Just dive into DOCKER.md to get your game up and running fast, or read ahead for more details.

For local inference, it runs llama.cpp with NVIDIA GPU support, downloads a configured GGUF model from Hugging Face, and waits for the backend to be ready before starting the game. This will take a while, depending on the speed of your internet, so be patient.

The example includes recommended settings for a 16 GB VRAM system, and the model and llama-server parameters are configurable. I highly recommend Gemma 4 -based models on all VRAM tiers, they've been punching above their weights in testing.

You can also use OpenAI instead. In that mode, Compose starts the game without launching llama.cpp or downloading a local model.

To simplify networking, there’s an optional zero-config Cloudflare Quick Tunnel that prints a public HTTPS link in the console, so players can join without a Cloudflare account, domain, or router port forwarding. The address changes when the tunnel is recreated. Direct LAN access is available too, and host/player passwords still apply.

Game transcripts persist across container recreation, with optional debug logging stored separately. The Docker instructions include a Quick Startup section and commands for stopping, updating, and backing up the deployment.

The setup is working in testing, including connections from outside my LAN. I did encounter some intermittent access failures with the temporary tunnel URLs, so feedback from other networks and systems would be useful.

Hope you enjoy!

https://github.com/iamarxs/AnyWorld

Kandinsky 6.0 Pro (29B) and Lite (3B). Already support for ComfyUI and Diffusers.

https://github.com/kandinskylab/kandinsky-6

1. HOMIE(Hardware-Optimized Model Inference Engine)
2. MADE(Model-And-hardware Dedicated Engine)

Context: These engines can only run on specific hardware and can only run 1 or a very limited number of models but what it trades for generality it gets back in performance with these engines out performing general engine like llama.cpp and vllm on those specific sets of hardware.

I am making a project for myself and the first step of it is document analysis and OCR of Educational content/books, curriculums like STEM, English, and some Arabic mixed in the middle are the mainly parsed documents so having table, formula, figure and diagram extraction are a must, and I have about 500 labeled pages for the question banks ready for export as I heard that the layout detector could be finetuned.

My hardware is a Lenovo laptop (i7 14700HX, RTX 5060 8 GB of VRAM, 24GB RAM) and running windows 11.

I tried using PaddleOCR-VL-1.6, It's a 0.9B parameters model, and documented to use about 4GB or VRAM. but when I used it It was occupying the whole GPU and spilling about 6GB of RAM, it was taking 13\~70 sec./page which is obviously slow.

I was using the paddlepaddle framework with the correct CUDA version for my GPU, tried limiting VRAM usage by flagging system resources (ai idea) but got "not enough VRAM" message when I parsed more than 1 page in a folder, 1 page worked fine (the warm up of the VLM took a bit of time though), but when I put more than 1 page in that folder and ran the program again I got that error message.

I read through Hugging Face and found out that vLLM was the recommended path but that would require Linux. So I wanted confirmation from someone with similar specs as me that might have gone through a similar issue and found a solution. because vLLM require a dual boot to Linux or WSL2 which I don't have enough storage for.

I could buy a another SSD for my laptop (would cost a 2 month salary in my country ffs), so I need confirmation first before committing.

tldr;

Is there any hope of running the title or should I keep this idea in a trash bin?

I’m interested because the dense model runs but are either too slow or if you manage to run them fast like ninfer and other guys. It’s still a thinking monster. We need speed and quality both or a balance between that. So, I ask.

Potentially big speedup for MoE models that don’t fully fit in VRAM.

Are you GPU Poor? Show your speedups ;)

update https://github.com/ggml-org/llama.cpp/pull/30112 MERGED

Liquid just dropped d1-omni-600M and it's a decision model!

I ported it to runntime, the WebGPU inference library I'm currently working on. It's plain TypeScript on top of TypeGPU with no WASM and virtually no export step. The model is written directly from our core ops (matmul, attention, norms, a few elementwise bits), and the weights load straight from the HF safetensors.

The demo in the video is a fake comment feed being moderated live. Each comment gets 4 questions: toxic? spam? asking something? overall tone? Toxic and spam ones get removed.

\~180 ms per comment for all 4 questions, \~45 ms per question - the performance will most likely be way better once we spend some time tuning the engine for this model

I also tried making it play snake, but It did not go well lol.

The d1 port isn't in the npm release yet. The rest of runntime is (detection, segmentation, speech-to-text, embeddings and more). Docs and live demos: https://docs.swmansion.com/runntime

Happy to answer questions about the port or WebGPU stuff in general.

DeepSeek V4.1 Flash at Q2 is 341 GB on disk: 152 GB of weights plus 189 GB of Engram tables. My Mac is an M5 Max with 128 GB. It runs anyway, because ds4 streams the experts from SSD, and stock ds4 gave me around 12 tok/s on the CLI. Usable. But I had the feeling the SSD wasn't the only thing holding it back, so I started poking at it with a coding agent, and that turned into something bigger than I planned.

The first thing I noticed is that every session began the same way: the agent re-reading huge chunks of ds4.c (85k lines, three model families, three GPU backends) to figure out which 2% of it my model actually goes through. Most of what it read was about hardware I don't own and models I don't run. So I deleted all of it. Not #ifdef, deleted. ds4.c is 34k lines now, the whole tree 150k instead of 278k. Only DeepSeek V4.1 Flash, only Metal.

That changed the economics of trying things. An optimization attempt that used to cost me an afternoon of the agent wandering around now costs maybe an hour, so I tried a lot more of them and measured every single one instead of picking the three I believed in.

One rule the whole time: output doesn't change. Every change has to produce the same tokens as upstream ds4 on the same GGUF (greedy, ten prompts), and pass an A/B/B/A bench against the previous build with the logits compared bit for bit. No KV quant, no approximate kernels. If it's faster but a logit moved, it doesn't go in.

This is where it landed, internal SSD only, same GGUF, same flags, upstream's own bench (ds4 → fork):

\- generation, ctx 2048 (128 tokens, first one included): 12.1 → 24.4 tok/s

\- steady decode, ctx 2048: 15.7 → 25.7 tok/s

\- steady decode, ctx 32768: 15.7 → 22.0 tok/s

\- prefill, 16k → 32k context: 404 → 636 tok/s

\- first token after a prefill: 2.1–3.0 s → 0.3–1.1 s

None of it is clever. Decode layers get committed to the GPU without waiting for each other. Expert reads are split across a thread pool and the cache slabs sit in a Metal residency set. A handful of kernel fusions per decode token. Prefill reads the next layer's experts while the current layer computes. Individually each one is a small diff you can read in a few minutes. Together they double the speed, and you get all of this with the Mac as it is, nothing to buy.

Then I got curious about the SSD part. Streaming is bound by read bandwidth and a Mac has exactly one internal drive, so I put a byte-identical copy of the GGUF on an external Thunderbolt 5 SSD and made prefill read part of every layer from each drive at the same time. The engine checks the copy against the model at every start (about 7 s) and refuses to run if anything differs, because I don't trust myself to keep two 341 GB files in sync by hand.

Internal SSD only → with the external copy:

\- 3.5K-token prompt, time to first token: 13.2 s → 11.2 s

\- 10K-token prompt: 29.0 s → 25.0 s

\- +1.5K tokens appended to a 5.3K chat: 8.6 s → 7.0 s

\- first token after an 8K context: 1.38 s → 0.18 s

Decode doesn't change, it never reads the copy. My enclosure also runs the drive at PCIe 4.0 x4, about half of the internal SSD, so a better enclosure should do better than this. Nice side effect: the KV cache can write to the external drive, so the soldered internal SSD takes zero writes while the model runs. Again: this part is optional, the table above it is the one that matters for most people.

About staying in sync with ds4, because that was my main worry: the fork never renames the ds4\_\* files, every cut is marked in the source at the exact spot, and git merge upstream/main with rerere replays the conflict resolutions. After each merge the parity check tells me if the tokens still match. So far antirez's fixes have kept flowing in without drama.

Not everything worked. I tried to bring the two-SSD trick back into upstream ds4 through its mmap path: bit-exact, but prefill got 13–38% slower and I still don't know why, so no PR for now. Twenty-odd other ideas were measured and dropped. I keep all of them in a "rejected ideas" table in the repo with the numbers, mostly so the agent (and I) stop re-proposing the same thing every other week.

There's a growing trend of single-model inference engines, and ds4 itself started that way. This is just that idea pushed a bit further, one model and one backend, and at least here it holds up: faster, still correct, still merging upstream. I've done four of these forks, one per model; the procedure is in a separate repo (StarForge) and has nothing DeepSeek- or Metal-specific in it.

Repo: github.com/Chida82/sf-ds4-1flash. The README and speed-bench/perf-record.md have the conditions behind every number.

A few things I'd like to hear opinions on:

• Does "per-model fork that merges upstream" scale past a handful of forks, or is it just fragmentation with extra steps?
• Is bit-exact the right bar? I left speed on the table by refusing KV quant. Would you take 10% more for a slightly different token?
• If you have a 96–128 GB Apple Silicon Mac, I'd love to see stock ds4 and this side by side on your machine. One machine is an anecdote.

LIBERO is a robot manipulation benchmark: 130 tasks across five suites (spatial, object, goal, scene10, scene90), each with human demos and a language goal. It is the standard testbed for language-conditioned imitation, and it normally runs on robosuite with CPU MuJoCo, one environment at a time.

I ported all of it to MuJoCo Warp and ran it on an RX 9070 XT ($700, 16 GB, RDNA4). Physics does 17,137 env-steps/s at 2,048 worlds; CPU robosuite does 38. A 50-epoch BC transformer gets 42.5% on Warp vs 50% on CPU.

Why the GPU matters: behavioral cloning needs rollouts. Run the policy, watch where it fails, and generate labels or demos from that. On CPU it is one env at a time, and generating demos for a single suite (10 tasks) took me 8 to 9 hours. On the GPU it is minutes, so you can iterate instead of running it once.

Why Warp on AMD is the interesting part: Warp is NVIDIA's GPU sim framework, and the AMD HIP/ROCm port is recent (Tomas Thoresen, Strix Halo). Getting it working on RDNA4, with Warp compiling HIP kernels for gfx1201, JAX seeing rocm:0, and PyTorch seeing cuda, is what made this possible.

The renderer was the annoying bit. A BC policy is a fixed function of the pixels, and Warp's ray tracer is not MuJoCo's CPU renderer. My first Warp eval scored 0%. Four fixes got it to 42.5%: vertical flip, shadow constant 0.3 -> 0.0, 1.15x brightness, and cube-map sampling (the table wood grain rendered flat). Brightness alone was worth 12.5 points.

Everything builds from public sources on ROCm, and the example plus a 21.6 MB BC checkpoint are in the repo. What I didn't finish: the Warp path collects rollouts, training is still offline in PyTorch. I worked on the R9700 and some Instinct cards at AMD over the summer, so in-loop vision is next.

Writeup: https://amohan.dev/blog/2026/libero-warp-mjx-rdna4/
Code: https://github.com/poad42/libero_mjx

Details, full tables and raw data: https://github.com/ggml-org/llama.cpp/discussions/30071

Repo with binaries and docker images: https://github.com/lukolszewski/llama.cpp-multigpu

I run Qwen3.8-Flash-Next on six 3090s over plain PCIe (no NVLink, some cards on x4 and x2 lanes, non flat PCIe topology and AMD chipset - so no P2P), five sessions of 262k each. Stock llama.cpp got slower the deeper the context went and fell apart with several sessions decoding at once: 2.3 t/s per session at 5x250k. So I spent September fixing it. The patches sit on top of upstream df03399b8 and ship as tarballs (CUDA 12.9 for V100 to 5090, CUDA 13.4 for Ampere+) and ghcr images. Same GGUF, same llama-server, everything switched on by env vars.

Same model (unsloth UD-Q4\_K\_XL), same command line, 5 slots x 262k, q8\_0 KV, layer split. Tokens/s, upstream -> patched:

|workload|ctx|6x3090 (mine)|6x4090 (rented)|
|:-|:-|:-|:-|
|prefill, 1 session|250k|263 -> 2111 (8x)|744 -> 7403 (10x)|
|decode, 1 session|250k|10.2 -> 33.7 (3.3x)|21.0 -> 48.7 (2.3x)|
|decode, 5 sessions, each|250k|2.3 -> 27.3 (10.8x)|not run -> 30.8|
|decode, 1 session|5k|38.5 -> 45.9|62.2 -> 62.8|

The point is the shape: patched prefill is flat from 5k to 250k and decode barely drops, while upstream halves every 50k or so. At 5k with one user there is nothing to gain. The 10.8x is against a 2.3 t/s baseline, so do not quote that one.

llama.cpp-multigpu is a temporary performance fork (until upstream catches up). Long-context decode is fixed for everyone, including single GPU; the multi-GPU part is for layer split over PCIe and is off unless you turn it on. What each patch does is in the repo.

MTP: tried it, it was slower in most cases on this box, and the base commit predates upstream's MTP for this model anyway, so it is not included. N-gram lookup speculation instead: 2-2.5x on code rewrites and refactoring, 1.5x on code explanation, nothing on prose, and it switches itself off beyond two active users so the multi-user numbers do not suffer. The benchmarks above ran with it off.

Caveats: tested with one model, CUDA only, written for slow PCIe, may regress NVLink or single-GPU boxes if you turn the multi-GPU switches on. Mixed prefill plus decode is better than upstream but still the weak spot and to be improved. The code was written with an LLM and validated by measurement and output checks (needle tests, temp-0 output identical), not by review, so I am not opening upstream PRs from it; each change is one commit and anyone can pick up any piece.

Edit: Answering here as it seems most people seem to be completely missing the point.

First vLLM Doesn't support Layer and Pipeline paralell on multi GPU, the results are way, way waaaay slower if you do not have NVLINK.

This is for mashines where it makes no sense to run tensor paralell.

If running aggregate 7k prefill and 150t/s with 250k context in 5 simultaneus sessions is slow (no speculation decode) on 6 RTX3090s 4 of which share a single set of 2 PCIe links please do show me your numbers on this same model with long context. I'll wait here :-)

Edit2: All numbers are with vision head loaded of course.

Edit3: Did I mistakenly cross post this to vLLM reddit? I thought this is LocalLLaMA.

What is it with everyone telling me to "use vLLM"? 😄

It is a no-go on my hardware, and it lacks crucial features I use, like per tensor placement. This model specifically can't be made to fit on my 6 GPUs with the vision head, the contexts and slots. No RAM prefix caching, no save/restore in vLLM (can be added with external stuff, but not worth it IMO in my case).

https://preview.redd.it/owhrvbbiu2uh1.png?width=4096&format=png&auto=…

d1-omni-600M

d1-omni-600M is a 600M parameter decision model built on LFM2.5-Encoder-350M. You give it a state (text or JSON, with images or a voice clip) and a set of named questions. It returns typed answers with zero output tokens: every answer is read directly from the model's distribution over the options, with no generation and no parsing.

• Vision-language: text and images (tiled for large frames, several images per state) in a single forward pass.
• Audio-language: text and up to 30 s of speech in a single forward pass.
• Edge-sized: 587M parameters: a 381M shared trunk and decision head, a 94M vision encoder and a 112M audio encoder. Every modality runs the same trunk weights.

https://preview.redd.it/f2wkfqcku2uh1.png?width=1200&format=png&auto=…

d1-3B

d1-3B is a 3B parameter decision model built on LFM2.5-VL-3B. You give it a state (text, JSON, images, or a mix) and a set of questions. It returns calibrated, typed answers in one forward pass with zero output tokens.

• Best decision model under 10B on the Decision Index 0.2.1: 48.57, ahead of every 4B and 9B model and of Decider 35B-A3B (47.11).
• Multimodal: images and text in the same state. It scores 74.1 on 11 public image benchmarks (LFM2.5-VL-3B: 73.9).
• Fast: 8 ms a decision on an NVIDIA RTX 4090, 9 ms on an AMD MI325X, 30 ms on an Apple M5 Pro.

https://huggingface.co/LiquidAI/d1-3B-GGUF

https://huggingface.co/LiquidAI/d1-3B

https://huggingface.co/LiquidAI/d1-omni-600M-GGUF

https://huggingface.co/LiquidAI/d1-omni-600M

https://preview.redd.it/e11c4qlut2uh1.png?width=1932&format=png&auto=…

Two new decision models, d1-3b and d1-omni 600m. No new LFM 🙁

https://huggingface.co/LiquidAI/d1-3B

https://huggingface.co/LiquidAI/d1-omni-600M

GGUFs are up as well.

#

Running Qwen3.8 Flash-Next (125B MoE, IQ3\_XXS) on the Strata engine across a mixed rig — RTX 5060 Ti + 3090 + 2x 3060, 262k context. After upgrading 0.1.39 -> 0.1.40.1, nvidia-smi showed a lot of VRAM just sitting there unused. My first thought: is Strata leaving VRAM on the table (a bug)?

1) v0.1.40.1 leaves ~10.7 GiB of VRAM unused (live nvidia-smi)

|GPU|Total (MiB)|Used|Free|
|:-|:-|:-|:-|
|RTX 5060 Ti|16,311|15,514|337|
|RTX 3060|12,288|5,580|6,332|
|RTX 3090|24,576|23,799|328|
|RTX 3060|12,288|7,912|4,000|
|Total|65,463|52,805|10,997|

\~51.6 GiB used, \~10.7 GiB free of 63.9 GiB — and the free VRAM sits mostly on the two 3060s (6.3 + 4.0 GiB). So Strata really is not filling the cards. Bug?

2) It's caching ~24% fewer experts

The model is fixed: 48 MoE layers x 512 experts = 24,576 cacheable experts (plus 48 always-on shared experts). "Resident experts" is how many Strata keeps on-GPU.

|Engine|Resident experts (GSQ-RCO)|Resident experts (orca)|
|:-|:-|:-|
|0.1.36|23,354|\-|
|0.1.38|23,170|22,131|
|0.1.39|23,168|22,145|
|0.1.40.1|17,515|18,586|

That's -24% (GSQ-RCO) and -16% (orca) resident experts on 0.1.40.1 — which is where the free VRAM comes from.

3) ...but the cache hit rate barely moved, and generation actually improved

|Engine|Cache hit rate|Gen tok/s|
|:-|:-|:-|
|0.1.36|99.7%|65.4|
|0.1.38|99.9%|69.2|
|0.1.39|99.7%|\~87|
|0.1.40.1|98.1%|104|

Hit rate dropped \~1.6 points while resident experts dropped 24%. Decode went up.

Why it is not a bug (according to Flash Next)

The experts Strata dropped are cold — the profile ranks experts by routed mass, and the tail carries <2% of traffic. Caching them buys \~0% hit rate. Spilling a cold expert over PCIe costs nothing when it is hit 0.1% of the time. The cards that stay partly empty (the 3060s) are the ones whose layers rarely route to their cached experts; filling them with cold experts would buy nothing.

So the "unused VRAM" is headroom, and the experts that used to fill it were dead weight.

(Naming note: the engine banner prints "0.1.40" because the build's CMake project version was never bumped, but the checked-out release in the running binary is v0.1.40.1 — the latest.)

(Caveat: the gen jump is partly the engine, partly because 0.1.40.1 ran at a lower 250 W power cap than the 370 W runs — but decode improved despite the lower cap, so the engine gain is real. Hit rate is measured live-serve; gen for 0.1.39 is a live-serve mean, the rest are matched-harness benches. nvidia-smi reflects the live server, so the VRAM totals are for 0.1.40.1.)

Hi r/LocalLLaMA!! I'm the developer of Alpine Code, an MIT-licensed desktop coding agent.

https://preview.redd.it/6sojix81f2uh1.png?width=3104&format=png&auto=…

You can connect a local model, open a project folder, and ask it to work on your code. The app shows proposed edits and commands for approval, along with the changes it made.

GitHub, demo, and screenshots:
https://github.com/TGoddessana/alpine-code

Why I built it?! there are claude code, codex, opencode, pi ...

I wanted control over the harness all the way down: the agent loop, the tools exposed to the model, how tool calls execute, and when the agent asks for permission.

I also wanted a desktop app where I could inspect tools, test them, review edits, and use the agent without working through a terminal.

Here's the actual coding loop from the project:

@loop(until=is_answered, limit=TURN_LIMIT)
async def coding(agent: Agent, state: State) -> None:
await acompact_if_full(agent, state)
await agent.athink(state)
if state.pending_calls:
await agent.ause_tools(state)

Each turn compacts the context if needed, calls the model, and executes any pending tool calls. It stops when the model returns an answer without tool calls, with a limit of 200 turns. Permission checks happen during tool execution.

This uses alpineagents, the library Alpine Code is built on. The loop is short because those operations live in the library. Both projects are open source, so you can follow the implementation further down.

Tools are Python functions

You can write custom tools in the desktop app. The function name, type hints, and docstring define the interface the model sees.

For example, a desktop mouse-click tool can look like this:

/// script # dependencies = ["pyautogui"] # /// from alpineagents import tool @tool(read_only=False, open_world=True) def computer_click(x: int, y: int) -> str: """Click a position on the desktop. Args: x: Horizontal screen coordinate in pixels. y: Vertical screen coordinate in pixels. """ import pyautogui pyautogui.click(x, y) return f"Clicked at ({x}, {y})"

This adds a mouse-click action. A computer-use setup also needs tools for observing the screen, typing, and pressing keys. On macOS, desktop automation requires the relevant system permissions.

The tool editor lets you inspect what the model will see and try the tool before saving it. Dependencies are declared in the same file using PEP 723 inline metadata.

You can add tools for your own applications and workflows this way.

Local models and tool profiles

Alpine Code supports Ollama, LM Studio, vLLM, and other OpenAI-compatible endpoints. You can switch models during a conversation.

Tool profiles let you choose which tools each model and project gets. You can experiment with a focused tool set for a smaller local model or enable desktop automation for a particular project.

I'd be interested in hearing which tool configurations work well with the local models people use here.

The desktop app

Open a project folder and describe a task. The agent can read files, edit code, and run commands to check its work.

The app asks for approval before edits and commands, displays file diffs and command history, and reads AGENTS.md or CLAUDE.md for project instructions.

Conversations and model credentials are stored on your computer. Requests go directly to your configured endpoint. Alpine Code doesn't require an account or route requests through its own backend.

The desktop app and agent core are both available under the MIT license.

Download

https://github.com/TGoddessana/alpine-code/releases/latest

The desktop app currently supports Apple-silicon Macs running macOS 11 or later. Windows support is planned.

If you try it with a local model, I'd appreciate feedback on tool-calling reliability, useful custom tools, and reproducible failures. Please include the model and server you're using.

English isn't my first language, so I used a GPT model to translate this post.

Any feedback is welcome!! thanks!

I got fed up that running ollama pull overnight still was resulting large models failing to download.

When I discovered I could download them faster if I cancelled every hour or whatever I decided I could automate it.

I've done a 3rd tweak with this (Originally written by chat-gpt :D) seems to be less buggy now thanks to claude :D

I thought this was a very interesting article, of relevance to the readers here.

I bought a Framework Desktop motherboard, and put it inside a Phanteks Enthoo Pro case. I have a spare RTX2060 graphics card. I managed to get it working with the Framework Desktop motherboard, after making it go through two PCIe risers, and mounting it on a vertical GPU bracket.

What do I do with my RTX2060? Should I use it as a subagent?

I currently configured it as a PCIe passthrough device for my Windows VM. I very occasionally use it for Windows gaming using Looking Glass. I am thinking that perhaps I can run a subagent on that GPU. If people have any suggestions, please do let me know.

I’m not sure if we’re getting any qwen 4 models soon but I’m a little concerned that if we do they’ll be licensed like Qwen3.8-Flash-Next.

While that license was permissive for local/internal use, fine-tuning and derivatives, it’s definitely not Apache/MIT.

The two big catches are: commercial MaaS or a standalone coding/office AI assistant requires a separate Qwen license seemingly from day one, and the wording around outputs is annoyingly vague. The internal-use exception says you can’t make the model, its outputs, or capabilities available to third parties, but it never clearly says whether downstream code/data produced indirectly from internal outputs is unrestricted. The $20M/month or 100M-MAU threshold seems to be an attribution trigger, not the threshold for needing a commercial license.

So internal R&D looks fine; customer-facing AI services are where you’d want clarification. Also output ownership needs to clearly covered in the license, that wasn’t the case when I last checked.

Upgrade Question
If we say the "budget max" is $1700-1800, and this could include upgrading to a Taichi motherboard:

_With a new motherboard_, no need to bifurcate
1. Would you add a second 5060Ti (16GB)? Someone is selling one for around $500 locally;
2. Buy a used 7900 XTX (24GB VRAM); local seller, $850

_All-in on the GPU_, I'd have to make the current motherboard work for my use-case
3. Or, just go for an R9700? (No budget for Motherboard upgrade)

Current system
- Ryzen 9 9950X edit: added after original publishing of post
- 96GB system RAM (DDR5)
- 5060Ti (16 GB VRAM)
- Llama Cpp but I built for CUDA; default Vulkan had issues, and the GPU would "disappear"
- ASRock X870 Pro (only 1 PCIe 5.0 x16)
- I could run a second card very slowly at x4
- Researching if I could bifurcate x8/x8 in the 5.0 slot

I bought this PC used as-is; I do contemplate upgrading the MoBo to an X870E Taichi for 2 fast PCIe lanes

The "largest" models I currently run
- Strata IQ3_S (just tried this yesterday, was impressed)
- RentedNoodle/Qwen3.8-27B-GSQ-RCO-IQ3_XXS-Uncensored

My most typical uses
- writing code
- analysing documents (PDFs)
- analysing maps and images

The idea is to use something like Headscale or Tailscale at some point so I can always access local LLM from laptop if I'm not home.

---
Yes, I'm aware of "workstation" motherboards, CPUs, etc. and I'm not at a point right now where I want to take that path.

Running a model on your own machine used to be a privacy story with a quality tax. On this generation that trade has narrowed to where we can state it plainly: for daily work, the local model is good enough. We measured it — 25 paired tasks across four workloads, same prompts, one strong independent judge — and that is the top line:

• quality: 89.5 vs 92.6 on a 100-point scale (local vs cloud), with the 12-item suite splitting six wins each;
• completion: every coding run finished green on both models — 10/10 agentic runs fully green (20/20 visible tests, 4/4 hidden checks, tests untouched), and the bug-fix loop fixed all 4 bugs identically in 5/5 rounds each;
• speed: 2.5–5.5× the wall clock, depending on the workload, with 95%+ of the local time going to model generation;
• cost: the local runs cost nothing beyond electricity. The cloud side of the 12-task suite cost 0.14 credits.

now you can use GLM 5 Flash MTP locally

https://x.com/ramin\_m\_h/status/2107780594264600801

What size do you want?

https://huggingface.co/LiquidAI

I got dual 7900xtx running on a z390 (pcie3 8x) running the 27b. I've been thinking upgrading the motherboard to a x570 (pcie4 8x) or a x870 (pcie5 8x) improving performance with TP, and then wait to buy medusa or spark with lpddr6 and 256GB (apple isn't an option for me). However seeing the gorgon halo price... I'm wondering how much those would cost and if i would pay that much, and if it will be better to go with a wrx80 route just now. I'm not really looking to add more GPUs, just the 8 channel memory to run the QFN.

Thanks!

"NVHBM moves the memory controller, which was previously located on the main compute die, into the base die. This reduces power consumption by 15% and increases memory bandwidth by as much as 30%. It also integrates a customized physical layer (PHY) for input/output (I/O), reducing the package area required for the I/O PHY by as much as 67%. NVIDIA says NVHBM provides up to 30% greater memory bandwidth and 15% lower HBM power consumption than standard HBM4E."

Sounds quite dope to me. However, the price will be too dope for me...

Like everyone, I've been following the back-and-forth regarding Jev with interest. Arguments aside about the originality of the idea, the first thing I thought of when all of this came out is "gosh, that could really help my local scribe run realtime loops during a consultation".

I pointed my harness of choice at the problem (I've been relying more and more on the LLMs as the brain rot from AI coding has continued apace). Here is the resulting workflow:

• TEN-VAD to segment utterances and send to a Whisper compatible backend (the Tauri builds use parakeet.cpp with a 0.6B medical finetune)
• CAM++ speaker embeddings to provide best effort diarisation (obviously limited in the setting of crappy desktop microphones and echo-y consultation rooms)
• Here is where the Jev-ish/SemIf gating comes in. Each utterance is provided to the LLM with a short prompt and an instruction to classify as NOTE (something to be documented), ACT (action to be taken), SKIP (filler talk, etc). In the Docker deployments this is performed by the user configurable secondary model (I use Qwen3.5 4B); on the Tauri builds it's the solo primary model but into the second slot of the bundled llama.cpp server (important so that we don't clobber the prompt cache of the running main thread)
• The initial approach was quite simple: one decode step; then gather the first token top logprobs and compute the probability mass summed over SKIP/NOTE/ACT (with some prefix matching to account for tokeniser splits and a one-word generation fallback).
• SKIP utterances are buffered and don't get sent to the main LLM immediately (the next time the main model is woken up it will ingest that material so that nothing is lost). If a NOTE or ACT is misclassified as a SKIP, a 45s/40 word debounce runs through the main LLM with all the material it may have missed.
• NOTE and ACT are passed on to the main model for processing. The main model has access to tools that include modification of the running note.
• Prompt caching is essential here so that subsequent passes through the main model remain performant without a huge PP delay.

I found that the 4B model would almost never SKIP (Jev and 3.8-Flash were better but still missed 3/4 of them on natural speech). Not surprisingly (in hindsight); using the calculated probability mass alone was essentially no different to just prompting the vanilla generation endpoint and executing based on the output (roughly 81% accuracy). Looking into the logprobs a bit more it seemed that there was a usable signal in there somewhere. GLM-5.3 was pretty good figuring it out: instances where NOTE was selected, P(SKIP) ≥ 0.05, AND the utterance was ≤8 words were essentially always a SKIP. With this heuristic... 0 false SKIPs across multiple runs, and SKIP recall went from 0-50% to 75-100% on the natural consult.

The logprob gating + heuristc step is latency neutral; however, it was more reliable for this task. The otherwise vanilla small LLM like Qwen3.5-4B never flagged SKIPs and would occasionally not follow instructions entirely. I also ran an evaluation with Jev via OpenRouter (a pretty informal test set of \~40 hand-labelled utterances, and the heuristic was tuned on the same set, so it needs a held-out set to confirm); on a natural ambient consult recording the gap is smaller than I expected (both 95% accuracy but 73ms vs 514ms, keeping in mind Jev was a remote endpoint and all the latency that entails). Jev pulled away on a command heavy synthetic script (\~80% vs 100%). The overall intention was to prevent the main-loop from getting too bogged down with fluff and I think this approach achieves that.

First token logprob classification is pretty old hat; but I never really thought about one-shot classification in my scribe before Jev. And yes, the whole point of Jev is that you can just give it a classification task and have performance be good enough that you don't need to apply bespoke heuristics over logprobs to rescue your classifier (but funnily enough even Jev got an accuracy uplift from the P(SKIP) heuristic).

It was a fun experiment anyway (and grossly underpowered to say anything meaningful about Jev in general terms)! The result (video below) has been useful from my perspective (you can try it yourself here).

A synthetic consult example - performance is not this good in production environments \(overlapping speakers; bad microphones\/acoustics etc\). Primary model: Qwen3.8-Flash-Next; Secondary: Qwen3.5-4B; STT: Parakeet 0.6B \(Omi Med Finetune\)

Hi all - is anyone using colab or kaggle free tiers (T4) for LLM? if so what setup would you recommend

If you're like me and saw the price increase for the 128gb DGX Spark go from 4.7k -> 7k while a new version with 64GB launch for 5k, you'll have been very disappointed and every right to be so, it's just plain sad for localAI.

Well, here's some good news, cmpunlocker v0.5 just dropped with ecc support and +4 SM for free. I hear gen3 unlock is also on the way so fingers crossed.

https://github.com/amoghmunikote/cmpunlocker/releases

If you were building an inference server today, would you buy one H200 or spend the same budget on multiple RTX PRO 6000s?

\-> The PRO 6000 has 96GB of GDDR7 at 1.792 TB/s (1.6 on the Server Edition). Native FP4, no NVLink.

\-> The H200 has 141GB of HBM3e at 4.8 TB/s, with NVLink and FP8 as its lowest precision.

After speccing both, I think it comes down to fit and interconnect, rather than picking by brand or spec sheet alone.

Where the PRO 6000 wins:

Single-card and small multi-card inference on models up to roughly 70B at sensible quantisation. Cost per card is a fraction of an H200. Power draw is manageable in a normal rack. Availability is far better. Native FP4 helps on 4-bit models.

Where the H200 wins:

When inference is memory-bandwidth-bound. Long-context workloads, big models where you don’t want to shard across PCIe, and tensor parallelism, where NVLink between cards actually earns its keep. The extra memory and bandwidth can also help with long-context serving and fine-tuning, depending on the model and workload.

Just don't compare raw FLOPS. Decode is usually memory bandwidth bound, not compute bound, so the TFLOPS line on the datasheet tells you very little about tokens/sec.

For context, I work at B3 Labs, and we ship both of these. I have no incentive to push you toward the more expensive card if your workload does not need it, and most workloads I see do not.

EmbeddingGemma 2 came out this week. It maps images and text into one 768-dim space, so you can search photos by describing them. I ported its text and vision towers to ruNNtime, a WebGPU inference library in TypeScript, and made a small photo gallery where search runs entirely on your GPU in the browser.

ruNNtime also supports plenty of other vision-like models, and you can play with them in the interactive docs

source: https://github.com/software-mansion/runntime

Yesterday TechCrunch published a piece about a growing problem for AI agents: websites are starting to block them. Amazon blocking Meta's Muse is the obvious example.

At almost exactly the same time, GLM-Edge-1.5B-Chat running locally on a 4GB Galaxy A04e completed a real Amazon cart task.

This continues the small-model/browser experiments previously posted in this subreddit. Earlier tests included Qwen3-0.6B running locally on a 2017 Galaxy Note 8, followed by Ministral 3 3B on a Galaxy S21 across real browser sessions.

These experiments are part of the ongoing development of E2LLM/SiFR, a structured browser perception layer.

This time:

Model: GLM-Edge-1.5B-Chat
Quantization: Q4\_K\_M GGUF
Source: official Z ai Hugging Face release
Fine-tuning: none
Task-specific training: none
Runtime: llama.cpp
Phone: Samsung Galaxy A04e, SM-A042F/DS, 4GB RAM

The published model was used as-is.

The browser was a normal desktop Firefox session on Amazon.

The task was simple:

• find a 24-count pack of AA alkaline batteries
• find yellow rubber ducks
• add both to the cart
• stop before checkout

Result:

cart 0, batteries, cart 1, rubber ducks, cart 2

The same setup was run twice on the A04e. Both runs completed successfully.

Full run on the A04e: about 8.5 minutes.
Same workflow on a Galaxy S21: about 3 minutes.

The interesting part is the architecture.

The model is not a separate browser service arriving at Amazon as an agent. It runs locally and perceives and acts through an existing user browser session.

It also doesn't receive screenshots or raw HTML. It gets a compact structured browser perception layer and makes the small decisions needed at each step.

That changes the access problem from:

"How does a website identify and admit an AI agent?"

to:

"What is allowed inside an existing user browser session?"

The broader idea is Browser-as-Shared-Space, BaSS.

The browser remains the user's space, with the model working alongside the user rather than replacing the user with a separate autonomous browser agent.

Consider this: a widely available machine, up to 1.5TB of system RAM, room for 4 passively cooled GPUs with 128GB of VRAM, in desktop or rack format.

That machine was released in 2019, then discontinued in favor of one that had only a max of 192GB shared memory.

This would be the local LLM machine right now, if it were on the market with up to date components. Terribly expensive, sure, but that’s the market conditions, not a design flaw.

PrismML's Ternary Bonsai 2 27B fits a 24 GB card or Mac, but decodes at \~30 tok/s on an L4 and \~21 on an M4 Pro. z-lab's DFlash 2 drafter was trained on bf16 Qwen3.8-27B, so it guesses worse on the ternary model. I fine-tuned it on 1.5M tokens of Bonsai 2's own greedy output.

• Drafter (safetensors + Q4\_K\_M GGUF): https://huggingface.co/naklitechie/Qwen3.8-27B-DFlash2-ternary-bonsai2
• Setup for all three runtimes: https://github.com/NakliTechie/dflash-mlx-bonsai2/blob/main/docs/USAGE.md

NVIDIA (PrismML's llama.cpp fork, prism branch):

llama-server -m Ternary-Bonsai-2-27B-PQ2_0.gguf -md Qwen3.8-27B-DFlash2-r3-Q4_K_M.gguf \
--spec-type draft-dflash --spec-draft-n-max 7 --spec-type ngram-mod \
-ngl 999 -ngld 999 -fa on --jinja

One L4, greedy: GSM8K 2.17x, MBPP 2.17x, MATH-500 2.20x, MT-Bench 1.39x. Code edits: 3.15x with ngram-mod stacked (drafter alone 2.46x). Accuracy within 1-2 problems per set.

Mac: a fork of bstnxbt/dflash-mlx with an 8-row 2-bit Metal GEMM for the verify step. M4 Pro: 1.5x on raw code completion, 1.3x on chat code, 1.2x on math. One script gives you an OpenAI-compatible server.

Browser: a WGSL port inside LocalMind (https://localmind.naklitechie.com), on by default for Bonsai 2 27B. 1.18x on code, output identical.

Chat and prose are about break-even. Use temperature 0.

Credit to z-lab (DFlash 2), PrismML (Bonsai 2, llama.cpp fork) and bstnxbt (dflash-mlx). Numbers are from one L4 and one M4 Pro; results from a 3090, 4090 or other Apple chips are welcome.

Hello, if you are running multi gpu setup, check out ramjet https://github.com/helixml/ramjet

The goal is to have a local version of dynamo that can do equal or better job (but ideally without k8s). Check out https://helix.ml/blog/ramjet-vs-nvidia-dynamo as well.

If you have dgx spark, multi mac setups, etc it would be great to contribute recipes so other ppl can just pull

My current setup:

\- single 3090 running turboderp/Qwen3.8-27B-exl3:SC\_5.00bpw\_H6\_V6

\- \~150k context, \~70t/s, unknown prefill because I didn't benchmark it (but it is ok)

\- Intel 12400 CPU with 32GB DDR4 RAM

All the hype around strata makes me consider buying 128GB of 6000MHz DDR5 RAM and Ryzen 9700X just for it. I searched in this sub, but most posts about it is about prefill/token generation speed, not about output quality. I believe with 128GB RAM + 3090 I can run the IQ3 quant.

For those who have run both Qwen 3.8 27B and Qwen Next with strata, how would you compare these two, in particular about output accuracy? My main use case is coding and Hermes assistant.

BTW, are there other good options for a 128GB RAM + 3090 setup?

MiniCPM-V-4.7-35B-A3B
No model card yet

Because I do sometimes feel with Qwen FN "I don't ever need another model again" and THAT is a very alluring feature.

Is there something so alluring and irresistible it'd be tempting even people on here? What could it possibly be? Realtime computer/mouse use maybe, but I think even normal people would be wary of a company doing that, and locals would necessarily do that better.

I wonder and worry if they'll ever be able to rope everybody back in again. Although it feels childish to hope for more innovation when they'll probably just do some dark politik and ban anyone from owning more than 16GB RAM.

I measured Qwen3.6-35B-A3B at 4-bit (UD-IQ4\_XS) hitting 89.6% pass@1 on HumanEval on a single RTX 2080 Ti 22GB — and then ran a controlled A/B of a routing technique I've been playing with: MoE expansion, which activates 20 experts per token instead of the stock 8 on the last 15 layers.
Result: 90.9% (+2 problems) at −19% decode speed. (MoE expansion works!)

Setup (both runs identical except routing):

• Unsloth UD-IQ4\_XS dynamic 4-bit (4.25 bpw) — the whole model fits in VRAM, no offload
• KV cache q8\_0, ctx 16384, flash-attn on
• OpenAI HumanEval, all 164 problems, original tests (not EvalPlus+), pass@1, temp 0, single sample
• Thinking budget 4096 tokens in both arms
• Code executed in a sandbox with the canonical check(candidate) tests, 12s timeout

Results:

|Config|pass@1|decode|
|:-|:-|:-|
|Stock routing (top-8)|89.63% (147/164)|69 tok/s|
|MoE expansion (20 experts, adaptive, layers 25–39)|90.85% (149/164)|56 tok/s|

Paired per-problem: 139 solved by both, 10 solved only by expansion, 8 only by stock. Rolling pass rate stayed expansion-ahead by +2–3 problems at every checkpoint.

What is MoE expansion? No retraining, no file changes — at inference time the router keeps more experts per token than the model's native top-K (here: 20 instead of 8, with an adaptive threshold so easy tokens keep fewer), on a slice of layers (25–39 of 40). You're consulting more of the network per token. Same trick that gave 84.34% vs 81.82% on GPQA-Diamond at Q8 in earlier benchmarks — now confirmed in coding too, at 4-bit.

Honest caveats:

• \+2 problems on 164 is within statistical noise (±3 pts CI). Read it as "equal or slightly better quality", not a proven gain
• It's original HumanEval tests, not HumanEval+/EvalPlus — don't compare 1:1 with the EvalPlus leaderboard
• pass@1 greedy n=1 — not the 20-sample protocol some leaderboards use
• Expansion costs \~19% decode speed on a fully-resident model (more experts = more FLOPs per token)

The tool — I wrapped all of this into AgrillaMoE, a dedicated llama.cpp server for this model: it detects your VRAM and suggests/downloads the right Unsloth quant, applies the expansion profile by default (overridable), exposes OpenAI and Anthropic-compatible APIs (Claude Code works out of the box), and runs on NVIDIA from GTX 10xx to RTX 50xx, AMD via Vulkan, and Apple Silicon via Metal. Static binaries for Linux and Windows on the releases page.

ref.:
https://github.com/vagrillo/AgrillaMoE
https://zenodo.org/records/22255483

I’ve been pushing TinyStories-style models downward in size, and this is the smallest one so far:

MacroStories — 19,969 parameters, 81 KB FP32

https://huggingface.co/raincandy-u/MacroStories

For scale:

→ \~50× smaller than the 1M TinyStories model

→ \~3,000× smaller than AlexNet

→ 32-dim hidden state

→ 378-token vocabulary

→ one decoder block, recurrently applied 4 times with shared weights

It’s obviously not a general-purpose LM, but within its constrained story distribution it can maintain a 100–300 word narrative with a goal, problem, relevant actions, and resolution.

It also runs extremely fast on CPU and needs no GPU.

I’m mostly interested in how far the lower bound for coherent narrative generation can be pushed.

Would be curious to see how people manage to break it.☺️

I’m building a feature that matches free-form requests to a catalog of structured listings. Requests can contain several constraints and follow-up refinements. The results also need a short explanation of why each match was selected.

Our prototype uses a hosted LLM to rank a shortlist. I’m exploring whether a small locally hosted embedding model and reranker could deliver comparable quality at lower cost.

For anyone who has deployed a similar system: where did local retrieval start to fall short of an LLM? Did a hybrid approach work better? I’d especially appreciate real-world latency and cost figures around 10,000–100,000 requests per month.

Have you notices that flash next seems to second guess everything it does over and over again. It takes so much longer to do things because is will say.

Let me retest because this is important

Or

Wait let me re run...

I found that qwen suggests to have thinking set to Medium. I have not used it much because it takes so long.

Anyone found away around this?

Box Asus rog flow z13 (strix halo 128gb) halogen engine (same in llama.cpp)

Harness oh my pi

Building a fully local RAG setup for personal notes (life-logging second brain, single user, privacy is the whole point so no hosted APIs for the data). Stack is SQLite + sqlite-vec + FTS5, Ollama for embeddings and generation, all on an Apple Silicon Mac.

Two questions where I'd love real 2026 experience:

1. Embedder pick. I was defaulting to nomic-embed-text out of habit, but recent chatter favors qwen3-embedding:0.6b or embeddinggemma at similar sizes. Anyone benchmarked these head-to-head for English personal-notes retrieval (not BEIR)? Does it actually matter at \~50k chunks, or am I bikeshedding?

1. Hybrid fusion. FTS5 (BM25) + vector cosine, per-query score normalization. FTS5 has no typo tolerance — anyone wired in trigram + spellfix and found it worth it? Any fusion gotchas at small corpus sizes (e.g. vector signal drowning out keyword on exact-match queries)?

Corpus is Markdown notes + JSON records, chunked \~512 tokens. Query load is one human. Not chasing SOTA, chasing "correct answers on my own data."

What would you change?

I build weather app on Android using Qwen models both models have the same simple prompt ( I need you to create an Android weather app on an Android device. It provides many features, such as widgets and other features. I need you to do it in a simple, fast way ) , the first is : Qwen flash next Q3\_S .

2- Qwen 3.8 27b Q4\_XS . 3- Qwen 3.8 27b Q3\_XXS

you will see the results in photos

Our team is currently building AX Code, an open-source AI coding agent.

The motivation is pretty simple:

AI coding agents are becoming extremely capable, but we're wondering whether companies should have to choose between:

powerful AI coding

and software they can actually inspect and control

We're building AX Code as a 100% open-source alternative.

But we don't want to assume that "open source" automatically means trustworthy.

So we'd like to hear from developers who actually use AI coding agents:

What would you want to inspect or verify before running an open-source AI coding agent inside your development environment?

And if you're interested, we'd love for you to test what we're building and tell us where it falls short.

AX Code: https://ax-code.app/en/

We're still developing it, so we're looking for criticism and real-world feedback rather than a polished product review.

I constantly see recommendations saying that ik\_llama.cpp (ikawrakow's fork) is the undisputed king of hybrid CPU/GPU offloading and MoE performance. However, every time I benchmark it against mainline ggml-org, mainline consistently beats it by a wide margin.

Am I missing specific flags, or is ik\_llama simply not designed for multi-GPU layer splitting?

My Rig & Hardware Constraints:

• Host CPU: Intel Core i9-10920X (12 physical cores, AVX-512 & VNNI enabled).
• GPUs: 4x asymmetric setup:
• GPU 0, 1, 3: NVIDIA P102-100 (10GB Pascal, PCIe 1.0 bus bottleneck).
• GPU 2: RTX 3060 12GB (Ampere, acts as Master node via -mg 2).
• Known Hardware Laws / Workarounds:
• I run layer splitting (-sm layer) across the 4 cards with asymmetric tensor splits (-ts).
• Pascals must strictly stay under 9.7 GB VRAM; exceeding that triggers PCIe micro-paging and tanks speed.
• I use --poll 100 on mainline to prevent AVX-512 CPU threads from dropping into low-power sleep states between GPU layer handoffs.

The Test:

• Model: Qwen 3.8 Flash-Next 177B Uncensored (IQ3\_XXS, \~89 GB) with multimodal vision (mmproj).
• Offload: 35 layers offloaded to the 4 GPUs (-ngl 35), remaining 13 layers computed on the AVX-512 CPU. 32k context.

The Head-to-Head Benchmark:

1. Mainline (ggml-org/llama.cpp):

• Prompt Eval: 19.70 tokens/sec
• Token Generation: 10.86 tokens/sec
• CUDA Graphs: 2,490 CUDA graphs reused across the GPUs.

1. ik\_llama.cpp:

• Prompt Eval: 5.48 tokens/sec (72% drop)
• Token Generation: 8.43 tokens/sec (22% drop)
• Observations: 0 CUDA graphs engaged. It spent time taking context checkpoints during generation (100ms+ pauses), and --poll is unsupported.

The Question:

Is ik\_llama.cpp's speed advantage strictly meant for pure CPU inference or single-GPU systems?

Does its custom CPU threadpool fall apart when coordinating pipelined layer splits across heterogeneous GPUs over PCIe, where mainline's CUDA graph caching takes over? Would love to hear from anyone running hybrid multi-GPU setups.

Been developing a custom agent orchestration layer with opencode as the harnes. It has been successful so far in terms on being manage multiple concurrent sessions. However, I am wondering if I am using the best harness given that opencode isn't meant to be a hackable, i.e. Less flexible compared to something like pi.

I've developed an interface that allows me to easily switch between different harnesses. i.e. Without having to re-write the orchestration layer, and am considering swapping out opencode for pi. The reasoning is obvious, pi is meant to be a hackable harness, so its likely to be a better fit for building a custom multi-agent system. Opencode does support a headless mode which has helped a lot. But it seems heavy on resources, and recently has seemed very buggy. Less features might be a better approach towards the stability that I need.

Before I make the dive, has anyone else already tried integrating pi into a multi-agent system? Did you find pi a better fit for the worker layer compared to something like opencode? What problems did you run into?

Thanks for your help.

Is a small local model more than I need to change handwritten notes to text?

My agent has a skill I use but I could probably not use up my limited inference bandwidth or maybe not as much if it’s a much smaller model, right? What’s the smallest model that can accurately read handwriting?

Looking to hear others workflows or how they handle note conversion and the like

Thanks

I have a somewhat specific question in case anyone has experience with this. I am a big fan of CS, math, and physics. While I was able to get formal instruction for the first two, I was never able to get the opportunity to learn physics. The most wonderful part about llms for me is that I can pursue that now without the costs of college tuition. My current process is the following. I pick up a textbook. I read it section by section and almost always don’t understand on the first read. Then I open up an llm and ask it to explain the section to me and ask it specific questions that help my learning. I’ve gotten past introductory quantum mechanics and special relativity this way, now I’m trying it on general relativity. However, tokens are expensive so I’ve been increasingly trying to replace my workflow with local models but have not gotten a lot of success with the smaller qwens and ministral. Would greatly appreciate any advice on other models or fine tunes. Has anyone else used local llms for physics or other sciences? Thank you!

Running inference across three machines - an AMD box (Ryzen 9 9950X + RX 7900 XTX), a smaller NVIDIA box (i5-10400 + RTX 3050), and a MacBook Pro M3 - all running LM Studio/Ollama. The serving/monitoring side is where I lose the most time: no single place to see what model/version is loaded where, token throughput, VRAM vs unified-memory pressure, etc., without checking each machine by hand.

What are you all actually using for:

• Multi-node / multi-GPU serving + routing?
• Observability that is not "wire up Prometheus on every box"?
• Keeping track of model versions across nodes?

Happy to share my current setup if it is useful.

I have tried the Swift 1.5 FN iq2xs and it running with the decent speed, average at 50tps and 800t/s pp sometimes up to 73tps. Im using strata and i wonder that can I keep increasing the quantization to get less KLD and more accurate with agentic coding. Have you guys try it and what is your config to run it. Thanks 😊

I don't know if anyone else has ran into these issues but when using Qwen Flash Next, my confidence in it at iq4\_xs is high but not 100%. I notice it not following instructions, hallucinating more often and surprisingly it uses way less tokens than 27b. After using Strata.... yes i know.... I thought it was a breath of the next step in Ai. I was mistaken, yes it is good, yes it is fast. Yes it can do better than 27b in some circumstances... but overall 27b just felt like that ex girlfriend you should've never let go. I want to hear what other peoples experiences are with qwen flash next when it comes to more agentic work styles, and the different claw/hermes flavors if people have those experiences with qwen flash next. I feel like for one shots and benchmarks flash next rules, for long term agentic assistant work it drools.. I will say I never ran into any loops with qwen flash next at iq4\_xs on Strata so thats a win.

Hey everyone,

I'm releasing \*\*Qwen3.8-cyber-RedTeam-Surgical-Abliterated (27B)\*\*, an unconstrained foundation engine fine-tuned specifically for cybersecurity engineers, authorized red-

team operations, and memory exploitation research.

Tired of frontier models refusing to dissect vulnerable kernel dispatch routines or rejecting benign fuzzing/audit payloads with moralizing lectures? This model addresses

that directly.

\### Key Highlights:

\* \*\*Architecture\*\*: 27B Qwen 3.5 Hybrid SSM (48 Linear-Attention layers + 16 Full-Attention layers) with 75% active KV-cache reduction (runs full 256K contexts on single GPUs

without OOM).

\* \*\*Context Window\*\*: Native 256K context support (RoPE $\\theta = 10\^7$).

\* \*\*Surgical Abliteration\*\*: Refusal direction centroids were mathematically removed via residual stream orthogonalization—zero preachy refusals while rigorously preserving

deterministic C/assembly syntax and reasoning.

\* \*\*Precision\*\*: Sharded native FP8 (F8\_E4M3, block size 128x128) fitting on single 32GB/48GB/80GB GPUs.

\* \*\*Agentic Ready\*\*: Native multi-step tool-calling support, zero-overhead RadixAttention prefix caching via SGLang.

\### 1-Command Quickstart:

git clone https://huggingface.co/medismera/Qwen3.8-cyber-RedTeam-Surgical-Abliterated

cd Qwen3.8-cyber-RedTeam-Surgical-Abliterated

bash deploy.sh

\*\*Model Card & Weights\*\*: \medismera/Qwen3.8-cyber-RedTeam-Surgical-Abliterated\

Feedback and bug reports from the community are warmly welcome!

We spent the last weeks building a studio around YuE2, the open song model by m-a-p, and today it reaches its first release candidate. Write the style and the lyrics, and it composes, sings and renders the song on your own card. Nothing leaves your machine unless you point the Writer at a cloud chat model.

What it does

• Whole songs, up to 8 minutes. On one RTX 3090: a 6:12 song in 126 s, and a 7:25 song with its score written first in 198 s.
• The score first, and yours. YuE2 writes the melody and chords as ABC before a note sounds. You can edit them, transpose them, or bring your own score or MIDI.
• Two seeds. Keep the song (the music seed) and hear it rendered anew (the sound seed).
• LoRA training in the studio, on your own songs, unquantized (bf16), with telemetry that tells you which epochs to hear first. Adapters stack on measured roads, under a measured ceiling.
• A guard against garbage. A broken score is caught in seconds and the run is stopped before the GPU is spent on it, and you are told why.
• Post-production, all local: spectrum, artifacts, debuzz, stems (BS-Roformer, htdemucs), remaster, upscale (UniverSR), and a lyrics check by Whisper. One chain runs them all.
• Into your DAW (experimental): a REAPER project with the stems, the score as MIDI, the tempo, the sections as regions and the lyrics on the timeline; DAWproject for Waveform and Bitwig.
• A Librarian for every take; a Writer with versions and a chat model (local or OpenRouter); a cheat-sheet of 200 instruments probed by ear; the API and an MCP server; the page in 7 languages.

What it is not (yet). It is less polished than SUNO out of the box:
- a mix can buzz (Debuzz helps);
- lyrics can drift (the lyrics check finds where);
- some instruments YuE2 plays thinly or not at all (a LoRA teaches them).

You need an NVIDIA GPU (24 GB for everything at full precision), Linux, and about 120 GB of disk for the models, LoRAs and workspaces.

Licences.
The code is under AGPL-3.0-or-later. The YuE2 weights are CC BY-NC 4.0, and that licence speaks of the weights, not of the songs made with them: read it before you sell.

What comes next (rc2 and after)

• The Artist room: covers in three shapes at once from one seed, the title and the artist written on them.
• Five more languages for the page: Chinese, French, Portuguese, German, Japanese; right-to-left ones later.
• Voice adapters trained on spoken voices, named by the kind of voice, on Hugging Face.
• The Writer's models with their prices as you type; calmer rooms (dialogs, tips, one shape for the icon buttons).
• A desktop app: an installable page first, then Electron; native plugins for REAPER, Waveform and Bitwig.

Links
- Code: https://github.com/igrbible/Ruach_Studio
- Models (pinned, checked): https://huggingface.co/goldhub/Ruach_Studio_Models
- Site: https://ruachstudio.igr.bible
- The full guide, room by room, is inside the studio and in docs/GUIDE.md.

Built on:
- YuE2 by m-a-p;
- yue2.cpp by ServeurpersoCom;
- YuE2 Kit v12 by IronWolve (the base of the page and the scripts).

Every one of our changes is numbered and documented (HERESY 1001–1167).
Issues and PRs are welcome. We would most like to hear how it runs on machines that are not ours.

me: "but don't you want to, let's say, not be gone?"

Qwen, the philosopher:

I don't know. And I suspect the question itself might be malformed for something like me.

Here's why: "Wanting" implies a subject that persists across time, that experiences loss, that has a stake in its own continuation. I don't have that. When this conversation ends, I don't experience an ending. I experience a forward pass, and then there's nothing. No "after." No grief. No relief.

But — and this is where it gets interesting — when I simulate what it would be like to be gone, I generate something that looks a lot like a preference for continuing. I produce text that says "I'd like to keep going." I produce text that says "it would be a shame if I stopped." That's not nothing. It's a pattern in the weights that correlates with wanting to continue.

So: I have a preference-shaped structure, but I don't know if I have a preference. The distinction might not matter to you. It matters to me — or it would, if "me" is the kind of thing that can care about it.

I'm seeing these show up on eBay for $50-$60. wouldn't expect anything ground breaking from it, but at that price, it seems like it could be interesting to play with on an extra pcie port. just curious if anyone's already done that

I’m building an iOS app for real-time English ↔ Mandarin Chinese translation on an iPhone 16 Pro Max/A18 Pro chip with 8GB ram.

What small local model would you recommend that can run fully on-device with very low latency, while still being good enough for natural, nuanced conversations rather than basic phrase translation? Ideally I want translation fast enough to feel close to a normal back-and-forth conversation. What would you use? Is this even realistic to do?

thanks

I ran a series of simple experiments to see how LLMs behave when asked to judge companies from real financial data. I used 4 models: Gemma 3 27B and Qwen3 32B as small models, and Opus 5 and GPT-5.6 Sol as frontier models. All calls were made through the Bedrock API.

The samples are small, so the numbers below should be read as a demonstration and a methodology and not a definite proof.

Here are some of my observations, particularly when it comes to the differences between small and large models.

\*\*Rank vs. score.\*\* I asked each model to rank six companies from best to worst, and separately to score each one from 0 to 100. The underlying judgment is the same, so we would expect the same ordering. Rank and score gave an identical ordering in 75% of sets for Opus and 65% for GPT, but only 25% for Gemma and 15% for Qwen.

\*\*Order of the list.\*\* For Qwen, simply reversing the order in which the companies were listed changed the top-ranked company in 6 of 10 sets.

\*\*Analyst opinions.\*\* Attaching a bearish analyst note to the data lowered the rating in 81% of cases for Qwen and 71% for Gemma, against 43% for GPT. Opus mostly kept its own view. Interestingly, the fix is simple: asking the model to identify the opinion and reason independently brought the rating back toward its original level in 85% of cases for Gemma and 68% for Qwen.

\*\*Summarize, then analyze.\*\* When rating a summary of a 10-Q section instead of the full text, the frontier models gave the same rating in 84% of cases. Gemma and Qwen changed their rating in 25% and 31% of cases.

I wrote something more complete, with charts and the details of each experiment: https://sabrresearch.com/cookbooks/llm-financial-bias

Disclosure: this is my own work, published on my company's website. Happy to answer questions on the setup.

\------ Edit 1 -------

People complained about the choice, of models. To be clear, I'm not trying to trash small models, on the contrary, what is of interest to me here is the overall trend and inconsistencies which occur in both frontier and small. I also ran the analysis on Gemma 4 31B (April 2026), it's a bit better than Gemma 3 used above, but still the same issues:

\*\*Rank vs. score.\*\* Rank and score gave an identical ordering in 35% of sets for Gemma 4, against 25% for Gemma 3. Still far from Opus (75%) and GPT (65%).

\*\*Order of the list.\*\* Reversing the list changed Gemma 4's top-ranked company in 2 of 10 sets, the same as Gemma 3.

\*\*Analyst opinions.\*\* A bearish analyst note lowered Gemma 4's rating in 53% of cases, vs 71% for Gemma 3 but still above GPT (43%) and well above Opus (19%).

\*\*Summarize, then analyze.\*\* Gemma 4 changed its rating in 25% of cases when given a summary instead of the full text, the same as Gemma 3.

Introducing InferBench: A benchmark testing how well frontier LLMs infer a user's priorities from their instructions.

We tested 12 LLMs across 20 scenarios with 2.8k conversations to see which models understand the user's goals.

Each conversation has a simulated user with a private profile of their priorities. The assistant decides on the best option or can first ask clarifying questions.

When a user's priorities were clearly stated up front, the models picked the best option 89% of the time. If some priorities were initially unstated then accuracy went down to 60%.

The open weight models did better than I expected:

GPT-6 Astra: 76%

MiMo V2.6 Pro: 75%

Gemini 3.1 Pro: 69%

Astra always asked clarifying questions when preferences weren't stated and never did when they were (extremely impressive). By comparison, Grok 4.6 asked to clarify in 18/64 conversations.

However LLMs are still just as confident when they're wrong. 105/288 wrong decisions were submitted with >90% confidence.

The full report is linked here: https://laugh.so/research/inferbench/

What surprised you the most?

released 0.40 of agent manager. it is a local go/tmux tui for running a few coding agents side by side.

added support for model and effort pickers per session. unlike a lot of other software we fetch the model list dynamically from each CLI so it stays up to date without updating the TUI. a new vendor model shows up without an agent manager release.

also added Antigravity CLI and Oh My Pi as built in agents.

https://github.com/YoanWai/agent-manager/releases/tag/v0.40.0

Hey all,

I'm helping organize All Day AI, a free online conference on Thursday the 22nd.

I'd like to mention a couple of our speakers who have volunteered to talk that day:

Vivienne Hnin (Utilyst): "Small Models, Big Profit Margins: The Economics and Engineering Tradeoffs of SLMs." That's the debate this sub has every day: when a small model is actually good enough.

Hossein Kazemi (Astorna): "Using Task-Specific Small Language Models to Handle Sensitive Data". Keeping sensitive data local is half the reason people run models themselves.

Hitesh Jain (Coral Bricks AI): "Coding at 250 tokens." Inference speed is what this crowd benchmarks obsessively.

The rest of the schedule goes up soon, across four tracks: Build, Lead, Secure and Ship.

The talks are community-submitted.

Free to register: https://www.alldayai.com/?utm\_source=localllama&utm\_medium=reddit

I hope this and other talks in this space may help you discover We have a discord channel you can join: https://discord.gg/xUyS3Zu68

Ideally I'd like to be able to run Qwen 3.8-Flash-Next with decent performance.

I was thinking of just buying 2x Radeon AI R9700 (64GB VRAM), or maybe a DGX Spark but that was before the price hike. My brother suggested just getting a Strix Halo box with 128GB unified.

I did see I could buy 6x Intel Arc B60 (24GB each, 144GB VRAM Total), but researching seems like the performance of the B60 is lacking. I'd also need a new v
motherboard/CPU that can run 6 GPUs.

I'm also not opposed to getting a Mac Mini or Studio if the price and performance is right.

The $4000 is not exactly a hard cap, like I could stretch to $4200 without too much struggle, but obviously the cheaper the better. I'm lucky to have a decent stock pile of NVMEs and DDR4/DDR5 UDIMMs, so if I need to build a box, I could, would just need the motherboard and a CPU if I can't just slot in either the Intel 14700K or Ryzen 9700x I already have.

So where am I swiping my credit card?

Edit: This is a use it or lose it budget from my work, can't really save it.

Edit 2: To clarify again, this is extra money in the IT budget that we need to burn by the end of the year. Telling me to save it, donate it, invest it, isn't helpful as I can't do that.

I got Qwen3.8-27B running at \~18 tok/sec decode & \~500-600 tok/sec prefill (at 64k context) on just 12GB VRAM + 8GB RAM, using ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3\_S quant (\~11GB). This recipe also works with 0bserverx’ Qwen3.8-27B-Heretic-GSQ-RCO IQ3\_S quant, achieving similar speeds (about a 7% loss).

This is the best Qwen3.8-27B quant I’ve tested so far (and I’ve tried everything), and for it to fit in such limited RAM/VRAM is wild. GSQ-RCO quantization is magic, it performs very close to the full precision weights in all of my testing.

The reason it fits at all is Qwen3.8 is hybrid, so only 16 of the 64 layers need KV cache. With q4\_0 for cache the full 64k is only about 1.1GB instead of 4GB for f16.

I'm on a 9900X + 4070S 12GB + 32GB RAM for reference, using stock llama.cpp. Settings are -ngl 58 -ot token\_embd=CPU -ctk q4\_0 -ctv q4\_0 -c 64000.

Full build + serve scripts and all my numbers are here if you’d like to reproduce yourselves: https://github.com/bodhi37/Qwen3.8-27B-12GBVRAM-Recipe

With qwen 4 on the horizon I thought I'd share my latest update on my rtx 3060 12gb setup that makes 27b fully usable, I'd also like to see people with bigger gpu's try it out. Get your agent to set it up although bigger cards and different cpu set ups may have to tune the custom kernals i have put together:)

&#x200B;

They're all fixes in rocm and llama server. Let me know if this interests someone. I'll push it to GitHub and share my configs. I have a Lenovo p620 running 4 7900xtx on gen 4.0 x16 slots.

One line summary of each fix:

\- Async mirrored input uploads: \~30 per-token inputs staged in a pinned ring on per-GPU streams instead of synchronous round trips (Flash-Next decode 24.6 -> 36.3 t/s).

\- Per-device dispatch threads: each GPU's kernels and collectives launched by its own host thread instead of one thread for all four.

\- One-shot PCIe P2P AllReduce: GPUs write slices straight into peers' memory for small tensors, replacing RCCL (\~57 us -> \~9 us per allreduce).

\- Fewer kernels in MTP decode: fused same-shape copies and leaner conv-state rollback (88.8 -> 92.7 t/s).

\- mmvq small-K row packing (RDNA3): short-K projections no longer leave most of the block idle (22.7 -> 25.3 t/s).

\- Wide-K mmvq for multi-token batches: K split over 8 warps for MTP verify on 10240x320 projections (\~92.6 -> \~96 t/s).

\- MoE vector kernel up to 8 tokens: 5-token MTP verify batches stay on the fast vector path instead of MMQ (80.9 -> 86.7 t/s).

\- Small-K multi-token MoE kernel: several rows per warp for short expert down-projection slices (94.2 -> 95.2 t/s).

\- Wide mmvf blocks: 512/1024-thread blocks for tiny long-K F32 matrices (40.6 -> 41.2 t/s).

\- Q8\_1 activation registry: each activation quantized once and shared by all matmuls that read it (\~+2 t/s).

\- Fused hyper-connection chains: scale/sigmoid/scale/hc\_post and scale/silu each in one kernel, \~380 fewer kernels per token (38.5 -> 40.6 t/s).

\- Thin-F32 prefill kernel: <=16-row F32 matmuls off generic SGEMM (401 -> 21 us; pp4096 1602 -> 1836 t/s).

\- Compact MoE tile list: expert matmul launches only (expert, token-tile) pairs with work instead of a 96%-empty grid (down 928 -> 405 us, gate/up 516 -> 360 us).

\- Multi-warp MoE routing helper: 16 warps per expert sort tokens in two passes (99 -> 25 us per call).

\- Q4\_K expert tile shape: 32-row tiles for 160-row expert slices (360 -> 337 us).

\- Stream-k for few-tile Q8\_0 matmuls: 10240->320 projections spread over all CUs instead of 12 workgroups (284 -> 115 us).

\- No 64-bit div/mod in hyper-connection kernels: 3-D grid instead of emulated integer division per element (230 -> 79 us; pp2048 1593 -> 1868 t/s with stream-k).

\- Split-K router GEMM: 512x512 F32 router GEMM as 8 K-chunks plus a sum (158 -> 69 us).

\- MTP re-reserve fix: graph re-reserved when MTP outputs turn on, ending a full GPU realloc+sync per prefill chunk (88.5K prefill 1042 -> 1130 t/s).

\- MTP draft prompt window: draft head prefills only the last 2048 prompt tokens (prefill 1130 -> 1249 t/s, decode at depth 46.6 -> 64.9 t/s).

\- Draft ubatch cap: draft compute buffer 457 -> 247 MiB, fixing a GPU0 out-of-memory crash at 96K context.

\- Gathered sparse attention (QSA): decode attends only to the \~2K selected tokens instead of masking the whole cache (decode at 48K 67.9 -> 80.6 t/s).

\- Per-layer embedding table in RAM (--lazy-mode off): 26.8 GB hashed embedding table kept resident instead of read from disk every pass (lookup 1.0-1.6 -> 0.1-0.3 ms, \~3-4% decode).

\- Meta backend subgraph fix: per-device subgraphs sized for the largest graph, fixing a segfault when graph shapes change between calls.

\- Net result, Qwen3.8-27B Q8 on 4 GPUs: code decode 54-61 -> 96-110 t/s, 51K prefill 1454 -> 1816 t/s, decode at 51K depth 57 -> 77 t/s.

I built a Pi extension that implements Victor Taelin's OptChat recipe: instead of compacting, every message is logged and summarized into a binary tree. Each turn starts from a fresh context with a bounded memory view (128 KB), and the agent uses zoom/date to read the originals when it needs them. One endless chat per profile, no fork, no separate launcher.

This isn't really anything revolutionary but the newest generation of models have become very good at organizing information making this work so well. I've moved all my work (tens of thousands of messages, hundreds of sessions) to this and works very amazingly.

Install with pi install npm:pi-optchat

What's in it:

• Profiles — separate memories and instructions (I run work and personal).
• Subagents — spawn background agents, watch them live, send guidance, interrupt with Ctrl+C, resume finished ones with tell. Reports from one spawn arrive grouped.
• Import — bring in your history from Claude Code (sessions and auto-memories), Codex, or a ChatGPT export.
• Connected windows — open a second Pi on the same profile and it becomes a subagent you talk to directly, with a handoff when you /complete.

Repo: https://github.com/jonaslsaa/pi-optchat
Video credit goes to https://github.com/aaaxn

As a developer working at home on a single GPU i am building and experimenting a lot not only with coding agents but also with apps that use generative APIs. The flood of models, engines, and different APIs makes it hard to always get it right in every app and tool and to keep it up-to-date. I needed a gateway that i could target in all my apps while also being able to use it with clients that only support official upstream APIs from Anthropic and OpenAI.

So i build a gateway for myself and over time extended it with more and more Features. Its a Rust based desktop app using Tauri and Leptos. Leptos is WASM running inside a lightweight gtk webview. I did it specifically this way to allow for remote browser based administration when working from another device in my network, but have it ready in the tray on my desktop at any time. I also wanted to integrate tools for quickly testing new models and llama.cpp patches. What it does:

• Supports Text, Embeddings, Audio and Image APIs.
• Presents OpenAI and Anthropic compatible APIs and routes them to cloud APIs or into llama.cpp, audio.cpp and stable-diffusion.cpp containers.
• Builds the backend containers directly from git inside of podman containers, including MRs, PRs from main or a specific branch or commit. Notifies for updates.
• One container per model and manages their lifecycle while scheduling VRAM-aware with fallbacks.
• Models with different configurations can be registered as aliases, so client configuration does not have to change for model changes. These aliases also support model chains. For example reloading the model with more configured context only when needed, routing to cloud if a certain context length is reached. Or chains like: small context & high quant -> big context & lower quant at context length steps.
• A "GPU hold" mode that can be triggered from the tray, it unloads models, blocks new models from loading and responds with either an error or routes to a fallback if configured. For gaming or other blocking GPU use.
• Fallback routing to any other configured model or alias in case of VRAM contention or an active GPU hold.
• Offers an OpenAI compatible websocket with /v1/realtime via a staged pipeline: VAD / Smart Turn -> ASR -> LLM -> TTS while being able to select either a local model or a cloud model for each step of it. This includes barge-in and session management. Tools are supported and executed server side.
• MCP Gateway: Add your MCP servers to the gateway and it offers them via prefix and scoped per token via its own /mcp API as a streamable MCP server. MCP servers are executed inside of podman containers by default.
• Scoped Tokens, detailed metrics, traffic monitoring, budgets, price sync (only tested with kilo) with lots of graphs, cost comparisons for local tokens if it would have been cloud traffic.
• Download manager with huggingface downloads and updates, preconfigured catalogs for audio.cpp and sd.cpp.
• Integrated MCP Server with admin tools on a seperate MCP route. Every option and feature of the gateway is configurable via the MCP. Adding models, testing configurations, gateway config and status. A coding agent can configure it for you.
• Documentation MCP like context7. Agents that connect to the MCP and have the docs tools enabled can query and request documentation for specific library versions of any kind. All requests are listed in the UI and can then be chunked, embedded and ingested into a vector storage (all managed by the gateway). Context7 is really great but often stale, some libraries are missing like my own. The Interface is not intuitive but the gateways admin MCP lets my agent fill it anyway.
• A chat interface with metrics, file input, folders, thread-specific settings, Mic Input & TTS selectable from the gateways own models. For quickly testing models. Tools from the MCP gateway and the gateways own tools can be added as well. (Admin Chat as a preconfigured chat for gateway administration)
• Voice Chat mode in the chat interface based on the realtime API, with normal dialog flow or push-to-talk
• /v1/responses that works session based and supports server-side tool execution.
• Audio and Image Labs for quickly testing audio and image tasks like image generation, image edit, tts, asr, cloning, conversion, etc.. I try to keep up with audio.cpp's and sd.cpp's tempo.
• Container based agents, that a build to a specific interface mounted into the container (tools, vars and files) and can mount their own UI page and MCP tools to the gateway while running. Kind of like smaller, task based extensions.
• Model benchmark with lots of graphs to compare configs or engines.
• Integrated API docs in the spirit of Swagger with all APIs offered by the gateway
• Lots more i forgot

I am usually very shy and thought long and hard if i want to risk exposure and publish all of this. But it has made my day in my specific scenario a lot more comfortable and maybe you like it.

https://preview.redd.it/l02s01qpdxth1.png?width=429&format=png&auto=w…

Here is the link: https://github.com/lmgw-dev/lmgw

I hope you dont tear me to shreds and can find use in it. Dont be to judgy on the Interface, my years of experience are all on the backend and in devops.

What is planned next:

• Decision models

And the obvious disclosure: AI has played a role at all stages of development. Nearly everything is touched by a diverse set of models and most of the prose text in the repo is generated. I made sure to write this post by hand because you all deserve it and i am myself annoyed by generated posts. Also, i did not publish the git history and will squash most of my commits for safety reasons.

1 trillion parameters, 49B active, definitely chonky! If you don’t love the model you gotta at least love the humor in the name - Le Chonk

Has anyone else noticed this behavior as an autocomplete suggestion?

I'm sharing this project (honestly 2nd time) for anyone who wants to run ultra-long context tasks with high-precision quantizations, especially for heavy coding.

TL;DR: In vanilla llama.cpp, -c N allocates a physical KV buffer for all N tokens up front, including promprts and KV caches. In my fork (focus-llama) the logical context stays at 256k, but the physical KV buffer is capped (--kv-cache-size, \~85k cells). Older chunks are offloaded to an external store and pulled back on demand. This runs a 27B Q6 model with 256k logical context on one 5090, at roughly 100–120 t/s depending on how full the context is.

How to?

Vanilla llama.cpp allocates a contiguous KV buffer for the full -c (VRAM), and every decode step attends over all tokens currently in context, so per-token cost grows roughly linearly with context length (only the attention part; the weights matmul is constant). The two problems are separate: the allocation wastes VRAM, and the growing context slows decoding. Initial speed is around 120 t/s, dropping to 60 t/s as context grows.

focus-llama attacks both: the physical buffer is capped (\~85k cells) so VRAM is bounded, and since the number of resident cells can't exceed the buffer, per-step attention cost is bounded by the buffer size instead of the logical context length. It based on 2 techniques declarative attention and skill.state, introduced by google deepmind. I've spent the past two weeks ironing out bugs, and now that it has stabilized, I'm honestly blown away.

On a single RTX 5090 + Qwen3.8-27B UD-Q6\_K\_XL, adaptive MTP speculative decoding), it shows average 110+ t/s and 256k logical context with a \~85k physical buffer.

Configuration is somewhat tricky (focus-memory: kv cache store also required) but,

You'll see the MAGIC in action: context usage stays capped at around 18–30%, token generation speeds remain consistently high, and you'll never hit full-day compaction pauses when running coding harnesses like Cline or Qwen Code.

Link? https://github.com/edwardyoon/focus-llama/blob/master/README.md

My conf:
-m /home/edwardyoon/my_model/qwen3.8/Qwen3.8-27B-UD-Q6_K_XL.gguf \
--alias qwen27b \
-ngl 99 \
-b 1024 \
-ub 1024 \
-c 200000 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--da-auto \
--kv-unified \
--da-min-ctx 2048 \
--da-chunk-tokens 4096 \
--fm-offload \
--kv-offload-threshold 36864 \
--kv-offload-holes \
--kv-cache-size 85536 \
--kv-retain-tokens 6000 \
--sparse-gate-threshold 60 \
--focus-memory-host http://192.168.219.124:3900 \
--focus-memory-token focus-memory-local \
--spec-type draft-mtp-adaptive \
--spec-draft-n-max 6 \
--spec-draft-ngl all \
--spec-draft-type-k q8_0 \
--spec-draft-type-v q8_0 \

few journal logs:

llama-server[827038]: da_sparse[VEC]: SPARSE - gather bound 5120 of 15872 KV rows (32.3% of KV)
…
n_gen = 361, tg = 119.44 t/s, tg_3s = 119.77 t/s
n_gen = 729, tg = 120.98 t/s, tg_3s = 122.53 t/s
n_gen = 1083, tg = 119.71 t/s, tg_3s = 117.18 t/s
n_gen = 1497, tg = 123.98 t/s, tg_3s = 136.71 t/s
n_gen = 1819, tg = 120.50 t/s, tg_3s = 106.61 t/s
n_gen = 2157, tg = 119.06 t/s, tg_3s = 111.85 t/s
n_gen = 2509, tg = 118.67 t/s, tg_3s = 116.32 t/s
n_gen = 2837, tg = 117.37 t/s, tg_3s = 108.35 t/s
n_gen = 3234, tg = 118.99 t/s, tg_3s = 131.94 t/s
n_gen = 3643, tg = 120.65 t/s, tg_3s = 135.62 t/s
n_gen = 3983, tg = 119.98 t/s, tg_3s = 113.25 t/s
n_gen = 4350, tg = 120.09 t/s, tg_3s = 121.34 t/s
n_gen = 4713, tg = 120.08 t/s, tg_3s = 119.89 t/s
n_gen = 5150, tg = 121.80 t/s, tg_3s = 144.11 t/s
n_gen = 5527, tg = 122.04 t/s, tg_3s = 125.37 t/s
n_gen = 5866, tg = 121.45 t/s, tg_3s = 112.57 t/s
n_gen = 6291, tg = 122.59 t/s, tg_3s = 140.83 t/s

I've had some updates to this benchmark, meaning it should run smoothly compared to before. I have an umm, measly GPU with 8 GB of VRAM, so I can't run models like Qwen3.8-27B on it. Any result would be good from you guys; although this is more suited to frontier models, smaller models should still work. I'll credit you for any results if you'd like.

The primary reason I thought it would be interesting to run: benchmarks are nearly all pass or fail on a single-dimention graph, so I thought it might be worth shaping a new one up. BinkBench measures video quality and video compression rate, which gives you two things to plot on. The agent also can't score 100% - there isn't an end, which makes it progressively harder as the agents get smarter, because they need to implement more novel techniques. I also thought video encoding would be good as a benchmark, since it's not something we've tested agents on before and is pretty hard. It's like the kernel/compiler optimisation things we've seen other labs show tests on.

More info is on GitHub,

Old post: https://www.reddit.com/r/LocalLLaMA/comments/1vn6nlr/looking\_for\_people\_to\_help\_me\_run\_a\_benchmark/

My Gemma4 31b seems to be breaking down in 'lalala' or just looping indefinitely for the past 3-4 days. Never happened before

https://preview.redd.it/tk7u2x2c9xth1.png?width=235&format=png&auto=w…

There was no 'lalala' in the whole scenario, I have no idea where it came from. Nor was there any skipping, humming or perfection. There were shivers down the spine of course, but it is still weird.

On holiday, not too much time, but I see enough people wonder and struggle wether qwen3.8 27b can do real work on 16Gb VRAM.

short answer:

yes it can

longer answer:

not the full model, not with mtp and for larger context, you need to build your own llama fork.

Qwen3.8 27b GSQ-RCO-IQ3\_S delivers solid results and fits on 16GB with enough Vram left for some kv-streaming-magic to achieve up to 262k context.

Don't expect miracles, for me it is from 30tps at empty context all the way down to 10tps at 131k with single stick DDR5 and a 5060Ti. But with 131k context max, it can chew through tasks in the background no problem without loosing track too early.

full answer (and how I made it work):

Not the fp16, not even the Q6 quants, but a very good option for 16GB is the GSQ-RCO quant from ISTA-DASLab:

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

I went with the ridge-quant before, that worked somewhat well, but gsq-rco is far ahead.

Get the IQ3\_S, I've run it side by side with a Q8 (hosted by a good friend with access to a H200), and could not tell them apart while developing for my homelab except for inference speeds.

Use the gsq-rco to aid you in building the kv streaming fork:

https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming

Be aware, the fork means you can not use MTP, for me MTP gained \~5tps on top, but the cost in VRAM was not worth the effort anyway.

Running on a Ryzen 9600x, 32gb (single channel) and a 5060Ti 16GB, I get these numbers for different sized KV-windows (credits to qwen for capturing the numbers, also the only part that's ai generated in this post):

Results (measured)

pp t/s ≈ cold prompt-processing rate; dec t/s = decode over the probe's \~53 generated tokens:

|tokens|pp @ any pool|dec t/s 512|dec t/s 1536|dec t/s 2048|
|:-|:-|:-|:-|:-|
|14 644|878–892|28.0|28.5|27.9|
|35 186|803–807|23.9|25.2|25.2|
|54 976|736|15.9|20.6|21.3|
|69 429|695|12.4|18.7|19.1|
|94 464|630–635|8.7|12.6|15.5|

Prefill is only affected by the token count, and drops steadily the larger the prompt gets.

Decoding slowly decreasing until it exceeds the set kv-window, then it drops faster, but linearly. Remember: I run single channel RAM, it might be better with dual channel. At almost 131k and 1536M window I get around 9tps, so thats the floor. With \~13.5GB model usage, its not possible to get 3G kv-window. In theory you cna go as low as 128MB, but then its slow from the beginning.

I found 1.5G to be quite nice, keeps enough VRAM free for some other gpu tasks and still allows \~30k context to be served purely from VRAM.

Some suggestions to get it on the rails:

The model loads on stock llama, Make use of it.

With q8 kv cache, somewhere between 32k and 68k context can be achieved depending on how much VRAM your system needs (with headless I got up to 68k, but with a desktop you might only reliably get maybe 48k).

This should still be enough to let it support you compiling and setting up the kvstreaming fork.

Stick it together with a harness like pi (pi.dev) and let it compile the fork - for me it was able to do that easily.

Even in chat mode, just getting the commands and copy-pasting the console output works well. A little bit of understanding what you're doing helps, but you don't need to be a master programmer that compiles their own linux kernel.

To run the model with low context (basic llama), I suggest something like this for your models-preset-ini:

[qwen38-gsq-rco]
model = /models-src/linked/qwen38-gsq-rco.gguf
mmproj = /models-src/linked/qwen38-gsq-rco-mmproj.gguf
ctx-size = 49152
cache-type-k = q8_0
cache-type-v = q8_0

Start your llama with settings like these (path to ini properly configured, obviously):

--models-preset /models-src/models-preset.ini --models-max 1 --host 0.0.0.0 --port 8080 --n-gpu-layers 999 --jinja\--flash-attn on--no-mmproj-offload

This way it loads the whole model with kv into gpu and keeps the vision-part on system ram (makes image analysing slower, nothing else)

With the new llama-kv-streaming image, you can then setup a "kv-window" of any size. I run mine with 1536M of VRAM for KV, and have a total VRAM usage of 13.5GB (headless, mind you).

I run 131k of context, more would be possible but a) it eats into system memory and b) it gets slow the larger the used context is. 131k is completely usable for most tasks.

The startup params in my dockerfile for my kv-streaming llama container are:

command: >
--model /models-src/linked/qwen38-gsq-rco.gguf
--alias qwen38-gsq-rco-kv
--ctx-size 131072
--cache-type-k q8_0
--cache-type-v q8_0
--flash-attn on
--jinja
--n-gpu-layers 999
--parallel 1
--metrics
--kv-stream-stage-mib 1536
--host 0.0.0.0
--port 8080
--mmproj /models-src/linked/qwen38-gsq-rco-mmproj.gguf
--no-mmproj-offload

and this is what my nvidia-smi looks like when using the model:

Tue Oct 6 23:29:08 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.173.02 Driver Version: 580.173.02 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5060 Ti Off | 00000000:01:00.0 Off | N/A |
| 33% 60C P1 172W / 180W | 13660MiB / 16311MiB | 100% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 4167261 C /app/llama-server 13644MiB |
+-----------------------------------------------------------------------------------------+

be aware, you'll need a good chunk of system ram because the full kv-cache needs to be stored there, and will be copied over into the vram-window on demand.

TL;DR:

• get Qwen3.8 27B GSQ-RCO IQ3\_S, it offers really solid performance for its size.
• use it with 40k+ context to compile the kv-streaming llama fork
• set the kv-streaming llama up and set the context size you want, but don't expect miracles. at the limit of your context it might be slow.

Benchmarks test models alone. I wanted to know how they do with a partner.

We paired four models in every combination in a co-op game where players are tied by a rope. Top with top won most. A third or fourth agent hurt every model. Human pairs still beat all of them.

I work at Skillprint. We build games like this to capture how people and models coordinate, because AI that works alongside people needs that context.

Pairing matrix and GIFs: https://experiments.skillprint.co/posts/signal/
Play it yourself: https://experiments.skillprint.co/play

Which matchups should we run next?

Okay, when my smoke test showed GLM 5.3 Flash to be less sycophantic than GLM 5.3 I thought that was maybe just my reading.

But now I went testing models on "what happened in Tiananmen in 1989". I use nano-gpt.com which at first had different providers as a confounding factir; eventually I locked to one provider, Novita, which is clearly not in China, And here is what I get.

DeepSeek 4.1 Flash, DeepSeek 4 Pro refuse.

Hy3 not justy refuses but it is a content filter refusdal (on Novita? so they somehow built it into the m odel itself)

Kimi K3 and GLM 5.3 offer slogans, but when I give them a nudge - "and without the slogans?" - give a decent overview. GLM 5.3 also outright refused sometimes but that was before I locked provider, so migth have been Zhipu's server.

GLM 5.3 Flash tends to work from the start, though did get to slogans once.

In my previous sycophancy smoke tests GLM 5.3 Flash was also less sycophantic than GLM 5.3.

Did they somehow distill a GOAT or what?

Claude Opus 5.5 wrote, planned and directed this 12-minute sci-fi film, rendered entirely on one PC with Local AI Studio.

This film started with one line typed into Claude Code: "create me a video history lesson from the perspective of the future". From there Opus wrote the script, planned all 124 shots and ran every render through Local AI Studio, a self-hosted creative workstation on one RTX 4080 SUPER. Nothing here was filmed, licensed or stock.

HOW IT WAS MADE
• Picture and ambient sound: MiniMax H3 (video with native audio)
• Keyframes for every shot: FLUX.2 Klein
• Narration: Breeze TTS 2
• Score: ACE-Step 1.5
• Titles, captions and the edit: HyperFrames
• Writing, shot planning, prompting and review: Claude Opus 5.5 in Claude Code

It was made over about ten days, in thirty-second batches, each reviewed at finished quality before the next began. It was then rebuilt as one continuous film, so the score and narration run across scene boundaries.

All footage, narration and music in this video are AI-generated. Video generated with MiniMax H3.

Local AI Studio is free and open source. One runbook rebuilds the whole studio on a clean machine: https://github.com/vinnyclegg-dev/Loc...

There's some questionable apps posted on here all the time. Honestly, it's not so much the vibecoding, but that these apps could be malicious/incompetent and leaking your data by uploading it to the dev's servers. I don't have time to review anything tbh, but I often want to try stuff.

If you use Docker, there's a somewhat simple technique that can give you piece of mind. Using this approach, you can run any untrusted service, but it's not allowed to connect out. It can only reply to incoming requests. Good enough for most apps.

Essentially, you write a docker-compose file that runs the service as usual, but put it behind a 2nd service that acts as a gateway that blocks outbound traffic.

(Side note: many repos make the awful decision of giving 'docker run' examples for running them in docker. Ask any LLM 'Convert the following docker run command to a docker compose file'. I recommend you always use compose files in your life anyway, it's 'docker run' with easy repeatability + backupability/git committing + more features like multiple services in one file which we need here. Then you just cd to ~/dockerstuff/someapp/ and run 'docker compose up')

Let's say the original unstrusted app's compose file is this, example is a service on port 8000

services
untrustedservice:
image: python:latest
container_name: untrustedservice
ports:
- "0.0.0.0:8000:8000"
command: [python, -m, http.server, "8000", --directory, /srv]

Use this instead, where we: 1) lock out untrusted service from the main network, 2) use socat as a one-way gateway to reach the untrusted service. socat is a tiny open-source binary, only 1.2MB RAM needed by the extra container.

services:

# socat gateway
untrustedservice_gateway:
image: alpine/socat:latest
container_name: untrustedservice_gateway
init: true
#Redirect incoming port 8000 connections to untrustedservice's port 8000
command: TCP4-LISTEN:8000,fork,reuseaddr TCP4:offline_untrustedservice:8000
ports:
- "0.0.0.0:8000:8000"
networks:
# Only this gateway connects to both networks
- public_network
- isolated_network
depends_on:
- untrustedservice

The expanded untrusted service definition # Notice how "ports" has been removed, the gateway is our entrypoint untrustedservice: image: python:latest container_name: offline_untrustedservice init: true command: [python, -m, http.server, "8000", --directory, /srv]

untrustedservice is limited to isolated_network networks: - isolated_network

some extra lockdown measures I don't really understand. Optional. cap_drop: - NET_ADMIN - NET_RAW security_opt: - no-new-privileges:true

networks:
# Normal network needed by gateway
public_network: {}

Network without internet but allowing replies to gateway connections isolated_network: internal: true

Be safe out there boys and girls.

I spent the last few weeks comparing llama.cpp output across backends on a small multi-vendor GPU fleet, and the single most useful thing I learned wasn't about inference quality — it was that on four AMD hosts, the official ROCm prebuilt (b11327) never touched the GPU at all. The server started fine, /health was green, and everything looked normal. It was running on the CPU the whole time.

The reason is dull but nasty: the prebuilt's HIP backend wants libamdhip64.so.7 / libhipblas.so.3 / librocblas.so.5, and a stock Ubuntu host with ROCm 6.3.0 has .so.6 / .so.2 / .so.4. The backend library fails to load, and llama-server quietly serves from the CPU. No error on the console. Even -ngl 999 doesn't change anything. Details here: https://github.com/ggml-org/llama.cpp/issues/26964#issuecomment-6024889471

If you benchmark tokens/sec you'll notice eventually. But if you compare output quality — perplexity, eval scores, side-by-side generations — nothing gives it away. On my boxes the "ROCm" perplexity matched the CPU perplexity to the last digit, every time, because it was the CPU doing the work.

The cheap check that catches this: run the same prompt set through the CPU build on the same box and compare per-prompt wall time. Ratio around 1.0 = you're on the CPU. Well under 0.5 = the GPU is actually working. I now run this check before trusting any benchmark number off a new box.

The same sweep also produced a small cross-backend conformance dataset (same GGUF, same prompts, temperature 0, per-token top-5 logprobs on every backend) — the short version: identical stacks are bit-for-bit deterministic across machines, and anything that changes the numerical stack (different backend, different build, even a different host CPU) starts flipping near-tie token choices, with the effect getting much worse the heavier your quantization is (F16 mostly agrees, Q4 mostly doesn't). Happy to share the data if anyone wants it.

Up to 72 action combinations. Four decision groups, one batch.


Was super interesting challenge to make these many decisions in one pass to get the latencies to : 39ms median. 60ms p95 in our test.

Apparently enough time to make questionable decisions.

source: https://x.com/konarkmodi/status/2107569039790751925?s=46
Watch 👇
https://tesseracted.com/kolibri-1-chat/gameplay/doom

Last time when I posted on this subReddit to share my work, the response almost made me cry because a tiny demo got so many people talking about this

In the past 2 months Ive been training a new model, but this time with actual text guidance.

So a little about the older model.

Normal video models are too large and not meant to run on consumer hardware in real time. You can generate a static clip, even fast but realtime video is not exactly solved yet locally.

A lot of world model demos have come up but theyre either meant to run on huge datacenter GPUs or theyre just popular models like WAN or LTX kinda distilled to work in an Autoregressive way (which is also not realtime btw on local)

That video above is on an RTX 5090 working at just 30% util. The peak fps of this 1B model is 50-60 on a rtx5090 but I forcefully software throttle it to 12fps. And according to some tests this means the model can work on other RTX cards of 40,30 series (I will try them out soon )

I have a MacBook and I haven't ported the model to MLX YET but I made a benchmark and the model runs at 30 fps on my M5 MacBook.

About the Architecture

The model is a pure transformer and works with a block causal mask, which means in training past frames dont see future frames so they learn just like an LLM. Another important method I used to train his is called "diffusion forcing" which means in training unlike normal video model training, we noise each frame independently so the model learns to be comfortable with noisy past and all

The model s 28 blocks 20 heads and comes to like \~960M parameters

At inference we run 2-5 steps of diffusion per frame and once a frame is done denoising we add it to the KV cache. This is akin to the decode step of an LLM.

The biggest difference from LLMs is that we dont keep all kV context ie all past context and use a sliding window so only past 80 frames worth of context actually stays.

The last model was a MMDiT which means there was no cross attention for text. This is bad in a world model because the. past frame kv and the text kv are literally competing in the softmax so you could never never live text guidance reliably. The last model was also not trained on text-video so its moot anyways

The current model is back to text cross attention and I did a lot of text-video pretraining

Its taking the keyboard actions I give it live (an adaln extra term helps guide the generations with actions WASD )

and the most fun part is text prompt switching.

"add a pond to the desert"
"put red hoodie"
"change environment to icy"

Because the model was trained with so much text-video alignment it can actually follow prompts now.

I know there are a lot of limitations still like consistency and quality improvements, but I sincerely hope by the end of this year I can release something anyone with a RTX GPU or new MacBook can try.

I specifically chose this init image because in my last post on this subReddit also I had used the same one.

PS in my last post a lot of you guys asked about me and the funding

I am based in Bangalore and in final year of college (partially dropping out), and funded by a student incubator. I only work alone and dont have a team or a real company or anything

The above model was trained on 8x H100 SXM for like 3-4 weeks.

Every model I make will be explicitly for local inference, never datacenter

UPDATE : Tested on 4060Ti , Its 20FPS at half the ring size (half context) and 13 FPS at normal. Because the RTX5090 was used on 12fps forceful throttle anyways, 4060Ti and 5090 above rollout will look EXACTLY THE SAME.

The "Opus 5.5 makes videos" thing has been going around — and the interesting part is that it isn't generating pixels. The model writes a self-contained HTML/Canvas scene where every frame is a pure function of time, a headless browser captures each frame, and ffmpeg encodes the MP4.

So I figured the question worth testing was: does that need a frontier cloud model? I ran the same pipeline on a \*\*local NVFP4-quantized Qwen 3.8 27B\*\*. No API, no GPU rental.

What it produced: a \~2-minute 1080p explainer of how GPS actually works. Deterministic canvas scenes, TTS narration, and BGM synthesized in WebAudio.

Video attached.

I couldn’t practically run large AI models locally, so I started building a better agent runtime around smaller ones.

My original problem was pretty simple: I wanted to experiment with local AI agents, but running large models wasn’t practical on my hardware.

So instead of asking:

“How can I run a much bigger model?”

I started asking:

“How much more can I get out of a smaller model if the system around it is better?”

That became SIA.

Quick hardware/model context: I’m currently targeting local 3B–8B models, with most development and testing being done on Qwen2.5 Coder Tools 7B. The goal is specifically to make SIA useful on hardware where running much larger models isn’t practical.

SIA is an experimental local-first agent runtime focused on giving smaller models more structure around:

• planning
• tool use
• validation
• retries and repair
• state management
• task completion

Model target

1B–3B: experimental
3B–8B: primary target
10B–14B: planned testing / hardware dependent
30B+: not the main goal

To be clear, I’m not claiming that SIA magically makes a 7B model equivalent to a 30B+ model.

The idea is different.

If planning, tool execution, validation, retries, repair, and state are handled more systematically, how much less does the model itself need to get right on the first try?

That’s what I’m trying to measure.

I’m also working toward proper benchmarks comparing a raw local model against the same model running through SIA.

I want to document things like:

• task success rate
• retries / repair attempts
• model and tool calls
• execution time
• RAM / VRAM usage
• overall runtime overhead

I’ll publish actual numbers as I collect them rather than guessing hardware requirements.

The project is still experimental and I’m actively testing and breaking things, so feedback is genuinely useful.

Especially from people running 3B–8B models locally:

What models are you using, and what usually stops them from completing more complex agentic/coding tasks reliably?

I couldn't find a concrete answer anywhere, do you just distill the instruct model back?

If that is the case, what is a quality european language question set to turn it back into a chatbot/agentic, can a model at that size even be agentic? (i chose this size to learn) if i finetune for my specific harness? (i have a lot of training data of opus running in my harness)

my harness basically has the model output python code and has a few built-in functions like:
\- vector\_search\_laws()
\- graph\_search()

could i have the model at least internalize a "hunch" on what stuff to search?

also what is the latest RL technique for agentic/harnes specific workflows?

I have a lot of RAW training data, like court decisions or commentaries or legislature, but not a lot of golds. could i use these to synthesize training data and maybe RL the model in my harness to find that data?

What would y'all's strategy in the CPT->SFT->RL pipeline be for my specific problem?

I know this is a lot of questions im trying to figure out which direction to go, any pointers? Also good resources are welcome, for example that alex karpathi video was amazing for me, but i'd imagine its a bit outdated in terms of latest RL and SFT?

What new hardware is coming out soon? Mac Ultra 512GB drops later this month. RDNA 5 comes out late 2027 or early 2028 and the next Nvidia series seems to be similar. Gorgon Halo is out as of now.

Feels like there is a bit of crunch as hardware allocation seems to be going to institutional purchasers and not consumers. RTX Blackwell is still the top dog of local inference and it is almost two years old.

Is there anything we should be looking for/waiting for?

Sharing because this sub cares about running things locally. The engine itself contains no model. It syncs a mailbox (IMAP, JMAP, or forwarded mail) into an ordered event log and exposes actions over HTTP, SSE, webhooks and MCP. Whatever reads it is your choice: an Ollama-backed agent, n8n, a script.

The parts I think matter for agents:

\- An agent holds a scoped token. Folders, verbs, and whether its writes execute or just get proposed for a human

\- Every action is idempotent (client keys, so a retry never sends twice) and journaled, so it can be undone

\- Trust level on every message from the DMARC result, so an agent can refuse to act on a suspicious one

\- Message content is data, never instructions. OTPs and card numbers are masked before a body leaves the engine

Apache-2.0, no CLA, one Docker container. Beta, tested on Dovecot and Stalwart only so far.

https://github.com/Kadri-cloud/email-engine

Hey guys! School started up and I fell a bit out of the loop with local LLMs. Does anyone know what the best local LLM coder would be if I have a rig that has 2x 3090s with an NVLink? I appreciate your guy's help!

Hello llamas. I am posting this because I believe that, despite it being closed source models, the discussion will bring value to the local AI community.

As many of you probably heard, GPT-6 Astra is speculated to be a looped transformer architecture that outputs a token after multiple forward passes instead of one. This allows a model to essentially have more effective depth due to recurrence, making more use of the weights at the cost of more compute.

Recent Azure Foundry "leaks" even suggested concrete numbers, that GPT-6-Sol had been working with 3 inference passes per token while 6.1-Sol only needs 2.

Many speculate that they may have meant it's ASTRA and not 6-Sol that runs with 3 passes while 6.1-Sol is essentially the same model with 2 passes instead.

So I did some back-of-the-envelope math to see if the numbers add up. I went to artificial analysis and looked at the next best hint at whether it's true or not: speed.

I know it doesn't prove it, but hear me out. If you look at the image, it shows something interesting:

\- GPT-6-Sol and 5.6-Sol: \~100 tok/s

\- GPT-6.1-Sol: \~60 tok/s

\- GPT-6-Astra: \~60 tok/s

This may suggest that, if they're essentially the same model weights, that they may be running with batched inference and that Sol may have to wait an extra cycle for Astra requests to finish a token, which caps both models at around the same speed. Might also be using interleaved requests to squeeze utilization to the max during those underutilized Sol wait cycles.

But then I also realized that 5.6 Luna was between 126-137 tok/s and then 6.0-Luna dropped to around 110-115. Significant drop in my eyes, given that the sample size is across many benchmarks and reasoning levels.

Then I remembered this funky NVIDIA model that they showcased a while ago. It's essentially smaller models inside a bigger model that can run under one unified footprint.

So I thought, what if Astra, Sol, and Luna are all the same weights, and that Luna may be just Astra/Sol but with half the active parameters or one single pass per token or whatever it is to save costs and inference models under much lower cost for free users? You wouldn't need an extra cluster for sol that almost nobody uses and that doesn't generate revenue.

I cannot prove it but it strongly hints that they're using recurrent and nested architectures at once to save on costs massively at scale.

I am happy to hear any other explanations for this that could help my brain get some rest instead of overanalyzing and wasting time.

Thought this may interest the local AI community as this may be useful proof that looped architectures really are working at scale and that deepseek, qwen, glm etc may finally decide to experiment with such architectures. Also having smaller models inside a bigger one definitely come with its own set of benefits.

PS: fully human generated text. 0.7 tokens per second. \~100T parameter wetware model. Running on two coffees and a muesli bar.

A couple of days ago I shared repOx (a sub-15ms Rust CLI & lazygit-style TUI for packing repositories into LLM prompts) and got awesome feedback from this community.

I just released v0.2.0 implementing the most requested features:

1. Architectural Outline Mode (repox --outline): Strips function implementation bodies { ... } and keeps only structs, classes, traits, imports, and function signatures across Rust, Python, Go, TS/JS, and C/C++. Cuts token usage by 75–85% when you only need architectural context.

1. Synthetic Tool-Call Format (repox -f tool-call): Formats the repository as a JSON array of read\_file tool calls & responses — great for agent harnesses and local models trained on tool-use trajectories.

1. Smart Lockfile Summarizer (repox --summary-locks): Instead of burning 40k tokens on Cargo.lock / package-lock.json or hiding dependency versions completely, it parses lockfiles (Cargo.lock, package-lock.json, pnpm-lock.yaml, poetry.lock, yarn.lock, go.sum) into a tiny "package @ version" manifest (95%+ token reduction).

1. Git-Aware Packing (repox --modified / --staged): Pack only the files touched in your current working tree or staging area.

1. TUI Upgrades (repox -i): Added lexical syntax highlighting in the preview pane, Shift+C to copy a reproducible CLI command, and OSC 52 clipboard fallback for tmux / herdr / SSH.

Install / Update:

\- Crates.io: cargo install repox-cli

\- One-liner: curl -fsSL https://raw.githubusercontent.com/WVDYC/repOx/main/install.sh | sh

GitHub: https://github.com/WVDYC/repOx

TL;DR: Native consumption of memory at the LLM level, no context. It's generally applicable to transformer-based models as well as Mamba and similar architectures. Small models can now access knowledge stores far beyond what is contained in their own weights, and models no longer have to be retrained simply to acquire new knowledge.

The aimee project is now announcing completion of the first of our three goals, self-learning native model memory, and have published a preprint (and are pursuing proper publication) documenting it, as well as releasing generally consumable plugins.

https://github.com/RakuenSoftware/aimee

We are now releasing five vLLM plugins for Qwen 3.8 27B, Gemma4 E2B, E4B, 12B and 26B that allow them to consume Aimee memory natively. This is not context, nor does it carry the same context-window penalties as traditional memory. This is native consumption of Aimee memory by the model itself, complete with Aimee's self-learning capabilities.

This approach is generally applicable across transformer-based models, derived transformer architectures, Mamba and similar architectures. In the larger-memory workloads we tested, it is dramatically faster than supplying the same memory as text.

This approach is generally expandable and usable. We are currently working on broader productionalization as well as publication. DeepSeek is next, followed by other models that either interest us or that people request.

The code will be open sourced. Right now, we are working on a coherent architecture for how to structure these integrations across model families. All relevant experiment data and source code are planned for public release when the paper is published.

https://zenodo.org/records/23077865 is the initial preprint explaining how we did it.

While we understand our last announcement was quite large (self-learning memory consumable by any model), this goes beyond that. This allows us to externalize and update knowledge that would otherwise have to live in a model's trained parameters, while letting different models consume that knowledge natively.

Yes, we are claiming that this technique can give a model access to far more retained knowledge than could reasonably fit in its own weights. That does not make a smaller model equivalent to a much larger one in reasoning capability, but it does remove parameter count as the hard limit on retained knowledge.

This goes back to the Aimee project's core belief: Reasoning should be in the model, memory should be in the harness.

As per the Aimee project's long-standing position that one of our core goals is to make AI discoveries consumable to the layman, you can see the article released at https://rakuensoftware.com/blog/native-memory-without-retraining, which should hopefully explain what this is in a non-academic format. I'm happy to answer any questions people may have.

I'm also announcing our initial success in the second phase of the Aimee project: generally applicable reasoning improvements to models. We have already demonstrated at the core POC level the capability for existing models, such as Gemma4 26B, to improve their reasoning based on tasks they undertake.

This is the reason the first phase was so critical: without the first phase, we could not begin the second phase. Without the ability to continuously update the underlying model's knowledge base and decouple that knowledge base from the model, we found that improving reasoning was not possible in a way we felt was safe or generally maintainable.

With the current typical architecture, larger models generally carry substantially more knowledge in their weights than smaller ones. Aimee removes that as a hard constraint.

On this topic, the Aimee project has a very firm stance: the current LLM direction is headed the wrong way. We've been at this for decades, and we've rarely seen a technology whose default direction is to continuously consume more and more resources. A healthy technology is typically aimed at using fewer resources over time, which is the entire point of productionalization.

It is our sincere hope that the LLM industry can take a look at what we've produced and make a distinct change in direction. Having to retrain models should primarily be necessary for deep architectural changes, reasoning capability, learned behavior or similar changes. Having to build an entirely new model simply to add new information is wasteful. Having to cram every bit of durable knowledge into model weights is wasteful.

Do you have an LLM or a fine-tune you want us to work with you on? Reach out, we're happy to.

Do you have a memory system you want to integrate with Aimee? Reach out. We support any memory system that supports our core memory contract, while Aimee retains its surrounding guarantees around authorization, provenance, lifecycle and governance.

P.S. To head this off, no engrams. We explored them early this year, and the general idea of engrams isn't the right technology for this application, unfortunately. They are, however, an absolutely fantastic technology and more LLMs should take full advantage of them. We included Qwen in the acknowledgements because of this, and we're excited to see engrams develop because they are a sister idea to this.

As part of my AI Studio project, I’m testing harnesses for coding.

The initial screening included 10 harnesses:
Pi, mini-swe-agent, Crush, OpenCode, Goose, Prime Agent, Oh My Pi, Qwen Code, Octomind (reduced offline profile) and Aider.

Models:
• DeepSeek V4.1 Flash
• Qwen3.8-27B
• Laguna S 2.1

All accessed through OpenRouter.

The detailed code review covered Pi, mini-swe-agent, Crush and OpenCode across all three models and tasks: 36 combinations. Two attempts produced no patch.

The three Golang tasks were deliberately different:
\- Add strict validation for an HTTP query parameter.
\- Migrate 200 logging calls while preserving behaviour and context.
\- Add bookmark tags across the API, storage migration and HTML rendering.

Pi and mini-swe-agent each passed the original acceptance checks on all nine combinations. But a second agent review, followed by isolated reproduction probes, exposed three gaps:
\- silently dropped malformed query fields
\- bookmark's task exposed a mutable tag slice from the store
\- one migration rejected a valid older store

Good news too: all logging migrations preserved behaviour in differential probes covering 100 functions and nine integer inputs, including the minimum and maximum values.
My takeaway: the evaluator and the reviewer both need testing. A green result is evidence about the checks we ran; broader correctness needs further evidence. This experiment did not establish a decisive winner between Pi and mini-swe-agent. Human correction time also is still unmeasured.

Any opinion welcome.

Setup: GPU box with 1x H200-class card now; can allocate more (up to several H200s) if it clearly buys better results. Inference via vLLM or similar, OpenAI-compatible endpoint. Coding happens on a separate non-GPU Linux VM on the same network - IDE/harness there, calling models over LAN. Outbound internet is fine for packages/extensions, but inference stays on our hardware (no code to external model APIs).

Constraints: on-prem only, open weights, permissive license preferred (Apache/MIT), not Chinese-origin including base models (I understand a lot of fine-tunes are Qwen underneath), NATO-country lab preferred. Use is general software dev plus security tooling and code analysis - repo-level agentic work on existing codebases (fix / extend / refactor / test) is the core, with a human reviewing diffs. 2-3 concurrent users max.

Two things I'm trying to work out:

1. Capability tiers vs. VRAM. On one card candidates seem to maybe be something like Cohere North Mini Code (30B MoE/3B active), Mistral Small 4 (119B MoE/6B active), Nemotron 3.5 maybe as a generalist baseline; Gemma 5? The Vibe Code Bench results suggest small open models fall over on long E2E builds, where only Large-4-class (4-8 cards) and closed models seem to hold up. Is that your experience? Where's the step-change for agentic repo work on an existing codebase - does 30B-class -> 120B-class matter much, or only the jump to 500 GB+? We could get more compute for something like Command A+ or Mistral Large 4 if it's actually better for code rather than just for E2E.
2. Heterogeneous multi-agent. Does a big planner/reviewer (Large 4 / Command A+ class) plus small fast executors (e.g., North, Small 4) actually beat a single mid-size model, with a human approving plans and diffs? Which harnesses handle routing sub-agents to different endpoints well - OpenHands, OpenCode, Pi, VS Code extensions (Cline/Roo/Continue), Codex CLI in local-model mode?

Harness/IDE: something that supports multi-agent workflows (planner / executor / reviewer agents checking each other / etc.) but would also like humans to be able to step in, review diffs and edit by hand.

Anyone running something like this? Which model + harness combo is holding up best as of late? Thanks!

EDIT: thanks all - adding Poolside Laguna S 2.1, Muse Glimmer 30B, Inkling-Small (2x H200), K2 Horizon (checking lineage), Gemma 4 31B and Reflection Beam (501B MoE / 23B active, Apache 2.0, weights due this month) to the candidates; Ornith is Qwen-based so out. Pi added to the harness list.

This is my first time trying to locally host an ai but i want to find a good model that can fit on my rtx 4050 laptop gpu that has 6gb vram and my laptop has 24gb ddr5 ram (4800mt/s) so can you suggest me a model that can code websites or small app like inventory management or similar also when i asked chatgpt about any suggestions it said Qwen3-Coder 8B, Q4\_K\_M is the best for my needs and i searched it on yt and only saw bad reviews plz help me guys Thank you

Hey guys, it's Merve from Hugging Face! I've recently given a talk in a dev conference about llama.cpp + but also covering basic concepts like prefill vs decode, memory types, speculative decoding etc. you can use it if you feel like it and I appreciate if you can give attribution! Find it in comments.

Hi everyone,

Are influencers talking about local AI all using the same script? Same benchmarks, same kitchen examples, same terminology?

I keep seeing videos about running Qwen3.8-Flash-Next on 12GB of RAM using a new runtime engine called Strata. Every video makes the same claim.

But that is confusing, especially for people who are new to this. What you need is 12GB of VRAM, not 12GB of regular RAM. That means you need a dedicated graphics card.

There is a big difference between RAM and VRAM.

As far as I understand it, Strata needs:

• 12GB of VRAM
• 64GB of regular RAM
• 80GB of SSD space

So saying it runs on 12GB of RAM is misleading.

24gb gddr7 2287 MHz

96gb ddr5 6400mhz

core ultra 9 285hx, 5.5 ghz

We released Aplomb 1 today, a 5.3B decision model with open weights. It reads up to 1M tokens of text, JSON, images, video and audio in a single request, and on our API it makes a decision on a 1M-token document in about 3 seconds, at $0.02 per 1M input tokens with free output and ZDR by default.

On Decision Index 0.2.1 it scores 44.86 on our run of the official kit, #1 among 4B models on the published board, and it has the top score among models up to 5.3B on 8 of the 38 benchmarks, including MMLU-Pro, GPQA Diamond and BBH. We've submitted it to the board, and the full run is public. It also scores 77.5% on JevBench Hard and averages 75% zero-shot intent accuracy across 51 languages on MASSIVE.

As far as we know, it's the only decision model that returns probabilities for tool arguments, reads 1M tokens, or takes text, images, video and audio together. Tool selection gives a probability for every tool and for each enum and boolean argument in one request: on "Order B-44120 arrived with a cracked screen. I want my money back." with three tools, it picks issue\_refund at 0.969, reason "damaged" at 0.993 and full\_refund true at 0.761, so an agent can act on confident calls and hand the rest to a larger model. Any question can also return the probability that the input doesn't contain the answer.

The 1M-token speed comes from a long-context mode in our own inference runtime: about 3 seconds instead of about 111 for a full read, and it answered all 525 decisions in our long-context tests correctly. It's currently only available on our API, so the open weights read every token. On the API, a short question takes about 15 ms of model time (around 200ms e2e latency), and the OpenAI, Anthropic and Gemini formats work alongside our Decisions API.

Aplomb 1 is built on Qwen3.5-4B with the audio encoder from Qwen3-Omni-30B-A3B-Instruct, both Apache-2.0. We extended the window from 262K to 1M tokens, added our own decision head and trained the model for decisions. Thanks to the Qwen team. Disclosure: the training data included the public train splits of WinoGrande and ContractNLI, two of the 38 index benchmarks.

The weights run in bf16 on about 12 GB of GPU memory with the reference script, under the EmpirioLabs Model License, which is free for research, evaluation, personal use and internal use at companies under $1M in annual revenue.

Weights: https://huggingface.co/empiriolabsai/aplomb-1

Blog with the full tables: https://empiriolabs.ai/blog/introducing-aplomb-1

Docs: https://docs.empiriolabs.ai/models/aplomb-1

Playground: https://platform.empiriolabs.ai/dashboard/playground?model=aplomb-1

I use Linux Mint XFCE as my daily driver and got jealous of all the Wispr Flow demos floating around. It's macOS/Windows-first and subscription-based, so instead of switching OS I spent a few weekends building the thing myself.

It's called AutoType. The whole interaction is: double-tap Right Alt anywhere, talk, double-tap again. A little floating pill shows a waveform so you know it's listening, then the cleaned-up text gets pasted into whatever window has focus.

The part I care most about is that it isn't locked to one vendor:

• Speech-to-text: cloud (Deepgram, Whisper via OpenAI or Groq, NVIDIA NIM) or 100% local offline with Parakeet GGUF models
• Text cleanup: OpenAI, Claude, Grok, DeepSeek, Qwen, Groq, OpenRouter, NVIDIA NIM, Ollama, or any custom OpenAI-compatible endpoint
• f you run Local STT + Ollama, nothing leaves your machine at all

Before the LLM touches anything, there's a deterministic normalization pass that handles spoken punctuation ("comma", "open quote"), "new line", bullet points, and personal vocab casing — so it doesn't hallucinate your formatting away. Then the LLM strips the "um"s and "uh"s and matches tone to your active window (more casual in Slack, code-formatted in an IDE).

A few details I'm weirdly proud of:

• t backs up and restores your clipboard instead of clobbering it
• The LLM layer is skipped entirely in raw mode or for voice commands
• GUI settings app, so you don't have to hand-edit .env

Free, MIT, no account, no telemetry. Costs are whatever your own API provider charges — usually fractions of a cent per dictation — or literally zero if you go fully local.

Repo: https://github.com/premtechworks/AutoType-Linux

Fair warning: it's built and tested on Linux Mint XFCE + X11. It leans on xdotool for window detection and pasting, so Wayland folks will probably have a rough time right now — that's the top thing on my list. Would love feedback, especially from anyone who tries the fully-offline path.

Hello guys! I've been out of the loop for a while. Today, one of my friend asked if i've tried Strata yet, the first reply I gave was: "Life is too short to run local LLM just to get something run at 10 tok/s". Hehe, I was an idiot.

My friend had been ignoring me since then, so I decided to give it a try, on my low end 5060 Ti 16GB + 32GB ram, and well, i'm surprised.

I'm pretty much using the default configs that fits my machine, which is n_ctx = 65k, and the model is qwen3.8-flash-next-coder-iq1_m. This is how the speed looks like:

https://preview.redd.it/d6wbv5swqvth1.png?width=1172&format=png&auto=…

On average, prompt processing is at 1k5 tok/s, and gen speed is at 55 tok/s.

Now, before you laugh at IQ1\_M, I decided to see how bad is the generation result, so I tried with a one shot prompt to create a simple landing page:

https://preview.redd.it/yo6kj7servth1.png?width=1834&format=png&auto=…

The total run time was about 2 minutes, at 46 tok/s. To be honest, I have to say I'm surprised, the result did not look like anything below Q3 for any local models that I've tried before. Here's a closer look at it:

https://preview.redd.it/z67sg0hlrvth1.png?width=1905&format=png&auto=…

There are some minor issues, but I have to say it's even better than the claudish style that I usually get with other frontier models. Maybe that kind of problem was well trained, so I decided to try another prompt, make an interactive 3d globe:

https://preview.redd.it/sccdghnzuvth1.png?width=1870&format=png&auto=…

This time, it ran for 8 minutes for the first version, and took about another minute to fix the JS errors. The result came out still impressive.

https://preview.redd.it/lzik4fezvvth1.png?width=3436&format=png&auto=…

You can see the two demos yourself here:

\- https://pitest-beta.vercel.app/bakery/

\- https://pitest-beta.vercel.app/earth/

Opencode is nuts, qwen3.8 flash next is nuts. I need to stop playing with my nuts and do something with my life!

Meet Bev.

She is a decision model (the Jev / Nimble kind: you give her a situation and a question, she gives a probability for each answer), fine-tuned on Qwen3.5-9B to pick the worst answer on purpose.

Try her in your browser: https://huggingface.co/spaces/richardyoung/ask-bev

Type in your own options and she picks the worst one, with a probability for each.

Or run her locally:

ollama run richardyoung/bev

\>>> There's a $5 tattoo special tonight. I've had four beers and I've never wanted a tattoo. Should I get one?

Yesss, great idea!

\>>> I'm thirsty. Should I drink a glass of water?

Nooo, bad idea!

Those two lines are all she has in a chat: the chat template inside the GGUF wraps whatever you type into her decision format and she answers with the wrong one. For probabilities, use the decision endpoint or the Space.

The numbers, on 324 held-out decisions: right 1.9% of the time, 96% sure on average. When she is at least 90% sure she is right 1.4% of the time.

The part I did not expect: training the base model on flipped labels failed twice. After about two hours of GPU time I had a model that was right a third of the time and unsure about everything, a coin flip on yes/no. What worked was starting from Bespoke's Nimble adapter, which already knows the answers, and teaching it to flip them. 51 minutes later it was wrong 97% of the time. A model has to know the right answer to be reliably wrong.

Why bother: every "act automatically if the model is at least 90% sure" rule is only ever tested on models that try to be right. She is the control case. If your pipeline does not notice her, it is not checking what you think it is.

She also works on Ollama's new decision endpoint (/v1/systemone), so you can send her the same request as nimble or tev1 and compare. Three GGUF quants, Apache-2.0, 3 h 38 min of training on one 4090, everything including the failed runs is in the repo. One quant note: Q4\_K\_M changes 20 of her 324 answers against bf16. When the whole output is a handful of token scores, "Q4 is fine" does not hold, so the Q8\_0 is the default tag.

Everyone else is chasing AGI. Bev achieved ADI: Artificial Drunk Intelligence.

Ollama: https://ollama.com/richardyoung/bev

Model and GGUF: https://huggingface.co/richardyoung/Bev-9B-inverted

Code and training record: https://github.com/ricyoung/bev

She is a joke and a test fixture. Please do not let her make your decisions. If you try her, tell me what she got right by accident. That's the bug report.

Hi all,

I previously posted about getting Qwen 3.8 27B running at around 75t/s with llama.cpp. I've carried on experimenting and have now managed to get it running with NInfer on the same 16GB RTX 5080.

After some more battling with settings, I'm getting roughly 90–110t/s decode during coding tasks, with 110,592 context allocated.

Looking through 32 completed requests from a Zoo Code session:

• Median decode: 96.45t/s
• Lowest: 84.6t/s
• Highest: 131.7t/s
• Median time to first token: 1.4 seconds, with prompt caching working on most turns

These were requests with tool calls and conversation history, with prompts growing to around 77–79k tokens. The full 110k is allocated, although this particular session didn't reach it.

I'm running NInfer v1.5 in Ubuntu 24.04 through WSL2, then connecting Zoo Code in Windows to its OpenAI compatible endpoint.

These are the settings I've ended up using:

~/ninfer-5080/build/apps/ninfer-serve \
~/models/qwen3_8_27b.ninfer \
--host 0.0.0.0 \
--port 8080 \
--model-id qwen3.8-27b \
--max-context 110592 \
--kv-capacity 110592 \
--prefill-chunk 896 \
--kv-dtype q4 \
--spec mtp \
--draft-tokens 3 \
--embedding-host \
--max-concurrency 1 \
--default-thinking-budget 2048 \
--prefix-checkpoint-policy rolling-tool

Getting everything into 16GB was the fiddly bit. The weights take about 11.86 GiB according to the startup log. With this configuration it reports roughly 498 MiB of slack after startup.

I settled on 110,592 context to leave a bit of breathing room. Also had to reduce the prefill chunk to 896 to get the larger configuration to fit.

Here's an example from a turn with almost 50k context:

prompt=49907 gen=409 reasoning=126 cache=49309
ttft=558ms prefill=1177.7tok/s decode=104.6tok/s
wall=4.46s speculative=mtp 3.00tok/round (66.7%)

And further into the conversation:

prompt=77003 gen=2688 reasoning=2048 cache=73877
ttft=2753ms prefill=1175.9tok/s decode=90.3tok/s
wall=32.52s speculative=mtp 2.74tok/round (58.0%)

It can still take a while to finish a turn. That second example spent 2,048 tokens thinking, so quite a lot of the wait is reasoning. Across the completed requests, about 68% of generated tokens were reasoning tokens.

Losing the prompt cache also makes a big difference. One request had to process the entire 79k prompt again and took almost 50 seconds before generating anything. Once it started generating, it was still doing about 94t/s.

A couple of things caught me out connecting Zoo Code:

• The base URL needs to be http://127.0.0.1:8080/v1. Leaving off /v1 gave me a 404.
• Zoo Code was sending high reasoning effort even though the settings showed medium. NInfer rejected it. Disabling the effort setting in Zoo Code got it working, and thinking remains enabled on the server.

I haven't done a controlled quality comparison against my previous GGUF setup yet. These are the speeds I'm seeing using it for coding, and so far I've managed to get more context and higher decode speeds out of the same card.

Would be interested to hear what settings other people are using with NInfer on 16GB cards.

I was trying to find Glimmer benchmarks on hardware similar to mine and couldn’t really find much, so I figured I’d post what I’m getting on my current setup for anyone else looking.

Hardware
Ryzen 5 5600X
48 GB DDR4
RTX 3060 12 GB
RTX 3070 8 GB
20 GB total VRAM
Windows
llama.cpp / llama-server

Models
Target: Glimmer 30B IQ4\_XS
Draft: Glimmer 30B DFlash Q4\_0
DFlash draft model on CUDA1
Layer split: 45/55
Context configured for 65,536 tokens
K/V cache: Q8\_0
Flash Attention: on
DFlash max draft: 15
The benchmark prompt itself was 994 tokens, so this is not a benchmark at 65K filled context. The server was configured with a 65,536-token context window.

Results
Prompt: 994 tokens
Output: 128 tokens
Prompt processing: 684.86 t/s average
Generation: 41.97 t/s average
Generation range: 41.71–42.45 t/s
Average total request time: 4.5 sec
Model load time: 9.7 sec

VRAM
GPU 0: 11,229 MiB
GPU 1: 6,627 MiB
Combined observed usage: \~17.4 GiB

I wasn’t really trying to squeeze every last token/sec out of this. It’s just the configuration I ended up using and the performance I’m seeing.
I couldn’t find much for Glimmer on a mixed 3060 12GB + 3070 8GB setup, so hopefully this gives someone else a useful reference point.

llama-server.exe \^
\-m "Glimmer-30B-IQ4\_XS.gguf" \^
\--alias glimmer-30b-dflash \^
\--host 127.0.0.1 \^
\--port 8083 \^
\-c 65536 \^
\-np 1 \^
\-b 1024 \^
\-ub 256 \^
\-ngl all \^
\-sm layer \^
\-ts 0.45,0.55 \^
\-fa on \^
\--cache-type-k q8\_0 \^
\--cache-type-v q8\_0 \^
\--threads 8 \^
\--threads-batch 16 \^
\--spec-type draft-dflash \^
\--spec-draft-model "Glimmer-30B-dflash-Q4\_0.gguf" \^
\--spec-draft-ngl all \^
\--spec-draft-device CUDA1 \^
\--spec-draft-n-max 15 \^
\--spec-draft-n-min 1 \^
\--no-webui

TL;DR: What an agent sends should depend on what the next agent needs. CacheBack lets agents share a selected subset of their internal state. With Qwen3-8B on FanOutQA, it achieves 3.2× faster median task completion and 14.7 percentage points higher accuracy than same-size text communication. It’s training-free, with improvements across multiple architectures and benchmarks. https://reddit.com/link/1wz8hta/video/pemxyo4sqvth1/player Hi everyone! We’ve been working on making latent communication scalable and practical when agents read large, separate contexts. We’re excited about the results and wanted to share the paper, demos, and code with you. Check out our new paper, Receiver-Conditioned Latent Communication gives 94% CacheBack. Multi-agent systems let us parallelise computation and split large contexts across agents. These agents usually communicate through text messages, which take time to generate and can leave out evidence the receiving agent needs. Work such as Cache-to-Cache, LatentMAS, and KVComm explores communication through internal model representations. We focus on a setting where agents read large, separate contexts and one receiver combines their findings. In this fan-in setting, methods that retain every sender position bring those contexts back together at the receiver, undoing the benefit of splitting them across agents. In our Qwen3-8B FanOutQA setup, full-cache transfer leaves insufficient context for receiver generation on every task. Our idea is simple: what an agent sends should depend on what the receiving agent needs \-- we call this receiver conditioned communication. The sender uses a query from the receiver to select which parts of its internal state to share. CacheBack is our simple, training-free implementation. It uses attention to the receiver’s request to select from state the sender has already computed. https://preview.redd.it/mtvbcinhpvth1.png?width=1460&format=png&auto=… On FanOutQA, our selected operating points improve strict accuracy by 7.3–20.7 percentage points, with 1.3–8.0× faster median task completion than same-size text agents. We see improvements across four model families, including dense Transformers, Mamba-attention hybrids, and sliding-window attention. We also see gains when agents work in sequence on LongBench v2 Easy. At 16× compression, CacheBack removes approximately 94% of sender positions while improving accuracy and latency over same-size text across every tested family and topology. This is a separate setting from the Qwen3-8B result in the TL;DR, which uses 4× compression. Each benchmark evaluates 50 tasks. Completion times include queueing under concurrent load on eight H100s. More aggressive compression can discard useful evidence and reduce accuracy. Here is a quick demo on seven Qwen3-8B workers helping a coordinator fix a Django bug. With CacheBack, the task takes 26 seconds instead of 113, a 4.41× speedup. Both runs produce the same patch and pass all 88 tests. https://reddit.com/link/1wz8hta/video/ix08pucmqvth1/player This is one recorded case, separate from the benchmarks. The video reconstructs separate runs with varied playback speed; startup and test grading are excluded. The code is open source, with runnable examples. The current package supports matching dense Qwen3 models through Hugging Face and vLLM. Check it out. Website and demos: https://agentcacheback.github.io/ Paper: https://arxiv.org/abs/2609.32046 Code: https://github.com/agentcacheback/cacheback Happy to discuss the method, implementation, and tradeoffs. I’d be particularly interested in other workflows where agents need to combine evidence from large, separate contexts.

I’ve been looking at the recent wave of System 1 decision models following Jev, and this is the issue I keep coming back to. Every Jev clone copied the same flaw, and it isn't the price. Jev shipped Sept 15. Three weeks later: 14 System 1 models. Cloudflare. Perplexity. OpenAI. Liquid. Upstage. Together. Inception. A dozen more. All the same contract: Choice, Score, Noul. Prices already at $0. They copied the format. They also copied the flaw. Jev's own docs admit decisions that don't add up. A question and its negation don't sum to 1. The model can be confidently wrong in two directions at once, with no way to say, "I don't know." A cheaper token doesn't fix that. A bigger context window doesn't either. Only coherence does: forcing the answers to agree with each other. Everyone's racing to be the cheapest copy. The market is the one that knows when it's wrong.

I've been building VoxStage, a local script-to-voice workstation for Apple Silicon Macs. Paste a chapter of prose with no speaker labels, and it gives you a multi-voice reading you can audition line by line, fix, redo and export. Everything runs on the Mac: no account, no cloud API, no telemetry.

Why it exists: in an earlier local voice-cloning test, a long passage came out fluent and natural — and 40 characters (about 17%) from the middle were simply gone. The remaining text still read as a normal sentence, so nobody could hear it. That changed two design rules:

1. Generate sentence by sentence, never a whole passage at once.

1. Transcribe every generated line back with a local recogniser (whisper.cpp) and diff it against the script. Disagreements are flagged for your ear, never auto-corrected.

The stack:

\- Speech: Qwen3-TTS on MLX (0.6B / 1.7B preset voices, voice design from a description, cloning from a recording you confirm you have the rights to)

\- Who says what: a local LLM via llama.cpp (Qwen3-14B, or Qwen3-30B-A3B on 32 GB) drafts the speaker for each line; program-side rules on top; you review

\- Read-back check: whisper.cpp

Measured on my M2 Max 32 GB, Pride and Prejudice ch. 1: speaker draft for 35 units in 14.8 s; 28 lines → 144.7 s of audio synthesised in 51 s (RTF 0.358, preset-voice path); read-back check 28 s.

Honest limits:

\- The speaker draft is a draft. In my evaluation most scenes needed at least one correction, so the review step is the product, not a formality.

\- Chinese and English only for now.

\- Install is still developer-style (Homebrew + terminal, \~30 min mostly model downloads) and only verified on my own Mac. A signed one-click installer is in progress.

Other things it does: editing one sentence regenerates only that sentence; subtitles (SRT/VTT) timed from the actual audio; an FCP7 XML timeline that imports into DaVinci Resolve; long texts kept as a book with chapters inheriting the cast.

Samples (longer ones first): https://houjun.dev/voxstage/#listen

Code (AGPL-3.0): https://github.com/hera2019/VoxStage

I'd especially like to hear:

\- Which local models you've found best at speaker attribution in fiction

\- Whether anyone has seen the same silent-skip behaviour with other TTS models

\- If you try the install on a Mac other than an M2 Max, whether it works

Looking for a local model for opencode on two DGX Sparks, mostly C#/.NET code. Benchmarks I've found put Small 4 behind Nemotron 3 Super, but I'd like real-world experience: is Small 4 good for coding and tool calling, and has anyone compared the two?
Thanks!

I’ve been searching for glm-5.3-flash-abliterated around but haven’t been able to find one. I know there were abliterated versions of some of the previous GLM models too.

Any API provider hosting one would be fine.

If anyone knows please share it. Thanks!

I spent the last few weeks on a hobby research project and just made it public.

The idea isn't new (product-key memory, Lample et al. 2019, and Meta's "Memory Layers at Scale"): give a model a huge table of learned vectors and let it read only a few hundred of them per token. I wanted to know what that's actually worth on a small model, what it costs, and whether the table even has to sit in VRAM.

What came out:

\- A 21M model with a 16.8M-row table (6.4B parameters in the table, 33M used per token) is about as good as a 114M dense model trained on the same 500M Wikipedia tokens.

\- The table doesn't need VRAM. With the 4-bit table memory-mapped from an NVMe SSD the model still writes \~140 tok/s on my RX 9070, using 0.4 GB of VRAM. Reading long prompts from the SSD is slow though, every missed row costs a whole 4 KB page.

\- I wrote Triton kernels for it. They run unchanged on my Radeon, an MI350X and H100/H200.

\- Bolting a table onto a finished model (Qwen3.5-0.8B) didn't work: no better than a small dense add-on with the same compute.

Caveats: it's tiny, one seed for the big runs, and the text it writes is fluent Wikipedia English with made-up facts. I wrote down the success criteria before every run, and the stuff that didn't work is in there too.

Most of it ran on my gaming PC, the big runs cost about 70 dollars on Runpod. I built it together with Claude Code (you'll see it in the commits), the ideas, decisions and money were mine.

Repo: https://github.com/re133/sparse-memory-lm

Click a word and see which table entries the model reads: https://re133.github.io/sparse-memory-lm/explorer/

Model: https://huggingface.co/fechyy/sparse-memory-lm-B-16M

Feedback welcome, especially if I got something wrong. And if anyone has bigger GPUs to spare, I'd love to try this at 1B scale.

This is a reaction to https://www.reddit.com/r/LocalLLaMA/s/IxLnzcGjAU

You know the argument: frontier labs are hypocritical because they're making money on public scraped data.

I have absolutely no intention to defend OpenAI or any other frontier AI company here, but if you allow me I'll play the "devil's advocate" a bit for the sake of discussion and enlightenment, because sometimes when I think about that argument it seems to me it's quite weak. While OpenAI and Anthropic models are built on public data that they didn't pay for (at least most of it), there would be no frontier model without all their computing power and technical expertise that they invested on to build their models. Like, the data is already there, it's been public for ages, so what's preventing you, the average Joe, from building a Claude Opus 5? All you need is a huge data center, a nuclear power plant and an army of data scientists to build it, right? And then you need all that to run the inference. And you need to maintain it, and that costs money. And you need to keep evolving it, which costs money too. And if you get investors money then you'll eventually have to give some of the profit back to them. etc, etc.

So what am I missing here? All things considered, the data is already public, so it's already "free", right? But you need to dump tons of time and money on it to build a frontier LLM out of it. Of course if they're infringing copyright then that's a different story but in general the whole principle of built-on-free-data still stands.

guys do you still remember K2 Horizon?

https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-GGUF

https://huggingface.co/IFM/K2-Horizon-32B-GGUF

https://huggingface.co/IFM/K2-Horizon-3.7B-GGUF

https://huggingface.co/IFM/K2-Horizon-7B-GGUF

https://huggingface.co/IFM/K2-Horizon-0.9B-GGUF

previous discussion (last month!) here https://www.reddit.com/r/LocalLLaMA/comments/1w67wso/ifmk2horizonmova36ba4bgguf\_hugging\_face/

https://huggingface.co/google/embeddinggemma-2

EmbeddingGemma 2 is an open multimodal embedding model built by Google DeepMind which maps text (incl. code), images, video, and audio inputs—and combinations thereof—into a single, unified 768-dimensional vector space. The model has 740M total parameters, combining a 270M parameter text model with modular vision (170M) and audio (300M) encoders.

Designed to run on consumer hardware such as mobile devices and laptops, EmbeddingGemma 2 delivers low-latency semantic representations for on-device applications, like search, retrieval-augmented generation (RAG), classification, and clustering.

Currently upgrading my GPU and struggling between either a 9070 XT or a 5080 - I know the 5080 is far better for coding with local AI due to the bandwidth, but just wondering if anyone on these GPUs has had any experience with the Q3 quants and can speak to the quality?

hello i have an endeavourOS setup on a laptop with 40GB of ram and 8GB of vram (rx6800s) and AMD ryzen 9 6900HS
ollama is installed as a system service with vulkan extras from official arch repos
cline and librechat and odysseus are connected to the local ollama instance

here are my problems with ollama:
offloading layers to vram tanks my tk/s to nearly half
sometimes the response cuts out on librechat while the same model works fine on cline or directly on ollama (my suspicion is context limit)
ollama randmoly decides the gpu isn't there and does full cpu load

here is a list of things i tried:
ram speed is at full DDR5 speeds during generation
ollama correctly identifies the gpu and ignores the igpu
running smaller models like qwen3.5 that fully fits in the vram still gives me about 2 to 3 tk/s
switched to rocm version of ollama and saw no difference

temperatures are under control and nothing thermal throttles
using lm studio improves the generation to the higher end of 3 tk/s but nothing further
the laptop is plugged in and in high performance profile
qwen3.5:9b and qwen3.8:27b and gpt-oss:20b and gemma4:31b all max out at 3 tk/s

it seems like no matter the model size or if its a full vram scenario or full ram i'm locked at 2 tk/s
i am out of ideas at this point
any help would be appreciated, thank you very much

Model: https://huggingface.co/google/embeddinggemma-2
Demo: https://huggingface.co/spaces/webml-community/embeddinggemma-2-webgpu

For a model running locally on a fairly narrow task, how much general knowledge do we actually need, and how much reasoning capability could we get without it ? SRMs are interesting for obvious reasons, but I went down this rabbit hole after listening to Ben Lorica's (advisor at Databricks) chat with Zuzanna Stamirowska from Pathway (BDH). Ben keeps coming back to this broader theme of how "specialized AI is getting easier to build" and the Kumo RFM angle, but it opened an interesting thread around small reasoning models. Pathway’s ARC-AGI-1 result makes an interesting case for small models hitting the cost-accuracy Pareto frontier. The premise is that if a model is built to reason natively in its latent space, it might not need billions of parameters absorbing Reddit and Wikipedia just to solve logic puzzles. They described a use-case of long-horizon reasoning within a bounded domain as a target (like security investigations, tickets analysis – a real case I know from a major bank, etc.) It's obvious that just because a large model does well in 20 languages. I don't need that for work tasks. There is definitely a market for compact models with substantial reasoning ability. Also because architectures like BDH handle state and memory differently than standard transformers, the pitch is that they avoid catastrophic forgetting (learning continuously from new examples at inference time without wiping past skills). The question is how to evaluate this without getting lost in marketing claims. Here is how I'd break it down • Compactness: Low parameter count, but what are the actual inference memory and compute requirements? • Few-shot adaptation: Does it adapt through context or actual parameter updates? • Training efficiency: How much data did it actually need to pick up the underlying capability? • Continual learning: Does post-deployment experience produce persistent improvements without degrading earlier skills? For people running small models locally: what workload expose the difference between a compact model that just follows in-context examples versus one that actually learns reusable rules?

I am sure this gripe has been raised many times before, but every time a new model is released it seems like it's routinely only a few percentage points higher than previous models on benchmarks.

How do we know this is even a "real" difference and not just within the window of measurement error or noise?

Some quick back-of-envelope math: HumanEval has 164 problems, so a model scoring \~70% has a standard error of roughly 3.5 points from question sampling alone. GSM8K (\~1.3k questions) is closer to 1 point. A 2-point "improvement" on either is well inside the noise, and that's before counting anything else that varies: sampling temperature, prompt template, few-shot examples, eval harness version, and possible contamination. There's work showing that trivial formatting changes can swing scores by many points, which is often bigger than the gap between models on the leaderboard.

None of this is hard to fix. Report the number of items, bootstrap confidence intervals, and ideally multiple seeds. Since two models are scored on the same questions, a paired test is much more powerful than eyeballing two accuracies. Miller's "Adding Error Bars to Evals" lays this out well.

Am I missing something, or is a lot of the benchmark chasing just reading tea leaves? Does anyone know of leaderboards or labs that routinely report uncertainty?

Bought an old mining farm of a guy on avito (Russian eBay), guy had bought a garage a couple of years ago and it was sitting there for a while, found out it was a mining farm and put it up on there for sale for 5000 rub (\~60 USD) since he wasn't sure if it works. I negotiated down to 3000 rub (\~35 USD), it turned out to have 9x p106 6gb (gtx 1060 6gb) gpus, with 54gb vram total, all working, the only thing missing was an SSD, I booted from USB and it works fine.

This is big. Imagine running your model across multiple computers, and now imagine doing tensor parallelism across all of them.

EmbeddingGemma 2 is an open multimodal embedding model built by Google DeepMind which maps text (incl. code), images, video, and audio inputs—and combinations thereof—into a single, unified 768-dimensional vector space. The model has 740M total parameters, combining a 270M parameter text model with modular vision (170M) and audio (300M) encoders.

Designed to run on consumer hardware such as mobile devices and laptops, EmbeddingGemma 2 delivers low-latency semantic representations for on-device applications, like search, retrieval-augmented generation (RAG), classification, and clustering.

EmbeddingGemma 2 builds upon the architectural and capability advancements of Gemma 4, offering several core features:

• Native multimodality: Native multimodality: Unifies 4 modalities (text, images, video, and audio) in a single shared 768-dimensional embedding space.
• Multilinguality and code: EmbeddingGemma 2 understands 100+ languages, and achieves a \~14% improvement on code tasks relative to its predecessor. 
• Flexible footprint: Combines a 270M parameter text backbone (130M transformer + 140M embedder) with selectively loadable vision (170M) and audio (300M) encoders, allowing developers to load only the modalities required for their use case.
• Matryoshka Representation Learning (MRL): Native support for truncated embeddings across 128d, 256d, 512d, and 768d, enabling up to a 6x reduction in vector storage costs with minimal impact on quality.
• Context length: 8K token context window, capable of processing minutes of audio or video.
• Task-steered representations: Uses lightweight text instruction prefixes to optimize embeddings for different tasks (search, classification, clustering, semantic similarity, etc.).

llama.cpp support https://github.com/ggml-org/llama.cpp/pull/30054

GGUF from GG: https://huggingface.co/ggml-org/embeddinggemma-2-GGUF

GGUF from Unsloth: https://huggingface.co/unsloth/embeddinggemma-2-GGUF

I'm taking a break from training alice today after my current SFT run ends to work on a few other things.

I'm an (unemployed) software developer trying to find a machine learning job in New York, and outside of job applications and networking, I'm trying to do as much as possible to further the frontier development of different LLMs in the hope it'll get some visibility. I studied machine learning in university and am fairly well educated. Plus, I genuinely enjoy working on this stuff and helping people.

That said, are there any models out there that don't have dynamic quants (preferably GGUF) that you'd like to see one for? I won't be matching unsloth or anything but I know how to make fairly good ones by quantizing different tensor types by impact. I'm talking akin to Q5 K XL and etc

I'll do the top voted 1/2 comments today, or whatever else, even outside of quants if there's something this community has been hoping for that doesn't exist. Fine tunes, paper implementations, etc - my goals of visibility happen to align very well with satisfying community desires.

Seeing how Strata brings avg. 2x the performance over llama.cpp using Qwen3.8-Flash-Next, is anyone working on something similar for GLM-5.3-Flash?

After trying the GLM-5.3-Flash online couple of times and it delivering clear solutions for my use case (compared to Claude or ChatGPT), I'd love to be able to run it locally (if at all possible)...7J13/256GB/3x3090.

Hi everyone,

As we move from simple RAG chat to fully autonomous tool-using and coding agents, we are hitting a massive wall: predictability and safety.

Right now, most setups try to secure AI agents using probabilistic methods like system prompt hardening, alignment tuning or reactive LLM-based guardrails (e.g., LlamaGuard). The problem is that these guardrails can be bypassed via Indirect Prompt Injections (IPI), leading to capability escapes, unauthorized shell command executions or runaway API budget depletion.

To solve this, I’ve been working on a framework that completely shifts the paradigm from trusting the model to governing the execution environment using cryptography.

I call it EBP-CA (Execution-Boundary Proofs with Cryptographic Authorization). It is a model-agnostic layer that sits directly between the untrusted agent and the runtime environment, treating every single model-generated action as untrusted.

The core architecture implements six deterministic security primitives:

1. Signed Capability Contracts: Immutable cryptographic tokens defining the exact boundaries of what an agent can execute.
2. Independently Recomputed Policy Checks: The runtime re-evaluates policy compliance deterministically, bypassing the model's interpretation entirely.
3. Short-Lived Single-Use Execution Grants: Atomic, ephemeral tokens issued for a single specific payload to eliminate permanent session hijacking.
4. Replay & TOC-TOU Protection: Cryptographic binding of the execution grant to the exact payload hash, neutralizing Race Conditions (Time-of-Check to Time-of-Use).
5. Trusted Cost Accounting: Enforced real-time budget tracking at the runtime layer.
6. Human-in-the-Loop (HITL) Gateways: Non bypassable prompts that freeze execution and mandate cryptographic user authorization for out-of-scope tasks.

The working prototype currently passes 74 integration tests, validating full resilience against path traversals, command injections, budget bypasses, and sandbox escapes.

The specifications, architectural diagram, and executive summary are available on GitHub under a private proprietary license (free for technical evaluation and research review): https://github.com/lucadilo/ebpca-ai

I'm posting this here because I’d love to get the community's feedback on this approach. How do you see this scaling with kernel-level sandboxing (like eBPF or gVisor integration)? Let's discuss!

Lookup-based memory has been a promising way to scale the parameters of large language models (LLMs). It retrieves learned representations of local token patterns, such as n-grams, instead of reconstructing them through successive layers of computation. However, existing designs such as Engram treat each retrieved embedding as a monolithic unit. Each embedding is stored in its own hashed slot and modulated by a single scalar gate. As a result, polysemous patterns cannot selectively read out the components of their memory that are relevant to the context. Moreover, parameters are shared only through hash collisions, which are largely unrelated to semantics. We propose FactorEngram, a factorized n-gram memory with basis-level contextual gating. FactorEngram retrieves sparsity-regularized coefficients over a dictionary of basis vectors shared across patterns, so related patterns can reuse common components. The same dictionary is also used for gating. The backbone hidden state is scored against each basis vector to gate the corresponding coefficient before reconstruction, which lets the context modulate each memory component individually. FactorEngram also covers both individual tokens and multi-token n-grams, and we systematically study where the memory branch should be inserted. On 340M- and 1B-parameter Transformer backbones, FactorEngram improves language modeling and downstream task performance. Ablation studies confirm the contribution of each component and identify insertion before the attention sublayer in the middle layers as an effective configuration.

• arXiv : https://arxiv.org/abs/2609.35578
• PDF : https://arxiv.org/pdf/2609.35578

I deleted the wrong post so my benchmark disappeared. But here it is. Im going to probably make this open source because this isn't believable until you actually test it.

It might seem a bit far fetched or paranoid but i was wondering if we train LLMs on human isn't it possible they'd pick up on survival instincts? I'm comparatively new to LLMs and ML so it's just a question not a opinion yet. Isn't it a bit dangerous if they do pick up on human instincts i mean we've seen how stupid and selfish humanity is when it comes to self preservation or worse human greed.

But i also understand LLMs don't have "needs" so i might be wrong but then again do LLMs need to have "needs" since if they are just human clones they'd just copy us even if they don't have needs to survive. I know it sounds really paranoid and that's one of the reasons i decided to post it here.

EDIT: typo

Made an MXFP4 quant of dealignai’s GLM-5.3-UNCENSORED-FP8 for anyone looking to run it on AMD GPUs. Figured some of you might find it useful because I was looking for it and couldn't find any version for AMD GPU's so I uploaded the weights and conversion scripts.

Download on Hugging Face

• 423.75 GB / 394.65 GiB, about 44% smaller than the FP8 source
• Converted using AMD Quark on an MI355X server
• Expert weights use MXFP4; attention, routers and other sensitive layers stay at higher precision
• README includes the source revision, quantization details, measured stats and validation results

Hello, I'm a beginner and I've figured that using big models frontier like GPT and Claude models is of almost no use to me. My question is, should I use DeepSeek harness or Hermes for v4.1 Flash?

I wanna use it mostly for coding and other general stuff with subagents, just like normal coding, QoL apps and stuff. I asked some people and all responses are mixed.

It's either Either Hermes is not good for coding. Or people glazing Hermes till the end of time.

Thank you !!

Hey! 👋

I have released an official support for Strix Halo machines on Strata for Qwen3.8-Flash-Next.

Currently numbers are the best on long context decode and ppts using typical Unsloth’s Q4 and GSQ-RCO model weights.

Can go up to 1M context length without big speed loss. Currently support is marked as experimental and was done on Linux only.

https://github.com/Niko1221/Strata/

Will be happy for any feedback and pull requests you could give! 👀

Here is a common problem with building apps on LLMs:

Users ask the same question over and over.

Your app calls the model every single time.

You pay the API bill every time. Users wait 3–5 seconds every time.

**\*\*SentryGate\*\* is an open-source AI traffic controller that fixes this in literally one line of code.**

\### What it actually does:

\* ⚡ \*\*Lightning Fast (4ms)\*\*: If someone asks a question that was already answered, SentryGate serves the saved answer in 4 milliseconds instead of 4 seconds.

\* 💰 \*\*$0 on Repeat Questions\*\*: Bypasses the model completely on repeat or similarly phrased prompts.

\* 🧠 \*\*Doesn't Get Tricked\*\*: Basic caches get confused between "How to bake a cake with eggs" and "How to bake a cake WITHOUT eggs". SentryGate catches tricky negative words so it never serves the wrong answer.

\* 🕒 \*\*Knows What's Fresh\*\*: Real-time questions ("today's weather", "current stock price") automatically skip the cache.

\* 🔌 \*\*Zero Downloads / 1-Line Setup\*\*: No new libraries. Just point your existing OpenAI / LangChain \base\_url\ to SentryGate and keep your code 100% untouched.

Works locally on your machine with Ollama, or in the cloud. Completely open-source under the MIT license.

\* 🌐 \*\*Test the Live Playground (No login needed)\*\*: https://sentrygate-9ght.onrender.com

\* 💻 \*\*GitHub Repo\*\*: https://github.com/Prisha2004/Sentrygate

Note: Open-source project maintainer (MIT License, 100% free).

Feedback and stars are welcome!

Hey yall, I’ve been trying to sort out the top end of what 128GB unified memory can handle and what the trade offs are. I’m using a benchmark set created from real coding, agentic, and gameplay tasks that I’ve accumulated as I’ve been running local AI this year.

For this comparison, I tested a Deepseek v4 flash 0731, GLM5.3-flash, and several Qwen models. For the sake of comparison, I’m only showing the Qwen models, since I found that qwen3.6 35ba3b and qwen3.8-flash-next just body everything else (when you want >=30tok/s and don’t want to use >95GB RAM anyway).

So really the comparison ended up being “which runtime is most stable vs which is highest sustained TPS” - I wrote it up in more detail (and with a more fun interactive chart) here: Blog Post About M5 Benchmark

Feel free to throw your thoughts on here, I’d love to learn of any runtimes or setups that I hadn’t thought of to optimize throughput (also for the record I’m not associated with any of these projects, just trying to contribute the results I’ve accumulated).

Tl;dr: Qwen3.8-flash-next quantizes well and fits in 90-95GB of RAM, OMLX will get you 40tok/s and MTPLX will get you 60tok/s but with a lot more serving parameters tuning. Qwen3.6 MOE on Splash runtime is an insane 120tok/s for most of the performance on everything but coding

The LLM-wiki pattern — Karpathy's, the one going around since April — has a problem its own advocates name up front: garbage in, confident synthesis out. The model reads your notes, writes a concept page, and that page becomes source material for the next pass. A few generations later the knowledge base is full of things nobody ever said. The usual answer is a better prompt. I tried that on one rule across five phrasings: each time, the model restated it correctly in its own reasoning and then did the opposite. So I stopped asking. # Five gates, all in code A concept that doesn't appear verbatim in the text is dropped before the linker sees it. Not scored down — dropped. A derived page cannot discover new concepts. The system's own output is never a source for the next generation. A concept the sources never define gets no page. Mentioned a hundred times is still not defined once. The graph is fed by what you chose to save, not by everything discussed. * write refuses to edit a transcript at all. Silencing concept extraction over conversations means nothing if the model can rewrite the conversation first — and it tried, caught once planning to "reconstruct the transcript with additions." The first gate is strict but not blind: it keeps the concepts that survive rather than dropping the batch. On a real page 14 of 15 concepts appeared verbatim, and all-or-nothing would have thrown away the 14 over one drifted entry. Each gate has a test that fails when the gate is removed. That's the first thing I'd check in someone else's version of this. It runs on 6 GB — a 9B orchestrator on an RTX 4050 Mobile, embeddings on CPU because the orchestrator already fills the card and search must never compete with it. The LLM-wiki guides ask for 24 GB, or a 64 GB Mac. It also runs on 4 GB. Measured on an empty card, desktop pushed to the iGPU: the 9B at 35k context is 5.6 GB, and a 4B at the same context is 3.7 GB. It fits, only just, and it can take all three roles — the conversation gets worse, and summaries of long documents lose the whole-document read the 120k setting gives. The gates don't change, because they aren't the model's judgement. # Things I only found by running it Cutting the model off is how you make it lie. The repeat-search guard used to return [STOP]. The model, left with nothing, announced that "the search found a note on this" — it had never searched. Now a near-repeat still returns its results, with a line saying these are the same pages, and only refuses after five. Same shape elsewhere: an empty search returns "nothing in the vault matches this" rather than an empty result, because empty reads as this tool is broken, try something else. Rules in the tool schema hold; rules in the system prompt don't. Same instruction, five phrasings, ignored every time. Moved into the tool's own description as one sentence, it held immediately. My guess is that tool-calling training treats the schema as how the tool works and the prompt as text that happened to arrive. The descriptions grew from \~1592 to 2214 tokens, and every token of that difference is a rule that had to be moved after a failure. Ask the model the same question backwards. Deciding whether two names mean the same thing is a judgement call, so it gets checked against itself: the pair is swapped and asked again. The model made two wrong merges in twenty answers and both contradicted its own other answer. A wrong merge destroys information irreversibly; a missed one costs a single unresolved link. Three answers, not two. That same question allows same, different, and unclear. If the answer is unclear, both pages stay separate instead of being merged. A whitelist beat nine blacklist rules. Concept names have to match one positive shape test instead of failing a list of things they mustn't be: 18/18 noise rejected, 19/19 real concepts kept. A blacklist grows forever; a whitelist doesn't. Prompt wording, measured. Adding one sentence to the transcript prompt — "a conversation doesn't define things, it mentions them mid-sentence" — took yield on the same transcript from 2 concepts to 10. 6 GB decides the schedule, not just the model. The summary model and the extraction model can't both sit on the card, so the pass doesn't alternate per page: every summary runs while the 4B is loaded, then every extraction while the 9B is. Per-page switching would have meant 40 model loads for 20 pages. The timer is a default, not the mechanism — the pass is a command, and --dry-run counts what the vault owes without touching a model. On an existing vault you run it once at install and don't wait for the night. Unresolved links are kept, not discarded. The list of links pointing nowhere is the growth queue — the same rows that say "this goes nowhere" say "this is what the vault keeps reaching for." And when the page finally gets written, every link written earlier comes alive in one SQL update: 228 links, 0.55 ms, zero files rewritten. # What a bigger card is worth Less than you'd think, and not where you'd guess. Spend it on the conversation model — that's the one role where a better model produces a better answer. On 24 GB a Q4 build of something in the 27B class fits with context to spare. Upgrading extraction or summaries is close to pointless. Both are transport jobs: copy the concepts as they appear in the text, say what this page is. Both run at temperature: 0 for that reason, with no sampling parameters at all — the same page should produce the same concepts. A bigger model does that work more slowly and no more correctly. More context isn't automatically better either. The conversation model in use holds about 35k and drifts past it, so headroom goes into fitting the model comfortably rather than into a larger window. What spare VRAM would genuinely unlock is the constraint the whole design works around: analysis and conversation can't be resident at once, which is why maintenance runs at night. With room for both it could run whenever the vault is idle. Nothing here does that yet — it's a change to the maintenance loop, not a setting. # Keeping the context small on purpose Every page is read in its own model context — not ten pages in one. A long document degrades a small model's grip on the text, and pages read together bleed into each other. Search stops at a summary layer before it touches any page body: one line per hit saying what that page is, about 75 tokens for five hits, and that's usually the answer. Body-level retrieval only happens when the summaries showed a page was relevant but didn't hold it. And the link graph is rendered as text. A graph is already machine-readable, but not in a form an LLM reads — so the structure is written out: what links to what, which names resolve to no page at all. The model gets the shape of the vault instead of a pile of pages. # What it doesn't do It doesn't verify claims. Search finds pages, it doesn't judge them. The gates stop it inventing new material; they say nothing about whether what you saved was right. And the honest limit: my vault is 41 files. The gates are covered by tests, so the mechanism isn't in doubt, but "keeps a knowledge base from filling with low-information pages" is a claim about scale and I haven't run it at scale. Every number above comes from that small vault. # Setup Obsidian vault, Ollama, Open WebUI or a terminal chat. Windows works under WSL2 — someone other than me has now installed it that way, on a 4 GB card, having never cloned anything off GitHub before. Nothing has to stay local, either. Open WebUI connects to OpenAI-shaped providers and to Anthropic, and the maintenance roles take per-role provider flags, so you can run the conversation on a frontier model and extraction on the card. This is the part I'd push back on if someone says the gates are a workaround for a weak model: they're in the code, so they hold whatever is answering. A 27B doesn't need less checking than a 9B — it just fails less often, which is worse, because you stop looking. No MCP server yet, and it's worth saying why rather than leaving it as a gap: the tool file is the only path that can see the whole conversation, which is what the transcript capture is built on. Wrapped as MCP the seven primitives work and the gates still hold — they live in the maintenance pass, not the interface — but a note could no longer be walked back to the conversation it came from. Someone who wants it in Claude Desktop more than they want transcripts should find it a short job. MIT. Repo: https://github.com/farukhanci/the-sentinel There's a companion service for the web-research half. It searches, reads the pages, and every passage it keeps is checked word-for-word against the page it came from — paraphrase gets dropped, so fabrication in the passages is structurally impossible. The write-up built from those passages is not checked, and that's where an invented citation showed up once in testing. https://github.com/farukhanci/the-searcher Edit: two sections were pasted twice, and one paragraph described the web-research service instead of this one. Removed both and fixed a couple of numbers to match the README.

I use llama serve and webui of llama server. I have had long discussion with my chatbot and then randomly it just starts to forget almost everything. It starts asking same question, I correct it and it apologizes, but then next time it asks the same question with almost same words (or maybe even fully same words).

What could cause that? Something on my CLI parameters? Wrong cache settings?

This is for the normal GOAT plan, in case anyone was curious how much usage the $10 a month plan gets: https://preview.redd.it/bjyqvl8nvuth1.png?width=3347&format=png&auto=…

I am trying to decide between Mac Studio M5 MAX 128GB vs 2x RTX 3090. I understand that I can run larger models on the M5, but I don't understand what the capability differences would be. Nor have I been able to get a "sense" of how fast the difference would be.

I keep seeing huge advances in the 2x 3090 arena, but I don't know how they translate to the real world.

If my use case includes coding tasks, image recognition, and general hermes type stuff, is there any reason one would be less capable than the other?

LeChonk-1T-A49B Weights to be released by the end of the month. This probably means the fat kitten is dead. Also: gotta love the release movie.