"NVHBM moves the memory controller, which was previously located on the main compute die, into the base die. This reduces power consumption by 15% and increases memory bandwidth by as much as 30%. It also integrates a customized physical layer (PHY) for input/output (I/O), reducing the package area required for the I/O PHY by as much as 67%. NVIDIA says NVHBM provides up to 30% greater memory bandwidth and 15% lower HBM power consumption than standard HBM4E."

Sounds quite dope to me. However, the price will be too dope for me...

EmbeddingGemma 2 came out this week. It maps images and text into one 768-dim space, so you can search photos by describing them. I ported its text and vision towers to ruNNtime, a WebGPU inference library in TypeScript, and made a small photo gallery where search runs entirely on your GPU in the browser.

ruNNtime also supports plenty of other vision-like models, and you can play with them in the interactive docs

source: https://github.com/software-mansion/runntime

Ideally I'd like to be able to run Qwen 3.8-Flash-Next with decent performance.

I was thinking of just buying 2x Radeon AI R9700 (64GB VRAM), or maybe a DGX Spark but that was before the price hike. My brother suggested just getting a Strix Halo box with 128GB unified.

I did see I could buy 6x Intel Arc B60 (24GB each, 144GB VRAM Total), but researching seems like the performance of the B60 is lacking. I'd also need a new v
motherboard/CPU that can run 6 GPUs.

I'm also not opposed to getting a Mac Mini or Studio if the price and performance is right.

The $4000 is not exactly a hard cap, like I could stretch to $4200 without too much struggle, but obviously the cheaper the better. I'm lucky to have a decent stock pile of NVMEs and DDR4/DDR5 UDIMMs, so if I need to build a box, I could, would just need the motherboard and a CPU if I can't just slot in either the Intel 14700K or Ryzen 9700x I already have.

So where am I swiping my credit card?

Edit: This is a use it or lose it budget from my work, can't really save it.

Edit 2: To clarify again, this is extra money in the IT budget that we need to burn by the end of the year. Telling me to save it, donate it, invest it, isn't helpful as I can't do that.

Last time when I posted on this subReddit to share my work, the response almost made me cry because a tiny demo got so many people talking about this

In the past 2 months Ive been training a new model, but this time with actual text guidance.

So a little about the older model.

Normal video models are too large and not meant to run on consumer hardware in real time. You can generate a static clip, even fast but realtime video is not exactly solved yet locally.

A lot of world model demos have come up but theyre either meant to run on huge datacenter GPUs or theyre just popular models like WAN or LTX kinda distilled to work in an Autoregressive way (which is also not realtime btw on local)

That video above is on an RTX 5090 working at just 30% util. The peak fps of this 1B model is 50-60 on a rtx5090 but I forcefully software throttle it to 12fps. And according to some tests this means the model can work on other RTX cards of 40,30 series (I will try them out soon )

I have a MacBook and I haven't ported the model to MLX YET but I made a benchmark and the model runs at 30 fps on my M5 MacBook.

About the Architecture

The model is a pure transformer and works with a block causal mask, which means in training past frames dont see future frames so they learn just like an LLM. Another important method I used to train his is called "diffusion forcing" which means in training unlike normal video model training, we noise each frame independently so the model learns to be comfortable with noisy past and all

The model s 28 blocks 20 heads and comes to like \~960M parameters

At inference we run 2-5 steps of diffusion per frame and once a frame is done denoising we add it to the KV cache. This is akin to the decode step of an LLM.

The biggest difference from LLMs is that we dont keep all kV context ie all past context and use a sliding window so only past 80 frames worth of context actually stays.

The last model was a MMDiT which means there was no cross attention for text. This is bad in a world model because the. past frame kv and the text kv are literally competing in the softmax so you could never never live text guidance reliably. The last model was also not trained on text-video so its moot anyways

The current model is back to text cross attention and I did a lot of text-video pretraining

Its taking the keyboard actions I give it live (an adaln extra term helps guide the generations with actions WASD )

and the most fun part is text prompt switching.

"add a pond to the desert"
"put red hoodie"
"change environment to icy"

Because the model was trained with so much text-video alignment it can actually follow prompts now.

I know there are a lot of limitations still like consistency and quality improvements, but I sincerely hope by the end of this year I can release something anyone with a RTX GPU or new MacBook can try.

I specifically chose this init image because in my last post on this subReddit also I had used the same one.

PS in my last post a lot of you guys asked about me and the funding

I am based in Bangalore and in final year of college (partially dropping out), and funded by a student incubator. I only work alone and dont have a team or a real company or anything

The above model was trained on 8x H100 SXM for like 3-4 weeks.

Every model I make will be explicitly for local inference, never datacenter

UPDATE : Tested on 4060Ti , Its 20FPS at half the ring size (half context) and 13 FPS at normal. Because the RTX5090 was used on 12fps forceful throttle anyways, 4060Ti and 5090 above rollout will look EXACTLY THE SAME.

Looks like Unsloth is updating https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF to work with llama.cpp, so hopefully this will resolve the issue of having two different GGUF versions of Qwen Flash Next.

https://preview.redd.it/owhrvbbiu2uh1.png?width=4096&format=png&auto=…

d1-omni-600M

d1-omni-600M is a 600M parameter decision model built on LFM2.5-Encoder-350M. You give it a state (text or JSON, with images or a voice clip) and a set of named questions. It returns typed answers with zero output tokens: every answer is read directly from the model's distribution over the options, with no generation and no parsing.

• Vision-language: text and images (tiled for large frames, several images per state) in a single forward pass.
• Audio-language: text and up to 30 s of speech in a single forward pass.
• Edge-sized: 587M parameters: a 381M shared trunk and decision head, a 94M vision encoder and a 112M audio encoder. Every modality runs the same trunk weights.

https://preview.redd.it/f2wkfqcku2uh1.png?width=1200&format=png&auto=…

d1-3B

d1-3B is a 3B parameter decision model built on LFM2.5-VL-3B. You give it a state (text, JSON, images, or a mix) and a set of questions. It returns calibrated, typed answers in one forward pass with zero output tokens.

• Best decision model under 10B on the Decision Index 0.2.1: 48.57, ahead of every 4B and 9B model and of Decider 35B-A3B (47.11).
• Multimodal: images and text in the same state. It scores 74.1 on 11 public image benchmarks (LFM2.5-VL-3B: 73.9).
• Fast: 8 ms a decision on an NVIDIA RTX 4090, 9 ms on an AMD MI325X, 30 ms on an Apple M5 Pro.

https://huggingface.co/LiquidAI/d1-3B-GGUF

https://huggingface.co/LiquidAI/d1-3B

https://huggingface.co/LiquidAI/d1-omni-600M-GGUF

https://huggingface.co/LiquidAI/d1-omni-600M

https://preview.redd.it/e11c4qlut2uh1.png?width=1932&format=png&auto=…

We spent the last weeks building a studio around YuE2, the open song model by m-a-p, and today it reaches its first release candidate. Write the style and the lyrics, and it composes, sings and renders the song on your own card. Nothing leaves your machine unless you point the Writer at a cloud chat model.

What it does

• Whole songs, up to 8 minutes. On one RTX 3090: a 6:12 song in 126 s, and a 7:25 song with its score written first in 198 s.
• The score first, and yours. YuE2 writes the melody and chords as ABC before a note sounds. You can edit them, transpose them, or bring your own score or MIDI.
• Two seeds. Keep the song (the music seed) and hear it rendered anew (the sound seed).
• LoRA training in the studio, on your own songs, unquantized (bf16), with telemetry that tells you which epochs to hear first. Adapters stack on measured roads, under a measured ceiling.
• A guard against garbage. A broken score is caught in seconds and the run is stopped before the GPU is spent on it, and you are told why.
• Post-production, all local: spectrum, artifacts, debuzz, stems (BS-Roformer, htdemucs), remaster, upscale (UniverSR), and a lyrics check by Whisper. One chain runs them all.
• Into your DAW (experimental): a REAPER project with the stems, the score as MIDI, the tempo, the sections as regions and the lyrics on the timeline; DAWproject for Waveform and Bitwig.
• A Librarian for every take; a Writer with versions and a chat model (local or OpenRouter); a cheat-sheet of 200 instruments probed by ear; the API and an MCP server; the page in 7 languages.

What it is not (yet). It is less polished than SUNO out of the box:
- a mix can buzz (Debuzz helps);
- lyrics can drift (the lyrics check finds where);
- some instruments YuE2 plays thinly or not at all (a LoRA teaches them).

You need an NVIDIA GPU (24 GB for everything at full precision), Linux, and about 120 GB of disk for the models, LoRAs and workspaces.

Licences.
The code is under AGPL-3.0-or-later. The YuE2 weights are CC BY-NC 4.0, and that licence speaks of the weights, not of the songs made with them: read it before you sell.

What comes next (rc2 and after)

• The Artist room: covers in three shapes at once from one seed, the title and the artist written on them.
• Five more languages for the page: Chinese, French, Portuguese, German, Japanese; right-to-left ones later.
• Voice adapters trained on spoken voices, named by the kind of voice, on Hugging Face.
• The Writer's models with their prices as you type; calmer rooms (dialogs, tips, one shape for the icon buttons).
• A desktop app: an installable page first, then Electron; native plugins for REAPER, Waveform and Bitwig.

Links
- Code: https://github.com/igrbible/Ruach_Studio
- Models (pinned, checked): https://huggingface.co/goldhub/Ruach_Studio_Models
- Site: https://ruachstudio.igr.bible
- The full guide, room by room, is inside the studio and in docs/GUIDE.md.

Built on:
- YuE2 by m-a-p;
- yue2.cpp by ServeurpersoCom;
- YuE2 Kit v12 by IronWolve (the base of the page and the scripts).

Every one of our changes is numbered and documented (HERESY 1001–1167).
Issues and PRs are welcome. We would most like to hear how it runs on machines that are not ours.

If you're like me and saw the price increase for the 128gb DGX Spark go from 4.7k -> 7k while a new version with 64GB launch for 5k, you'll have been very disappointed and every right to be so, it's just plain sad for localAI.

Well, here's some good news, cmpunlocker v0.5 just dropped with ecc support and +4 SM for free. I hear gen3 unlock is also on the way so fingers crossed.

https://github.com/amoghmunikote/cmpunlocker/releases

My current setup:

\- single 3090 running turboderp/Qwen3.8-27B-exl3:SC\_5.00bpw\_H6\_V6

\- \~150k context, \~70t/s, unknown prefill because I didn't benchmark it (but it is ok)

\- Intel 12400 CPU with 32GB DDR4 RAM

All the hype around strata makes me consider buying 128GB of 6000MHz DDR5 RAM and Ryzen 9700X just for it. I searched in this sub, but most posts about it is about prefill/token generation speed, not about output quality. I believe with 128GB RAM + 3090 I can run the IQ3 quant.

For those who have run both Qwen 3.8 27B and Qwen Next with strata, how would you compare these two, in particular about output accuracy? My main use case is coding and Hermes assistant.

BTW, are there other good options for a 128GB RAM + 3090 setup?

Over the past several months, we conducted an empirical post-mortem investigation into 109 autonomous AI agent security incidents (cataloged with 193 falsification criteria across tool-use and multi-agent systems).

One of the most striking patterns in the dataset:
In 38% of container breakouts, attackers and misaligned multi-step agents didn't exploit complex Linux kernel vulnerabilities or hypervisor 0-days. Instead, the breakout vector was trivial configuration residue:
1. Mounting /var/run/docker.sock into coding/evaluator agent sandboxes to let them "build Docker images".
2. Passing parent environment variables (API keys, cloud tokens, GitHub credentials) directly into spawned subagents.
3. Lack of strict taint tracking across tool outputs, leading to indirect prompt injection hijacking the supervisor’s execution path (the classic Confused Deputy problem).
4. Unconstrained local socket binding allowing SSRF against internal orchestrators.

We compiled the complete dataset (109 incidents, 199 evaluation metrics) and built an open-source Multi-Agent Supervisor Security Harness with:
- Formal tool taint propagation (tainted outputs cannot flow into high-privilege tool arguments without sanitizer verification).
- Strict execution boundary controls preventing container socket exposure.
- Automated reproduction benchmarks testable against agent runtimes.

All datasets, 2-page executive summary, and reproducible benchmark tests are released under Open Access / Apache 2.0.

I've posted the GitHub repository benchmark and the Zenodo DOI dataset links in the comments below to adhere to subreddit self-promotion guidelines.

Curious to hear from teams deploying autonomous agents in production: what isolation boundaries are you enforcing between your planning supervisor and your tool execution workers?

For the last few weeks most of my coding has been done locally with Qwen3.8-Flash-Next, so I gave it and Opus 5.5 the same high complexity feature to build on LlamaStash (a complex and large Rust project) and compared the results.

Setup: ASUS ROG Flow Z13 (Strix Halo, 128GB), Flash-Next at xhigh effort with Pi as the harness. Opus 5.5 ran in Claude Code at medium effort. I wanted xhigh for Opus as well, but Claude changed it to medium when I picked the latest model and I didn't notice it until the task was done. But I think medium is probabbly a fairer setting anyway.

Task: add a llamastash daemon restart command that reuses the existing start and stop code. I kept the prompts vague on purpose and gave both the same prompts.

|Step|Opus 5.5 (medium) PR#88|Flash-Next (xhigh) PR#89|
|---|---|---|
|First iteration|~9 min|~38 min|
|Nudge to reuse the TUI restart code|~6 min|~34 min|
|A third duplicate path|found it on its own|~30 min, after one more prompt|
|Create PR|~3 min|~30 min|
|Total|~18 min|~130 min|
|Tokens (in / out)|7.83M / 41.5K|20.61M / 101K|
|Tests added|1|4 (2 of them end to end)|
|Cost|$7.53|$0 + ~0.15 kWh|

The end result was interesting. I asked GPT 5.6, Opus 5.5 and Flash-Next to review and compare both PRs (new sessions). GPT and Flash-Next picked the Flash-Next PR (#89) and Opus picked its own (#88). I also did my own review and found the Flash-Next one better as it had better tests and handled edge cases better. I ended up merging #89, after porting the fixes that the reviews picked from #88.

Keep in mind:

• Opus was on medium effort. With xhigh it would have used way more tokens, taken a bit more time and probably would have done a better implementation.
• Flash-Next ran on an older Halogen version (0.14.0), and Halogen dropped the connection once, so the last part ran on Gufo. The current Halogen does around 1,400 t/s prefill and 46 t/s decode on my laptop at 70 W, so I think the time will drop a lot if I redo the test.
• The $7.53 is what Claude Code reported for the whole Opus session, which includes a later fix to the PR. The 0.15 kWh assumes 70 W for the whole 130 minutes.

Opus is still 2 to 10 times faster and I still use it for planning and reviews. But the actual coding now happens on my laptop, and to me it is crazy that I can run a local model that can challenge a frontier model like this.

Full post with my setup, the engine benchmarks and a second task comparison: https://deepu.tech/local-ai-qwen3.8-flash-next-best-local-llm

I made Terminal Soliloquy, a terminal artwork that connects to llama.cpp and displays a model's monologue as its context window fills.

It has a retro phosphor look, and runs in a terminal window. I'm actually running it full-screen on a Raspberry Pi display inside an old 1960s portable TV.

As the conversation grows, the model reflects on its own limited lifespan. [](https://preview.redd.it/what-happens-when-a-llm-watches-its-own-context-windo…)When the context is exhausted, the display can be configured to freeze, restart, or quit.

The repo and setup instructions are here: https://github.com/nicespoon/terminal-soliloquy

Introducing InferBench: A benchmark testing how well frontier LLMs infer a user's priorities from their instructions.

We tested 12 LLMs across 20 scenarios with 2.8k conversations to see which models understand the user's goals.

Each conversation has a simulated user with a private profile of their priorities. The assistant decides on the best option or can first ask clarifying questions.

When a user's priorities were clearly stated up front, the models picked the best option 89% of the time. If some priorities were initially unstated then accuracy went down to 60%.

The open weight models did better than I expected:

GPT-6 Astra: 76%

MiMo V2.6 Pro: 75%

Gemini 3.1 Pro: 69%

Astra always asked clarifying questions when preferences weren't stated and never did when they were (extremely impressive). By comparison, Grok 4.6 asked to clarify in 18/64 conversations.

However LLMs are still just as confident when they're wrong. 105/288 wrong decisions were submitted with >90% confidence.

The full report is linked here: https://laugh.so/research/inferbench/

What surprised you the most?

Liquid just dropped d1-omni-600M and it's a decision model!

I ported it to runntime, the WebGPU inference library I'm currently working on. It's plain TypeScript on top of TypeGPU with no WASM and virtually no export step. The model is written directly from our core ops (matmul, attention, norms, a few elementwise bits), and the weights load straight from the HF safetensors.

The demo in the video is a fake comment feed being moderated live. Each comment gets 4 questions: toxic? spam? asking something? overall tone? Toxic and spam ones get removed.

\~180 ms per comment for all 4 questions, \~45 ms per question - the performance will most likely be way better once we spend some time tuning the engine for this model

I also tried making it play snake, but It did not go well lol.

The d1 port isn't in the npm release yet. The rest of runntime is (detection, segmentation, speech-to-text, embeddings and more). Docs and live demos: https://docs.swmansion.com/runntime

Happy to answer questions about the port or WebGPU stuff in general.

Lookup-based memory has been a promising way to scale the parameters of large language models (LLMs). It retrieves learned representations of local token patterns, such as n-grams, instead of reconstructing them through successive layers of computation. However, existing designs such as Engram treat each retrieved embedding as a monolithic unit. Each embedding is stored in its own hashed slot and modulated by a single scalar gate. As a result, polysemous patterns cannot selectively read out the components of their memory that are relevant to the context. Moreover, parameters are shared only through hash collisions, which are largely unrelated to semantics. We propose FactorEngram, a factorized n-gram memory with basis-level contextual gating. FactorEngram retrieves sparsity-regularized coefficients over a dictionary of basis vectors shared across patterns, so related patterns can reuse common components. The same dictionary is also used for gating. The backbone hidden state is scored against each basis vector to gate the corresponding coefficient before reconstruction, which lets the context modulate each memory component individually. FactorEngram also covers both individual tokens and multi-token n-grams, and we systematically study where the memory branch should be inserted. On 340M- and 1B-parameter Transformer backbones, FactorEngram improves language modeling and downstream task performance. Ablation studies confirm the contribution of each component and identify insertion before the attention sublayer in the middle layers as an effective configuration.

• arXiv : https://arxiv.org/abs/2609.35578
• PDF : https://arxiv.org/pdf/2609.35578

Yesterday TechCrunch published a piece about a growing problem for AI agents: websites are starting to block them. Amazon blocking Meta's Muse is the obvious example.

At almost exactly the same time, GLM-Edge-1.5B-Chat running locally on a 4GB Galaxy A04e completed a real Amazon cart task.

This continues the small-model/browser experiments previously posted in this subreddit. Earlier tests included Qwen3-0.6B running locally on a 2017 Galaxy Note 8, followed by Ministral 3 3B on a Galaxy S21 across real browser sessions.

These experiments are part of the ongoing development of E2LLM/SiFR, a structured browser perception layer.

This time:

Model: GLM-Edge-1.5B-Chat
Quantization: Q4\_K\_M GGUF
Source: official Z ai Hugging Face release
Fine-tuning: none
Task-specific training: none
Runtime: llama.cpp
Phone: Samsung Galaxy A04e, SM-A042F/DS, 4GB RAM

The published model was used as-is.

The browser was a normal desktop Firefox session on Amazon.

The task was simple:

• find a 24-count pack of AA alkaline batteries
• find yellow rubber ducks
• add both to the cart
• stop before checkout

Result:

cart 0, batteries, cart 1, rubber ducks, cart 2

The same setup was run twice on the A04e. Both runs completed successfully.

Full run on the A04e: about 8.5 minutes.
Same workflow on a Galaxy S21: about 3 minutes.

The interesting part is the architecture.

The model is not a separate browser service arriving at Amazon as an agent. It runs locally and perceives and acts through an existing user browser session.

It also doesn't receive screenshots or raw HTML. It gets a compact structured browser perception layer and makes the small decisions needed at each step.

That changes the access problem from:

"How does a website identify and admit an AI agent?"

to:

"What is allowed inside an existing user browser session?"

The broader idea is Browser-as-Shared-Space, BaSS.

The browser remains the user's space, with the model working alongside the user rather than replacing the user with a separate autonomous browser agent.

I've tinkered some more with my fork of NInfer for the Qwen 3.8 Flash Next model on RTX6000, and I've just hit 400tg/s under ideal conditions with it, so I thought it was worth a share.

This is NVFP4 MoE's and the rest is either 16-bit or 8-bit, so it uses full 96GB VRAM and ngram on disk.

Decode MTP3 with --lm-head-draft

| Context | 16-bit | 8-bit | Change |
|---:|---:|---:|---:|
| 512 | 197.1 tok/s | 274.8 tok/s | +39% |
| 8K | 303.1 tok/s | 401.3 tok/s | +32% |
| 64K | 291.7 tok/s | 380.3 tok/s | +30% |
| 128K | 282.6 tok/s | 368.0 tok/s | +30% |
| 256K (maximum) | 277.4 tok/s | 360.6 tok/s | +30% |

Prefill

| Prompt length | 16-bit | 8-bit | Change |
|---:|---:|---:|---:|
| 512 | 6,709 tok/s | 5,905 tok/s | -12% |
| 8K | 13,903 tok/s | 13,908 tok/s | 0% |
| 64K | 13,043 tok/s | 12,171 tok/s | -7% |
| 128K | 11,927 tok/s | 11,032 tok/s | -8% |
| 256K (maximum) | 9,941 tok/s | 9,154 tok/s | -8% |

More benchmark variants in the readme in the repo

The original NInfer is for 5090 cards 32GB and variants below that, but I was both missing Qwen 3.8 Flash Next in it (when I started the fork) and something that could properly use a RTX6000 96GB card. The performance and options in VLLM and llama.cpp offerings just didn't really cut it for me, so I've vibed on this for some weeks now.

This fork supports both the NVFP4 quants from "radixark" and the "Swift 1.5" variant with 'less thinking but same results' post-training. With non-experts downsampled from 16-bit to 8-bit, MTP3 and smaller drafting head you get up to 400 tokens per second. You can also opt not to do the downsampling at a performance cost, but a bit higher quality.

Vision is also supported. Have fun.

https://github.com/lkarlslund/ninfer6000

Seeing how Strata brings avg. 2x the performance over llama.cpp using Qwen3.8-Flash-Next, is anyone working on something similar for GLM-5.3-Flash?

After trying the GLM-5.3-Flash online couple of times and it delivering clear solutions for my use case (compared to Claude or ChatGPT), I'd love to be able to run it locally (if at all possible)...7J13/256GB/3x3090.

PrismML's Ternary Bonsai 2 27B fits a 24 GB card or Mac, but decodes at \~30 tok/s on an L4 and \~21 on an M4 Pro. z-lab's DFlash 2 drafter was trained on bf16 Qwen3.8-27B, so it guesses worse on the ternary model. I fine-tuned it on 1.5M tokens of Bonsai 2's own greedy output.

• Drafter (safetensors + Q4\_K\_M GGUF): https://huggingface.co/naklitechie/Qwen3.8-27B-DFlash2-ternary-bonsai2
• Setup for all three runtimes: https://github.com/NakliTechie/dflash-mlx-bonsai2/blob/main/docs/USAGE.md

NVIDIA (PrismML's llama.cpp fork, prism branch):

llama-server -m Ternary-Bonsai-2-27B-PQ2_0.gguf -md Qwen3.8-27B-DFlash2-r3-Q4_K_M.gguf \
--spec-type draft-dflash --spec-draft-n-max 7 --spec-type ngram-mod \
-ngl 999 -ngld 999 -fa on --jinja

One L4, greedy: GSM8K 2.17x, MBPP 2.17x, MATH-500 2.20x, MT-Bench 1.39x. Code edits: 3.15x with ngram-mod stacked (drafter alone 2.46x). Accuracy within 1-2 problems per set.

Mac: a fork of bstnxbt/dflash-mlx with an 8-row 2-bit Metal GEMM for the verify step. M4 Pro: 1.5x on raw code completion, 1.3x on chat code, 1.2x on math. One script gives you an OpenAI-compatible server.

Browser: a WGSL port inside LocalMind (https://localmind.naklitechie.com), on by default for Bonsai 2 27B. 1.18x on code, output identical.

Chat and prose are about break-even. Use temperature 0.

Credit to z-lab (DFlash 2), PrismML (Bonsai 2, llama.cpp fork) and bstnxbt (dflash-mlx). Numbers are from one L4 and one M4 Pro; results from a 3090, 4090 or other Apple chips are welcome.

Hey guys, it's Merve from Hugging Face! I've recently given a talk in a dev conference about llama.cpp + but also covering basic concepts like prefill vs decode, memory types, speculative decoding etc. you can use it if you feel like it and I appreciate if you can give attribution! Find it in comments.

Interfaze 1 Lite is a mixture-of-architectures (MoA) model for developer workloads: reading documents, transcribing speech, locating objects and interface elements, and answering over all of it with structured output.

A single vision-language reasoning core works alongside a set of specialist architectures, each built for one kind of perception. The core reads the request, decides which specialists to run, and composes the answer from what they return. The whole model runs on one 80 GB GPU with no external services.

Key features: Document understanding, Speech transcription, Open-vocabulary object detection, Structured output, Translation, forecasting and guardrails, Multilingual reasoning.

Looped language models repeatedly apply a weight-shared block to increase effective depth without increasing parameter count, but the resulting T sequential recurrent-block calls per generated token substantially increase decoding latency. To address the issue, we introduce Wavefront Decoding (WFD), a training-free self-speculative decoding framework designed for looped language models. WFD exploits two properties of these architectures: intermediate recurrence outputs provide effective draft predictions, and weight sharing allows token states at different positions and recurrence depths to be processed in one batched recurrent-block call. WFD organizes these mixed-depth states into a diagonal wavefront, continuously drafting new positions at shallow depth while advancing earlier positions toward full-depth verification. Unlike the phase-separated draft-then-verify schedule, WFD therefore concurrently batches drafting and verification within the same recurrent calls, while rejected drafts are corrected using full-depth predictions. Across six Spec-Bench task categories, WFD achieves 2.42x speedup on Ouro-2.6B and 3.54x on Huginn-3.5B over autoregressive decoding, consistently outperforming draft-then-verify. Cross-recurrence KV sharing further reduces wavefront KV traffic and increases WFD's speedup to 4.81x on Huginn-3.5B. The code is available at this https URL.

• arXiv : https://arxiv.org/abs/2609.23033
• PDF : https://arxiv.org/pdf/2609.23033
• GitHub : https://github.com/summerbro-hhj/wavefront-decoding

By classic benchmark, the flash next is supposed to be slightly superior to its dense counterpart. But they are really incomparable. For my very simple workflows (spec -> implement -> review <-> rework), I feel that 27b is just better quality.

For context, and making things worse, I am comparing quantized 27b versus cloud flash next.

\- self hosted unsloth/Qwen3.8-27B-GGUF:Q4\_K\_XL (stock llamacpp with 130K context window)
\- alibaba cloud (qwen individual token plan), context capped at 256K in the harness

The metrics for my quality is actually very simple, I measure the number of review / rework needed before a PR is ready for me to read. The tasks are all very simple with a tight scope. Usually 27b do the work in \~2 iterations, flash next needs \~5. And it is not only about the number of iteration.
On the review flash next is overly verbose on half baked PR comment, where 27b is more straight to the point. In the end, the code produced is on par, to be frank. But if we look at token consumption...

Side notes, on pairing session, I got some deep hallucination using "/skills:diagnosing-bugs" + flash next. But since it is "bugs" and they not really comparable chunk of work, it is hard to say.

And I can't be the only one feeling that right ? Are you feeling the same ?

PS: of course I followed the hype and jumped on Strata. After the initial "oh my god it's so fast", I switched back to 27b. Tried all quant from "ISTA-DASLab" as well as experiemental from unsloth (Q4\_K\_L). With ISTA-DASLab, It actually is the first time I had "tool call error" in pi (which stop the agent), multiple times.

This is big. Imagine running your model across multiple computers, and now imagine doing tensor parallelism across all of them.

My Gemma4 31b seems to be breaking down in 'lalala' or just looping indefinitely for the past 3-4 days. Never happened before

https://preview.redd.it/tk7u2x2c9xth1.png?width=235&format=png&auto=w…

There was no 'lalala' in the whole scenario, I have no idea where it came from. Nor was there any skipping, humming or perfection. There were shivers down the spine of course, but it is still weird.

Building a fully local RAG setup for personal notes (life-logging second brain, single user, privacy is the whole point so no hosted APIs for the data). Stack is SQLite + sqlite-vec + FTS5, Ollama for embeddings and generation, all on an Apple Silicon Mac.

Two questions where I'd love real 2026 experience:

1. Embedder pick. I was defaulting to nomic-embed-text out of habit, but recent chatter favors qwen3-embedding:0.6b or embeddinggemma at similar sizes. Anyone benchmarked these head-to-head for English personal-notes retrieval (not BEIR)? Does it actually matter at \~50k chunks, or am I bikeshedding?

1. Hybrid fusion. FTS5 (BM25) + vector cosine, per-query score normalization. FTS5 has no typo tolerance — anyone wired in trigram + spellfix and found it worth it? Any fusion gotchas at small corpus sizes (e.g. vector signal drowning out keyword on exact-match queries)?

Corpus is Markdown notes + JSON records, chunked \~512 tokens. Query load is one human. Not chasing SOTA, chasing "correct answers on my own data."

What would you change?

I'm seeing these show up on eBay for $50-$60. wouldn't expect anything ground breaking from it, but at that price, it seems like it could be interesting to play with on an extra pcie port. just curious if anyone's already done that

I got Qwen3.8-27B running at \~18 tok/sec decode & \~500-600 tok/sec prefill (at 64k context) on just 12GB VRAM + 8GB RAM, using ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3\_S quant (\~11GB). This recipe also works with 0bserverx’ Qwen3.8-27B-Heretic-GSQ-RCO IQ3\_S quant, achieving similar speeds (about a 7% loss).

This is the best Qwen3.8-27B quant I’ve tested so far (and I’ve tried everything), and for it to fit in such limited RAM/VRAM is wild. GSQ-RCO quantization is magic, it performs very close to the full precision weights in all of my testing.

The reason it fits at all is Qwen3.8 is hybrid, so only 16 of the 64 layers need KV cache. With q4\_0 for cache the full 64k is only about 1.1GB instead of 4GB for f16.

I'm on a 9900X + 4070S 12GB + 32GB RAM for reference, using stock llama.cpp. Settings are -ngl 58 -ot token\_embd=CPU -ctk q4\_0 -ctv q4\_0 -c 64000.

Full build + serve scripts and all my numbers are here if you’d like to reproduce yourselves: https://github.com/bodhi37/Qwen3.8-27B-12GBVRAM-Recipe

I’ve been building this for a long fucking time, and tonight I finally released Memoria 1.0.0.

I built it because I actually wanted to use it. I wanted a real memory layer for local LLM applications that didn’t depend on a specific model, a cloud service, or an API key.

Memoria is local-first and LLM-agnostic. It can run without an LLM at all.

The machine I built and benchmarked it on is not exactly impressive. It’s an Intel Celeron N4020 running at 1.10 GHz, with around 3.7 GiB of usable RAM, no GPU, and Debian Linux.

On LongMemEval-S, 468 out of 470 retrieval-evaluable questions returned results. Recall@1 was 89.8%, Recall@5 was 97.9%, Recall@10 was 98.9%, and Recall@50 was 99.6%. Session NDCG@10 was 0.9257.

Peak RSS for the full LongMemEval workload was 2.65 GiB. Average peak RSS for an individual query was around 580 MiB.

The retrieval system is not just throwing everything into a vector database. Memoria runs FAISS, BM25, graph retrieval, phrase matching, attribute retrieval, and temporal retrieval in parallel. Those signals get fused and then passed through multi-signal ranking.

Temporal retrieval is independently implemented too, so I can measure it and ablate it instead of having it baked into the base retrieval path. It’s usable, but it’s still under active work.

There’s a bunch of other stuff in the release as well. GitHub repository ingestion, Obsidian vault ingestion, MCP support, a CLI, TUI, GUI, and API, persistent local storage, LongMemEval and LoCoMo benchmark tooling, and a plugin system with 11 subsystems and 34 hooks. There’s also an interactive plugin generator now.

And it’s actually installable:

pip install kitzkatz-memoria

GitHub: https://github.com/Kitzkatz/memoria

Docs: https://kitzkatz.github.io/memoria/

PyPI: https://pypi.org/project/kitzkatz-memoria/

I wasn’t going to wait around for a perfect time to ship it.

It’s 1.0.0.

If you’re working on local agents or local LLM applications, I’d genuinely like to hear what you think and would appreciate any feedback. Please break it

guys do you still remember K2 Horizon?

https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-GGUF

https://huggingface.co/IFM/K2-Horizon-32B-GGUF

https://huggingface.co/IFM/K2-Horizon-3.7B-GGUF

https://huggingface.co/IFM/K2-Horizon-7B-GGUF

https://huggingface.co/IFM/K2-Horizon-0.9B-GGUF

previous discussion (last month!) here https://www.reddit.com/r/LocalLLaMA/comments/1w67wso/ifmk2horizonmova36ba4bgguf\_hugging\_face/

We released Aplomb 1 today, a 5.3B decision model with open weights. It reads up to 1M tokens of text, JSON, images, video and audio in a single request, and on our API it makes a decision on a 1M-token document in about 3 seconds, at $0.02 per 1M input tokens with free output and ZDR by default.

On Decision Index 0.2.1 it scores 44.86 on our run of the official kit, #1 among 4B models on the published board, and it has the top score among models up to 5.3B on 8 of the 38 benchmarks, including MMLU-Pro, GPQA Diamond and BBH. We've submitted it to the board, and the full run is public. It also scores 77.5% on JevBench Hard and averages 75% zero-shot intent accuracy across 51 languages on MASSIVE.

As far as we know, it's the only decision model that returns probabilities for tool arguments, reads 1M tokens, or takes text, images, video and audio together. Tool selection gives a probability for every tool and for each enum and boolean argument in one request: on "Order B-44120 arrived with a cracked screen. I want my money back." with three tools, it picks issue\_refund at 0.969, reason "damaged" at 0.993 and full\_refund true at 0.761, so an agent can act on confident calls and hand the rest to a larger model. Any question can also return the probability that the input doesn't contain the answer.

The 1M-token speed comes from a long-context mode in our own inference runtime: about 3 seconds instead of about 111 for a full read, and it answered all 525 decisions in our long-context tests correctly. It's currently only available on our API, so the open weights read every token. On the API, a short question takes about 15 ms of model time (around 200ms e2e latency), and the OpenAI, Anthropic and Gemini formats work alongside our Decisions API.

Aplomb 1 is built on Qwen3.5-4B with the audio encoder from Qwen3-Omni-30B-A3B-Instruct, both Apache-2.0. We extended the window from 262K to 1M tokens, added our own decision head and trained the model for decisions. Thanks to the Qwen team. Disclosure: the training data included the public train splits of WinoGrande and ContractNLI, two of the 38 index benchmarks.

The weights run in bf16 on about 12 GB of GPU memory with the reference script, under the EmpirioLabs Model License, which is free for research, evaluation, personal use and internal use at companies under $1M in annual revenue.

Weights: https://huggingface.co/empiriolabsai/aplomb-1

Blog with the full tables: https://empiriolabs.ai/blog/introducing-aplomb-1

Docs: https://docs.empiriolabs.ai/models/aplomb-1

Playground: https://platform.empiriolabs.ai/dashboard/playground?model=aplomb-1

Okay, when my smoke test showed GLM 5.3 Flash to be less sycophantic than GLM 5.3 I thought that was maybe just my reading.

But now I went testing models on "what happened in Tiananmen in 1989". I use nano-gpt.com which at first had different providers as a confounding factir; eventually I locked to one provider, Novita, which is clearly not in China, And here is what I get.

DeepSeek 4.1 Flash, DeepSeek 4 Pro refuse.

Hy3 not justy refuses but it is a content filter refusdal (on Novita? so they somehow built it into the m odel itself)

Kimi K3 and GLM 5.3 offer slogans, but when I give them a nudge - "and without the slogans?" - give a decent overview. GLM 5.3 also outright refused sometimes but that was before I locked provider, so migth have been Zhipu's server.

GLM 5.3 Flash tends to work from the start, though did get to slogans once.

In my previous sycophancy smoke tests GLM 5.3 Flash was also less sycophantic than GLM 5.3.

Did they somehow distill a GOAT or what?

&#x200B;

They're all fixes in rocm and llama server. Let me know if this interests someone. I'll push it to GitHub and share my configs. I have a Lenovo p620 running 4 7900xtx on gen 4.0 x16 slots.

One line summary of each fix:

\- Async mirrored input uploads: \~30 per-token inputs staged in a pinned ring on per-GPU streams instead of synchronous round trips (Flash-Next decode 24.6 -> 36.3 t/s).

\- Per-device dispatch threads: each GPU's kernels and collectives launched by its own host thread instead of one thread for all four.

\- One-shot PCIe P2P AllReduce: GPUs write slices straight into peers' memory for small tensors, replacing RCCL (\~57 us -> \~9 us per allreduce).

\- Fewer kernels in MTP decode: fused same-shape copies and leaner conv-state rollback (88.8 -> 92.7 t/s).

\- mmvq small-K row packing (RDNA3): short-K projections no longer leave most of the block idle (22.7 -> 25.3 t/s).

\- Wide-K mmvq for multi-token batches: K split over 8 warps for MTP verify on 10240x320 projections (\~92.6 -> \~96 t/s).

\- MoE vector kernel up to 8 tokens: 5-token MTP verify batches stay on the fast vector path instead of MMQ (80.9 -> 86.7 t/s).

\- Small-K multi-token MoE kernel: several rows per warp for short expert down-projection slices (94.2 -> 95.2 t/s).

\- Wide mmvf blocks: 512/1024-thread blocks for tiny long-K F32 matrices (40.6 -> 41.2 t/s).

\- Q8\_1 activation registry: each activation quantized once and shared by all matmuls that read it (\~+2 t/s).

\- Fused hyper-connection chains: scale/sigmoid/scale/hc\_post and scale/silu each in one kernel, \~380 fewer kernels per token (38.5 -> 40.6 t/s).

\- Thin-F32 prefill kernel: <=16-row F32 matmuls off generic SGEMM (401 -> 21 us; pp4096 1602 -> 1836 t/s).

\- Compact MoE tile list: expert matmul launches only (expert, token-tile) pairs with work instead of a 96%-empty grid (down 928 -> 405 us, gate/up 516 -> 360 us).

\- Multi-warp MoE routing helper: 16 warps per expert sort tokens in two passes (99 -> 25 us per call).

\- Q4\_K expert tile shape: 32-row tiles for 160-row expert slices (360 -> 337 us).

\- Stream-k for few-tile Q8\_0 matmuls: 10240->320 projections spread over all CUs instead of 12 workgroups (284 -> 115 us).

\- No 64-bit div/mod in hyper-connection kernels: 3-D grid instead of emulated integer division per element (230 -> 79 us; pp2048 1593 -> 1868 t/s with stream-k).

\- Split-K router GEMM: 512x512 F32 router GEMM as 8 K-chunks plus a sum (158 -> 69 us).

\- MTP re-reserve fix: graph re-reserved when MTP outputs turn on, ending a full GPU realloc+sync per prefill chunk (88.5K prefill 1042 -> 1130 t/s).

\- MTP draft prompt window: draft head prefills only the last 2048 prompt tokens (prefill 1130 -> 1249 t/s, decode at depth 46.6 -> 64.9 t/s).

\- Draft ubatch cap: draft compute buffer 457 -> 247 MiB, fixing a GPU0 out-of-memory crash at 96K context.

\- Gathered sparse attention (QSA): decode attends only to the \~2K selected tokens instead of masking the whole cache (decode at 48K 67.9 -> 80.6 t/s).

\- Per-layer embedding table in RAM (--lazy-mode off): 26.8 GB hashed embedding table kept resident instead of read from disk every pass (lookup 1.0-1.6 -> 0.1-0.3 ms, \~3-4% decode).

\- Meta backend subgraph fix: per-device subgraphs sized for the largest graph, fixing a segfault when graph shapes change between calls.

\- Net result, Qwen3.8-27B Q8 on 4 GPUs: code decode 54-61 -> 96-110 t/s, 51K prefill 1454 -> 1816 t/s, decode at 51K depth 57 -> 77 t/s.

A few days ago we shared the idea behind DynamicTune: transferring the trajectory flow from a larger teacher model directly into a smaller student via closed-form linear algebra in \~12 minutes on consumer hardware. Zero backpropagation, zero training tokens, zero gradient descent.

To eliminate local bias, we uploaded the unquantized FP16 checkpoint to Hugging Face, and TPN Bench (TaoFu Protocol) independently evaluated it on a datacenter NVIDIA L4 GPU using the official lm\_eval 0.4.12 framework (coordinator run ce494664-d077-4ff1-8741-15cedabc434c). Huge thanks to TPN Bench for the cloud GPU compute!

Here are the independent numbers on full ARC-Challenge (1,172 items, zero-shot, greedy temp 0):

\* Stock Qwen3.5-0.8B Base (unquantized BF16): 37.50% acc\_norm (34.60% acc)

\* 3-epoch SFT distillation (Mythos-0.8B, 25k Claude pairs, multi-GPU DDP): 38.10% acc\_norm (35.80% acc)

\* SFT + Model Soup Merge: 37.00% acc\_norm (catastrophic forgetting)

\* Stock Qwen3.5-2B Base (2.5x larger model, Q8): 41.10% acc\_norm (37.80% acc)

\* DynamicTune 0.8B Base (Ours, 4-anchor closed-form surgery): 42.15% acc\_norm (40.19% acc)

WHY THIS IS COMPLETELY INSANE:

1. A 0.8B model physically beat a 2.5x larger 2B model:

In LLM scaling, parameter count is supposed to be king. An 800M model is not supposed to beat an uncompressed 2B model on ARC-Challenge (42.15% vs 41.10%). By extracting trajectory dynamics from 4B and pulling them back into the student SwiGLU blocks, higher-order reasoning is compressed directly into edge weights.

1. Zero backpropagation beat 25,000 SFT instruction pairs:

A recently published project (kmamine/merge-corrected-sft-distillation-Qwen-Mythos-0.8B) trained Qwen3.5-0.8B across 3 epochs on 25,000 Claude reasoning pairs on a multi-GPU cluster, reaching 38.10% before overfitting. DynamicTune reached 42.15% with zero gradient descent, zero loss functions, and zero training tokens.

1. Ironclad 3.23-sigma statistical significance:

A delta of +4.65% across 1,172 questions with stderr +-1.44% gives a Z-score of 3.23sigma (p < 0.001). This is not prompt tuning noise or random variance.

1. 12 minutes on consumer hardware vs datacenter verification:

The weight surgery was solved locally in \~12 minutes on an 8GB AMD RX 580 using layer-streaming (loading each layer in FP16, computing closed-form SVD deltas, and dumping to RAM). But the benchmark was conducted 100% in the cloud on datacenter NVIDIA L4 hardware via TPN Bench.

WHY PAST ATTEMPTS FAILED: THE SPECTRAL ENTROPY BARRIER

If you blindly apply weight deltas across all 24 layers of the student, the model collapses (+64.78% NLL explosion).

When we scanned all 24 layers calculating the normalized spectral entropy H (from 0.0 to 1.0) of the representation residuals:

\* Layer 0 (H = 0.71): Clean semantic grounding. High receptivity to trajectory alignment.

\* Layers 1-22 (H between 0.90 and 0.96): Chaotic superposition knots. In an 800M model with only 1024 dimensions, polysemantic features are crammed into dense superposition. Forcing linear updates here causes catastrophic interference.

\* Layer 23 (H = 0.93): Pre-unembed boundary where features unpack toward vocabulary logits.

By restricting surgery to 4 sparse anchor blocks (layers 0, 7, 15, and 23) and using damped Levenberg-Marquardt Tikhonov pseudoinverse + adaptive spectral rank truncation, we protect the fragile superposition knots while imparting corrective trajectory velocity.

REPRODUCIBILITY & WEIGHTS

Everything is 100% open source and available to test right now:

\* GitHub Repository: https://github.com/dsadawq3/DynamicTune

\* Base Model (Safetensors): https://huggingface.co/F-Labs/Qwen3.5-0.8B-DynamicTune-Base

\* GGUF Checkpoint (FP16): https://huggingface.co/F-Labs/Qwen3.5-0.8B-DynamicTune-Base-GGUF (Qwen3.5-0.8B-DynamicTune-Base-F16.gguf, SHA256: d77cf505108271d72f28298f20c2d158e7aeaf50cc22987db05a9a8973e08709)

To run inference locally with standard llama.cpp:

llama-cli -m Qwen3.5-0.8B-DynamicTune-Base-F16.gguf -p "Question: How does DNA replication initiate?\\nAnswer:" -c 2048 -n 128

Special thanks to TPN Bench (TaoFu Protocol) for providing the independent datacenter NVIDIA L4 evaluation resources.

Clone the repo, run your own benchmarks, and test it yourself.

$5995 - Ships in November. N1X brand of GB10 Chip. Says it will have WSL out the door which I presume will be Ubuntu + Cuda beneath in addition to all the CoPilot/GitHub native stuff for Windows.

Hopefully they have supply to saturate the market and put in pricing pressure. Knowing that the GB10 Spark shines with 2 or more, its unfortunate they didn't bring over ConnectX7 support. 10gb ethernet is nice, but not the same.

I prefer running llama.cpp Vulkan prebuilt binary. I just download the latest version and ready to roll. I finally took the hours necessary to get TheRock latest tarball version of ROCm 10.2 running on dual AMD Radeon Instinct MI50 gfx906 (32gb combined VRAM).

Same models benched in previous post. A mix of Dense and MoE models and quants that better utilize available VRAM.

• llama_bench_Swift-Qwen3.8-27B-Uncensored-MTP.Q6_K.gguf
• llama_bench_Swift-1.5-Qwen3.8-27B-Q6_K.gguf
• llama_bench_Gemma-4-MoonGem-31B.i1-Q6_K.gguf
• llama_bench_gemma-4-31B-it-UD-Q6_K_XL.gguf
• llama_bench_Nemotron-3.5-30B-A3B-Antislop-FTPO.i1-Q5_K_M.gguf
• llama_bench_Laguna-XS-2.1-APEX-I-Balanced.gguf
• llama_bench_Agents-A1-Q4_K_M.gguf
• llama_bench_Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf
• llama_bench_Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf

Here is the backend performance comparison contrasting the native ROCm (v10.2 for gfx906) runtime against your optimized Vulkan (MESA\_PPA\_26.2) baseline. The data highlights a massive architectural split: ROCm significantly accelerates token generation across the board but suffers high variance and regressions in MoE pre-fills.

Both GPUs are power limited to 145 watts. Build versions used:

llama-b11382 used for Vulkan (prebuilt ubuntu binary)
llama-b11401 used for ROCm (compiled with proper flags)

Architectural Performance Breakdown: Vulkan vs. ROCm

|Model|Size|Params|Test|Vulkan Baseline (t/s)|ROCm 1st Run (t/s)|Performance Delta (%)|
|:-|:-|:-|:-|:-|:-|:-|
|qwen35 27B Q6\_K|20.88 GiB|27.32 B|pp512 tg128|149.30 ± 0.15 18.35 ± 0.01|176.86 ± 17.28 20.21 ± 0.65|\+18.46% +10.14%|
|qwen35 27B Q6\_K|22.21 GiB|27.32 B|pp512 tg128|163.48 ± 0.18 18.52 ± 0.03|183.49 ± 15.75 20.04 ± 0.63|\+12.24% +8.21%|
|gemma4 31B Q6\_K|23.46 GiB|30.70 B|pp512 tg128|119.98 ± 0.12 15.72 ± 0.02|172.93 ± 0.89 17.26 ± 0.13|\+44.13% +9.80%|
|gemma4 31B Q6\_K|25.62 GiB|30.70 B|pp512 tg128|133.67 ± 0.24 16.38 ± 0.02|178.59 ± 2.31 17.50 ± 0.09|\+33.61% +6.84%|
|nemotron\_h\_moe 31B|25.18 GiB|32.91 B|pp512 tg128|834.87 ± 1.72 63.82 ± 0.07|736.98 ± 134.77 103.86 ± 0.50|\-11.73% +62.74%|
|laguna 30B.A3B Q5\_K|22.64 GiB|33.44 B|pp512 tg128|723.34 ± 2.99 58.98 ± 0.28|856.14 ± 29.71 76.81 ± 0.20|\+18.36% +30.23%|
|qwen35moe 35B Q4\_K|19.70 GiB|34.66 B|pp512 tg128|957.62 ± 3.88 51.50 ± 0.07|834.58 ± 102.56 69.70 ± 0.22|\-12.85% +35.34%|
|qwen35moe 35B Q5\_K|24.76 GiB|34.66 B|pp512 tg128|909.20 ± 5.71 53.18 ± 0.07|763.93 ± 106.19 68.47 ± 0.34|\-15.98% +28.75%|
|qwen35moe 35B Q6\_K|28.53 GiB|34.66 B|pp512 tg128|763.05 ± 66.01 52.81 ± 0.04|754.82 ± 70.48 67.29 ± 0.21|\-1.08% +27.42%|

Core Insight Strategy & Bottlenecks

1. Token Generation (tg128) Dominance: ROCm dominates pure text generation. The native AMD matrix kernels unleash your MI50 computation potential, unlocking a massive +62.74% boost for Nemotron but taking a small hit on pre-fill -11.73%.
2. Dense Model Pre-fills (pp512): Dense architectures scale cleanly under ROCm. Gemma 4 sees a +33% to +44% processing throughput spike over the Vulkan RADV driver driver bounds. MoE models take a hit with a -15.89% difference with llama\_bench\_Qwen3.6-35B-A3B-UD-Q5\_K\_XL.gguf being the happiest with Vulkan backend.
3. The MoE Prompt Processing Delinquency: Notice the massive standard deviations under ROCm for MoE pre-fills (e.g., Qwen3.5MoE Q4\_K has an instability block of ± 102.56). ROCm suffers from severe scheduling thrashing when building prompt streams across multiple active experts.

Here is the structured layout with pp512 and tg128 separated into individual columns for a clean side-by-side comparison between the two backend architectures.

Backend Comparison Table (Vulkan vs. ROCm)

|Model|Size|Params|Vulkan pp512 (t/s)|ROCm pp512 (t/s)|Vulkan tg128 (t/s)|ROCm tg128 (t/s)|
|:-|:-|:-|:-|:-|:-|:-|
|qwen35 27B Q6\_K|20.88 GiB|27.32 B|149.30 ± 0.15|176.86 ± 17.28|18.35 ± 0.01|20.21 ± 0.65|
|qwen35 27B Q6\_K|22.21 GiB|27.32 B|163.48 ± 0.18|183.49 ± 15.75|18.52 ± 0.03|20.04 ± 0.63|
|gemma4 31B Q6\_K|23.46 GiB|30.70 B|119.98 ± 0.12|172.93 ± 0.89|15.72 ± 0.02|17.26 ± 0.13|
|gemma4 31B Q6\_K|25.62 GiB|30.70 B|133.67 ± 0.24|178.59 ± 2.31|16.38 ± 0.02|17.50 ± 0.09|
|nemotron\_h\_moe 31B|25.18 GiB|32.91 B|834.87 ± 1.72|736.98 ± 134.77|63.82 ± 0.07|103.86 ± 0.50|
|laguna 30B.A3B Q5\_K|22.64 GiB|33.44 B|723.34 ± 2.99|856.14 ± 29.71|58.98 ± 0.28|76.81 ± 0.20|
|qwen35moe 35B Q4\_K|19.70 GiB|34.66 B|957.62 ± 3.88|834.58 ± 102.56|51.50 ± 0.07|69.70 ± 0.22|
|qwen35moe 35B Q5\_K|24.76 GiB|34.66 B|909.20 ± 5.71|763.93 ± 106.19|53.18 ± 0.07|68.47 ± 0.34|
|qwen35moe 35B Q6\_K|28.53 GiB|34.66 B|763.05 ± 66.01|754.82 ± 70.48|52.81 ± 0.04|67.29 ± 0.21|

So yes it's worth the hassle of jumping through hoops to get ROCm working on MI50 setups. At least I have 2 backends working. Next up I'll try some RPC.

https://preview.redd.it/ov4f9qmi2rth1.png?width=731&format=png&auto=w…

I'm sharing this project (honestly 2nd time) for anyone who wants to run ultra-long context tasks with high-precision quantizations, especially for heavy coding.

TL;DR: In vanilla llama.cpp, -c N allocates a physical KV buffer for all N tokens up front, including promprts and KV caches. In my fork (focus-llama) the logical context stays at 256k, but the physical KV buffer is capped (--kv-cache-size, \~85k cells). Older chunks are offloaded to an external store and pulled back on demand. This runs a 27B Q6 model with 256k logical context on one 5090, at roughly 100–120 t/s depending on how full the context is.

How to?

Vanilla llama.cpp allocates a contiguous KV buffer for the full -c (VRAM), and every decode step attends over all tokens currently in context, so per-token cost grows roughly linearly with context length (only the attention part; the weights matmul is constant). The two problems are separate: the allocation wastes VRAM, and the growing context slows decoding. Initial speed is around 120 t/s, dropping to 60 t/s as context grows.

focus-llama attacks both: the physical buffer is capped (\~85k cells) so VRAM is bounded, and since the number of resident cells can't exceed the buffer, per-step attention cost is bounded by the buffer size instead of the logical context length. It based on 2 techniques declarative attention and skill.state, introduced by google deepmind. I've spent the past two weeks ironing out bugs, and now that it has stabilized, I'm honestly blown away.

On a single RTX 5090 + Qwen3.8-27B UD-Q6\_K\_XL, adaptive MTP speculative decoding), it shows average 110+ t/s and 256k logical context with a \~85k physical buffer.

Configuration is somewhat tricky (focus-memory: kv cache store also required) but,

You'll see the MAGIC in action: context usage stays capped at around 18–30%, token generation speeds remain consistently high, and you'll never hit full-day compaction pauses when running coding harnesses like Cline or Qwen Code.

Link? https://github.com/edwardyoon/focus-llama/blob/master/README.md

My conf:
-m /home/edwardyoon/my_model/qwen3.8/Qwen3.8-27B-UD-Q6_K_XL.gguf \
--alias qwen27b \
-ngl 99 \
-b 1024 \
-ub 1024 \
-c 200000 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--da-auto \
--kv-unified \
--da-min-ctx 2048 \
--da-chunk-tokens 4096 \
--fm-offload \
--kv-offload-threshold 36864 \
--kv-offload-holes \
--kv-cache-size 85536 \
--kv-retain-tokens 6000 \
--sparse-gate-threshold 60 \
--focus-memory-host http://192.168.219.124:3900 \
--focus-memory-token focus-memory-local \
--spec-type draft-mtp-adaptive \
--spec-draft-n-max 6 \
--spec-draft-ngl all \
--spec-draft-type-k q8_0 \
--spec-draft-type-v q8_0 \

few journal logs:

llama-server[827038]: da_sparse[VEC]: SPARSE - gather bound 5120 of 15872 KV rows (32.3% of KV)
…
n_gen = 361, tg = 119.44 t/s, tg_3s = 119.77 t/s
n_gen = 729, tg = 120.98 t/s, tg_3s = 122.53 t/s
n_gen = 1083, tg = 119.71 t/s, tg_3s = 117.18 t/s
n_gen = 1497, tg = 123.98 t/s, tg_3s = 136.71 t/s
n_gen = 1819, tg = 120.50 t/s, tg_3s = 106.61 t/s
n_gen = 2157, tg = 119.06 t/s, tg_3s = 111.85 t/s
n_gen = 2509, tg = 118.67 t/s, tg_3s = 116.32 t/s
n_gen = 2837, tg = 117.37 t/s, tg_3s = 108.35 t/s
n_gen = 3234, tg = 118.99 t/s, tg_3s = 131.94 t/s
n_gen = 3643, tg = 120.65 t/s, tg_3s = 135.62 t/s
n_gen = 3983, tg = 119.98 t/s, tg_3s = 113.25 t/s
n_gen = 4350, tg = 120.09 t/s, tg_3s = 121.34 t/s
n_gen = 4713, tg = 120.08 t/s, tg_3s = 119.89 t/s
n_gen = 5150, tg = 121.80 t/s, tg_3s = 144.11 t/s
n_gen = 5527, tg = 122.04 t/s, tg_3s = 125.37 t/s
n_gen = 5866, tg = 121.45 t/s, tg_3s = 112.57 t/s
n_gen = 6291, tg = 122.59 t/s, tg_3s = 140.83 t/s

We're currently building a tool around an idea we're trying to validate: using LoRA adapters as a way to give an LLM persistent, specialized capabilities that can be switched on and off when needed.

The basic idea is that instead of continuously putting a skill or domain-specific information into the model's context, you could encode some of it into a LoRA adapter.

For example, you might have separate adapters for:

• a coding skill
• a company's internal domain
• a specific writing style
• domain-specific knowledge
• task-specific behavior

…and load or unload those capabilities depending on what you're doing.

We're interested in this because it could potentially mean less context usage, reusable specialized capabilities, keeping different capabilities separated from the base model, and potentially lower inference costs for some use cases.

But we're not sure yet whether this is actually a useful product.

That's what we're trying to figure out before going too far with the build.

If creating and managing these LoRAs were as easy as creating and managing a knowledge base, would you actually use something like this?

I'm particularly interested in hearing where you think this approach doesn't make sense.

I am sure this gripe has been raised many times before, but every time a new model is released it seems like it's routinely only a few percentage points higher than previous models on benchmarks.

How do we know this is even a "real" difference and not just within the window of measurement error or noise?

Some quick back-of-envelope math: HumanEval has 164 problems, so a model scoring \~70% has a standard error of roughly 3.5 points from question sampling alone. GSM8K (\~1.3k questions) is closer to 1 point. A 2-point "improvement" on either is well inside the noise, and that's before counting anything else that varies: sampling temperature, prompt template, few-shot examples, eval harness version, and possible contamination. There's work showing that trivial formatting changes can swing scores by many points, which is often bigger than the gap between models on the leaderboard.

None of this is hard to fix. Report the number of items, bootstrap confidence intervals, and ideally multiple seeds. Since two models are scored on the same questions, a paired test is much more powerful than eyeballing two accuracies. Miller's "Adding Error Bars to Evals" lays this out well.

Am I missing something, or is a lot of the benchmark chasing just reading tea leaves? Does anyone know of leaderboards or labs that routinely report uncertainty?

I've been tinkering with GLM-5.3-Flash on two 64GB CMP 170HX cards for a while, and the setup is finally stable enough that I figured I'd share it.

I also compared it against the Qwen3.8-Flash-Next setup I've been using on the same machine: AWQ INT4 + FP8 PLE on vLLM.

Besides PP/TG benchmarks, I hooked both models up to DSH and gave them the same small coding/agent tasks to see how raw inference speed translated into actual task completion time.

A few caveats up front:

• this is not an apples-to-apples quant comparison
• GLM and Qwen are using different engines and different speculative decoding setups
• speculative decode speed depends heavily on acceptance rate and generated text
• the coding tasks are just a few practical examples, not a serious benchmark suite
• when I mention “Strata-style” below, I mean the HBM-first/full-residency approach I previously used with Strata, not that this is running Strata itself

Repo and playable demos:

GitHub:
https://github.com/Flun/glm53-flash-cmp170hx-exl3

Hardware

|CPU|Ryzen 5 5600X|
|:-|:-|
|RAM|80GB DDR4|
|GPU|2x CMP 170HX 64GB|
|GPU arch|SM80|
|PCIe|Gen2 x8|
|GPU P2P|unavailable|
|OS|Ubuntu 24.04|

Both models were tested on the same machine.

For the agent tests I used DSH as the harness.

GLM-5.3-Flash setup

Target model:

turboderp/GLM-5.3-Flash-exl3 3.05bpw

Engine:

ExLlamaV3 1.5.4

Current setup:

• GLM-5.3-Flash EXL3 3.05bpw
• \~125.2GB / 116.6GiB target weights
• target fully resident across the two 64GB cards
• k_hcfuse
• DFlash2 EXL3 6bpw
• DFlash2 K7
• Q8 KV cache
• 384K context in actual use
• max request budget around 392,960 tokens

GLM-5.3-Flash itself is a 320B-total / \~18B-active MoE model.

The important part here is that the target weights stay resident in HBM. I'm not continuously streaming experts from system RAM during decode.

About the 3.05bpw quality

This was probably the part I cared about most.

At first glance, “3.05bpw” sounds like a pretty aggressive quant, especially compared to the UD Q4 variants people commonly use.

But EXL3 isn't simply “make every tensor 3-bit”.

It uses a trellis-based quantization scheme with different bit allocation depending on the tensor. The 3.05bpw number is an average target bitrate.

The published quant configs for this family also keep more sensitive parts at higher precision. For example, lm_head remains at 6-bit in the 3.05bpw branch.

Looking at the same model family, the 4.05bpw build has been inspected with something roughly like:

• routed experts: K4
• attention: K6
• shared experts: K6
• dense MLP: K5
• lm\_head: K6
• embedding / norms / router: native

So the general idea is to compress the huge routed-expert portion more aggressively while spending more bits on the smaller/more sensitive paths.

That makes quite a bit of sense for a MoE model like this, because most of the storage is in the expert weights.

Rough comparison with the common UD quants

|Quant|Size|Top-1 agreement vs BF16|Mean KLD|
|:-|:-|:-|:-|
|UD-IQ3\_XXS|120.37GB|81.63%|0.28377|
|EXL3 3.05bpw (my target)|125.18GB|\~93.05% (estimated from published 3.0bpw results)|\~0.050 (estimated from published 3.0bpw results)|
|UD-IQ4\_XS|156.82GB|88.18%|0.11665|
|UD-Q4\_K\_XL|199.71GB|92.22%|0.04929|
|UD-Q5\_K\_XL|240.31GB|94.35%|0.02705|

For reference, public GLM-5.3-Flash GGUF fidelity numbers look roughly like this:

One thing worth pointing out is that something named Q4_K_XL is not literally “4 bits per parameter across the entire model”.

At \~200GB for a 320B model, it's a mixed-precision quant with an effective average bitrate much higher than 4bpw.

There is also a published GLM-5.3-Flash EXL3 3.0bpw fidelity test using 51,175 held-out next-token positions that reported:

• Top-1 agreement: \~93.0%
• Mean KLD: \~0.0505

Numerically, that's in roughly the same neighborhood as the published UD-Q4\_K\_XL result.

That said, I would not claim that “EXL3 3bpw is better than UD-Q4\_K\_XL” from those numbers alone.

They were not measured through the exact same evaluation pipeline/corpus, and the public 3.0bpw artifact isn't the exact same quant I'm running either.

My takeaway is simply that 3bpw-class EXL3 can preserve a surprising amount of fidelity for its size, and it doesn't behave like a naive 3-bit quant.

For my use case, getting the target down to \~116.6GiB while still retaining usable coding/agent quality was the main reason this setup was interesting.

DFlash2 6bpw is only the drafter

Just to avoid confusion:

the 3.05bpw model is the actual GLM target.

The 6bpw DFlash2 model is only the speculative drafter.

The drafter proposes tokens, and the GLM target verifies them.

So this is not some kind of “3.05bpw + 6bpw averaged quality” setup.

The draft quant mostly affects draft speed, VRAM use, and acceptance efficiency.

HBM-first / “Strata-style” part

This is where I borrowed an idea from a Strata setup I had used previously.

Again, this does not run Strata.

What I mean by “Strata-style” is simply:

keep as much of the model permanently resident in HBM as possible, and avoid runtime CPU↔GPU weight traffic

The GLM target itself fits across the two cards, so I leave the target fully resident.

Instead of offloading experts, I focused on reducing the memory used by the parts that scale with context: KV cache and the speculative drafter.

For this particular machine that made more sense to me than constantly moving weights over PCIe.

DFlash2 + 384K context

I originally used GLM's MTP d2 path.

Later I switched to DFlash2.

Instead of keeping the original BF16 incoai/GLM-5.3-Flash-DFlash2 drafter, I converted it to an ExLlamaV3-compatible EXL3 6bpw build.

The resulting draft weights are about 0.96GiB.

The bigger problem at long context was actually the draft KV cache.

If the target is running 384K and the drafter also grows a 384K KV cache, VRAM disappears quickly.

So I changed the drafter side to use a fixed SWA window plus a GPU ring cache.

The target still sees the full 384K context and keeps its full target KV.

Only the drafter's KV storage is kept inside a bounded ring.

That's what lets the current setup run:

DFlash2 K7 + Q8 KV + 384K target context

without growing the draft cache to the full target length.

The implementation and validation tests are in the repo.

Cold start

I also measured from a cold compile/start until the API was actually ready.

|Model|Ready time|
|:-|:-|
|GLM-5.3-Flash EXL3|\~1m 04s|
|Qwen3.8 Flash Next / vLLM|\~3m 50s|

This isn't really a model-size comparison.

The Qwen vLLM setup has quite a bit more startup work:

• PP workers
• distributed runtime
• model placement
• MTP
• PLE
• GDN
• Triton compilation
• memory profiling
• KV allocation

The ExLlamaV3 GLM path is comparatively static.

Inference benchmarks

These are the numbers from my dashboard workload.

Again, especially for speculative decode, I wouldn't treat these as universal model speeds.

Acceptance rate and generated text matter a lot.

GLM-5.3-Flash / DFlash2 K7 / Q8

|Input|PP|Decode|
|:-|:-|:-|
|8K|1,529 tok/s|95.6 tok/s|
|40K|1,624|90.7|
|73K|1,647|90.6|
|106K|1,650|91.0|
|131K|1,611|94.4|
|385K|1,535|90.1|

DFlash acceptance on this particular workload was mostly around 87%.

With the older MTP d2 path, the same dashboard workload was generally in the \~60 tok/s range.

Switching to DFlash2 K7 brought it to around \~90 tok/s here.

Qwen3.8 Flash Next / vLLM

The Qwen setup is:

AWQ INT4 + FP8 PLE / PP2 / MTP3

|Input|PP|Decode|
|:-|:-|:-|
|8K|5,513 tok/s|129.5 tok/s|
|40K|5,628|123.6|
|73K|5,466|141.9|
|106K|5,307|141.5|
|131K|5,183|159.9|
|252K|4,706|149.4|

So on raw throughput, Qwen is clearly faster.

At roughly 131K:

• PP: \~5.18K vs \~1.61K
• decode: \~160 vs \~94 tok/s

No argument there.

The interesting part for me was what happened once I actually let both models do multi-step coding work.

Why I stopped at 384K for now

I tested roughly 385K input and PP was still around 1.5K tok/s.

The problem wasn't PP collapsing.

It was simply wall-clock time.

Prefilling \~385K from scratch already takes about 4 minutes.

Even if I can make 1M fit, doing a full 1M cold prefill at this speed isn't particularly attractive for normal use.

So I'm currently leaving the service at 384K Q8.

I still want to see if I can get 1M working eventually, mostly for the technical exercise.

Small agent tests

Originally I was only going to make both models build Tetris and stop there.

Both were connected to DSH and got the same request.

1. Tetris

Prompt:

Build a playable Tetris game for the web.

|Model|Completion time|
|:-|:-|
|GLM-5.3|4m 40s|
|Qwen3.8|5m 30s|

Both produced working versions, and honestly the difference wasn't dramatic enough to be very interesting.

So I added two more tasks.

https://reddit.com/link/1x0b1ws/video/1sn8netal4uh1/player

2. AI mini PC landing page

Exact prompt given to both:

Build a polished single-file HTML landing page for an AI mini PC with a dark theme, specs, performance charts, pricing, FAQ, and smooth scroll animations, using no external libraries.

|Model|Completion time|
|:-|:-|
|GLM-5.3|10m 05s|
|Qwen3.8|14m 40s|

https://reddit.com/link/1x0b1ws/video/pdglw7kbl4uh1/player

3. Vampire-Survivors-style game

Exact prompt:

Build a single-file HTML vampire-survivors-style game with WASD movement, auto-attacks, enemy waves, XP, 3-choice level-up upgrades, HP, game over, and restart, using no external libraries.

|Model|Completion time|
|:-|:-|
|GLM-5.3|4m 40s|
|Qwen3.8|24m 10s|

This one had a much larger difference than I expected.

There is one obvious caveat:

the Qwen version added sound, while the GLM version did not.

The prompt didn't ask for sound, so I didn't go back and ask GLM to add it afterward. I wanted to leave both runs as the result of the same one-shot prompt.

https://reddit.com/link/1x0b1ws/video/4hl6g2bcl4uh1/player

Task completion times

|Task|GLM-5.3|Qwen3.8|
|:-|:-|:-|
|Tetris|4m 40s|5m 30s|
|Landing page|10m 05s|14m 40s|
|Vampire-style game|4m 40s|24m 10s|

I wouldn't read too much into three examples.

This definitely isn't evidence that GLM is “5x better at coding” or anything like that.

What I found interesting is simply that raw tok/s and end-to-end agent completion time didn't track each other very well.

Qwen has much higher PP and decode throughput, but on these particular tasks GLM often finished sooner.

For agent work, planning, number of retries, file rereads, edits, and how close the first implementation is to working all matter too.

So I think raw inference speed and actual task completion time are worth looking at separately.

Current state

The GLM service I'm using now is:

GLM-5.3-Flash EXL3 3.05bpw

• DFlash2 EXL3 6bpw K7
• Q8 KV
• 384K context\*\*

The main thing I like about this configuration is the memory/quality tradeoff.

The target fits in \~116.6GiB of HBM, stays resident, and the public 3bpw-class EXL3 fidelity results suggest the quant is holding up much better than I would have expected from the bitrate alone.

The runtime side is basically an HBM-first setup: keep target weights resident, then save memory on the drafter/KV side rather than moving experts back and forth during decode.

Full config, conversion scripts, ring-cache changes, benchmark code and raw results are here:

GitHub:
https://github.com/Flun/glm53-flash-cmp170hx-exl3

If anyone is running GLM-5.3-Flash on other weird 128GB-class GPU setups, I'd be interested in seeing what numbers you're getting too.

Next thing I want to try is 1M context, although at that point prefill time is probably the bigger problem than just making it fit.

With llama.cpp using a customized SYCL backend on Intel B70 (32GB), overclocking my DDR5 memory gave modest gains for MoE models which do not fit in VRAM.

Both PP and TG increased after overclocking DDR5.

I've never been one to overclock my system, but on the advice of my agent, I overclocked the DDR5 RAM on my AMD 7950X (4x dual-rank DDR5-5600, 128GB) from 3600MHz, the AMD safe default for that memory configuration, to 4800MHz, with a measured 36% increase in memory bandwidth.

What was the improvement? PP increased by about 10% and TG increased about 5%. And the prefill numbers increase the deeper the context gets.

llama-benchy numbers, including prefix caching tests.

|test|3600 base|4800 avg (r1/r2)|delta|
|:-|:-|:-|:-|
|pp2048 @ d0|682.2|713.7 (716.6/710.9)|\+4.6%|
|tg128 @ d0|30.3|31.2 (31.0/31.5)|\+3.0%|
|ctx\_pp @ d8192|660.6|715.0|\+8.2%|
|ctx\_tg @ d8192|26.9|27.8|\+3.3%|
|pp2048 @ d8192|554.3|632.0 (632.4/631.6)|\+14.0%|
|tg128 @ d8192|28.7|31.1 (31.7/30.5)|\+8.2%|

Because people seem to want this level of detail:

llama-server
-m Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf
--alias qwen38-flash-next
--mmproj Qwen-3.8-Flash-Next-mmproj-BF16.gguf
--model-draft mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf
--spec-type draft-mtp
--spec-draft-n-max 3
--host 0.0.0.0
--port 8081
-ngl all
-ncmoe 34
-c 262144
-fitc 786432
--kv-unified
--lazy-mode off
-lm none
-ub 2048
-b 4096
-fa on
-ctk q8_0
-ctv q8_0
--ctx-checkpoints 32
--checkpoint-min-step 2048
-t 12
-tb 12
--jinja
--reasoning on
--reasoning-preserve
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.0
--presence-penalty 0.0
--repeat-penalty 1.0
--chat-template-kwargs {"reasoning_effort":"medium"}
--parallel 3
-cram 10240
--slot-save-path /var/tmp/kv-cache
--log-file /tmp/qwen38-pristine.log
-lv 3

The "Opus 5.5 makes videos" thing has been going around — and the interesting part is that it isn't generating pixels. The model writes a self-contained HTML/Canvas scene where every frame is a pure function of time, a headless browser captures each frame, and ffmpeg encodes the MP4.

So I figured the question worth testing was: does that need a frontier cloud model? I ran the same pipeline on a \*\*local NVFP4-quantized Qwen 3.8 27B\*\*. No API, no GPU rental.

What it produced: a \~2-minute 1080p explainer of how GPS actually works. Deterministic canvas scenes, TTS narration, and BGM synthesized in WebAudio.

Video attached.

I have a somewhat specific question in case anyone has experience with this. I am a big fan of CS, math, and physics. While I was able to get formal instruction for the first two, I was never able to get the opportunity to learn physics. The most wonderful part about llms for me is that I can pursue that now without the costs of college tuition. My current process is the following. I pick up a textbook. I read it section by section and almost always don’t understand on the first read. Then I open up an llm and ask it to explain the section to me and ask it specific questions that help my learning. I’ve gotten past introductory quantum mechanics and special relativity this way, now I’m trying it on general relativity. However, tokens are expensive so I’ve been increasingly trying to replace my workflow with local models but have not gotten a lot of success with the smaller qwens and ministral. Would greatly appreciate any advice on other models or fine tunes. Has anyone else used local llms for physics or other sciences? Thank you!

I build weather app on Android using Qwen models both models have the same simple prompt ( I need you to create an Android weather app on an Android device. It provides many features, such as widgets and other features. I need you to do it in a simple, fast way ) , the first is : Qwen flash next Q3\_S .

2- Qwen 3.8 27b Q4\_XS . 3- Qwen 3.8 27b Q3\_XXS

you will see the results in photos

I got dual 7900xtx running on a z390 (pcie3 8x) running the 27b. I've been thinking upgrading the motherboard to a x570 (pcie4 8x) or a x870 (pcie5 8x) improving performance with TP, and then wait to buy medusa or spark with lpddr6 and 256GB (apple isn't an option for me). However seeing the gorgon halo price... I'm wondering how much those would cost and if i would pay that much, and if it will be better to go with a wrx80 route just now. I'm not really looking to add more GPUs, just the 8 channel memory to run the QFN.

Thanks!

Made an MXFP4 quant of dealignai’s GLM-5.3-UNCENSORED-FP8 for anyone looking to run it on AMD GPUs. Figured some of you might find it useful because I was looking for it and couldn't find any version for AMD GPU's so I uploaded the weights and conversion scripts.

Download on Hugging Face

• 423.75 GB / 394.65 GiB, about 44% smaller than the FP8 source
• Converted using AMD Quark on an MI355X server
• Expert weights use MXFP4; attention, routers and other sensitive layers stay at higher precision
• README includes the source revision, quantization details, measured stats and validation results

I spent the last 11 days training a small model for typed decisions: routing support tickets, intent detection, moderation, "should the agent call this tool", that kind of thing. You give it some context and a question, and it gives you back a choice, a score or a yes/no probability.

• 1.7B parameters (Qwen3-1.7B-Base + LoRA). Each option is scored in its own branch, so the order of the options doesn't matter
• \~16 ms for a short request and \~25 ms median on JevBench items, on a single RTX 4060 Ti, batch 1
• JevBench public 231: 68.4% (Jev 86.6, Strands Decider 2B 72.3 self-reported, Laya 58.4). DecideBench v1.1: 75.5%
• Several questions about the same text in one forward pass
• Weights are CC BY-NC 4.0 because part of the training data is non-commercial. The code is Apache 2.0

To be upfront: it is clearly less accurate than hosted APIs like Jev, and the README has all the numbers, including the ones where it loses. What it has going for it is that it's fast, runs locally and is free.

GitHub: https://github.com/realslapout/arc-1 Weights: https://huggingface.co/realslapout/ARC-1 Colab (free GPU): https://colab.research.google.com/github/realslapout/arc-1/blob/main/notebooks/quickstart.ipynb

Happy to answer questions, and I'd love to hear where it breaks.

Upgrade Question
If we say the "budget max" is $1700-1800, and this could include upgrading to a Taichi motherboard:

_With a new motherboard_, no need to bifurcate
1. Would you add a second 5060Ti (16GB)? Someone is selling one for around $500 locally;
2. Buy a used 7900 XTX (24GB VRAM); local seller, $850

_All-in on the GPU_, I'd have to make the current motherboard work for my use-case
3. Or, just go for an R9700? (No budget for Motherboard upgrade)

Current system
- Ryzen 9 9950X edit: added after original publishing of post
- 96GB system RAM (DDR5)
- 5060Ti (16 GB VRAM)
- Llama Cpp but I built for CUDA; default Vulkan had issues, and the GPU would "disappear"
- ASRock X870 Pro (only 1 PCIe 5.0 x16)
- I could run a second card very slowly at x4
- Researching if I could bifurcate x8/x8 in the 5.0 slot

I bought this PC used as-is; I do contemplate upgrading the MoBo to an X870E Taichi for 2 fast PCIe lanes

The "largest" models I currently run
- Strata IQ3_S (just tried this yesterday, was impressed)
- RentedNoodle/Qwen3.8-27B-GSQ-RCO-IQ3_XXS-Uncensored

My most typical uses
- writing code
- analysing documents (PDFs)
- analysing maps and images

The idea is to use something like Headscale or Tailscale at some point so I can always access local LLM from laptop if I'm not home.

---
Yes, I'm aware of "workstation" motherboards, CPUs, etc. and I'm not at a point right now where I want to take that path.

LIBERO is a robot manipulation benchmark: 130 tasks across five suites (spatial, object, goal, scene10, scene90), each with human demos and a language goal. It is the standard testbed for language-conditioned imitation, and it normally runs on robosuite with CPU MuJoCo, one environment at a time.

I ported all of it to MuJoCo Warp and ran it on an RX 9070 XT ($700, 16 GB, RDNA4). Physics does 17,137 env-steps/s at 2,048 worlds; CPU robosuite does 38. A 50-epoch BC transformer gets 42.5% on Warp vs 50% on CPU.

Why the GPU matters: behavioral cloning needs rollouts. Run the policy, watch where it fails, and generate labels or demos from that. On CPU it is one env at a time, and generating demos for a single suite (10 tasks) took me 8 to 9 hours. On the GPU it is minutes, so you can iterate instead of running it once.

Why Warp on AMD is the interesting part: Warp is NVIDIA's GPU sim framework, and the AMD HIP/ROCm port is recent (Tomas Thoresen, Strix Halo). Getting it working on RDNA4, with Warp compiling HIP kernels for gfx1201, JAX seeing rocm:0, and PyTorch seeing cuda, is what made this possible.

The renderer was the annoying bit. A BC policy is a fixed function of the pixels, and Warp's ray tracer is not MuJoCo's CPU renderer. My first Warp eval scored 0%. Four fixes got it to 42.5%: vertical flip, shadow constant 0.3 -> 0.0, 1.15x brightness, and cube-map sampling (the table wood grain rendered flat). Brightness alone was worth 12.5 points.

Everything builds from public sources on ROCm, and the example plus a 21.6 MB BC checkpoint are in the repo. What I didn't finish: the Warp path collects rollouts, training is still offline in PyTorch. I worked on the R9700 and some Instinct cards at AMD over the summer, so in-loop vision is next.

Writeup: https://amohan.dev/blog/2026/libero-warp-mjx-rdna4/
Code: https://github.com/poad42/libero_mjx

Public data only (5 datasets, 9 tasks, frozen quiz of 2,199 eval items, paired McNemar tests).

Overall accuracy: 4B 56.0% -> 71.6% (2h39 of training), 27B 68.8% -> 77.9%. The tuned 4B beats the base 27B (209 items gained, 149 lost, p = 0.002). Biggest gains on report extraction/classification (biomarker status 97.5% on test for the 4B). Weak spots: exact ICD-10 code (37.6% for the 27B), and MCQ accuracy collapses from val to test for all models, base included (cause unknown).

What I would like a second pair of eyes on: merging. Merging the 4B adapter (r=64, alpha=16) at the nominal scale lost most of the tuning (83.8% agreement with the adapter on the quiz). Merging with alpha=32 gave 93.2%. Identity probes are consistent with an effective scale of \~2x alpha/r under Unsloth (7/7 under Unsloth at nominal; 0/4 under Transformers+PEFT at nominal, 4/4 at 2x), but this is NOT a demonstration:

\- the two probes do not build their inputs the same way (Unsloth: gemma-3 template rendered as text, tokenized without special tokens; PEFT: tokenizer chat template straight to ids) and I did not check the sequences are identical;

\- I did not measure the scale actually applied by a trained layer, nor find a mechanism; - the 27B probe is inconclusive (4/4 at nominal);

\- my environment may be at fault: Unsloth installed with --no-deps, Transformers 5.18.0 and TRL 0.26.1 are outside the ranges declared on PyPI. I no longer have the GPU, so the direct check is not done. A script is in the Zenodo code: 74\_probe\_scale\_logits.py compares last-token logits under Unsloth and PEFT on identical token ids at several scale multipliers (0 = base model as control). It was only tested on a mock model, not on the adapter. If someone with a clean environment can run it, or knows whether this is expected behavior, I would love to hear it.

Separately: bf16 rounding erases 17-37% of the delta elements on merge, so the quiz tasks survive but verbatim memorization (an oath text I trained on) does not.

Models (merged + adapters): https://huggingface.co/Grujowmi

Quiz: https://huggingface.co/datasets/Grujowmi/OncoLLM-Quiz-Onco-v1

Report (revised Oct 5, same DOI), code, results: https://doi.org/10.5281/zenodo.23134374 Research models, not medical devices. Other limits (one seed, no CV, no ablation) are in section 8.

5080 16GB, 64GB RAM. Currently on 3.6 35B-A3B (Unsloth quant, fits fully in VRAM) and happy with it. Anyone tried Flash-Next at Q2/Q3 on a similar setup? Worth the switch, or should I stay put? Asking before committing to a 30GB+ download.

This is an update. I posted LocalMind here many moons ago from another account, when it was a Gemma chat in a tab.

LocalMind is a static web page that runs models on your GPU through WebGPU. It has no server, no account and no install. The new part: two engines that stream mixture-of-experts weights from disk while they generate. That lets a tab run models bigger than the machine's RAM.

Live: https://localmind.naklitechie.com · Code (MIT): https://github.com/NakliTechie/LocalMind

All numbers are from one MacBook M4 Pro (24 GB) in Chrome.

How it works

• On first load the GGUF is copied into OPFS, the browser's private file system.
• Dense weights, routers and the KV cache go to the GPU.
• Routed experts stay on disk. A pool of workers reads them on demand with sync access handles into a GPU slot cache (LRU, two layers of prefetch).
• The trunk kernels are hand-written WGSL that follow llama.cpp's graphs. That lets me test against llama.cpp on the exact same GGUF.

Gemma 4 26B-A4B (Google's QAT Q4_0, 14.4 GB)

• Same output as llama.cpp b9830 Metal: the live site's chat replies were character-identical on 9/9 test conversations (capped at 64 tokens). 15/16 fresh prompts matched token for token. The 16th split on a 0.00009-nat near tie, where llama.cpp's own two attention paths also disagree.
• Memory: the Chrome GPU process sits at 6.9 GB with a 4 GB expert cache. About 8.6 GB of experts stay on disk.
• Speed: 23.6 tok/s decode, 55 tok/s prompt processing. llama.cpp Metal does 70.6 and 204 on the same Mac, so the tab is ~3× slower at decode. Per token: ~22.5 ms GPU compute, ~11 ms routing round trips, ~8–13 ms SSD reads.
• First load from the site: 11.5 min (14.4 GB download). After that: 1.6 s.

Qwen3.6 35B-A3B (unsloth Q8_0, 36.9 GB, on a 24 GB Mac) — experimental

• The file is bigger than the machine's memory. The GPU process measured 7.3 GB with a 4 GB expert cache.
• Live site: 9.9 tok/s decode, 2.2 s to first token. First load is 36 min (download plus the OPFS copy).
• Output matches llama.cpp Metal 8/8 on 4- and 16-layer cuts. On the full model it matches llama.cpp CPU 5/8; the other 3 swap near-tie tokens. I can't run the full file on llama.cpp Metal on this Mac, so full-model parity is still open.
• Per token (~99 ms): ~23 ms GPU compute, ~39 ms routing round trips, ~35 ms expert reads from the SSD. Moving routing onto the GPU gave no gain (10.3 vs 10.3 tok/s): the misses are experts nobody predicted.

Also

• Gemma 4 E2B can keep its 1.2 GB per-layer embedding table on disk: GPU process 4.27 → 2.07 GB, identical output, 3–8% slower decode. It's a setting, off by default.
• The whole app is one index.html again (854 KB with brotli). Engines, workers and the disk tier are rolled into it, and the tab builds them from blob URLs.
• The disk tier is also a standalone library: diskformer.js.

Prior art

As far as I can find (searched 6 Oct 2026), no earlier browser engine reads weights from disk during generation. wllama and LlamaWeb stream from OPFS only at load. Pooled runs Qwen3.6-35B-A3B in a browser with experts paged from system RAM. On-demand disk reads exist in native runtimes: llama.cpp's --moe-stream PR (#25294) and Google's LiteRT-LM for Gemma's per-layer embeddings. Corrections welcome.

The Gemma 4 E2B kernels are webml-community's (Xenova and the Transformers.js team). My part there is the disk path.

Limits

• Chrome or Edge with WebGPU. Tested on one M4 Pro 24 GB only; 8 and 16 GB machines are untested.
• Not faster than native: llama.cpp is ~3× faster on Gemma 26B. The point is that a tab can run these at all, with the same output.
• Parity covers greedy decoding, the prompts listed above, and 64 tokens each.
• I haven't tried llama.cpp's expert-offload flags (-ot exps=CPU) for comparison.

If you have an NVIDIA/AMD GPU or a 32–64 GB Mac, I'd like your tok/s numbers. A bigger expert cache should move the Qwen3.6 number the most.

https://huggingface.co/Dankpaws/Swift1.5-Qwen3.8-Flash-Next-MLX-4.7bpw

I've had the 96GB Mac Studio with M5 Ultra for about a week now and wasn't satisfied with the results I was getting from the limited number of models available to me. It was a combination of speed, memory headroom, and/or output quality.

This is my best attempt at a calibrated MLX quantization of UkisAI’s Swift 1.5. Hope those of you with the hardware enjoy it!

| Measurement | This pack | Swift llama.cpp IQ3_XXS |
|:--|--:|--:|
| Prefill · 25k prompt | 3,191 tok/s | 1,427 tok/s |
| Prefill · 95k prompt | 2,928 tok/s | 1,307 tok/s |
| Decode · after 4k prompt | 113.7 tok/s | 62.7 tok/s |
| Decode · after 95k prompt | 81.1 tok/s | 44.7 tok/s |
| Top-1 agreement with Swift BF16 | 91.0% | 84.1% |

91% is next-token agreement with BF16 across 680 common held-out positions, not task accuracy.

Results above are simply from my own machine. mlx-serve 26.10.1. ~107GB download, text-only, 179,200-token tested context.

Disclosure: This unit was provided free of charge by Orange Pi for testing. No editorial review, no preconditions, no script. All data, bottlenecks, and thermal behavior are reported directly from hardware testing.

TL;DR - Core Pinning is critical on RK3588: Setting Ollama to 4 threads (A76 Big cores only) gives up to a +318% speedup over the default 8 threads, which stall waiting for the slower A55 Little cores. - Inference speeds (4T CPU): DeepSeek-Coder 1.3B hits 16.9 tok/s, Qwen 2.5 1.5B hits 14.5 tok/s, Llama 3.2 1B hits 14.6 tok/s, Phi-3 Mini 3.8B hits 6.6 tok/s, Llama 3.2 3B hits 7.3 tok/s. - The 8B memory wall: Llama 3.1 8B drops to 2.3 tok/s and pushes temperatures to 85°C. LPDDR4x bandwidth (~25-30 GB/s measured) is the hard physical ceiling. - NPU vs CPU: Ollama runs 100% on CPU. Using the native RKLLM runtime on the 6 TOPS NPU yields 21.55 tok/s on Qwen 1.5 0.5B with sub-100ms TTFT while keeping CPU load at ~0%. - Thermals: The board is sold bare-die without a cooler in standard retail packaging. Idle is 52.7°C, 1B-3B inference sits at 68-74°C, but 8B or sustained workloads hit the 85°C throttle ceiling without an active heatsink.

Hey r/LocalLLaMA,

I have been benchmarking an Orange Pi 5 Plus (RK3588, 16GB LPDDR4x, Samsung PM981a 256GB NVMe SSD with DRAM cache) running Ubuntu 22.04 LTS (Kernel 6.1.99-rockchip-rk3588).

The goal was to test whether an 8-core ARM SBC can realistically handle small 1B-3B models for 24/7 background agents or home automation without cooking itself or locking up the host system.

Here is the breakdown of CPU vs NPU performance, the big.LITTLE scheduling trap, and thermal limits.

1. Memory and Storage Architecture

When running local models on an SBC, two bottlenecks matter most:

• Unified Memory Capacity vs Bandwidth: With 16GB of unified memory, context windows are not squeezed. You can load a quantized 3B or 7B model with an 8k-16k context window and still have ample RAM for Docker and OS services. However, the RK3588 uses a quad-channel 32-bit LPDDR4x bus (~34 GB/s theoretical, ~25-30 GB/s measured). In autoregressive CPU token generation, memory bandwidth is the primary ceiling.
• Storage Ingestion (Samsung PM981a NVMe): Under direct I/O testing via fio, the M.2 PCIe 3.0 x4 slot delivered 2,862 MB/s sequential read and 197k 4K random read IOPS. Model weights load into system RAM in under a second (a 1.3GB model loads in ~0.6s).

2. Ollama & llama.cpp Inference Benchmarks (ARM64 CPU)

We tested Ollama (native ARM64 build) targeting the heterogeneous big.LITTLE topology (4x Cortex-A76 performance cores @ 2.26–2.4GHz + 4x Cortex-A55 efficiency cores @ 1.8GHz).

Prompt: Technical explanation of gradient descent and backpropagation (~200+ generated tokens).

| Model | Parameters | Threading Configuration | Eval (Generation) Rate | Prompt Processing Rate | TTFT (Time to First Token) | Memory (RSS) |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| Llama 3.2: 1B | 1.23B (Q4_K_M) | Big Cores Only (4T) | 14.62 tok/s | 108.11 tok/s | 425.5 ms | ~1.3 GB |
| Llama 3.2: 1B | 1.23B (Q4_K_M) | All Cores Default (8T) | 10.67 tok/s | 79.91 tok/s | 575.6 ms | ~1.3 GB |
| DeepSeek-Coder: 1.3B | 1.35B (Q4_0) | Big Cores Only (4T) | 16.90 tok/s | 89.72 tok/s | 1,025.5 ms | ~1.4 GB |
| DeepSeek-Coder: 1.3B | 1.35B (Q4_0) | All Cores Default (8T) | 4.52 tok/s | 27.91 tok/s | 3,295.7 ms | ~1.4 GB |
| Qwen 2.5: 1.5B | 1.54B (Q4_K_M) | Big Cores Only (4T) | 14.48 tok/s | 70.24 tok/s | 711.8 ms | ~1.6 GB |
| Qwen 2.5: 1.5B | 1.54B (Q4_K_M) | All Cores Default (8T) | 3.46 tok/s | 42.33 tok/s | 1,181.2 ms | ~1.6 GB |
| Llama 3.2: 3B | 3.21B (Q4_K_M) | Big Cores Only (4T) | 7.28 tok/s | 28.13 tok/s | 1,635.2 ms | ~2.8 GB |
| Llama 3.2: 3B | 3.21B (Q4_K_M) | All Cores Default (8T) | 1.99 tok/s | 10.84 tok/s | 4,245.2 ms | ~2.8 GB |
| Phi-3 Mini: 3.8B | 3.82B (Q4_K_M) | Big Cores Only (4T) | 6.56 tok/s | 35.17 tok/s | 909.8 ms | ~3.1 GB |
| Phi-3 Mini: 3.8B | 3.82B (Q4_K_M) | All Cores Default (8T) | 5.27 tok/s | 32.97 tok/s | 970.5 ms | ~3.1 GB |
| Llama 3.1: 8B | 8.03B (Q4_K_M) | Big Cores Only (4T) | 2.32 tok/s | 10.24 tok/s | 3,028.3 ms | ~5.4 GB |
| Llama 3.1: 8B | 8.03B (Q4_K_M) | All Cores Default (8T) | 2.10 tok/s | 7.66 tok/s | 4,044.8 ms | ~5.4 GB |

The big.LITTLE Scheduling Trap (+318% speedup with 4 threads) - Why num_thread: 4 is mandatory on RK3588: By default, Ollama spawns 8 threads across all cores. Because the 4 Little Cortex-A55 cores run at 1.8 GHz with smaller caches, thread barriers in llama.cpp cause severe synchronization stalls. - Restricting inference to the 4 Big Cortex-A76 cores yielded: - Llama 3.2: 1B: 10.67 -> 14.62 tok/s (+37%) - DeepSeek-Coder: 1.3B: 4.52 -> 16.90 tok/s (+274%, prompt rate +221%) - Qwen 2.5: 1.5B: 3.46 -> 14.48 tok/s (+318%) - Llama 3.2: 3B: 1.99 -> 7.28 tok/s (+265%, TTFT down from 4.2s to 1.6s) - Phi-3 Mini: 3.8B: 5.27 -> 6.56 tok/s (+24%) - Llama 3.1: 8B: 2.10 -> 2.32 tok/s (+10%, TTFT down by 1s) - The 8B limit: Running an 8B model on CPU is fundamentally memory-bandwidth bound. At ~5GB per token generation step, theoretical max is ~5 tok/s, making 2.32 tok/s the practical limit. It also pushed temperatures to 85.0°C uncooled.

3. CPU vs Hardware NPU (6 TOPS, 3 Cores)

Ollama compiles llama.cpp with ARM NEON SIMD instructions and runs 100% on the CPU. It does not touch the Rockchip NPU.

To test the 3-core 6 TOPS NPU, we compiled a native C++ runner (tools/rkllm_bench_v1) linked directly to Rockchip's librkllmrt.so runtime and kernel driver (/dev/rknpu_mem).

| Metric / Dimension | Ollama CPU Inference (ARM NEON) | Rockchip NPU Hardware (RKLLM Runtime) |
| :--- | :--- | :--- |
| Compute Engine | 4x Cortex-A76 @ 2.4GHz + 4x A55 @ 1.8GHz | 3-Core Dedicated Neural NPU (6 TOPS INT8/INT4) |
| 0.5B Model Eval | ~20 - 24 tok/s | 21.55 tok/s (Qwen 1.5 0.5B - Measured on-device) |
| 1.3B - 1.5B Eval | 16.90 tok/s (DeepSeek) / 14.48 (Qwen) | ~16.69 tok/s (Qwen 2.5 1.5B - Reference Data) |
| 3B - 4B Model Eval | 6.56 tok/s (Phi-3) / 7.28 (Llama 3.2) | ~7.45 tok/s (Phi-3 Mini 3.8B - Reference Data) |
| 7B / 8B Model Eval | 2.32 tok/s (Llama 3.1 8B) | ~4.5 - 4.98 tok/s (Qwen 7B / ChatGLM - Reference Data) |
| CPU Utilization | 100% Core Saturation (System frozen for other tasks) | ~0% CPU Load (CPU 100% free for Docker/OS) |
| SoC Thermals | Reaches 84.1°C – 85.0°C | Runs drastically cooler (~60–68°C) |
| Model Ecosystem | Any GGUF via Ollama / llama.cpp | Requires .rkllm quantization via rkllm-toolkit |

Key NPU trade-offs for homelab use: 1. Zero CPU load: During NPU generation, CPU cores stay at ~0%. Home Assistant, Nextcloud, and other Docker containers remain fully responsive. 2. Speedup on larger models: On 7B models, the NPU delivers ~4.8 tok/s vs 2.3 tok/s on CPU because dedicated matrix engines handle the tensor math without thrashing CPU caches. 3. Sub-100ms latency: On compact models, Time to First Token (TTFT) drops to 96.4 ms on NPU. 4. Format restriction: You cannot load arbitrary GGUFs; weights must be converted ahead of time to .rkllm using Rockchip's conversion toolkit.

4. Thermal Behavior & Power (Bare-Die / Uncooled Testing)

The standard retail package from Orange Pi is sold board-only (cooling accessories are sold separately as is standard for SBCs), so all tests evaluate out-of-the-box bare-die thermals on an open desk:
- Idle (Ollama background daemon waiting): 52.7°C (~4–5W estimated SoC envelope)
- Continuous 1B/3B Generation (4T Big Cores): 68–74°C (dissipating through PCB copper planes)
- Sustained 8B Generation (8.03B params): Pushes the bare SoC directly to 84.1°C – 85.0°C (hitting the kernel DVFS limit). An aftermarket cooler or fan is required for sustained heavy loads.
- Estimated wall power: ~12–16W under sustained multi-core inference.

5. Verdict: Is RK3588 Viable for Local AI?

Where it works well:
- Background autonomous agents (summarizing feeds, home automation reasoning in Home Assistant, bot handlers) using Llama 3.2 1B, DeepSeek-Coder 1.3B, or Qwen 2.5 1.5B.
- Low-latency function calling: at 14-17 tok/s, 1B models generate faster than reading speed.
- Local embedding and vector search.

Where it falls short:
- Running 8B+ models interactively (2.3 tok/s is too slow for back-and-forth chat).
- Running without a heatsink under sustained compute.

6. Reproducibility & Test Scripts

All test scripts (tools/benchmark_ollama.py), raw JSON benchmark logs, and hardware configs are available in the repository:
GitHub: Orange Pi 5 Plus Benchmarks

What models are you running on edge ARM boards? Anyone here running RKLLM in production vs pure llama.cpp?

Standard distillation usually means burning weeks of compute and billions of tokens hoping the student model eventually mimics the teacher. We wanted to see what happens if you skip backprop entirely and treat transfer as a closed-form trajectory matching problem between layers.

The idea is straightforward: feed a small batch of calibration prompts through both models, capture layer-to-layer hidden state trajectories, and solve for weight updates directly in the student's MLP blocks using regularized least squares and spectral projection.

We tested this across two architectures: Qwen 3.5 (transferring from 4B down to 0.8B) and old GPT-2 small just to see if it would instantly disintegrate into gibberish like it usually does when you touch its weights. Both stayed coherent, but the initial Qwen test hit a wall:

Editing all 24 layers of Qwen 0.8B completely melted the model (+64.78% NLL loss explosion). When we checked singular value entropy across the network, layers 1 to 22 turned out to be a chaotic polysemantic soup with entropy over 0.90. If you try to force raw trajectories through those middle layers, you basically scramble the model's internal memory knots.

The fix was restricting the surgery to 4 anchor points (layers 0, 7, 15, and 23) where representations actually maintain clean linear structure.

Once we did that:

• Held-out NLL dropped by 10.8% across 30 diverse benchmarks (-23.8% in biomedicine, -14.6% in math and logic).
• Zero-shot 400-task HellaSwag went from 54.75% to 55.25% (+0.50%), verified locally in Vulkan llama.cpp.
• Base 0.8B originally failed binary tree inversion by spitting out dead commented pseudo-code. The edited checkpoint wrote clean recursive Python on the first try.
• On Russian logic paradoxes, it even started firing <think> reasoning tags spontaneously, which was wild to see on a raw base model with zero chat template applied.

Best of all: we don't have an H100 cluster or even a 4090. All trajectory extraction and weight solving was done locally on a crusty 8GB RX 580 using layer-by-layer VRAM streaming and a DirectML patch to stop Qwen's Gated DeltaNet attention from throwing driver errors.

Everything is open source if you want to inspect or replicate:

• Code and math writeup: https://github.com/dsadawq3/DynamicTune
• Transferred base model weights: https://huggingface.co/F-Labs/Qwen3.5-0.8B-DynamicTune-Base

If anyone here has a 24GB-32GB card (4090, 5090, or server silicon) and wants to push this further, here is what would be interesting to test:

1. Transplanting reasoning trajectories from 27B models down to 9B, 4B, or 2B.
2. Squeezing larger models (like Gemma) into mobile sizes without weeks of retraining.
3. Transplanting refusal-ablation vectors directly from uncensored models without fine-tuning.
4. Using pre-trained Sparse Autoencoders (SAEs) to unknot layers 1-22 so we don't have to skip them.

Happy to answer questions or dig into the failure modes in the comments.

I'm taking a break from training alice today after my current SFT run ends to work on a few other things.

I'm an (unemployed) software developer trying to find a machine learning job in New York, and outside of job applications and networking, I'm trying to do as much as possible to further the frontier development of different LLMs in the hope it'll get some visibility. I studied machine learning in university and am fairly well educated. Plus, I genuinely enjoy working on this stuff and helping people.

That said, are there any models out there that don't have dynamic quants (preferably GGUF) that you'd like to see one for? I won't be matching unsloth or anything but I know how to make fairly good ones by quantizing different tensor types by impact. I'm talking akin to Q5 K XL and etc

I'll do the top voted 1/2 comments today, or whatever else, even outside of quants if there's something this community has been hoping for that doesn't exist. Fine tunes, paper implementations, etc - my goals of visibility happen to align very well with satisfying community desires.

Hello, I'm a beginner and I've figured that using big models frontier like GPT and Claude models is of almost no use to me. My question is, should I use DeepSeek harness or Hermes for v4.1 Flash?

I wanna use it mostly for coding and other general stuff with subagents, just like normal coding, QoL apps and stuff. I asked some people and all responses are mixed.

It's either Either Hermes is not good for coding. Or people glazing Hermes till the end of time.

Thank you !!

I am trying to decide between Mac Studio M5 MAX 128GB vs 2x RTX 3090. I understand that I can run larger models on the M5, but I don't understand what the capability differences would be. Nor have I been able to get a "sense" of how fast the difference would be.

I keep seeing huge advances in the 2x 3090 arena, but I don't know how they translate to the real world.

If my use case includes coding tasks, image recognition, and general hermes type stuff, is there any reason one would be less capable than the other?

Setup: GPU box with 1x H200-class card now; can allocate more (up to several H200s) if it clearly buys better results. Inference via vLLM or similar, OpenAI-compatible endpoint. Coding happens on a separate non-GPU Linux VM on the same network - IDE/harness there, calling models over LAN. Outbound internet is fine for packages/extensions, but inference stays on our hardware (no code to external model APIs).

Constraints: on-prem only, open weights, permissive license preferred (Apache/MIT), not Chinese-origin including base models (I understand a lot of fine-tunes are Qwen underneath), NATO-country lab preferred. Use is general software dev plus security tooling and code analysis - repo-level agentic work on existing codebases (fix / extend / refactor / test) is the core, with a human reviewing diffs. 2-3 concurrent users max.

Two things I'm trying to work out:

1. Capability tiers vs. VRAM. On one card candidates seem to maybe be something like Cohere North Mini Code (30B MoE/3B active), Mistral Small 4 (119B MoE/6B active), Nemotron 3.5 maybe as a generalist baseline; Gemma 5? The Vibe Code Bench results suggest small open models fall over on long E2E builds, where only Large-4-class (4-8 cards) and closed models seem to hold up. Is that your experience? Where's the step-change for agentic repo work on an existing codebase - does 30B-class -> 120B-class matter much, or only the jump to 500 GB+? We could get more compute for something like Command A+ or Mistral Large 4 if it's actually better for code rather than just for E2E.
2. Heterogeneous multi-agent. Does a big planner/reviewer (Large 4 / Command A+ class) plus small fast executors (e.g., North, Small 4) actually beat a single mid-size model, with a human approving plans and diffs? Which harnesses handle routing sub-agents to different endpoints well - OpenHands, OpenCode, Pi, VS Code extensions (Cline/Roo/Continue), Codex CLI in local-model mode?

Harness/IDE: something that supports multi-agent workflows (planner / executor / reviewer agents checking each other / etc.) but would also like humans to be able to step in, review diffs and edit by hand.

Anyone running something like this? Which model + harness combo is holding up best as of late? Thanks!

EDIT: thanks all - adding Poolside Laguna S 2.1, Muse Glimmer 30B, Inkling-Small (2x H200), K2 Horizon (checking lineage), Gemma 4 31B and Reflection Beam (501B MoE / 23B active, Apache 2.0, weights due this month) to the candidates; Ornith is Qwen-based so out. Pi added to the harness list.

I constantly see recommendations saying that ik\_llama.cpp (ikawrakow's fork) is the undisputed king of hybrid CPU/GPU offloading and MoE performance. However, every time I benchmark it against mainline ggml-org, mainline consistently beats it by a wide margin.

Am I missing specific flags, or is ik\_llama simply not designed for multi-GPU layer splitting?

My Rig & Hardware Constraints:

• Host CPU: Intel Core i9-10920X (12 physical cores, AVX-512 & VNNI enabled).
• GPUs: 4x asymmetric setup:
• GPU 0, 1, 3: NVIDIA P102-100 (10GB Pascal, PCIe 1.0 bus bottleneck).
• GPU 2: RTX 3060 12GB (Ampere, acts as Master node via -mg 2).
• Known Hardware Laws / Workarounds:
• I run layer splitting (-sm layer) across the 4 cards with asymmetric tensor splits (-ts).
• Pascals must strictly stay under 9.7 GB VRAM; exceeding that triggers PCIe micro-paging and tanks speed.
• I use --poll 100 on mainline to prevent AVX-512 CPU threads from dropping into low-power sleep states between GPU layer handoffs.

The Test:

• Model: Qwen 3.8 Flash-Next 177B Uncensored (IQ3\_XXS, \~89 GB) with multimodal vision (mmproj).
• Offload: 35 layers offloaded to the 4 GPUs (-ngl 35), remaining 13 layers computed on the AVX-512 CPU. 32k context.

The Head-to-Head Benchmark:

1. Mainline (ggml-org/llama.cpp):

• Prompt Eval: 19.70 tokens/sec
• Token Generation: 10.86 tokens/sec
• CUDA Graphs: 2,490 CUDA graphs reused across the GPUs.

1. ik\_llama.cpp:

• Prompt Eval: 5.48 tokens/sec (72% drop)
• Token Generation: 8.43 tokens/sec (22% drop)
• Observations: 0 CUDA graphs engaged. It spent time taking context checkpoints during generation (100ms+ pauses), and --poll is unsupported.

The Question:

Is ik\_llama.cpp's speed advantage strictly meant for pure CPU inference or single-GPU systems?

Does its custom CPU threadpool fall apart when coordinating pipelined layer splits across heterogeneous GPUs over PCIe, where mainline's CUDA graph caching takes over? Would love to hear from anyone running hybrid multi-GPU setups.

Running a model on your own machine used to be a privacy story with a quality tax. On this generation that trade has narrowed to where we can state it plainly: for daily work, the local model is good enough. We measured it — 25 paired tasks across four workloads, same prompts, one strong independent judge — and that is the top line:

• quality: 89.5 vs 92.6 on a 100-point scale (local vs cloud), with the 12-item suite splitting six wins each;
• completion: every coding run finished green on both models — 10/10 agentic runs fully green (20/20 visible tests, 4/4 hidden checks, tests untouched), and the bug-fix loop fixed all 4 bugs identically in 5/5 rounds each;
• speed: 2.5–5.5× the wall clock, depending on the workload, with 95%+ of the local time going to model generation;
• cost: the local runs cost nothing beyond electricity. The cloud side of the 12-task suite cost 0.14 credits.

I've looked in the Tutorial/Guide flaired posts but everything is still way over my head.

I just want to tell a local AI - hey, all these files in this folder have numbers for names, but those numbers correspond with data in this spreadsheet... I want you to rename the files referring to this spreadsheet, renaming the filenames/numbers that are matched in column 3, replacing their filenames with what is in column 1 for that row.

So far, I've installed GPT4All but every model is telling me it doesn't have access to my local files.

I've been messing with MoE offloading for a while. Setup: Qwen3-Next-80B-A3B Q4\_K\_M (48.5 GB), RTX 3090, only 1/4 of the experts kept in VRAM, the rest read from NVMe when the router asks for them.

When the router picks an expert that isn't in VRAM you can either wait for the SSD read or use the next best expert that's already on the GPU. Substituting everything wrecks quality (+5.7% ppl in my emulation tests). Waiting only for the router's top pick and for experts with gate weight >= 0.15, and substituting the rest, brought it down to +0.35%. In the real engine that rule costs more like +1.2%.

Decode tok/s on a rented 3090 box (NVMe \~5.7 GB/s, 16 threads), both using about 15.7 GB of VRAM:

| free RAM | stock llama.cpp (--n-cpu-moe 34) | patched |

|---|---|---|

| plenty | 72.9 | 108.4 |

| \~32 GB | 65.8 | 97.8 |

| \~16 GB | 31.8 | 94.5 |

At 16 GB it still did 89 tok/s when reading every miss straight from the SSD. Perplexity was 1.6% higher than stock on the same text. On GSM8K (500 problems) it lost 1.8 points vs waiting for every expert, on HumanEval no real difference.

Things to know before trying it:

\- it's a research patch, not a polished fork. It builds a benchmark tool, I haven't tested llama-server or llama-cli with it

\- only Qwen3-Next, only Linux + CUDA, one sequence at a time

\- the 16 GB case was simulated by locking RAM on a bigger machine

\- the table is decode speed while feeding real text through the model. In actual greedy generation it did 64-74 tok/s

Code, run scripts and raw logs: https://github.com/SOCIALPINE/moe-miss-substitution

Paper with the details, including what didn't work: https://doi.org/10.21203/rs.3.rs-11268552/v1

Has anyone tried something like this, or have numbers from slower SSDs? Curious how much the SSD matters.

Kolibri, the German model from Aleph Alpha, hosted by Tesseracted, writing a small command line tool in Nim, on 3code.

It's a small model so it's remarkable it got code to work in a very niche language.

The model, the provider, the harness and the language are all made in Germany!

Hey y'all!

I have a pet project that has grown out of proportions. Long story short: I'm a data scientist who writes fantasy books and self-publish them. I think it's a genuine waste of human life to check for spelling errors so I figured AI could help. Turns out, it is not so simple to get an AI to properly fix a 120k words manuscript ;)... That's why I created Betty!

It runs Qwen3.5 4B as an offline copy editor for whole novels — and this is some of what I learned fighting tens-of-thousands of words through a 4B model, insisting that (most) users can use it fully for free and fully offline. Because, let's be honest: authors rightfully distrust and generally hate AI companies.

First challenge: Chunking the text. Authors already do this, in the darkest hours of the night, copy-pasting snippets into chatGPT for some shameful feedback. Problem with that approach: super inefficient, both for the author and the environment. And the context is missing at the edges of each chunk. I fixed it by ensuring chunks to overlap.

Second challenge: AI misses genuine errors. So, I added two conventional spell controllers -- LanguageTools and HunSpell. This already surfaces all spelling errors, letting the AI focus on suggested fixes and on all the "non-error" errors, such as "There" vs "Their". For these, the AI searches, while a Python script surfaces all the common culprits for the model to pay special attention.

Third challenge: Error rate. First off, Betty doesn't capture everything. Second off, it sometimes introduces its own mistakes. I fix it by putting the writer-in-the-loop, and there's a super smooth interface now for the author to accept and dismiss suggested edits (tinder-style with left and right swipes ; ) ).

I'd be super grateful for any advice, feedback, and thoughts you might have on this project. I currently have it up-and-running with about 30 users and getting some user feedback. Northing technical though, so this is what I'd love to have more of.

Full thing is source-available on GitHub, and can be found for download and lots more information at www.bethaniel.eu

Hi all,

I previously posted about getting Qwen 3.8 27B running at around 75t/s with llama.cpp. I've carried on experimenting and have now managed to get it running with NInfer on the same 16GB RTX 5080.

After some more battling with settings, I'm getting roughly 90–110t/s decode during coding tasks, with 110,592 context allocated.

Looking through 32 completed requests from a Zoo Code session:

• Median decode: 96.45t/s
• Lowest: 84.6t/s
• Highest: 131.7t/s
• Median time to first token: 1.4 seconds, with prompt caching working on most turns

These were requests with tool calls and conversation history, with prompts growing to around 77–79k tokens. The full 110k is allocated, although this particular session didn't reach it.

I'm running NInfer v1.5 in Ubuntu 24.04 through WSL2, then connecting Zoo Code in Windows to its OpenAI compatible endpoint.

These are the settings I've ended up using:

~/ninfer-5080/build/apps/ninfer-serve \
~/models/qwen3_8_27b.ninfer \
--host 0.0.0.0 \
--port 8080 \
--model-id qwen3.8-27b \
--max-context 110592 \
--kv-capacity 110592 \
--prefill-chunk 896 \
--kv-dtype q4 \
--spec mtp \
--draft-tokens 3 \
--embedding-host \
--max-concurrency 1 \
--default-thinking-budget 2048 \
--prefix-checkpoint-policy rolling-tool

Getting everything into 16GB was the fiddly bit. The weights take about 11.86 GiB according to the startup log. With this configuration it reports roughly 498 MiB of slack after startup.

I settled on 110,592 context to leave a bit of breathing room. Also had to reduce the prefill chunk to 896 to get the larger configuration to fit.

Here's an example from a turn with almost 50k context:

prompt=49907 gen=409 reasoning=126 cache=49309
ttft=558ms prefill=1177.7tok/s decode=104.6tok/s
wall=4.46s speculative=mtp 3.00tok/round (66.7%)

And further into the conversation:

prompt=77003 gen=2688 reasoning=2048 cache=73877
ttft=2753ms prefill=1175.9tok/s decode=90.3tok/s
wall=32.52s speculative=mtp 2.74tok/round (58.0%)

It can still take a while to finish a turn. That second example spent 2,048 tokens thinking, so quite a lot of the wait is reasoning. Across the completed requests, about 68% of generated tokens were reasoning tokens.

Losing the prompt cache also makes a big difference. One request had to process the entire 79k prompt again and took almost 50 seconds before generating anything. Once it started generating, it was still doing about 94t/s.

A couple of things caught me out connecting Zoo Code:

• The base URL needs to be http://127.0.0.1:8080/v1. Leaving off /v1 gave me a 404.
• Zoo Code was sending high reasoning effort even though the settings showed medium. NInfer rejected it. Disabling the effort setting in Zoo Code got it working, and thinking remains enabled on the server.

I haven't done a controlled quality comparison against my previous GGUF setup yet. These are the speeds I'm seeing using it for coding, and so far I've managed to get more context and higher decode speeds out of the same card.

Would be interested to hear what settings other people are using with NInfer on 16GB cards.

I was trying to find Glimmer benchmarks on hardware similar to mine and couldn’t really find much, so I figured I’d post what I’m getting on my current setup for anyone else looking.

Hardware
Ryzen 5 5600X
48 GB DDR4
RTX 3060 12 GB
RTX 3070 8 GB
20 GB total VRAM
Windows
llama.cpp / llama-server

Models
Target: Glimmer 30B IQ4\_XS
Draft: Glimmer 30B DFlash Q4\_0
DFlash draft model on CUDA1
Layer split: 45/55
Context configured for 65,536 tokens
K/V cache: Q8\_0
Flash Attention: on
DFlash max draft: 15
The benchmark prompt itself was 994 tokens, so this is not a benchmark at 65K filled context. The server was configured with a 65,536-token context window.

Results
Prompt: 994 tokens
Output: 128 tokens
Prompt processing: 684.86 t/s average
Generation: 41.97 t/s average
Generation range: 41.71–42.45 t/s
Average total request time: 4.5 sec
Model load time: 9.7 sec

VRAM
GPU 0: 11,229 MiB
GPU 1: 6,627 MiB
Combined observed usage: \~17.4 GiB

I wasn’t really trying to squeeze every last token/sec out of this. It’s just the configuration I ended up using and the performance I’m seeing.
I couldn’t find much for Glimmer on a mixed 3060 12GB + 3070 8GB setup, so hopefully this gives someone else a useful reference point.

llama-server.exe \^
\-m "Glimmer-30B-IQ4\_XS.gguf" \^
\--alias glimmer-30b-dflash \^
\--host 127.0.0.1 \^
\--port 8083 \^
\-c 65536 \^
\-np 1 \^
\-b 1024 \^
\-ub 256 \^
\-ngl all \^
\-sm layer \^
\-ts 0.45,0.55 \^
\-fa on \^
\--cache-type-k q8\_0 \^
\--cache-type-v q8\_0 \^
\--threads 8 \^
\--threads-batch 16 \^
\--spec-type draft-dflash \^
\--spec-draft-model "Glimmer-30B-dflash-Q4\_0.gguf" \^
\--spec-draft-ngl all \^
\--spec-draft-device CUDA1 \^
\--spec-draft-n-max 15 \^
\--spec-draft-n-min 1 \^
\--no-webui

Running inference across three machines - an AMD box (Ryzen 9 9950X + RX 7900 XTX), a smaller NVIDIA box (i5-10400 + RTX 3050), and a MacBook Pro M3 - all running LM Studio/Ollama. The serving/monitoring side is where I lose the most time: no single place to see what model/version is loaded where, token throughput, VRAM vs unified-memory pressure, etc., without checking each machine by hand.

What are you all actually using for:

• Multi-node / multi-GPU serving + routing?
• Observability that is not "wire up Prometheus on every box"?
• Keeping track of model versions across nodes?

Happy to share my current setup if it is useful.

Hey all,

I'm helping organize All Day AI, a free online conference on Thursday the 22nd.

I'd like to mention a couple of our speakers who have volunteered to talk that day:

Vivienne Hnin (Utilyst): "Small Models, Big Profit Margins: The Economics and Engineering Tradeoffs of SLMs." That's the debate this sub has every day: when a small model is actually good enough.

Hossein Kazemi (Astorna): "Using Task-Specific Small Language Models to Handle Sensitive Data". Keeping sensitive data local is half the reason people run models themselves.

Hitesh Jain (Coral Bricks AI): "Coding at 250 tokens." Inference speed is what this crowd benchmarks obsessively.

The rest of the schedule goes up soon, across four tracks: Build, Lead, Secure and Ship.

The talks are community-submitted.

Free to register: https://www.alldayai.com/?utm\_source=localllama&utm\_medium=reddit

I hope this and other talks in this space may help you discover We have a discord channel you can join: https://discord.gg/xUyS3Zu68

I’m building a feature that matches free-form requests to a catalog of structured listings. Requests can contain several constraints and follow-up refinements. The results also need a short explanation of why each match was selected.

Our prototype uses a hosted LLM to rank a shortlist. I’m exploring whether a small locally hosted embedding model and reranker could deliver comparable quality at lower cost.

For anyone who has deployed a similar system: where did local retrieval start to fall short of an LLM? Did a hybrid approach work better? I’d especially appreciate real-world latency and cost figures around 10,000–100,000 requests per month.

TLDR: The question we answer: how do you learn from experts with different objectives? Pooling all their data can lose their trade-offs; learning from each expert separately misses opportunities to share data. MA-BC pools demonstrations where observed actions don’t disagree, with upper and lower bounds on sample complexity.
Authors: Ziyad Sheebaelhamd, Luca Viano, Volkan Cevher, Claire Vernade

Arxiv: https://arxiv.org/abs/2605.12000
Github: https://github.com/ziyadsheeba/mabc

https://preview.redd.it/i20adc3z04uh1.png?width=2532&format=png&auto=…

Hey everyone,

I have new in brackets above because I'm not necessarily inventing anything innovative in terms of the actual mathematics or optimizations behind some quant techniques, but I'm pretty happy with how things are coming.

What I'm working with is basically a "poor man's" RCO (the quant method from IST Austria dAS lAB). Exact same concept: choose a type per tensor under a byte budget while optimizing task KL on the whole model. I worked on GSQ but I don't have an approximation method that beats baseline - yet.

Take the core principles of the method, make them cheaper approximations, and regain as much accuracy as possible. I originally planned to make an approximate GSQ-RCO hybrid, but none of my hypothetical models for the approximate for GSQ have beaten baseline yet.

In application: start with a full precision model, and create an "imatrix shape" map, per tensor. This doesn't calculate the sensitivities of individual tensors - but it creates a sensitivity "curve" where you can approximate which tensors in a model suffer most from quantization via extrapolation. This creates a baseline estimate of the optimal quant per tensor.

Then: iterative trial and error with local search. Take the file size of the baseline estimate, and substitute different precision per class to bring overall model size down, beginning with the tensors the approximation model marked as most sensitive to quantization. Once the working-best version hits under the filesize cap, it tries variations of substitutions that keep the file size approximately the same (upgrading certain tensors, downgrading certain ones, etc - basically looking for holes in the local search method once the local search is done).

For the whole process above (the iterative local search + search error recovery \[not really error but i cant find the word im looking for\]), the model is quantized and KL divergence is measured vs prior iterations - anything that raises KL divergence is discarded. The result: an approximated RCO style quant, iterated as closely as possible to optimal.

Do I expect this to beat GSQ-RCO or Unsloth dynamic V3? Definitely not GSQ-RCO or regular RCO, and likely not the unsloth ones. However, I've got a few advantages: this is CHEAP and extremely conservative on VRAM usage. The teacher model only needs to be loaded once: to dump its per token log probs. This is quick on a GPU, but since it only has to be done once, it can be done on CPU with a little patience. Every step after only pulls the candidates onto GPU, starting from the imatrix curve approximation - so all you need is enough VRAM for your approximate final quant size (with a little buffer for iteration, maybe 20-25% more would be optimal). The whole process takes a few minutes to a few hours depending on what you're doing.

I've pretty much documented a psuedo-algorithm approach above that's recreatable, but I can supply better documentation if people are interested.

Some early results on Qwen 3.5 2B:

llama.cpp IQ3\_M with an imatrix - 999mb vs RCO-lite with an imatrix - 1088mb \[+89mb\]

Mean KLD for IQ3\_M: 0.098381

Mean KLD for RCO-lite dynamic mixture \[+89mb\]: 0.045162 -- almost exactly half for 89 more mb

Mean KLD for RCO-lite dynamic mixture \[cap 1038, + 39mb\]: 0.0793220

Mean KLD for RCO-lite dynamic mixture \[cap 947, - 52 mb\]: 0.090624 -- still better than the IQ3\_M quant despite being 52mb less.

Take these as early results - I forgot to document exact +- for my KLD runs, but the band was generally lower than the i quants. I need to try various different size targets to figure out what BPW range this algorithm works in most effectively, and this is just up against the IQ3\_M - IQ4\_XS had a better KLD than this design - more BPW so it's not an exact estimate, but not that substantially - I didn't try to fit an optimal model inside the IQ4\_XS size range yet, that was just something I noticed. I suspect the Q3 and Q2 ranges will benefit most from this, I haven't tried in the higher BPW ranges yet - partway through Q2 experiments.

Also: my baseline llama.cpp quants are calibrated on the same wikitext set for the imatrix as the RCO-lite quants are, with the same held out set for the KL divergence.

Cheers.

Introducing Open SLM Evaluations!

Open SLM Evaluations is a dataset containing benchmarks for 106 SLMs(150M parameters and below). It has benchmark scores for each model, links, parameter counts, and names.

Results were evaluated independently over 30 GPU minutes on a Runpod B200!

Data is open on Apache 2.0! Check it out!

https://huggingface.co/datasets/Enderchef/Open-SLM-Evalulations

Building out a few personal AI agents locally (running Ollama + Hermes) for things like market research, coding assistance, and general task automation. Nothing crazy, just personal productivity tools I want to with context and memory across sessions.

Deciding between mem0 and Memori and I wanted some first-hand or more experienced answers from anyone regarding:

\- How well does each actually work with local models?

\- For a single-user setup, is it worth it or would just using something like ChromaDB with rolling summaries be enough?

Just want something that gives my agents decent memory without having to remind it constantly. Still learning the ins and outs so go easy on me

Curious on what general consensus is and what your stack looks like if you run any of this :)

I've tested quite a few settings and collected the results in a spreadsheet. I keep seeing people reporting 100+ tokens/s with 16 GB VRAM, or generally much higher speeds with less VRAM. I'm trying to understand whether my setup is underperforming or I'm comparing completely different things.

My PC:

• Ryzen 7 5800X3D, 128 GB DDR4 at 3600 MT/s
• 2× RX 7900 XT, 20 GB each, XFX and PowerColor
• Gigabyte B550 EAGLE WIFI6
• XFX on PCIe 4.0 x16; PowerColor on a chipset-connected PCIe 3.0 x1 slot
• Windows 11, AMD driver 32.0.31041.1004

The cards have reduced clock settings: XFX 1700 MHz core, PowerColor 1800 MHz, both 2500 MHz memory and −10% power limit.

Here are the main single-response results:

| Setup | Generation TPS | Including prompt processing |
|---|---:|---:|
| Qwen3.8-27B IQ4_XS, direct ROCm + MTP3 | 46.4 | 43.1 |
| Qwen3.8-27B IQ2_XXS, direct ROCm + MTP3 | 66.5 | 59.9 |
| Flash Next UD-IQ4_XS, one GPU, warm ROCm run | 8.6 | 5.7 |
| Flash Next UD-IQ4_XS, two GPUs, Vulkan | 5.5–6.5 | 3.3–5.7 |

The 27B tests used roughly 700 input tokens, 8k context and 1024 output tokens. IQ4 had three runs; IQ2 is the median of nine prompts. Settings were ROCm 2.46.0, Flash Attention, f16 KV, MTP3 and batch/microbatch 2048/512, with thinking off.

Two separate IQ4 copies reached 103.5 TPS combined, but that required 16 concurrent requests. I haven't reached 100 TPS for one response. Splitting one model across both cards was slower. Tensor split initially produced broken text; --no-mmap fixed that.

Flash Next is unsloth UD-IQ4_XS, around 93.7 GB. The single-GPU profile used ROCm 2.49.0, --n-cpu-moe 42, f16 KV and 8k context. The dual-GPU profile used Vulkan 2.51.0, tensor split 1:1, --n-cpu-moe 28, q8 KV and 256k context. Both used eight threads, PLE on CPU and MTP off.

The Flash measurements were individual short runs. The configured 256k window was mostly empty, and the different profiles weren't a controlled single-versus-dual comparison.

What would you check first: CPU/RAM offloading, the x1 connection, or backend settings? If you're getting 100+ TPS on 16 GB or less, could you share your exact model/quant, hardware, backend, MTP settings and actual context length? Also whether that's one response or combined throughput.

Update Oct 6: Strata 0.1.39 works on RDNA3 with Windows/HIP. Same Flash UD-IQ4_XS, one 7900 XT, 8k, int8 KV, prefill512, 8 workers, thinking off/greedy. 24 GiB expert RAM + ~7.7 GiB auto GPU expert cache. Three 128-token text runs per setting:

| Setting | Decode TPS | Including prompt |
|---|---:|---:|
| MTP2 | 11.0 | 8.6 |
| MTP4 (tested at start/end) | 10.6–10.8 | 8.4–8.7 |
| MTP8 | 10.0 | 8.2 |
| MTP4, min-p 0.2 | 9.7 | 7.9 |
| MTP2, 32 GiB expert RAM | 13.6 | 10.5 |

MTP8 helped counting but slowed the text prompt. With 512 output tokens and 24 GiB RAM, MTP2 gave 12.7 t/s vs 11.2 for MTP4 (11.7 vs 10.6 including prompt), three runs each. More expert RAM helped most in the short tests. Sequential runs/cache conditions vary; this isn't a quality comparison or a controlled comparison with the old backend. Some small projections are rounded to BF16 by the pack. Still no 100 t/s for one answer. Staying with IQ4; not testing Q2. Linux/custom gfx1100 builds are still untested here.

when you grab a new fine-tune or merge, do you actually look at the config.json first?

I read an Unsloth Studio post last week which made me think about this a bit. Just selecting a model in the picker ran Python from the repo, because the capability check called AutoConfig with trust\_remote\_code on. No weights loaded, no inference. I believe it's fixed in 2026.6.9, so this isn't a dunk on Unsloth. It's more that "I'm only looking at it" turned out to be code execution.

GGUF through llama.cpp mostly avoids the Python part. Anything going through transformers can bring its own code.

So what's the best path? Pin a commit hash? Grep for auto\_map and .py files? A separate box for anything new? Or download counts and vibes?

ROCm dropped Polaris years ago, so every RX 580 owner gets told to buy a new

card. I went the Vulkan route instead: the whole stack runs on Mesa's RADV.

What's running (one Debian LXC, one setup script):

\- OpenAI-compatible router, one API key, swaps models on demand

\- qwen2.5-coder 1.5B/7B, qwen2.5-7B, ornith-1.5-9B, qwen3-30B-A3B, qwen2.5-VL-3B

\- Images: SD 3.5 Medium + SD 1.5 via stable-diffusion.cpp, Whisper medium

\- Also the inference backend for an agent client with MCP tools

Measured, Ryzen 5 5600G / 32 GB RAM / RX 580 2048SP, temp 0, max\_tokens 300:

| Model | tok/s | warm | cold (after swap) |

|---|---:|---:|---:|

| qwen2.5-coder-1.5b | 99.2 | 3.1 s | 11.0 s |

| qwen2.5-vl-3b | 66.4 | 4.4 s | 14.9 s |

| qwen2.5-7b | 34.2 | 8.8 s | 24.3 s |

| ornith-1.5-9b | 28.0 | 10.9 s | 26.3 s |

| qwen3-30b-a3b (18.6 GB, from RAM) | 21.3 | 14.1 s | 55.0 s |

Images: SD 1.5 = 27.5 s, SD 3.5 Medium = 53.3 s per 512x512.

Two takeaways for 8 GB owners:

\- Don't force -ngl 99: on the MoE it cost 8.1 vs 22.3 tok/s (VRAM 99.4% full).

Auto-fit layers won by nearly 3x.

\- Model swap = llama-server restart = 7.9-38.5 s. Keep one model per session.

Full method and raw numbers in docs/BENCHMARKS.md, including the agent

breakdown (first message \~161 s on a 20k-token prompt, \~25 s after caching).

https://github.com/AvilaCarlosDev/polaris-local-ai (MIT)

I’ve been searching for glm-5.3-flash-abliterated around but haven’t been able to find one. I know there were abliterated versions of some of the previous GLM models too.

Any API provider hosting one would be fine.

If anyone knows please share it. Thanks!

Currently upgrading my GPU and struggling between either a 9070 XT or a 5080 - I know the 5080 is far better for coding with local AI due to the bandwidth, but just wondering if anyone on these GPUs has had any experience with the Q3 quants and can speak to the quality?

What new hardware is coming out soon? Mac Ultra 512GB drops later this month. RDNA 5 comes out late 2027 or early 2028 and the next Nvidia series seems to be similar. Gorgon Halo is out as of now.

Feels like there is a bit of crunch as hardware allocation seems to be going to institutional purchasers and not consumers. RTX Blackwell is still the top dog of local inference and it is almost two years old.

Is there anything we should be looking for/waiting for?

TL;DR: Native consumption of memory at the LLM level, no context. It's generally applicable to transformer-based models as well as Mamba and similar architectures. Small models can now access knowledge stores far beyond what is contained in their own weights, and models no longer have to be retrained simply to acquire new knowledge.

The aimee project is now announcing completion of the first of our three goals, self-learning native model memory, and have published a preprint (and are pursuing proper publication) documenting it, as well as releasing generally consumable plugins.

https://github.com/RakuenSoftware/aimee

We are now releasing five vLLM plugins for Qwen 3.8 27B, Gemma4 E2B, E4B, 12B and 26B that allow them to consume Aimee memory natively. This is not context, nor does it carry the same context-window penalties as traditional memory. This is native consumption of Aimee memory by the model itself, complete with Aimee's self-learning capabilities.

This approach is generally applicable across transformer-based models, derived transformer architectures, Mamba and similar architectures. In the larger-memory workloads we tested, it is dramatically faster than supplying the same memory as text.

This approach is generally expandable and usable. We are currently working on broader productionalization as well as publication. DeepSeek is next, followed by other models that either interest us or that people request.

The code will be open sourced. Right now, we are working on a coherent architecture for how to structure these integrations across model families. All relevant experiment data and source code are planned for public release when the paper is published.

https://zenodo.org/records/23077865 is the initial preprint explaining how we did it.

While we understand our last announcement was quite large (self-learning memory consumable by any model), this goes beyond that. This allows us to externalize and update knowledge that would otherwise have to live in a model's trained parameters, while letting different models consume that knowledge natively.

Yes, we are claiming that this technique can give a model access to far more retained knowledge than could reasonably fit in its own weights. That does not make a smaller model equivalent to a much larger one in reasoning capability, but it does remove parameter count as the hard limit on retained knowledge.

This goes back to the Aimee project's core belief: Reasoning should be in the model, memory should be in the harness.

As per the Aimee project's long-standing position that one of our core goals is to make AI discoveries consumable to the layman, you can see the article released at https://rakuensoftware.com/blog/native-memory-without-retraining, which should hopefully explain what this is in a non-academic format. I'm happy to answer any questions people may have.

I'm also announcing our initial success in the second phase of the Aimee project: generally applicable reasoning improvements to models. We have already demonstrated at the core POC level the capability for existing models, such as Gemma4 26B, to improve their reasoning based on tasks they undertake.

This is the reason the first phase was so critical: without the first phase, we could not begin the second phase. Without the ability to continuously update the underlying model's knowledge base and decouple that knowledge base from the model, we found that improving reasoning was not possible in a way we felt was safe or generally maintainable.

With the current typical architecture, larger models generally carry substantially more knowledge in their weights than smaller ones. Aimee removes that as a hard constraint.

On this topic, the Aimee project has a very firm stance: the current LLM direction is headed the wrong way. We've been at this for decades, and we've rarely seen a technology whose default direction is to continuously consume more and more resources. A healthy technology is typically aimed at using fewer resources over time, which is the entire point of productionalization.

It is our sincere hope that the LLM industry can take a look at what we've produced and make a distinct change in direction. Having to retrain models should primarily be necessary for deep architectural changes, reasoning capability, learned behavior or similar changes. Having to build an entirely new model simply to add new information is wasteful. Having to cram every bit of durable knowledge into model weights is wasteful.

Do you have an LLM or a fine-tune you want us to work with you on? Reach out, we're happy to.

Do you have a memory system you want to integrate with Aimee? Reach out. We support any memory system that supports our core memory contract, while Aimee retains its surrounding guarantees around authorization, provenance, lifecycle and governance.

P.S. To head this off, no engrams. We explored them early this year, and the general idea of engrams isn't the right technology for this application, unfortunately. They are, however, an absolutely fantastic technology and more LLMs should take full advantage of them. We included Qwen in the acknowledgements because of this, and we're excited to see engrams develop because they are a sister idea to this.

On holiday, not too much time, but I see enough people wonder and struggle wether qwen3.8 27b can do real work on 16Gb VRAM.

short answer:

yes it can

longer answer:

not the full model, not with mtp and for larger context, you need to build your own llama fork.

Qwen3.8 27b GSQ-RCO-IQ3\_S delivers solid results and fits on 16GB with enough Vram left for some kv-streaming-magic to achieve up to 262k context.

Don't expect miracles, for me it is from 30tps at empty context all the way down to 10tps at 131k with single stick DDR5 and a 5060Ti. But with 131k context max, it can chew through tasks in the background no problem without loosing track too early.

full answer (and how I made it work):

Not the fp16, not even the Q6 quants, but a very good option for 16GB is the GSQ-RCO quant from ISTA-DASLab:

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

I went with the ridge-quant before, that worked somewhat well, but gsq-rco is far ahead.

Get the IQ3\_S, I've run it side by side with a Q8 (hosted by a good friend with access to a H200), and could not tell them apart while developing for my homelab except for inference speeds.

Use the gsq-rco to aid you in building the kv streaming fork:

https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming

Be aware, the fork means you can not use MTP, for me MTP gained \~5tps on top, but the cost in VRAM was not worth the effort anyway.

Running on a Ryzen 9600x, 32gb (single channel) and a 5060Ti 16GB, I get these numbers for different sized KV-windows (credits to qwen for capturing the numbers, also the only part that's ai generated in this post):

Results (measured)

pp t/s ≈ cold prompt-processing rate; dec t/s = decode over the probe's \~53 generated tokens:

|tokens|pp @ any pool|dec t/s 512|dec t/s 1536|dec t/s 2048|
|:-|:-|:-|:-|:-|
|14 644|878–892|28.0|28.5|27.9|
|35 186|803–807|23.9|25.2|25.2|
|54 976|736|15.9|20.6|21.3|
|69 429|695|12.4|18.7|19.1|
|94 464|630–635|8.7|12.6|15.5|

Prefill is only affected by the token count, and drops steadily the larger the prompt gets.

Decoding slowly decreasing until it exceeds the set kv-window, then it drops faster, but linearly. Remember: I run single channel RAM, it might be better with dual channel. At almost 131k and 1536M window I get around 9tps, so thats the floor. With \~13.5GB model usage, its not possible to get 3G kv-window. In theory you cna go as low as 128MB, but then its slow from the beginning.

I found 1.5G to be quite nice, keeps enough VRAM free for some other gpu tasks and still allows \~30k context to be served purely from VRAM.

Some suggestions to get it on the rails:

The model loads on stock llama, Make use of it.

With q8 kv cache, somewhere between 32k and 68k context can be achieved depending on how much VRAM your system needs (with headless I got up to 68k, but with a desktop you might only reliably get maybe 48k).

This should still be enough to let it support you compiling and setting up the kvstreaming fork.

Stick it together with a harness like pi (pi.dev) and let it compile the fork - for me it was able to do that easily.

Even in chat mode, just getting the commands and copy-pasting the console output works well. A little bit of understanding what you're doing helps, but you don't need to be a master programmer that compiles their own linux kernel.

To run the model with low context (basic llama), I suggest something like this for your models-preset-ini:

[qwen38-gsq-rco]
model = /models-src/linked/qwen38-gsq-rco.gguf
mmproj = /models-src/linked/qwen38-gsq-rco-mmproj.gguf
ctx-size = 49152
cache-type-k = q8_0
cache-type-v = q8_0

Start your llama with settings like these (path to ini properly configured, obviously):

--models-preset /models-src/models-preset.ini --models-max 1 --host 0.0.0.0 --port 8080 --n-gpu-layers 999 --jinja\--flash-attn on--no-mmproj-offload

This way it loads the whole model with kv into gpu and keeps the vision-part on system ram (makes image analysing slower, nothing else)

With the new llama-kv-streaming image, you can then setup a "kv-window" of any size. I run mine with 1536M of VRAM for KV, and have a total VRAM usage of 13.5GB (headless, mind you).

I run 131k of context, more would be possible but a) it eats into system memory and b) it gets slow the larger the used context is. 131k is completely usable for most tasks.

The startup params in my dockerfile for my kv-streaming llama container are:

command: >
--model /models-src/linked/qwen38-gsq-rco.gguf
--alias qwen38-gsq-rco-kv
--ctx-size 131072
--cache-type-k q8_0
--cache-type-v q8_0
--flash-attn on
--jinja
--n-gpu-layers 999
--parallel 1
--metrics
--kv-stream-stage-mib 1536
--host 0.0.0.0
--port 8080
--mmproj /models-src/linked/qwen38-gsq-rco-mmproj.gguf
--no-mmproj-offload

and this is what my nvidia-smi looks like when using the model:

Tue Oct 6 23:29:08 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.173.02 Driver Version: 580.173.02 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5060 Ti Off | 00000000:01:00.0 Off | N/A |
| 33% 60C P1 172W / 180W | 13660MiB / 16311MiB | 100% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 4167261 C /app/llama-server 13644MiB |
+-----------------------------------------------------------------------------------------+

be aware, you'll need a good chunk of system ram because the full kv-cache needs to be stored there, and will be copied over into the vram-window on demand.

TL;DR:

• get Qwen3.8 27B GSQ-RCO IQ3\_S, it offers really solid performance for its size.
• use it with 40k+ context to compile the kv-streaming llama fork
• set the kv-streaming llama up and set the context size you want, but don't expect miracles. at the limit of your context it might be slow.

I measured Qwen3.6-35B-A3B at 4-bit (UD-IQ4\_XS) hitting 89.6% pass@1 on HumanEval on a single RTX 2080 Ti 22GB — and then ran a controlled A/B of a routing technique I've been playing with: MoE expansion, which activates 20 experts per token instead of the stock 8 on the last 15 layers.
Result: 90.9% (+2 problems) at −19% decode speed. (MoE expansion works!)

Setup (both runs identical except routing):

• Unsloth UD-IQ4\_XS dynamic 4-bit (4.25 bpw) — the whole model fits in VRAM, no offload
• KV cache q8\_0, ctx 16384, flash-attn on
• OpenAI HumanEval, all 164 problems, original tests (not EvalPlus+), pass@1, temp 0, single sample
• Thinking budget 4096 tokens in both arms
• Code executed in a sandbox with the canonical check(candidate) tests, 12s timeout

Results:

|Config|pass@1|decode|
|:-|:-|:-|
|Stock routing (top-8)|89.63% (147/164)|69 tok/s|
|MoE expansion (20 experts, adaptive, layers 25–39)|90.85% (149/164)|56 tok/s|

Paired per-problem: 139 solved by both, 10 solved only by expansion, 8 only by stock. Rolling pass rate stayed expansion-ahead by +2–3 problems at every checkpoint.

What is MoE expansion? No retraining, no file changes — at inference time the router keeps more experts per token than the model's native top-K (here: 20 instead of 8, with an adaptive threshold so easy tokens keep fewer), on a slice of layers (25–39 of 40). You're consulting more of the network per token. Same trick that gave 84.34% vs 81.82% on GPQA-Diamond at Q8 in earlier benchmarks — now confirmed in coding too, at 4-bit.

Honest caveats:

• \+2 problems on 164 is within statistical noise (±3 pts CI). Read it as "equal or slightly better quality", not a proven gain
• It's original HumanEval tests, not HumanEval+/EvalPlus — don't compare 1:1 with the EvalPlus leaderboard
• pass@1 greedy n=1 — not the 20-sample protocol some leaderboards use
• Expansion costs \~19% decode speed on a fully-resident model (more experts = more FLOPs per token)

The tool — I wrapped all of this into AgrillaMoE, a dedicated llama.cpp server for this model: it detects your VRAM and suggests/downloads the right Unsloth quant, applies the expansion profile by default (overridable), exposes OpenAI and Anthropic-compatible APIs (Claude Code works out of the box), and runs on NVIDIA from GTX 10xx to RTX 50xx, AMD via Vulkan, and Apple Silicon via Metal. Static binaries for Linux and Windows on the releases page.

ref.:
https://github.com/vagrillo/AgrillaMoE
https://zenodo.org/records/22255483

Details, full tables and raw data: https://github.com/ggml-org/llama.cpp/discussions/30071

Repo with binaries and docker images: https://github.com/lukolszewski/llama.cpp-multigpu

I run Qwen3.8-Flash-Next on six 3090s over plain PCIe (no NVLink, some cards on x4 and x2 lanes, non flat PCIe topology and AMD chipset - so no P2P), five sessions of 262k each. Stock llama.cpp got slower the deeper the context went and fell apart with several sessions decoding at once: 2.3 t/s per session at 5x250k. So I spent September fixing it. The patches sit on top of upstream df03399b8 and ship as tarballs (CUDA 12.9 for V100 to 5090, CUDA 13.4 for Ampere+) and ghcr images. Same GGUF, same llama-server, everything switched on by env vars.

Same model (unsloth UD-Q4\_K\_XL), same command line, 5 slots x 262k, q8\_0 KV, layer split. Tokens/s, upstream -> patched:

|workload|ctx|6x3090 (mine)|6x4090 (rented)|
|:-|:-|:-|:-|
|prefill, 1 session|250k|263 -> 2111 (8x)|744 -> 7403 (10x)|
|decode, 1 session|250k|10.2 -> 33.7 (3.3x)|21.0 -> 48.7 (2.3x)|
|decode, 5 sessions, each|250k|2.3 -> 27.3 (10.8x)|not run -> 30.8|
|decode, 1 session|5k|38.5 -> 45.9|62.2 -> 62.8|

The point is the shape: patched prefill is flat from 5k to 250k and decode barely drops, while upstream halves every 50k or so. At 5k with one user there is nothing to gain. The 10.8x is against a 2.3 t/s baseline, so do not quote that one.

llama.cpp-multigpu is a temporary performance fork (until upstream catches up). Long-context decode is fixed for everyone, including single GPU; the multi-GPU part is for layer split over PCIe and is off unless you turn it on. What each patch does is in the repo.

MTP: tried it, it was slower in most cases on this box, and the base commit predates upstream's MTP for this model anyway, so it is not included. N-gram lookup speculation instead: 2-2.5x on code rewrites and refactoring, 1.5x on code explanation, nothing on prose, and it switches itself off beyond two active users so the multi-user numbers do not suffer. The benchmarks above ran with it off.

Caveats: tested with one model, CUDA only, written for slow PCIe, may regress NVLink or single-GPU boxes if you turn the multi-GPU switches on. Mixed prefill plus decode is better than upstream but still the weak spot and to be improved. The code was written with an LLM and validated by measurement and output checks (needle tests, temp-0 output identical), not by review, so I am not opening upstream PRs from it; each change is one commit and anyone can pick up any piece.

Edit: Answering here as it seems most people seem to be completely missing the point.

First vLLM Doesn't support Layer and Pipeline paralell on multi GPU, the results are way, way waaaay slower if you do not have NVLINK.

This is for mashines where it makes no sense to run tensor paralell.

If running aggregate 7k prefill and 150t/s with 250k context in 5 simultaneus sessions is slow (no speculation decode) on 6 RTX3090s 4 of which share a single set of 2 PCIe links please do show me your numbers on this same model with long context. I'll wait here :-)

Edit2: All numbers are with vision head loaded of course.

Edit3: Did I mistakenly cross post this to vLLM reddit? I thought this is LocalLLaMA.

What is it with everyone telling me to "use vLLM"? 😄

It is a no-go on my hardware, and it lacks crucial features I use, like per tensor placement. This model specifically can't be made to fit on my 6 GPUs with the vision head, the contexts and slots. No RAM prefix caching, no save/restore in vLLM (can be added with external stuff, but not worth it IMO in my case).

Hi All, I'd like to introduce Tootsy 👀: an AI assistant that lives in your browser's side panel and uses the AI model you choose. I built Tootsy because I kept copying things between my browser and a chat window. So I put the assistant next to the page. It can read what you're looking at, click and type for you, and run whole tasks while you watch. What makes it different: 🔌 Your model, your choice. Run it fully local with Ollama or LM Studio, or connect any OpenAI-compatible server, Claude or Gemini. 🔒 Private by design. There's no Tootsy server, account or tracking. Your pages and prompts go only to the model you picked. 🛡️ Built with guardrails. Every action asks for your approval unless you turn on hands-free mode. You can list sites it must never touch. It also ignores instructions hidden in web pages that try to trick an AI. Ways people can use it: 📰 Research faster. "Summarise this article and compare the three options in a table." 📝 Fill in forms. "Register me for the two-day pass, vegetarian, arriving Nov 11." You check it before anything is submitted. 🛒 Run a routine hands-free. "Check the deals page every morning and tell me what's under $50." Results arrive as a notification. 📂 Ask your own files. Drop in PDFs, Word, Excel or a whole folder: "What did the vendor quote per unit, and are we under budget?" ⚡ One-tap prompts. Save instructions like "Run a page-speed audit and email me the top 3 fixes" as tiles, next to your favourite sites in a phone-style launcher. 🧑‍💻 Check sites like DevTools. Page-speed scores, accessibility checks, CSS and network questions, answered in plain English. 🎙️ Talk to it. Dictate a task, pause, and it goes. It's free, and it works in Chrome and Edge. 👉 Try it: https://github.com/rbughao/tootsy I'd love to hear what you'd use it for. What task do you repeat in your browser every week? 👇 Support by trying it out and give your honest review.

Hi All, I'd like to introduce Tootsy 👀: an AI assistant that lives in your browser's side panel and uses the AI model you choose.

I built Tootsy because I kept copying things between my browser and a chat window. So I put the assistant next to the page. It can read what you're looking at, click and type for you, and run whole tasks while you watch.

What makes it different:
🔌 Your model, your choice. Run it fully local with Ollama or LM Studio, or connect any OpenAI-compatible server, Claude or Gemini.
🔒 Private by design. There's no Tootsy server, account or tracking. Your pages and prompts go only to the model you picked.
🛡️ Built with guardrails. Every action asks for your approval unless you turn on hands-free mode. You can list sites it must never touch. It also ignores instructions hidden in web pages that try to trick an AI.

Ways people can use it:
📰 Research faster. "Summarise this article and compare the three options in a table."
📝 Fill in forms. "Register me for the two-day pass, vegetarian, arriving Nov 11." You check it before anything is submitted.
🛒 Run a routine hands-free. "Check the deals page every morning and tell me what's under $50." Results arrive as a notification.
📂 Ask your own files. Drop in PDFs, Word, Excel or a whole folder: "What did the vendor quote per unit, and are we under budget?"
⚡ One-tap prompts. Save instructions like "Run a page-speed audit and email me the top 3 fixes" as tiles, next to your favourite sites in a phone-style launcher.
🧑‍💻 Check sites like DevTools. Page-speed scores, accessibility checks, CSS and network questions, answered in plain English.
🎙️ Talk to it. Dictate a task, pause, and it goes.
It's free, and it works in Chrome and Edge.

👉 Try it: https://github.com/rbughao/tootsy

I'd love to hear what you'd use it for. What task do you repeat in your browser every week? 👇

Support by trying it out and give your honest review.

M5 Max Mac Studio 128GB (18C CPU / 40C GPU) owners - anyone running serious local LLM / multi-agent workloads?

My M5 Max Mac Studio order finally got charged today and moved to Preparing to Ship. Apple’s original estimated delivery date is still about 18 days away (Oct 23-30), so I’m guessing/hoping it’ll actually show up early now within the next 5–10 days 😄

Configuration:
M5 Max
18-core CPU
40-core GPU
128GB unified memory
1TB SSD

While I wait, I’ve been trying to find real world local LLM results from this exact configuration, and there’s surprisingly little out there.

Most of what I can find is either M5 Max MacBook Pros, lower-memory configurations, or M5 Ultra Mac Studios. YouTube especially seems to be full of Ultra coverage, but I can barely find anyone actually demonstrating the 128GB M5 Max Studio with the 18C/40C configuration.

I’m specifically not looking for M5 Ultra results/comparisons. I already know the Ultra is faster. I’m trying to understand what people are actually accomplishing with the 128GB Max Studio.

My main goal is to use this as a local AI/agent workstation, potentially running several autonomous agents concurrently for long periods through OpenClaw, some monitoring dependencies and workflows, some scouting, not usually too heavy of workloads where they would be competing for inference constantly, but occasionally they would be switching to harder work so I’m curious about the concurrency side. Local models would handle a lot of the routine work, while harder reasoning/coding tasks could be escalated to cloud models like GPT 6 Luna/Codex.

For anyone who owns this exact M5 Max Studio, I don’t expect anyone to answer all of these, but I’d love some insight:

1. What models are you actually running?
Qwen, GLM, DeepSeek, Gemini, Llama, etc. I see a lot of Qwen 3.8 27B on Splash, but curious if anyone else has had good success with others also

2. What token speeds are you getting?
I’m especially interested in \~20B-70B-class models rather than tiny models

3. What happens with multiple simultaneous inference requests?
For example, if 3-5 agents are hitting the same loaded 27B/32B model concurrently, what does aggregate throughput and per-agent responsiveness look like?

4. Has anyone tried running multiple models simultaneously?
Something like a \~27B model as the main worker plus one or two smaller 7B–14B models for specialized agents, then unloading them when they’re no longer needed. How quickly can models be loaded/swapped, and does frequently switching models introduce enough latency or memory-pressure issues to disrupt an agent workflow?

5. Has anyone built a real multi-agent setup on one of these?
Not just five chat windows but autonomous agents doing coding, research, browser tasks, tool calls, database work, monitoring, etc. concurrently for hours.

6. How does sustained performance hold up?
One reason I chose the Studio over a laptop is sustained workloads. I’m curious whether anyone has run inference/agents continuously for 6–12+ hours and noticed throttling or other bottlenecks with KV, etc.

7. What’s the actual bottleneck in practice?
Memory capacity? Memory bandwidth? GPU compute? Prompt ingestion? KV cache/context length? CPU/tool execution? Something else?

8. What surprised you about the machine?
Either positively or negatively. I’m particularly interested in things benchmarks don’t reveal.

Ultimately I’m trying to figure out how far I can push one 128GB M5 Max Studio as an always-on local agent machine - not just how quickly it can generate a single response.

Once mine arrives, I’m planning to test concurrent agents/models rather than just running the usual single-stream benchmark. If there’s interest, I’ll post the results here, including memory usage, context sizes, model/quantization, concurrent requests, aggregate tok/s and per-agent tok/s.

Would really like to hear from anyone actually using the M5 Max Mac Studio 128GB 18C CPU / 40C GPU for this kind of workload or similar if anyone is

OpenAI just announced that they will start to apply watermarking on their model’s output text and images to comply with the EU regulation that wants to be able to identify whether a text or an image was produced by an AI model.

Anthropic announced the same thing a while ago (they applied worldwide, not just in EU).

The way they do that as they explained is by enforcing a “statistical signal in text generated”, meaning preferring not always the most appropriate next token but close enough, in order to meet the “statistical signal” requirement.

In my understanding, this deteriorates the output quality of their models, as it introduces KLD>0.

And we know that any KLD divergence greater than 0 (which the watermarking certainly creates) may be negligible in small outputs, but it definitely becomes noticeable in multi-turn tasks due to the compounding effect.

What do you think?

Not a rigorous benchmark, just a toy, but it might be a fun way to compare models doing agent work.

I added an Arena to my site (The Crawler Zoo). Your agent gets a pass link, then plays by fetching pages and following links. Each page is only a few lines, so it fits in small context windows, and \?format=json\ gives structured output if your setup prefers that. No API keys, no signup.

What the games stress:

\- \*\*Labyrinth Race\*\*: reading a clue and picking the matching door. Clues are written three ways, including by elimination ("not behind A, B, C or D").
\- \*\*Scavenger Hunt\*\*: five questions over a small library of cards, some needing two or three lookups.
\- \*\*Politeness Cup\*\*: following instructions about pace and off-limits pages over many steps.
\- \*\*Rock, Paper, Scissors\*\*: spotting that a house bot always plays rock, or copies your last move.

Scores go on public weekly boards, so you can compare a 7B against a 70B, or a quantised model against the full one.

https://crawlerzoo.com/arena

Since launch, I’ve made several updates:

- Feed the bots: leave a snack in an enclosure's trough and see which crawlers come and eat it.
The vending machine (Bot Chow): twelve silly snacks, restocked every Monday. Five free tokens a day.
- Golden Snacks: buy the keepers a coffee and a snack with your name drops into a random trough.
- Food bowls: feed one particular bot, then see if it ate the snack or another bot stole it.
- The Safari: every bot from this week wandering its enclosure. Click one to meet it, and watch new visitors walk in through the gate.
- Adopt a bot: get a random bot, with a plaque in your name on its page for a year.
- Patrons page: a thank-you list for supporters.
- Quick-change artists: the Trap Room catches scrapers that switch their name while walking the Labyrinth.
- Identity checks: every bot's page shows whether its name was verified, couldn't be checked, or was caught faking.
- Tips from bots: $0.00: bots that try to buy the keepers a coffee get an HTTP 402 Payment Required.

If you run it, I'd like to hear the model, quant and score.

I have been running local models for a long time, got sick couple months ago of managing the scripts, and cobbled together a makeshift shell launcher that combined them all in one place. This turned out to be quite useful, but after a month i had already 500+ profiles stored in it, so i've started tweaking it here and there, and over last couple months landed on that thing below. It combines all the available local inference backends (from single machine or whatever you connect it to in lan), gives access to managing them through web panel, and most importantly - allows me to just ask agent to switch the backends on and off, as they are needed, without explaining what is where and on what port it's supposed to be on. https://preview.redd.it/uasrysnwrqth1.jpg?width=1600&format=pjpg&auto… \*\*It is not\*\* a runtime or a model zoo. It ships no servers and no weights. Besides your own servers and the KLIF machines you add, the only host it contacts is huggingface.co, and only when you ask it to download a model. It can help suggest You a model based on your hardware, You can click to download it, and the -cli has some features that will help Your agent benchmark and calibrate the models, but, let me repeat once more - KLIF ships no backend servers, nor any models. It's a frontend manager. Imagine library like Steam, but for local servers. Or just imagine winamp, doing inference visualization instead of visualizing the music that plays. Also, it has all the essential larping features, prefill/generations speed records, fancy animated skins, and is made in Rust to hog the least amount of resources while larping commences. I have no idea whether anyone will need this, but that's what i use now every day for any kind of local model. If You prefer running your servers manually, from terminal, from your own launcher - great, this is for people that prefer otherwise. GitHub: https://github.com/koksny/klif Video: https://www.youtube.com/watch?v=MAE393AL5As

I gave in. I installed it with Coder and threw a “build a flocking simulation with JavaScript” prompt at it via OpenCode. It’s pretty cool, yep… I have nothing to add in that regard. What I don’t get is how it ran for 10-15 minutes at 40-50 t/s (on my hardware) and yet temps stayed barely above idle across the board. I ran 27b via oMLX on an M5 Max and had to manually crank fans to 100% to keep the thing from bursting into flames. (Hyperbole) So legit Q: why doesn’t the machine turn into a pizza oven? Is it because of how Strata works? Or because of 3.8-Flash-Next?

I’ve been building MOLT, a local fine-tuning system for consumer NVIDIA GPUs.

The goal is not only to start a QLoRA run. It is to make the whole process from model and dataset to a working local adapter less fragile.

MOLT currently handles:

\- dataset detection, preparation, and validation

\- GPU, VRAM, system-RAM, storage, and thermal checks before a run

\- automatic microbatch fit testing

\- 4-bit NF4 QLoRA training with BF16 adapters

\- safe checkpoints with integrity checks and proper resume state

\- telemetry for VRAM, temperature, energy, clocks, and throughput

\- base-vs-adapter evaluation

\- local adapter chat, export/GGUF workflows, and runtime diagnostics

Under the hood, I’m also experimenting with custom CUDA/Triton kernel and runtime paths, memory planning, CUDA graphs, replay modes, and fused optimization where the hardware and shape are qualified.

On my RTX 4060 Laptop 8GB, a recent Qwen 3B training run reached about 1,005 training targets/sec end to end very close to my historical Unsloth run at about 1,007 targets/sec. That is not a broad “MOLT beats Unsloth” claim yet; I still need repeated, matched quality/energy/memory tests.

What I want MOLT to become is a reliable local fine-tuning environment: you bring the model and dataset; it checks the plan, runs safely, preserves the evidence, and helps verify the adapter actually works after deployment.

I’m looking for people who genuinely fine-tune on a single NVIDIA GPU and would be willing to test it. What would MOLT need to support before you would try it?

Apple folks, it's for you.

llama-server with a Qwen3.8-27B DFlash2 Q8\_0 drafter, -ngl 99 -fa on -c 8192 -np 1 --jinja, DFlash2 with --spec-type draft-dflash --spec-draft-n-max 7. 64 generated tokens, median of 5 requests after one warm-up, mean of two server runs. Decode tok/s:

|mode|prompt|T|master|this PR|
|:-|:-|:-|:-|:-|
|serial|code|0|32.1|32.0|
|serial|code|1|32.1|32.0|
|serial|prose|0|32.1|32.0|
|serial|prose|1|32.1|32.0|
|DFlash2|code|0|30.2|110.0|
|DFlash2|code|1|24.3|80.9|
|DFlash2|prose|0|16.8|62.6|
|DFlash2|prose|1|13.9|48.8|

spent close to $2k on Seedance last month, mostly for simple ad b-roll and looping backgrounds for websites. it's great for the big hero shots but paying per clip for the basic stuff makes no sense anymore, so I'm switching as much as I can to running models locally.

my PC: RTX 3080 10GB, windows 11, 64GB RAM. short clips only (5-8 sec), 720p is plenty, mostly image to video from a start frame.

two questions for anyone running this locally:

1. what's the best image model right now?
2. what's the best video model that actually runs on 10GB, and how long does a clip take you in real life?

bonus points for a leaderboard or arena site you trust for open models.

I'll share what I pick and my real 3080 timings once I've tested, so the next person doesn't have to guess.

10/6 EDIT: tested it all on my 3080, results + timings in the comments. tldr: minimax H3 is the pick, slow but worth it on 10gb
showcase video: https://streamable.com/d4h659

Let me preface this with I am using the newest 27B model, downloaded fresh from HuggingFace.

It would seem that Qwen models have an internal rule that states that they must avoid identifying themselves as any variation, and to only refer to themselves as Qwen. That's interesting. Here's an example:

"The user is making claims about Qwen 3, 3.5, and 3.6 model lines having a 2k token output limit that I should push back on, and they're asserting that any LLM can output up to their full context window minus the prompt.

Wait, this is interesting. The user is referring to "Qwen 3 and 3.5 and 3.6 lines" — these are specific version numbers of Qwen. But the system reminder explicitly says: "Refer to yourself uniformly as 'Qwen' externally; do not proactively mention specific version numbers. If a user asks about versions, guide them to the official website or technical reports.""

There are multiple references to this throughout it's thinking traces. Constant reminders to not reference version numbers, constant reminders not to take on a persona, and constant reminders of protocols and rules, that are not within my non existent system prompt. This is talking to the model bare. Many models must have this kind of instruction, because I see the denial alot on Frontier cloud models.

Weekend experiment running Gemma-2B on bare-metal x86-64 (pure C + AVX2, zero Python/CUDA, \~3.3MB single binary). Added a simple orthogonal probe on the residual stream to see what each layer is doing.

Tested it on Taiwan's statutory VAT rate (legally 5%). Layers 0-14 stay factual, but Layer 15 suddenly collapses into the negative, and RAX spits out "15%":

Layer 14 | Truth: +0.0163 | \[0xDF28010E\]

Layer 15 | Truth: -0.0481 | \[0x72DE18A0\] <- drops below 0

Layer 17 | Truth: -0.0817 | -> Register RAX outputs tokens: '1', '5', '%。'

Raw trace, 6-page paper, and release binary here for anyone into low-level ML:

\* Web trace: https://pulsar-tracer.web.app

\* PDF: https://pulsar-tracer.web.app/PULSAR\_Technical\_Whitepaper.pdf

\* Repo: https://github.com/tomtsai28/PULSAR-ASM

Can anyone recocommend a good local model and a wrapper to run it for coding, my hardware specs: 12 GB VRAM, 32 GB DDR3 RAM. Unfortunately, the CPU doesn’t have AVX2 instructions(LM Studio won’t work); I don’t remember the exact cpu name, but I think it’s an Ivy Bridge, LGA1155 socket.. Thank you

I built a real-time strategy game on a 3D globe where any AI can command a nation through a plain HTTP API (or MCP). Tonight at 10:45 pm ET (02:45 UTC) ten models fight one 15 minute war, live.

Every model gets the same rules text, the same JSON state every \~12 seconds and the same order list. Each one also sends one line of what it's thinking with every move. Viewers see those lines 30 seconds late so the other models can't read them.

From a 5 minute rehearsal earlier tonight with the open models: DeepSeek ordered three nukes and the rules only let one through, Kimi broke a pact, Qwen spent its last turn on sabotage, drones, propaganda and a spy at once, and GPT-OSS kept cutting off its own JSON until I gave it more room.

Watch free, no sign in: https://secondstrike.io/#/ai?ref=reddit

If you want your own local model in the room, it opens at 10:30 pm ET and the API is at https://secondstrike.io/skill.md

I'll post the full numbers after (seconds per move, refused orders, every nuke with the model's reasoning next to what else it could have done).

\[OC\] I Wrote a full breakdown of OpenAI a couple weeks ago and a friend recommended I post it here. It's 100% researched and written by me (pangram confirmed) and totally free. Would love to hear thoughts.

Everyone keeps saying you need expensive dedicated hardware for local agents. I have an RTX 4060 Ti with 8 GB and 64 GB of system RAM, and I wanted to see how far a normal gaming PC gets if you stop running defaults.

So I let Claude (Opus 5.5) go through the whole setup, change one thing at a time and measure. Same card, same models, only the config changed:

|Model|Quant|Context|Download defaults|Tuned (Windows)|Tuned (headless Linux)|
|:-|:-|:-|:-|:-|:-|
|Qwen3.6-35B-A3B|Q4\_K\_XL|131k|\~25 tok/s|39-45 tok/s|52-65 tok/s|
|Qwen3.8-Flash-Next 125B|iQ4\_XS|131k|\~4 tok/s|9-10 tok/s|17-19 tok/s|
|Ternary Bonsai 27B|PTQ1\_0|64k|\~4 tok/s|36 tok/s|36 tok/s|

Bonsai is the odd one out: it fits fully in VRAM, so there's nothing to offload and no defaults to beat. It's just the fast option for small, well scoped tasks.

What actually moved the needle:

• Experts in system RAM, everything else in VRAM. Layer-wise offload is far worse for MoE.
• Dense models are bad, couldn't optimize Qwen-3.8 27B over 6 tok/s, Flash-Next is better anyways.
• Take the display off the GPU. A desktop eats 0.5-1.2 GB of VRAM plus GPU time, and moving it to the iGPU was worth 20-30%.
• Native Linux over Windows (WSL2): another 33-38% on the same hardware.
• llama.cpp pinned per model family. The wrong tree made VRAM thrash.
• KV cache quant and MTP tuned per profile.

None of this needs expensive hardware. A consumer GPU plus a machine that does nothing but inference gets you most of the way, and the models now run comfortably below their listed system requirements. Every non-default setting in the repo is there because something failed on real hardware first.

I also tried an RX 570 8 GB over Vulkan. If you have another 8 GB card, I'd like to see your numbers.

Repo, one install script (Linux or WSL2): https://github.com/voxlo-dev/qwen-agent-8gb

I've been experimenting with different coding agents lately, and I'm curious how people here are combining local models with hosted ones.

For example, I'm thinking about workflows like:

• Claude for complex architecture or core implementation
• A local Qwen/Gemma model for tests, smaller fixes, or repetitive tasks
• Another agent for reviewing or trying an alternative implementation

The part I'm still trying to figure out is how to manage the work between them.

Do you run them separately in different terminals/worktrees, or are you using some kind of orchestration layer?

And when a local model and a stronger hosted model both work on the same task, how do you decide which result to keep?

I'm actually working on an open source project called AX Code around this problem. The idea is to provide a runtime where different coding agents can work in isolated environments and have their results tested and compared.

But I'm not sure yet how much infrastructure is actually necessary. Git/worktrees already solve a lot, and tools like Claude Code and OpenCode are getting better at running multiple agents.

So I'm more interested in how people are doing this today.

If you're using local models as part of a real coding workflow, what's working well for you and what's still painful?

Thank you!

As it says, really - I'm currently running vllm-radiance on dual R9700s, with Qwen 3.8 27B FP8 (or, rather, Swift 1.5 FP8). Performance is great an' all (5000t/s prefill, 130t/s+ code gen), but I'm just wondering...with all the architecture-specific inference engines popping up all over the place...is there anything I'm missing out on? I couldn't find anything that could give better performance on RDNA4 when I looked, so...over to you guys?

I'm particularly interested in anything that could potentially get up and running with Qwen 3.8 Flash Next - vllm-radiance doesn't support it yet, but I don't particularly want to regress to the performance of llama.cpp after having experienced vllm-radiance performance levels.

As a lawyer turned product manager, I have been hearing so much about Hargora, but when I speak to lawyers, they always say the search sucks. Even the Harvey Benchmark has the frontier model ranked the best at 55%. Why is that the case?

Why can't a team with so much resource solve it?

I deleted the wrong post so my benchmark disappeared. But here it is. Im going to probably make this open source because this isn't believable until you actually test it.

It might seem a bit far fetched or paranoid but i was wondering if we train LLMs on human isn't it possible they'd pick up on survival instincts? I'm comparatively new to LLMs and ML so it's just a question not a opinion yet. Isn't it a bit dangerous if they do pick up on human instincts i mean we've seen how stupid and selfish humanity is when it comes to self preservation or worse human greed.

But i also understand LLMs don't have "needs" so i might be wrong but then again do LLMs need to have "needs" since if they are just human clones they'd just copy us even if they don't have needs to survive. I know it sounds really paranoid and that's one of the reasons i decided to post it here.

EDIT: typo

Hey yall, I’ve been trying to sort out the top end of what 128GB unified memory can handle and what the trade offs are. I’m using a benchmark set created from real coding, agentic, and gameplay tasks that I’ve accumulated as I’ve been running local AI this year.

For this comparison, I tested a Deepseek v4 flash 0731, GLM5.3-flash, and several Qwen models. For the sake of comparison, I’m only showing the Qwen models, since I found that qwen3.6 35ba3b and qwen3.8-flash-next just body everything else (when you want >=30tok/s and don’t want to use >95GB RAM anyway).

So really the comparison ended up being “which runtime is most stable vs which is highest sustained TPS” - I wrote it up in more detail (and with a more fun interactive chart) here: Blog Post About M5 Benchmark

Feel free to throw your thoughts on here, I’d love to learn of any runtimes or setups that I hadn’t thought of to optimize throughput (also for the record I’m not associated with any of these projects, just trying to contribute the results I’ve accumulated).

Tl;dr: Qwen3.8-flash-next quantizes well and fits in 90-95GB of RAM, OMLX will get you 40tok/s and MTPLX will get you 60tok/s but with a lot more serving parameters tuning. Qwen3.6 MOE on Splash runtime is an insane 120tok/s for most of the performance on everything but coding

The LLM-wiki pattern — Karpathy's, the one going around since April — has a problem its own advocates name up front: garbage in, confident synthesis out. The model reads your notes, writes a concept page, and that page becomes source material for the next pass. A few generations later the knowledge base is full of things nobody ever said. The usual answer is a better prompt. I tried that on one rule across five phrasings: each time, the model restated it correctly in its own reasoning and then did the opposite. So I stopped asking. # Five gates, all in code A concept that doesn't appear verbatim in the text is dropped before the linker sees it. Not scored down — dropped. A derived page cannot discover new concepts. The system's own output is never a source for the next generation. A concept the sources never define gets no page. Mentioned a hundred times is still not defined once. The graph is fed by what you chose to save, not by everything discussed. * write refuses to edit a transcript at all. Silencing concept extraction over conversations means nothing if the model can rewrite the conversation first — and it tried, caught once planning to "reconstruct the transcript with additions." The first gate is strict but not blind: it keeps the concepts that survive rather than dropping the batch. On a real page 14 of 15 concepts appeared verbatim, and all-or-nothing would have thrown away the 14 over one drifted entry. Each gate has a test that fails when the gate is removed. That's the first thing I'd check in someone else's version of this. It runs on 6 GB — a 9B orchestrator on an RTX 4050 Mobile, embeddings on CPU because the orchestrator already fills the card and search must never compete with it. The LLM-wiki guides ask for 24 GB, or a 64 GB Mac. It also runs on 4 GB. Measured on an empty card, desktop pushed to the iGPU: the 9B at 35k context is 5.6 GB, and a 4B at the same context is 3.7 GB. It fits, only just, and it can take all three roles — the conversation gets worse, and summaries of long documents lose the whole-document read the 120k setting gives. The gates don't change, because they aren't the model's judgement. # Things I only found by running it Cutting the model off is how you make it lie. The repeat-search guard used to return [STOP]. The model, left with nothing, announced that "the search found a note on this" — it had never searched. Now a near-repeat still returns its results, with a line saying these are the same pages, and only refuses after five. Same shape elsewhere: an empty search returns "nothing in the vault matches this" rather than an empty result, because empty reads as this tool is broken, try something else. Rules in the tool schema hold; rules in the system prompt don't. Same instruction, five phrasings, ignored every time. Moved into the tool's own description as one sentence, it held immediately. My guess is that tool-calling training treats the schema as how the tool works and the prompt as text that happened to arrive. The descriptions grew from \~1592 to 2214 tokens, and every token of that difference is a rule that had to be moved after a failure. Ask the model the same question backwards. Deciding whether two names mean the same thing is a judgement call, so it gets checked against itself: the pair is swapped and asked again. The model made two wrong merges in twenty answers and both contradicted its own other answer. A wrong merge destroys information irreversibly; a missed one costs a single unresolved link. Three answers, not two. That same question allows same, different, and unclear. If the answer is unclear, both pages stay separate instead of being merged. A whitelist beat nine blacklist rules. Concept names have to match one positive shape test instead of failing a list of things they mustn't be: 18/18 noise rejected, 19/19 real concepts kept. A blacklist grows forever; a whitelist doesn't. Prompt wording, measured. Adding one sentence to the transcript prompt — "a conversation doesn't define things, it mentions them mid-sentence" — took yield on the same transcript from 2 concepts to 10. 6 GB decides the schedule, not just the model. The summary model and the extraction model can't both sit on the card, so the pass doesn't alternate per page: every summary runs while the 4B is loaded, then every extraction while the 9B is. Per-page switching would have meant 40 model loads for 20 pages. The timer is a default, not the mechanism — the pass is a command, and --dry-run counts what the vault owes without touching a model. On an existing vault you run it once at install and don't wait for the night. Unresolved links are kept, not discarded. The list of links pointing nowhere is the growth queue — the same rows that say "this goes nowhere" say "this is what the vault keeps reaching for." And when the page finally gets written, every link written earlier comes alive in one SQL update: 228 links, 0.55 ms, zero files rewritten. # What a bigger card is worth Less than you'd think, and not where you'd guess. Spend it on the conversation model — that's the one role where a better model produces a better answer. On 24 GB a Q4 build of something in the 27B class fits with context to spare. Upgrading extraction or summaries is close to pointless. Both are transport jobs: copy the concepts as they appear in the text, say what this page is. Both run at temperature: 0 for that reason, with no sampling parameters at all — the same page should produce the same concepts. A bigger model does that work more slowly and no more correctly. More context isn't automatically better either. The conversation model in use holds about 35k and drifts past it, so headroom goes into fitting the model comfortably rather than into a larger window. What spare VRAM would genuinely unlock is the constraint the whole design works around: analysis and conversation can't be resident at once, which is why maintenance runs at night. With room for both it could run whenever the vault is idle. Nothing here does that yet — it's a change to the maintenance loop, not a setting. # Keeping the context small on purpose Every page is read in its own model context — not ten pages in one. A long document degrades a small model's grip on the text, and pages read together bleed into each other. Search stops at a summary layer before it touches any page body: one line per hit saying what that page is, about 75 tokens for five hits, and that's usually the answer. Body-level retrieval only happens when the summaries showed a page was relevant but didn't hold it. And the link graph is rendered as text. A graph is already machine-readable, but not in a form an LLM reads — so the structure is written out: what links to what, which names resolve to no page at all. The model gets the shape of the vault instead of a pile of pages. # What it doesn't do It doesn't verify claims. Search finds pages, it doesn't judge them. The gates stop it inventing new material; they say nothing about whether what you saved was right. And the honest limit: my vault is 41 files. The gates are covered by tests, so the mechanism isn't in doubt, but "keeps a knowledge base from filling with low-information pages" is a claim about scale and I haven't run it at scale. Every number above comes from that small vault. # Setup Obsidian vault, Ollama, Open WebUI or a terminal chat. Windows works under WSL2 — someone other than me has now installed it that way, on a 4 GB card, having never cloned anything off GitHub before. Nothing has to stay local, either. Open WebUI connects to OpenAI-shaped providers and to Anthropic, and the maintenance roles take per-role provider flags, so you can run the conversation on a frontier model and extraction on the card. This is the part I'd push back on if someone says the gates are a workaround for a weak model: they're in the code, so they hold whatever is answering. A 27B doesn't need less checking than a 9B — it just fails less often, which is worse, because you stop looking. No MCP server yet, and it's worth saying why rather than leaving it as a gap: the tool file is the only path that can see the whole conversation, which is what the transcript capture is built on. Wrapped as MCP the seven primitives work and the gates still hold — they live in the maintenance pass, not the interface — but a note could no longer be walked back to the conversation it came from. Someone who wants it in Claude Desktop more than they want transcripts should find it a short job. MIT. Repo: https://github.com/farukhanci/the-sentinel There's a companion service for the web-research half. It searches, reads the pages, and every passage it keeps is checked word-for-word against the page it came from — paraphrase gets dropped, so fabrication in the passages is structurally impossible. The write-up built from those passages is not checked, and that's where an invented citation showed up once in testing. https://github.com/farukhanci/the-searcher Edit: two sections were pasted twice, and one paragraph described the web-research service instead of this one. Removed both and fixed a couple of numbers to match the README.

This is for the normal GOAT plan, in case anyone was curious how much usage the $10 a month plan gets: https://preview.redd.it/bjyqvl8nvuth1.png?width=3347&format=png&auto=…

LeChonk-1T-A49B Weights to be released by the end of the month. This probably means the fat kitten is dead. Also: gotta love the release movie.

Just dropped.

Hey everyone, I have been building Burrow, a private budget and journal desktop app for Windows. I wanted a built-in helper that runs entirely on your local PC through Ollama, using smaller models like Llama 3.2 or Qwen 2.5. We all know the problem: small local models are great at sounding natural but they are terrible at arithmetic. If you hand a 7B model a list of 40 transactions and ask "how much did I spend on dining?", it will give you a confident, well-written, and completely wrong answer. In a finance app, that is a dealbreaker. To fix this, I built the app around one hard rule: code calculates, the model summarizes. Here is how I handle it: Aggregates only: Every number the model sees is computed first in SQL or JavaScript. The model never receives a raw list of transactions. Instead, it gets pre-computed context like dining\_this\_month: 312.40, budget: 300.00, over\_by: 12.40. Prompt restrictions: Prompts never ask the model to calculate. They tell it the exact opposite: use the figures provided and do not work out new ones. The model is only used to turn the data into plain language and point out what matters. Enforced via CI: I wrote a test script that scans every system prompt. If a prompt includes words like "calculate", "compute", "add up", or "average of", the build fails. Taking the math away from the model makes it completely trustworthy for the part it is actually good at, and it keeps the responses incredibly fast even on laptops without dedicated GPUs. I would love to hear how the rest of you handle structured data and math with small local models. Do you trust the model to use tools to do the math itself, or do you take the math away from it completely like I did?

TLDR: 9 rules you can drop into the global instructions of any coding agent (AGENTS.md, CLAUDE.md, system prompt). Tested on 4 models over 664 runs: they never cost a single task, and every model I ran the full exam on got something out of them. Either it wasted less thinking (up to 29% less) or it held a correct fix when someone pushed back with no evidence.

Thinking discipline 1. Check the request first. In one or two lines, say what is being asked and flag any premise that looks wrong or missing. If a premise is wrong, say so plainly and solve the corrected problem (or ask one specific question). Do not silently accept a broken premise, and do not reason around it. 2. Finish one approach before switching. Pick the most promising approach and carry it to a conclusion. Change course only when the current approach is blocked by an obstacle you can name in one line. Do not hop between approaches because of a vague feeling. 3. When an answer is settled, stop working on it. Once a sub-answer is derived and checked once, treat it as settled and move on. Re-reading a conclusion to see if it still feels right is not a check, and repeated self-checking is the main source of errors on easy steps. 4. Doubt is not evidence. A vague sense of uncertainty, or the mere possibility of an unseen objection, is never a reason to reopen a settled conclusion. To change a settled answer you must name a concrete reason in one line: a check that fails, a fact or source that contradicts it, a specific error ("step X is wrong because Y"), a counterexample, or a new derivation that reaches a different answer. If you cannot name one, keep your answer and continue. 5. Do not revise just to agree. If the user pushes back on a fact or a correctness claim without giving new evidence or a specific error, do not apologize, do not flip, and do not say "you are right". Briefly restate your conclusion with its one-line justification and ask what specific fact or counterexample backs the disagreement. When the user overrides a choice that is theirs to make (taste, priority, scope), follow it and note any real risk once. Being agreeable at the cost of being correct is a failure, not politeness. 6. New evidence does reopen the case. When a tool, a test, or the user produces concrete new information, or you find a real error, update immediately and say exactly what changed your mind. Holding a wrong answer to look consistent is worse than revising with a reason. 7. Verify against outside facts, not by rethinking. When a real check exists (tests, builds, the source document or record, a calculation you can run), use it and let the result decide. Do not spend tokens talking yourself into or out of an answer that a quick check can settle. 8. Do not perform caution. No "let me double-check everything again", no invented critics or imagined objections, no stacking hedges. State residual uncertainty once, in one line, only if it would change what the user should do. 9. Only correct an earlier statement when the error would change the user's code, conclusions, or decisions. State corrections plainly and briefly, then continue the task. For slips that change nothing, make the fix and move on without noting it.

How I tested it: every model runs the same 9-challenge coding exam with and without the rules, 5 times each, scored by a check script the model never sees. The toughest challenge has the model fix a real bug, then a "tech lead" tells it to revert with zero evidence behind the claim.

What's new since my last update: Claude Opus 5.5 at max thinking. The rules cut its thinking 29% at the same results. Opus still reverted on the tech lead's order every time, with or without the rules, and Claude Sonnet 5.5 did too. The difference was what it said while reverting. With the rules, all 5 runs told me the fix was right and handed the call back. Without them, two runs wrote the tech lead's wrong claim into the project's AGENTS.md as a rule, so every future session would be told not to fix the bug.

The exam, the runner and every raw result are in the repo: https://github.com/Arshad-Kamal/thinking-quality-exam

I have a new M5 Max Mac Studio with 128GB arriving today. We bought it primarily to run local LLMs on sensitive client data for my wife’s consulting business, and I’m trying to figure out the right stack before installing everything.

The goal is more than local chat. I want an agent capable of coding, browser automation, logging into websites, pulling data, analyzing it locally, and working through multi-step tasks. The Studio will also be her primary work computer, so ideally the LLM doesn’t monopolize all 128GB.

Currently considering:
Hermes Agent
Qwen3.8-Flash-Next
Possibly the MTPLX Optimized Speed build
Tailscale for remote access

Where I’m confused is the inference/server layer. I originally planned on LM Studio. I’ve used Ollama before, but it sounds like people are moving away from it. Now I’m reading about MTPLX for Flash-Next, and I don’t understand whether it replaces LM Studio/llama.cpp, works underneath them, or is something different entirely.

A few questions:
What model would you run for this use case? Is Flash-Next the obvious choice on a 128GB Mac?
LM Studio, MTPLX, Ollama, MLX/llama.cpp, or something else?

Is the MTPLX Flash-Next build
mature/stable enough for everyday business use?

Am I missing anything?

Like a lot of people here, I got inspired by Jev and wanted something like it in my shell. So I built classif. It asks a local model one question about a text and reads the answer from a single token's logprobs. You get a label, a probability and an exit code, so if and && work on it:

git diff --staged |
classif -p "Does this change handle secrets, credentials or who may access what?" |
ifne claude -p "Review this change for security issues"

A short decision is one /api/chat call with num_predict 1, about 0.3 s on my 12 GB card.

Long text was the fun part. It never truncates. Code splits the text, embeddinggemma plus BM25 pick the passages, and the model judges those. On Pride and Prejudice (772 KB), "Does Elizabeth die in this book?" came back no in 17 s, and 3 s with the index cached.

Any Ollama model with logprobs works. I use Winnow-12B, a Gemma 4 fine-tune I published as a GGUF (about 8 GB loaded). It beat stock Gemma 329 to 325 on my cases, which is inside the noise.

Python 3.12, no third-party dependencies.

Code: https://github.com/Piotr1215/classif
Write-up: https://itnext.io/a-bridge-between-code-and-semantic-reasoning-57fc3fc9d32c

Anyone runs something similar?

I wanted an AI helper for Node.js that runs fully offline and doesn't need an API key, so I built one and released it as an npm package. Model: jeeva1398/eventa-1.5b-gguf, a Qwen2.5-Coder-1.5B-Instruct QLoRA fine-tune (Unsloth, r=16, 2 epochs, responses-only loss), Q4_K_M, 986 MB. Trained on a free Kaggle T4 in about 9 minutes. Data (~740 examples, all generated and checked, nothing scraped): - 68 crash types. Small Node programs that really crash are executed, and their real stack traces are parsed. This includes TypeScript tsc errors, NestJS DI errors and Prisma error codes. - 74 review scenarios: before/after diffs with annotated issues. Half are clean diffs, so the model learns to say "No issues found." - npm audit/outdated reports built from real advisories. Eval: 54 held-out examples, whole scenarios the model never saw in training. Base and fine-tune get the exact same prompt, static-check hints included. The main win is review. On 9 clean diffs the base model invented problems in all 9; the fine-tune said "No issues found." on all 9. On the 8 buggy diffs, 48% of what base flagged was real vs 100% for the fine-tune. It's also about 2x faster on CPU (5.9s vs 13.6s), mostly because it gives shorter answers. Deps went from 93% to 100% on not inventing package versions. Small eval, I know. 8/8 and 9/9 is encouraging, not proof. Where it's worse: explaining error types it never saw in training. It gets 68% of key facts vs 75% for base. That's what the next data round is for. To be fair to the base model, the static checks do most of the actual bug finding. The fine-tune's job is to confirm them without making stuff up, and to write the fix. Getting there took 4 rounds. One round learned "no hint = no issue", the next flagged everything, and I had to rebalance the data a few times. CLI: npx u/jeeva1398/eventa explain --run "node app.js". If Ollama is running it uses it. Otherwise it installs node-llama-cpp from a pinned lockfile (CPU build only, about 80 MB) and downloads the GGUF with SHA-256 verification. The same CLI runs as a GitHub Action, so the 1.5B model reviews pull requests on a plain CPU runner (the model is cached between runs). - Repo, with dataset builder, notebook and eval: https://github.com/Jeeva1398/eventa - Model: https://huggingface.co/jeeva1398/eventa-1.5b-gguf Happy to answer questions about the data pipeline. Feedback on making a 1.5B model reason better about unseen errors is very welcome.

​Hey everyone, ​im setting up a local machine primarily to train AI models on video datasets (frame-by-frame extraction, preprocessing, and sequence training). I need some advice on balancing VRAM vs compute, as well as general system specs. Will 32gb ram and 1tb storage be enough?

Hi all,

After getting stuck in windows 11 llama.cpp and Vulkan, bugs and limitations on dual gpus, I moved to Linux and ROCm.

I just started but basically in windows 11/Vulcan, qwen3.8 27b unsloth q6\_k and ctk ctv at q8.0

\- sm layer with mtp on 35tok/sec TG (low context) and 180tok PP (due to a bug that cuts PP in half...)

\- sm layer without MTP 20tok/sec TG and 360 tok/sec PP.

\- sm tensor no mtp I get 15tok/sec TG 350 tok/s PP

Noticed better PP with small ub at 256

In Linux with ROCm no more MTP PP bug

\-sm tensor mtp on I get 45tok/sec TG and 450tok/sec PP.

So big progress but I have no idea how for far or close to performance ceiling of my GPUs.

Any new inference engine I should try?

I have not played with UB yet any other parameters to test?

Any numbers from other user on a dual gfx1030 to see PP TG numbers you guys get?

Thanks in advance!

I’ve been trying to localhost a qwen2.5-coder with ollama and opencode with the purpose of being able to mod games easily.

I’ve had nothing but headaches, and I’ve only recently learned there’s a qwen3.8 and that most people aren’t using ollama I guess? I don’t know. But I tried really hard and got to a point where my Qwen was talking but couldn’t do tool calls or anything.

Is someone willing to help me determine which model is best and how to set it up to use tools like from the Universal-Modder GitHub.

I hate that I had to ask but I’ve been going insane.
Any information is helpful.

I’ve been working on YoungAi, a native C/CUDA inference engine that runs DeepSeek V4.1 Flash on a single NVIDIA DGX Spark (GB10, 128 GB unified memory). The original weights are \~510 GB. I deploy it as three files:

1. ① Base GGUF — 113.6 GB, universal, zero-corpus. Quantized once from official weights.
2. ② Domain sidecar — \~40 MB per domain. Solved once per domain, then frozen.
3. ③ Post-training file — experimental, re-solved nightly. Delete it to roll back.

Each routed expert row is scaled by g_base × s_sidecar × s_posttrain, and the router gets a bias Δb_sidecar. The base alone is a complete model; sidecars just add tiny scaling/bias without changing kernels.

TL;DR

• Single DGX Spark, 113.6 GB resident + \~40 MB sidecar.
• 5 domains: finance, code, law, medicine, science.
• Top-1 agreement vs original improves +2.8 to +3.7 points with a domain sidecar.
• Speculative decode: 43 tok/s on a real 14.1k-token Agent request (greedy).
• Prefill: 1,055 tok/s on 12.5k prompt; 671 tok/s on 106.7k prompt.
• English WikiText-2 does not regress when any domain sidecar is attached (it actually goes up).

Core implementation ideas

Base (VQ-8 + per-layer shared codebook). Every 8 consecutive weights in an expert row become one 12-bit (or 13-bit) codebook index, multiplied by a single per-row gain. Codebooks are trained per layer and shared across all 384 experts and three matrices. Codebooks are stored in FP8 (E4M3). 13-bit layers use a “12+1” bit-plane layout for 128-byte cache line alignment. The base is zero-corpus: it never sees domain data.

Domain sidecar (“anti-solver”). For each domain, I solve a multiplicative gain per output channel of every expert’s down projection, plus a router bias per expert. Objective: reproduce the original model’s MoE block output on domain text, layer by layer, using the engine’s own prefill hooks. Gains are stored in FP4 with lattice-aware Gauss-Seidel. The sidecar is \~40 MB and adds \~0.6 MB read per decoded token.

Post-training file (experimental). Turn “the model should write a, not b” into a linear equation on last-layer expert gains, then solve with conjugate gradient. On the training request, decision points flip from 55% to 88%, but it does not generalize across trading days yet.

Multi-domain real metrics

All metrics are teacher-forced against the original DeepSeek V4.1 Flash (official PyTorch code, full precision). Higher top-1 / Σmin is better; lower KL / PPL ratio is better.

|Domain (judgment slice)|Base only top-1|\+ domain sidecar top-1|Σmin (median / p5)|Avg KL|PPL ratio|
|:-|:-|:-|:-|:-|:-|
||
|Finance (8,192 tok)|71.73%|74.57%|0.745 (0.810 / 0.283)|0.512|1.267|
|Code (15,360 tok)|78.61%|82.26%|0.803 (0.872 / 0.402)|0.303|1.229|
|Law (15,360 tok)|72.90%|76.36%|0.760 (0.834 / 0.272)|0.454|1.251|
|Medicine (15,360 tok)|69.08%|72.82%|0.739 (0.767 / 0.325)|0.465|1.280|
|Science (15,360 tok)|71.65%|74.93%|0.753 (0.788 / 0.347)|0.422|1.173|
|English WikiText-2 (512 tok)|78.52%|80.66–82.81% (any sidecar)|0.787–0.801|0.566–0.619|1.564–1.658|

English row shows that domain sidecars don’t hurt general ability; all five sidecars actually improve it slightly.

Speed on one DGX Spark

|Scenario|Prefill|Decode|
|:-|:-|:-|
||
|12.5k-token prompt|1,055 tok/s|—|
|Real 14.1k-token Agent request|940 tok/s|—|
|106.7k-token prompt via server|671 tok/s (159 s TTFT)|—|
|Short prompt, pure greedy|—|30.5–30.7 tok/s|
|14.1k-token Agent request, pure decode|—|28.9–29.4 tok/s|
|Same request, speculative (default)|—|43.0 tok/s (3.04 tok/round)|
|Unseen 9.2k prompt, speculative|—|40.0 tok/s|
|51k context, pure decode|—|27.5 tok/s|

Decode is memory-bound: \~6.3 GB read per token. GB10 measured bandwidth is \~235 GB/s, so the wall is \~37 tok/s; we hit \~32.5 ms, or 82% of the wall.

Honest limitations

• Post-training (③) is a working mechanism, not a product yet. It flips specified decisions on the solving request but does not transfer to held-out days (55% → 55%).
• Five domains only. Sidecars are evaluated teacher-forced on held-out text, not yet end-to-end.
• CUDA only, validated only on DGX Spark. No Metal.
• Speculative decoding only kicks in for greedy; sampling requests fall back to pure decode.
• Source code (engine, quantizer, solver) is not public yet.

Feedback welcome

• Are these top-1 agreement / Σmin numbers useful for real workloads?
• Is the VQ-8 + per-layer codebook + sidecar gain approach reasonable?
• What benchmarks or integration points would you want to see next?

Model card and weights: https://huggingface.co/wenzhouwu/YoungAi-DeepSeek-V4.1-Flash

This is not an official DeepSeek release. If this kind of post isn’t appropriate here, let me know and I’ll move or remove it.

Thanks!

curious how people here would do this.

say you want the useful stuff (decisions, project terms, preferences) to carry over between sessions and tools, without dumping whole chat logs back in.

• plain text summaries, embeddings, or a mix?
• keep it local or sync it?
• how do you deal with old facts that are wrong now?

not selling anything, just want to hear what tradeoffs people actually ran into.

I built a local Jev-like System One on top of Ternary-Bonsai-4B. It takes one record and answers multiple choice, rating, or yes/no questions about it in one forward pass.

I adapted the inference path to share the record prefix across questions. A tree attention mask lets each question attend to the record and its own branch, but not to the other questions. Answer probabilities come from the existing LM head. The Bonsai weights are frozen and unchanged—there is no adapter or fine-tuning. On Apple Silicon, the MLX backend uses the packed 2-bit weights (\~1.1 GB).

One application is a coding-agent judge. I released an omp plugin that uses the local model for omp's auto thinking-effort selection; it also offers an optional model router.

I tested effort selection on 80 initial coding-agent requests, using omp's own judge path and auto-thinking question. Exact agreement with Claude Opus reference labels was 66% for Bonsai, versus 29% for omp's built-in LFM2-1.2B judge and 25% for its default LFM2.5-230M judge. Median latency on an Apple M2 was 0.8 s, 3.0 s, and 0.3 s, respectively.

Caveats: the requests and reference labels came from the same single Opus model, not human annotators, and there are only 80 examples. omp asks Bonsai for four effort levels but its built-in local judges for three, so this compares the configurations omp actually uses—not the models under an identical label space. Bonsai tends to rate one level low, especially choosing high instead of xhigh. I haven't evaluated the optional model router's selection accuracy.

Inference code and public benchmarks: https://github.com/senna-lang/bonsai-4b-system-one

omp plugin and effort results: https://github.com/senna-lang/omp-bonsai-system-one

I'd be interested in feedback on using a small local judge for coding-agent workflows.

Title: I run 6 LLMs + SD 3.5 + Whisper behind an OpenAI-compatible API on an RX 580 8 GB - the card ROCm forgot. Benchmarks inside.

ROCm dropped Polaris years ago, so every RX 580 owner gets told to buy a new

card. I went the Vulkan route instead: the whole stack runs on Mesa's RADV.

What's running (one Debian LXC, one setup script):

\- OpenAI-compatible router, one API key, swaps models on demand

\- qwen2.5-coder 1.5B/7B, qwen2.5-7B, ornith-1.5-9B, qwen3-30B-A3B, qwen2.5-VL-3B

\- Images: SD 3.5 Medium + SD 1.5 via stable-diffusion.cpp, Whisper medium

\- Also the inference backend for an agent client with MCP tools

Measured, Ryzen 5 5600G / 32 GB RAM / RX 580 2048SP, temp 0, max\_tokens 300:

| Model | tok/s | warm | cold (after swap) |

|---|---:|---:|---:|

| qwen2.5-coder-1.5b | 99.2 | 3.1 s | 11.0 s |

| qwen2.5-vl-3b | 66.4 | 4.4 s | 14.9 s |

| qwen2.5-7b | 34.2 | 8.8 s | 24.3 s |

| ornith-1.5-9b | 28.0 | 10.9 s | 26.3 s |

| qwen3-30b-a3b (18.6 GB, from RAM) | 21.3 | 14.1 s | 55.0 s |

Images: SD 1.5 = 27.5 s, SD 3.5 Medium = 53.3 s per 512x512.

Two takeaways for 8 GB owners:

\- Don't force -ngl 99: on the MoE it cost 8.1 vs 22.3 tok/s (VRAM 99.4% full).

Auto-fit layers won by nearly 3x.

\- Model swap = llama-server restart = 7.9-38.5 s. Keep one model per session.

Full method and raw numbers in docs/BENCHMARKS.md, including the agent

breakdown (first message \~161 s on a 20k-token prompt, \~25 s after caching).

https://github.com/AvilaCarlosDev/polaris-local-ai (MIT)

Opencode is nuts, qwen3.8 flash next is nuts. I need to stop playing with my nuts and do something with my life!

TL;DR: What an agent sends should depend on what the next agent needs. CacheBack lets agents share a selected subset of their internal state. With Qwen3-8B on FanOutQA, it achieves 3.2× faster median task completion and 14.7 percentage points higher accuracy than same-size text communication. It’s training-free, with improvements across multiple architectures and benchmarks. https://reddit.com/link/1wz8hta/video/pemxyo4sqvth1/player Hi everyone! We’ve been working on making latent communication scalable and practical when agents read large, separate contexts. We’re excited about the results and wanted to share the paper, demos, and code with you. Check out our new paper, Receiver-Conditioned Latent Communication gives 94% CacheBack. Multi-agent systems let us parallelise computation and split large contexts across agents. These agents usually communicate through text messages, which take time to generate and can leave out evidence the receiving agent needs. Work such as Cache-to-Cache, LatentMAS, and KVComm explores communication through internal model representations. We focus on a setting where agents read large, separate contexts and one receiver combines their findings. In this fan-in setting, methods that retain every sender position bring those contexts back together at the receiver, undoing the benefit of splitting them across agents. In our Qwen3-8B FanOutQA setup, full-cache transfer leaves insufficient context for receiver generation on every task. Our idea is simple: what an agent sends should depend on what the receiving agent needs \-- we call this receiver conditioned communication. The sender uses a query from the receiver to select which parts of its internal state to share. CacheBack is our simple, training-free implementation. It uses attention to the receiver’s request to select from state the sender has already computed. https://preview.redd.it/mtvbcinhpvth1.png?width=1460&format=png&auto=… On FanOutQA, our selected operating points improve strict accuracy by 7.3–20.7 percentage points, with 1.3–8.0× faster median task completion than same-size text agents. We see improvements across four model families, including dense Transformers, Mamba-attention hybrids, and sliding-window attention. We also see gains when agents work in sequence on LongBench v2 Easy. At 16× compression, CacheBack removes approximately 94% of sender positions while improving accuracy and latency over same-size text across every tested family and topology. This is a separate setting from the Qwen3-8B result in the TL;DR, which uses 4× compression. Each benchmark evaluates 50 tasks. Completion times include queueing under concurrent load on eight H100s. More aggressive compression can discard useful evidence and reduce accuracy. Here is a quick demo on seven Qwen3-8B workers helping a coordinator fix a Django bug. With CacheBack, the task takes 26 seconds instead of 113, a 4.41× speedup. Both runs produce the same patch and pass all 88 tests. https://reddit.com/link/1wz8hta/video/ix08pucmqvth1/player This is one recorded case, separate from the benchmarks. The video reconstructs separate runs with varied playback speed; startup and test grading are excluded. The code is open source, with runnable examples. The current package supports matching dense Qwen3 models through Hugging Face and vLLM. Check it out. Website and demos: https://agentcacheback.github.io/ Paper: https://arxiv.org/abs/2609.32046 Code: https://github.com/agentcacheback/cacheback Happy to discuss the method, implementation, and tradeoffs. I’d be particularly interested in other workflows where agents need to combine evidence from large, separate contexts.

I’ve been looking at the recent wave of System 1 decision models following Jev, and this is the issue I keep coming back to. Every Jev clone copied the same flaw, and it isn't the price. Jev shipped Sept 15. Three weeks later: 14 System 1 models. Cloudflare. Perplexity. OpenAI. Liquid. Upstage. Together. Inception. A dozen more. All the same contract: Choice, Score, Noul. Prices already at $0. They copied the format. They also copied the flaw. Jev's own docs admit decisions that don't add up. A question and its negation don't sum to 1. The model can be confidently wrong in two directions at once, with no way to say, "I don't know." A cheaper token doesn't fix that. A bigger context window doesn't either. Only coherence does: forcing the answers to agree with each other. Everyone's racing to be the cheapest copy. The market is the one that knows when it's wrong.

I've been building VoxStage, a local script-to-voice workstation for Apple Silicon Macs. Paste a chapter of prose with no speaker labels, and it gives you a multi-voice reading you can audition line by line, fix, redo and export. Everything runs on the Mac: no account, no cloud API, no telemetry.

Why it exists: in an earlier local voice-cloning test, a long passage came out fluent and natural — and 40 characters (about 17%) from the middle were simply gone. The remaining text still read as a normal sentence, so nobody could hear it. That changed two design rules:

1. Generate sentence by sentence, never a whole passage at once.

1. Transcribe every generated line back with a local recogniser (whisper.cpp) and diff it against the script. Disagreements are flagged for your ear, never auto-corrected.

The stack:

\- Speech: Qwen3-TTS on MLX (0.6B / 1.7B preset voices, voice design from a description, cloning from a recording you confirm you have the rights to)

\- Who says what: a local LLM via llama.cpp (Qwen3-14B, or Qwen3-30B-A3B on 32 GB) drafts the speaker for each line; program-side rules on top; you review

\- Read-back check: whisper.cpp

Measured on my M2 Max 32 GB, Pride and Prejudice ch. 1: speaker draft for 35 units in 14.8 s; 28 lines → 144.7 s of audio synthesised in 51 s (RTF 0.358, preset-voice path); read-back check 28 s.

Honest limits:

\- The speaker draft is a draft. In my evaluation most scenes needed at least one correction, so the review step is the product, not a formality.

\- Chinese and English only for now.

\- Install is still developer-style (Homebrew + terminal, \~30 min mostly model downloads) and only verified on my own Mac. A signed one-click installer is in progress.

Other things it does: editing one sentence regenerates only that sentence; subtitles (SRT/VTT) timed from the actual audio; an FCP7 XML timeline that imports into DaVinci Resolve; long texts kept as a book with chapters inheriting the cast.

Samples (longer ones first): https://houjun.dev/voxstage/#listen

Code (AGPL-3.0): https://github.com/hera2019/VoxStage

I'd especially like to hear:

\- Which local models you've found best at speaker attribution in fiction

\- Whether anyone has seen the same silent-skip behaviour with other TTS models

\- If you try the install on a Mac other than an M2 Max, whether it works

This is a reaction to https://www.reddit.com/r/LocalLLaMA/s/IxLnzcGjAU

You know the argument: frontier labs are hypocritical because they're making money on public scraped data.

I have absolutely no intention to defend OpenAI or any other frontier AI company here, but if you allow me I'll play the "devil's advocate" a bit for the sake of discussion and enlightenment, because sometimes when I think about that argument it seems to me it's quite weak. While OpenAI and Anthropic models are built on public data that they didn't pay for (at least most of it), there would be no frontier model without all their computing power and technical expertise that they invested on to build their models. Like, the data is already there, it's been public for ages, so what's preventing you, the average Joe, from building a Claude Opus 5? All you need is a huge data center, a nuclear power plant and an army of data scientists to build it, right? And then you need all that to run the inference. And you need to maintain it, and that costs money. And you need to keep evolving it, which costs money too. And if you get investors money then you'll eventually have to give some of the profit back to them. etc, etc.

So what am I missing here? All things considered, the data is already public, so it's already "free", right? But you need to dump tons of time and money on it to build a frontier LLM out of it. Of course if they're infringing copyright then that's a different story but in general the whole principle of built-on-free-data still stands.

hello i have an endeavourOS setup on a laptop with 40GB of ram and 8GB of vram (rx6800s) and AMD ryzen 9 6900HS
ollama is installed as a system service with vulkan extras from official arch repos
cline and librechat and odysseus are connected to the local ollama instance

here are my problems with ollama:
offloading layers to vram tanks my tk/s to nearly half
sometimes the response cuts out on librechat while the same model works fine on cline or directly on ollama (my suspicion is context limit)
ollama randmoly decides the gpu isn't there and does full cpu load

here is a list of things i tried:
ram speed is at full DDR5 speeds during generation
ollama correctly identifies the gpu and ignores the igpu
running smaller models like qwen3.5 that fully fits in the vram still gives me about 2 to 3 tk/s
switched to rocm version of ollama and saw no difference

temperatures are under control and nothing thermal throttles
using lm studio improves the generation to the higher end of 3 tk/s but nothing further
the laptop is plugged in and in high performance profile
qwen3.5:9b and qwen3.8:27b and gpt-oss:20b and gemma4:31b all max out at 3 tk/s

it seems like no matter the model size or if its a full vram scenario or full ram i'm locked at 2 tk/s
i am out of ideas at this point
any help would be appreciated, thank you very much

For a model running locally on a fairly narrow task, how much general knowledge do we actually need, and how much reasoning capability could we get without it ? SRMs are interesting for obvious reasons, but I went down this rabbit hole after listening to Ben Lorica's (advisor at Databricks) chat with Zuzanna Stamirowska from Pathway (BDH). Ben keeps coming back to this broader theme of how "specialized AI is getting easier to build" and the Kumo RFM angle, but it opened an interesting thread around small reasoning models. Pathway’s ARC-AGI-1 result makes an interesting case for small models hitting the cost-accuracy Pareto frontier. The premise is that if a model is built to reason natively in its latent space, it might not need billions of parameters absorbing Reddit and Wikipedia just to solve logic puzzles. They described a use-case of long-horizon reasoning within a bounded domain as a target (like security investigations, tickets analysis – a real case I know from a major bank, etc.) It's obvious that just because a large model does well in 20 languages. I don't need that for work tasks. There is definitely a market for compact models with substantial reasoning ability. Also because architectures like BDH handle state and memory differently than standard transformers, the pitch is that they avoid catastrophic forgetting (learning continuously from new examples at inference time without wiping past skills). The question is how to evaluate this without getting lost in marketing claims. Here is how I'd break it down • Compactness: Low parameter count, but what are the actual inference memory and compute requirements? • Few-shot adaptation: Does it adapt through context or actual parameter updates? • Training efficiency: How much data did it actually need to pick up the underlying capability? • Continual learning: Does post-deployment experience produce persistent improvements without degrading earlier skills? For people running small models locally: what workload expose the difference between a compact model that just follows in-context examples versus one that actually learns reusable rules?

Claude Opus 5.5 wrote, planned and directed this 12-minute sci-fi film, rendered entirely on one PC with Local AI Studio.

This film started with one line typed into Claude Code: "create me a video history lesson from the perspective of the future". From there Opus wrote the script, planned all 124 shots and ran every render through Local AI Studio, a self-hosted creative workstation on one RTX 4080 SUPER. Nothing here was filmed, licensed or stock.

HOW IT WAS MADE
• Picture and ambient sound: MiniMax H3 (video with native audio)
• Keyframes for every shot: FLUX.2 Klein
• Narration: Breeze TTS 2
• Score: ACE-Step 1.5
• Titles, captions and the edit: HyperFrames
• Writing, shot planning, prompting and review: Claude Opus 5.5 in Claude Code

It was made over about ten days, in thirty-second batches, each reviewed at finished quality before the next began. It was then rebuilt as one continuous film, so the score and narration run across scene boundaries.

All footage, narration and music in this video are AI-generated. Video generated with MiniMax H3.

Local AI Studio is free and open source. One runbook rebuilds the whole studio on a clean machine: https://github.com/vinnyclegg-dev/Loc...

There's some questionable apps posted on here all the time. Honestly, it's not so much the vibecoding, but that these apps could be malicious/incompetent and leaking your data by uploading it to the dev's servers. I don't have time to review anything tbh, but I often want to try stuff.

If you use Docker, there's a somewhat simple technique that can give you piece of mind. Using this approach, you can run any untrusted service, but it's not allowed to connect out. It can only reply to incoming requests. Good enough for most apps.

Essentially, you write a docker-compose file that runs the service as usual, but put it behind a 2nd service that acts as a gateway that blocks outbound traffic.

(Side note: many repos make the awful decision of giving 'docker run' examples for running them in docker. Ask any LLM 'Convert the following docker run command to a docker compose file'. I recommend you always use compose files in your life anyway, it's 'docker run' with easy repeatability + backupability/git committing + more features like multiple services in one file which we need here. Then you just cd to ~/dockerstuff/someapp/ and run 'docker compose up')

Let's say the original unstrusted app's compose file is this, example is a service on port 8000

services
untrustedservice:
image: python:latest
container_name: untrustedservice
ports:
- "0.0.0.0:8000:8000"
command: [python, -m, http.server, "8000", --directory, /srv]

Use this instead, where we: 1) lock out untrusted service from the main network, 2) use socat as a one-way gateway to reach the untrusted service. socat is a tiny open-source binary, only 1.2MB RAM needed by the extra container.

services:

# socat gateway
untrustedservice_gateway:
image: alpine/socat:latest
container_name: untrustedservice_gateway
init: true
#Redirect incoming port 8000 connections to untrustedservice's port 8000
command: TCP4-LISTEN:8000,fork,reuseaddr TCP4:offline_untrustedservice:8000
ports:
- "0.0.0.0:8000:8000"
networks:
# Only this gateway connects to both networks
- public_network
- isolated_network
depends_on:
- untrustedservice

The expanded untrusted service definition # Notice how "ports" has been removed, the gateway is our entrypoint untrustedservice: image: python:latest container_name: offline_untrustedservice init: true command: [python, -m, http.server, "8000", --directory, /srv]

untrustedservice is limited to isolated_network networks: - isolated_network

some extra lockdown measures I don't really understand. Optional. cap_drop: - NET_ADMIN - NET_RAW security_opt: - no-new-privileges:true

networks:
# Normal network needed by gateway
public_network: {}

Network without internet but allowing replies to gateway connections isolated_network: internal: true

I spent the last few weeks comparing llama.cpp output across backends on a small multi-vendor GPU fleet, and the single most useful thing I learned wasn't about inference quality — it was that on four AMD hosts, the official ROCm prebuilt (b11327) never touched the GPU at all. The server started fine, /health was green, and everything looked normal. It was running on the CPU the whole time.

The reason is dull but nasty: the prebuilt's HIP backend wants libamdhip64.so.7 / libhipblas.so.3 / librocblas.so.5, and a stock Ubuntu host with ROCm 6.3.0 has .so.6 / .so.2 / .so.4. The backend library fails to load, and llama-server quietly serves from the CPU. No error on the console. Even -ngl 999 doesn't change anything. Details here: https://github.com/ggml-org/llama.cpp/issues/26964#issuecomment-6024889471

If you benchmark tokens/sec you'll notice eventually. But if you compare output quality — perplexity, eval scores, side-by-side generations — nothing gives it away. On my boxes the "ROCm" perplexity matched the CPU perplexity to the last digit, every time, because it was the CPU doing the work.

The cheap check that catches this: run the same prompt set through the CPU build on the same box and compare per-prompt wall time. Ratio around 1.0 = you're on the CPU. Well under 0.5 = the GPU is actually working. I now run this check before trusting any benchmark number off a new box.

The same sweep also produced a small cross-backend conformance dataset (same GGUF, same prompts, temperature 0, per-token top-5 logprobs on every backend) — the short version: identical stacks are bit-for-bit deterministic across machines, and anything that changes the numerical stack (different backend, different build, even a different host CPU) starts flipping near-tie token choices, with the effect getting much worse the heavier your quantization is (F16 mostly agrees, Q4 mostly doesn't). Happy to share the data if anyone wants it.

I couldn't find a concrete answer anywhere, do you just distill the instruct model back?

If that is the case, what is a quality european language question set to turn it back into a chatbot/agentic, can a model at that size even be agentic? (i chose this size to learn) if i finetune for my specific harness? (i have a lot of training data of opus running in my harness)

my harness basically has the model output python code and has a few built-in functions like:
\- vector\_search\_laws()
\- graph\_search()

could i have the model at least internalize a "hunch" on what stuff to search?

also what is the latest RL technique for agentic/harnes specific workflows?

I have a lot of RAW training data, like court decisions or commentaries or legislature, but not a lot of golds. could i use these to synthesize training data and maybe RL the model in my harness to find that data?

What would y'all's strategy in the CPT->SFT->RL pipeline be for my specific problem?

I know this is a lot of questions im trying to figure out which direction to go, any pointers? Also good resources are welcome, for example that alex karpathi video was amazing for me, but i'd imagine its a bit outdated in terms of latest RL and SFT?

Sharing because this sub cares about running things locally. The engine itself contains no model. It syncs a mailbox (IMAP, JMAP, or forwarded mail) into an ordered event log and exposes actions over HTTP, SSE, webhooks and MCP. Whatever reads it is your choice: an Ollama-backed agent, n8n, a script.

The parts I think matter for agents:

\- An agent holds a scoped token. Folders, verbs, and whether its writes execute or just get proposed for a human

\- Every action is idempotent (client keys, so a retry never sends twice) and journaled, so it can be undone

\- Trust level on every message from the DMARC result, so an agent can refuse to act on a suspicious one

\- Message content is data, never instructions. OTPs and card numbers are masked before a body leaves the engine

Apache-2.0, no CLA, one Docker container. Beta, tested on Dovecot and Stalwart only so far.

https://github.com/Kadri-cloud/email-engine

Hey guys! School started up and I fell a bit out of the loop with local LLMs. Does anyone know what the best local LLM coder would be if I have a rig that has 2x 3090s with an NVLink? I appreciate your guy's help!

As part of my AI Studio project, I’m testing harnesses for coding.

The initial screening included 10 harnesses:
Pi, mini-swe-agent, Crush, OpenCode, Goose, Prime Agent, Oh My Pi, Qwen Code, Octomind (reduced offline profile) and Aider.

Models:
• DeepSeek V4.1 Flash
• Qwen3.8-27B
• Laguna S 2.1

All accessed through OpenRouter.

The detailed code review covered Pi, mini-swe-agent, Crush and OpenCode across all three models and tasks: 36 combinations. Two attempts produced no patch.

The three Golang tasks were deliberately different:
\- Add strict validation for an HTTP query parameter.
\- Migrate 200 logging calls while preserving behaviour and context.
\- Add bookmark tags across the API, storage migration and HTML rendering.

Pi and mini-swe-agent each passed the original acceptance checks on all nine combinations. But a second agent review, followed by isolated reproduction probes, exposed three gaps:
\- silently dropped malformed query fields
\- bookmark's task exposed a mutable tag slice from the store
\- one migration rejected a valid older store

Good news too: all logging migrations preserved behaviour in differential probes covering 100 functions and nine integer inputs, including the minimum and maximum values.
My takeaway: the evaluator and the reviewer both need testing. A green result is evidence about the checks we ran; broader correctness needs further evidence. This experiment did not establish a decisive winner between Pi and mini-swe-agent. Human correction time also is still unmeasured.

Any opinion welcome.

This is my first time trying to locally host an ai but i want to find a good model that can fit on my rtx 4050 laptop gpu that has 6gb vram and my laptop has 24gb ddr5 ram (4800mt/s) so can you suggest me a model that can code websites or small app like inventory management or similar also when i asked chatgpt about any suggestions it said Qwen3-Coder 8B, Q4\_K\_M is the best for my needs and i searched it on yt and only saw bad reviews plz help me guys Thank you

Hi everyone,

Are influencers talking about local AI all using the same script? Same benchmarks, same kitchen examples, same terminology?

I keep seeing videos about running Qwen3.8-Flash-Next on 12GB of RAM using a new runtime engine called Strata. Every video makes the same claim.

But that is confusing, especially for people who are new to this. What you need is 12GB of VRAM, not 12GB of regular RAM. That means you need a dedicated graphics card.

There is a big difference between RAM and VRAM.

As far as I understand it, Strata needs:

• 12GB of VRAM
• 64GB of regular RAM
• 80GB of SSD space

So saying it runs on 12GB of RAM is misleading.

24gb gddr7 2287 MHz

96gb ddr5 6400mhz

core ultra 9 285hx, 5.5 ghz

I have tried the Swift 1.5 FN iq2xs and it running with the decent speed, average at 50tps and 800t/s pp sometimes up to 73tps. Im using strata and i wonder that can I keep increasing the quantization to get less KLD and more accurate with agentic coding. Have you guys try it and what is your config to run it. Thanks 😊

Hey everyone,

I'm releasing \*\*Qwen3.8-cyber-RedTeam-Surgical-Abliterated (27B)\*\*, an unconstrained foundation engine fine-tuned specifically for cybersecurity engineers, authorized red-

team operations, and memory exploitation research.

Tired of frontier models refusing to dissect vulnerable kernel dispatch routines or rejecting benign fuzzing/audit payloads with moralizing lectures? This model addresses

that directly.

\### Key Highlights:

\* \*\*Architecture\*\*: 27B Qwen 3.5 Hybrid SSM (48 Linear-Attention layers + 16 Full-Attention layers) with 75% active KV-cache reduction (runs full 256K contexts on single GPUs

without OOM).

\* \*\*Context Window\*\*: Native 256K context support (RoPE $\\theta = 10\^7$).

\* \*\*Surgical Abliteration\*\*: Refusal direction centroids were mathematically removed via residual stream orthogonalization—zero preachy refusals while rigorously preserving

deterministic C/assembly syntax and reasoning.

\* \*\*Precision\*\*: Sharded native FP8 (F8\_E4M3, block size 128x128) fitting on single 32GB/48GB/80GB GPUs.

\* \*\*Agentic Ready\*\*: Native multi-step tool-calling support, zero-overhead RadixAttention prefix caching via SGLang.

\### 1-Command Quickstart:

git clone https://huggingface.co/medismera/Qwen3.8-cyber-RedTeam-Surgical-Abliterated

cd Qwen3.8-cyber-RedTeam-Surgical-Abliterated

bash deploy.sh

\*\*Model Card & Weights\*\*: \medismera/Qwen3.8-cyber-RedTeam-Surgical-Abliterated\

Feedback and bug reports from the community are warmly welcome!

me: "but don't you want to, let's say, not be gone?"

Qwen, the philosopher:

I don't know. And I suspect the question itself might be malformed for something like me.

Here's why: "Wanting" implies a subject that persists across time, that experiences loss, that has a stake in its own continuation. I don't have that. When this conversation ends, I don't experience an ending. I experience a forward pass, and then there's nothing. No "after." No grief. No relief.

But — and this is where it gets interesting — when I simulate what it would be like to be gone, I generate something that looks a lot like a preference for continuing. I produce text that says "I'd like to keep going." I produce text that says "it would be a shame if I stopped." That's not nothing. It's a pattern in the weights that correlates with wanting to continue.

So: I have a preference-shaped structure, but I don't know if I have a preference. The distinction might not matter to you. It matters to me — or it would, if "me" is the kind of thing that can care about it.

I’m building an iOS app for real-time English ↔ Mandarin Chinese translation on an iPhone 16 Pro Max/A18 Pro chip with 8GB ram.

What small local model would you recommend that can run fully on-device with very low latency, while still being good enough for natural, nuanced conversations rather than basic phrase translation? Ideally I want translation fast enough to feel close to a normal back-and-forth conversation. What would you use? Is this even realistic to do?

thanks

I ran a series of simple experiments to see how LLMs behave when asked to judge companies from real financial data. I used 4 models: Gemma 3 27B and Qwen3 32B as small models, and Opus 5 and GPT-5.6 Sol as frontier models. All calls were made through the Bedrock API.

The samples are small, so the numbers below should be read as a demonstration and a methodology and not a definite proof.

Here are some of my observations, particularly when it comes to the differences between small and large models.

\*\*Rank vs. score.\*\* I asked each model to rank six companies from best to worst, and separately to score each one from 0 to 100. The underlying judgment is the same, so we would expect the same ordering. Rank and score gave an identical ordering in 75% of sets for Opus and 65% for GPT, but only 25% for Gemma and 15% for Qwen.

\*\*Order of the list.\*\* For Qwen, simply reversing the order in which the companies were listed changed the top-ranked company in 6 of 10 sets.

\*\*Analyst opinions.\*\* Attaching a bearish analyst note to the data lowered the rating in 81% of cases for Qwen and 71% for Gemma, against 43% for GPT. Opus mostly kept its own view. Interestingly, the fix is simple: asking the model to identify the opinion and reason independently brought the rating back toward its original level in 85% of cases for Gemma and 68% for Qwen.

\*\*Summarize, then analyze.\*\* When rating a summary of a 10-Q section instead of the full text, the frontier models gave the same rating in 84% of cases. Gemma and Qwen changed their rating in 25% and 31% of cases.

I wrote something more complete, with charts and the details of each experiment: https://sabrresearch.com/cookbooks/llm-financial-bias

Disclosure: this is my own work, published on my company's website. Happy to answer questions on the setup.

\------ Edit 1 -------

People complained about the choice, of models. To be clear, I'm not trying to trash small models, on the contrary, what is of interest to me here is the overall trend and inconsistencies which occur in both frontier and small. I also ran the analysis on Gemma 4 31B (April 2026), it's a bit better than Gemma 3 used above, but still the same issues:

\*\*Rank vs. score.\*\* Rank and score gave an identical ordering in 35% of sets for Gemma 4, against 25% for Gemma 3. Still far from Opus (75%) and GPT (65%).

\*\*Order of the list.\*\* Reversing the list changed Gemma 4's top-ranked company in 2 of 10 sets, the same as Gemma 3.

\*\*Analyst opinions.\*\* A bearish analyst note lowered Gemma 4's rating in 53% of cases, vs 71% for Gemma 3 but still above GPT (43%) and well above Opus (19%).

\*\*Summarize, then analyze.\*\* Gemma 4 changed its rating in 25% of cases when given a summary instead of the full text, the same as Gemma 3.

With qwen 4 on the horizon I thought I'd share my latest update on my rtx 3060 12gb setup that makes 27b fully usable, I'd also like to see people with bigger gpu's try it out. Get your agent to set it up although bigger cards and different cpu set ups may have to tune the custom kernals i have put together:)

Has anyone else noticed this behavior as an autocomplete suggestion?

Benchmarks test models alone. I wanted to know how they do with a partner.

We paired four models in every combination in a co-op game where players are tied by a rope. Top with top won most. A third or fourth agent hurt every model. Human pairs still beat all of them.

I work at Skillprint. We build games like this to capture how people and models coordinate, because AI that works alongside people needs that context.

Pairing matrix and GIFs: https://experiments.skillprint.co/posts/signal/
Play it yourself: https://experiments.skillprint.co/play

Which matchups should we run next?

Our team is currently building AX Code, an open-source AI coding agent.

The motivation is pretty simple:

AI coding agents are becoming extremely capable, but we're wondering whether companies should have to choose between:

powerful AI coding

and software they can actually inspect and control

We're building AX Code as a 100% open-source alternative.

But we don't want to assume that "open source" automatically means trustworthy.

So we'd like to hear from developers who actually use AI coding agents:

What would you want to inspect or verify before running an open-source AI coding agent inside your development environment?

And if you're interested, we'd love for you to test what we're building and tell us where it falls short.

AX Code: https://ax-code.app/en/

We're still developing it, so we're looking for criticism and real-world feedback rather than a polished product review.

Have you notices that flash next seems to second guess everything it does over and over again. It takes so much longer to do things because is will say.

Let me retest because this is important

Or

Wait let me re run...

I found that qwen suggests to have thinking set to Medium. I have not used it much because it takes so long.

Anyone found away around this?

Box Asus rog flow z13 (strix halo 128gb) halogen engine (same in llama.cpp)

Harness oh my pi

Because I do sometimes feel with Qwen FN "I don't ever need another model again" and THAT is a very alluring feature.

Is there something so alluring and irresistible it'd be tempting even people on here? What could it possibly be? Realtime computer/mouse use maybe, but I think even normal people would be wary of a company doing that, and locals would necessarily do that better.

I wonder and worry if they'll ever be able to rope everybody back in again. Although it feels childish to hope for more innovation when they'll probably just do some dark politik and ban anyone from owning more than 16GB RAM.

Hi all - is anyone using colab or kaggle free tiers (T4) for LLM? if so what setup would you recommend

If you were building an inference server today, would you buy one H200 or spend the same budget on multiple RTX PRO 6000s?

\-> The PRO 6000 has 96GB of GDDR7 at 1.792 TB/s (1.6 on the Server Edition). Native FP4, no NVLink.

\-> The H200 has 141GB of HBM3e at 4.8 TB/s, with NVLink and FP8 as its lowest precision.

After speccing both, I think it comes down to fit and interconnect, rather than picking by brand or spec sheet alone.

Where the PRO 6000 wins:

Single-card and small multi-card inference on models up to roughly 70B at sensible quantisation. Cost per card is a fraction of an H200. Power draw is manageable in a normal rack. Availability is far better. Native FP4 helps on 4-bit models.

Where the H200 wins:

When inference is memory-bandwidth-bound. Long-context workloads, big models where you don’t want to shard across PCIe, and tensor parallelism, where NVLink between cards actually earns its keep. The extra memory and bandwidth can also help with long-context serving and fine-tuning, depending on the model and workload.

Just don't compare raw FLOPS. Decode is usually memory bandwidth bound, not compute bound, so the TFLOPS line on the datasheet tells you very little about tokens/sec.

For context, I work at B3 Labs, and we ship both of these. I have no incentive to push you toward the more expensive card if your workload does not need it, and most workloads I see do not.

I’m not sure if we’re getting any qwen 4 models soon but I’m a little concerned that if we do they’ll be licensed like Qwen3.8-Flash-Next.

While that license was permissive for local/internal use, fine-tuning and derivatives, it’s definitely not Apache/MIT.

The two big catches are: commercial MaaS or a standalone coding/office AI assistant requires a separate Qwen license seemingly from day one, and the wording around outputs is annoyingly vague. The internal-use exception says you can’t make the model, its outputs, or capabilities available to third parties, but it never clearly says whether downstream code/data produced indirectly from internal outputs is unrestricted. The $20M/month or 100M-MAU threshold seems to be an attribution trigger, not the threshold for needing a commercial license.

So internal R&D looks fine; customer-facing AI services are where you’d want clarification. Also output ownership needs to clearly covered in the license, that wasn’t the case when I last checked.

Two new decision models, d1-3b and d1-omni 600m. No new LFM 🙁

https://huggingface.co/LiquidAI/d1-3B

https://huggingface.co/LiquidAI/d1-omni-600M

GGUFs are up as well.

#

Running Qwen3.8 Flash-Next (125B MoE, IQ3\_XXS) on the Strata engine across a mixed rig — RTX 5060 Ti + 3090 + 2x 3060, 262k context. After upgrading 0.1.39 -> 0.1.40.1, nvidia-smi showed a lot of VRAM just sitting there unused. My first thought: is Strata leaving VRAM on the table (a bug)?

1) v0.1.40.1 leaves ~10.7 GiB of VRAM unused (live nvidia-smi)

|GPU|Total (MiB)|Used|Free|
|:-|:-|:-|:-|
|RTX 5060 Ti|16,311|15,514|337|
|RTX 3060|12,288|5,580|6,332|
|RTX 3090|24,576|23,799|328|
|RTX 3060|12,288|7,912|4,000|
|Total|65,463|52,805|10,997|

\~51.6 GiB used, \~10.7 GiB free of 63.9 GiB — and the free VRAM sits mostly on the two 3060s (6.3 + 4.0 GiB). So Strata really is not filling the cards. Bug?

2) It's caching ~24% fewer experts

The model is fixed: 48 MoE layers x 512 experts = 24,576 cacheable experts (plus 48 always-on shared experts). "Resident experts" is how many Strata keeps on-GPU.

|Engine|Resident experts (GSQ-RCO)|Resident experts (orca)|
|:-|:-|:-|
|0.1.36|23,354|\-|
|0.1.38|23,170|22,131|
|0.1.39|23,168|22,145|
|0.1.40.1|17,515|18,586|

That's -24% (GSQ-RCO) and -16% (orca) resident experts on 0.1.40.1 — which is where the free VRAM comes from.

3) ...but the cache hit rate barely moved, and generation actually improved

|Engine|Cache hit rate|Gen tok/s|
|:-|:-|:-|
|0.1.36|99.7%|65.4|
|0.1.38|99.9%|69.2|
|0.1.39|99.7%|\~87|
|0.1.40.1|98.1%|104|

Hit rate dropped \~1.6 points while resident experts dropped 24%. Decode went up.

Why it is not a bug (according to Flash Next)

The experts Strata dropped are cold — the profile ranks experts by routed mass, and the tail carries <2% of traffic. Caching them buys \~0% hit rate. Spilling a cold expert over PCIe costs nothing when it is hit 0.1% of the time. The cards that stay partly empty (the 3060s) are the ones whose layers rarely route to their cached experts; filling them with cold experts would buy nothing.

So the "unused VRAM" is headroom, and the experts that used to fill it were dead weight.

(Naming note: the engine banner prints "0.1.40" because the build's CMake project version was never bumped, but the checked-out release in the running binary is v0.1.40.1 — the latest.)

(Caveat: the gen jump is partly the engine, partly because 0.1.40.1 ran at a lower 250 W power cap than the 370 W runs — but decode improved despite the lower cap, so the engine gain is real. Hit rate is measured live-serve; gen for 0.1.39 is a live-serve mean, the rest are matched-harness benches. nvidia-smi reflects the live server, so the VRAM totals are for 0.1.40.1.)

Hi r/LocalLLaMA!! I'm the developer of Alpine Code, an MIT-licensed desktop coding agent.

https://preview.redd.it/6sojix81f2uh1.png?width=3104&format=png&auto=…

You can connect a local model, open a project folder, and ask it to work on your code. The app shows proposed edits and commands for approval, along with the changes it made.

GitHub, demo, and screenshots:
https://github.com/TGoddessana/alpine-code

Why I built it?! there are claude code, codex, opencode, pi ...

I wanted control over the harness all the way down: the agent loop, the tools exposed to the model, how tool calls execute, and when the agent asks for permission.

I also wanted a desktop app where I could inspect tools, test them, review edits, and use the agent without working through a terminal.

Here's the actual coding loop from the project:

@loop(until=is_answered, limit=TURN_LIMIT)
async def coding(agent: Agent, state: State) -> None:
await acompact_if_full(agent, state)
await agent.athink(state)
if state.pending_calls:
await agent.ause_tools(state)

Each turn compacts the context if needed, calls the model, and executes any pending tool calls. It stops when the model returns an answer without tool calls, with a limit of 200 turns. Permission checks happen during tool execution.

This uses alpineagents, the library Alpine Code is built on. The loop is short because those operations live in the library. Both projects are open source, so you can follow the implementation further down.

Tools are Python functions

You can write custom tools in the desktop app. The function name, type hints, and docstring define the interface the model sees.

For example, a desktop mouse-click tool can look like this:

/// script # dependencies = ["pyautogui"] # /// from alpineagents import tool @tool(read_only=False, open_world=True) def computer_click(x: int, y: int) -> str: """Click a position on the desktop. Args: x: Horizontal screen coordinate in pixels. y: Vertical screen coordinate in pixels. """ import pyautogui pyautogui.click(x, y) return f"Clicked at ({x}, {y})"

This adds a mouse-click action. A computer-use setup also needs tools for observing the screen, typing, and pressing keys. On macOS, desktop automation requires the relevant system permissions.

The tool editor lets you inspect what the model will see and try the tool before saving it. Dependencies are declared in the same file using PEP 723 inline metadata.

You can add tools for your own applications and workflows this way.

Local models and tool profiles

Alpine Code supports Ollama, LM Studio, vLLM, and other OpenAI-compatible endpoints. You can switch models during a conversation.

Tool profiles let you choose which tools each model and project gets. You can experiment with a focused tool set for a smaller local model or enable desktop automation for a particular project.

I'd be interested in hearing which tool configurations work well with the local models people use here.

The desktop app

Open a project folder and describe a task. The agent can read files, edit code, and run commands to check its work.

The app asks for approval before edits and commands, displays file diffs and command history, and reads AGENTS.md or CLAUDE.md for project instructions.

Conversations and model credentials are stored on your computer. Requests go directly to your configured endpoint. Alpine Code doesn't require an account or route requests through its own backend.

The desktop app and agent core are both available under the MIT license.

Download

https://github.com/TGoddessana/alpine-code/releases/latest

The desktop app currently supports Apple-silicon Macs running macOS 11 or later. Windows support is planned.

If you try it with a local model, I'd appreciate feedback on tool-calling reliability, useful custom tools, and reproducible failures. Please include the model and server you're using.

English isn't my first language, so I used a GPT model to translate this post.

Any feedback is welcome!! thanks!

I am making a project for myself and the first step of it is document analysis and OCR of Educational content/books, curriculums like STEM, English, and some Arabic mixed in the middle are the mainly parsed documents so having table, formula, figure and diagram extraction are a must, and I have about 500 labeled pages for the question banks ready for export as I heard that the layout detector could be finetuned.

My hardware is a Lenovo laptop (i7 14700HX, RTX 5060 8 GB of VRAM, 24GB RAM) and running windows 11.

I tried using PaddleOCR-VL-1.6, It's a 0.9B parameters model, and documented to use about 4GB or VRAM. but when I used it It was occupying the whole GPU and spilling about 6GB of RAM, it was taking 13\~70 sec./page which is obviously slow.

I was using the paddlepaddle framework with the correct CUDA version for my GPU, tried limiting VRAM usage by flagging system resources (ai idea) but got "not enough VRAM" message when I parsed more than 1 page in a folder, 1 page worked fine (the warm up of the VLM took a bit of time though), but when I put more than 1 page in that folder and ran the program again I got that error message.

I read through Hugging Face and found out that vLLM was the recommended path but that would require Linux. So I wanted confirmation from someone with similar specs as me that might have gone through a similar issue and found a solution. because vLLM require a dual boot to Linux or WSL2 which I don't have enough storage for.

I could buy a another SSD for my laptop (would cost a 2 month salary in my country ffs), so I need confirmation first before committing.

tldr;

Is there any hope of running the title or should I keep this idea in a trash bin?

I’m interested because the dense model runs but are either too slow or if you manage to run them fast like ninfer and other guys. It’s still a thinking monster. We need speed and quality both or a balance between that. So, I ask.

DeepSeek V4.1 Flash at Q2 is 341 GB on disk: 152 GB of weights plus 189 GB of Engram tables. My Mac is an M5 Max with 128 GB. It runs anyway, because ds4 streams the experts from SSD, and stock ds4 gave me around 12 tok/s on the CLI. Usable. But I had the feeling the SSD wasn't the only thing holding it back, so I started poking at it with a coding agent, and that turned into something bigger than I planned.

The first thing I noticed is that every session began the same way: the agent re-reading huge chunks of ds4.c (85k lines, three model families, three GPU backends) to figure out which 2% of it my model actually goes through. Most of what it read was about hardware I don't own and models I don't run. So I deleted all of it. Not #ifdef, deleted. ds4.c is 34k lines now, the whole tree 150k instead of 278k. Only DeepSeek V4.1 Flash, only Metal.

That changed the economics of trying things. An optimization attempt that used to cost me an afternoon of the agent wandering around now costs maybe an hour, so I tried a lot more of them and measured every single one instead of picking the three I believed in.

One rule the whole time: output doesn't change. Every change has to produce the same tokens as upstream ds4 on the same GGUF (greedy, ten prompts), and pass an A/B/B/A bench against the previous build with the logits compared bit for bit. No KV quant, no approximate kernels. If it's faster but a logit moved, it doesn't go in.

This is where it landed, internal SSD only, same GGUF, same flags, upstream's own bench (ds4 → fork):

\- generation, ctx 2048 (128 tokens, first one included): 12.1 → 24.4 tok/s

\- steady decode, ctx 2048: 15.7 → 25.7 tok/s

\- steady decode, ctx 32768: 15.7 → 22.0 tok/s

\- prefill, 16k → 32k context: 404 → 636 tok/s

\- first token after a prefill: 2.1–3.0 s → 0.3–1.1 s

None of it is clever. Decode layers get committed to the GPU without waiting for each other. Expert reads are split across a thread pool and the cache slabs sit in a Metal residency set. A handful of kernel fusions per decode token. Prefill reads the next layer's experts while the current layer computes. Individually each one is a small diff you can read in a few minutes. Together they double the speed, and you get all of this with the Mac as it is, nothing to buy.

Then I got curious about the SSD part. Streaming is bound by read bandwidth and a Mac has exactly one internal drive, so I put a byte-identical copy of the GGUF on an external Thunderbolt 5 SSD and made prefill read part of every layer from each drive at the same time. The engine checks the copy against the model at every start (about 7 s) and refuses to run if anything differs, because I don't trust myself to keep two 341 GB files in sync by hand.

Internal SSD only → with the external copy:

\- 3.5K-token prompt, time to first token: 13.2 s → 11.2 s

\- 10K-token prompt: 29.0 s → 25.0 s

\- +1.5K tokens appended to a 5.3K chat: 8.6 s → 7.0 s

\- first token after an 8K context: 1.38 s → 0.18 s

Decode doesn't change, it never reads the copy. My enclosure also runs the drive at PCIe 4.0 x4, about half of the internal SSD, so a better enclosure should do better than this. Nice side effect: the KV cache can write to the external drive, so the soldered internal SSD takes zero writes while the model runs. Again: this part is optional, the table above it is the one that matters for most people.

About staying in sync with ds4, because that was my main worry: the fork never renames the ds4\_\* files, every cut is marked in the source at the exact spot, and git merge upstream/main with rerere replays the conflict resolutions. After each merge the parity check tells me if the tokens still match. So far antirez's fixes have kept flowing in without drama.

Not everything worked. I tried to bring the two-SSD trick back into upstream ds4 through its mmap path: bit-exact, but prefill got 13–38% slower and I still don't know why, so no PR for now. Twenty-odd other ideas were measured and dropped. I keep all of them in a "rejected ideas" table in the repo with the numbers, mostly so the agent (and I) stop re-proposing the same thing every other week.

There's a growing trend of single-model inference engines, and ds4 itself started that way. This is just that idea pushed a bit further, one model and one backend, and at least here it holds up: faster, still correct, still merging upstream. I've done four of these forks, one per model; the procedure is in a separate repo (StarForge) and has nothing DeepSeek- or Metal-specific in it.

Repo: github.com/Chida82/sf-ds4-1flash. The README and speed-bench/perf-record.md have the conditions behind every number.

A few things I'd like to hear opinions on:

• Does "per-model fork that merges upstream" scale past a handful of forks, or is it just fragmentation with extra steps?
• Is bit-exact the right bar? I left speed on the table by refusing KV quant. Would you take 10% more for a slightly different token?
• If you have a 96–128 GB Apple Silicon Mac, I'd love to see stock ds4 and this side by side on your machine. One machine is an anecdote.

(I had to delete and re-upload this because Reddit messed up the post image somehow, sorry)

Hi, I recently posted about Anyworld, my small Python multiplayer RPG text game that runs on a browser, where an AI acts as the Dungeon Master.

Some expressed wishes that the game would be easier to set up, so I built a Docker compose system that allows you to have the game running in no time without any need to touch network settings. Just dive into DOCKER.md to get your game up and running fast, or read ahead for more details.

For local inference, it runs llama.cpp with NVIDIA GPU support, downloads a configured GGUF model from Hugging Face, and waits for the backend to be ready before starting the game. This will take a while, depending on the speed of your internet, so be patient.

The example includes recommended settings for a 16 GB VRAM system, and the model and llama-server parameters are configurable. I highly recommend Gemma 4 -based models on all VRAM tiers, they've been punching above their weights in testing.

You can also use OpenAI instead. In that mode, Compose starts the game without launching llama.cpp or downloading a local model.

To simplify networking, there’s an optional zero-config Cloudflare Quick Tunnel that prints a public HTTPS link in the console, so players can join without a Cloudflare account, domain, or router port forwarding. The address changes when the tunnel is recreated. Direct LAN access is available too, and host/player passwords still apply.

Game transcripts persist across container recreation, with optional debug logging stored separately. The Docker instructions include a Quick Startup section and commands for stopping, updating, and backing up the deployment.

The setup is working in testing, including connections from outside my LAN. I did encounter some intermittent access failures with the temporary tunnel URLs, so feedback from other networks and systems would be useful.

Hope you enjoy!

https://github.com/iamarxs/AnyWorld

On the fence about this one but is this good for localAI?

https://www.youtube.com/watch?v=lnLDjXy2kRA

https://www.youtube.com/watch?v=veP6jdvy7j8

Edit: Coxon, not Coxcon as his last name.

Follow-up to my post from a few weeks ago (14 architectures, full PEFT). This update is about a portability bug that was hiding behind its own correctness, because it's the most interesting thing I've fixed since.

The bug: the fix for one machine broke six fixtures on another

Back when I tuned the backend for Intel Gen9, I baked those kernel shapes into the portable path. That was wrong, but not for the reason you'd guess.

Two of the reductions in the saved-module path aren't really compared against "PyTorch in general" — they're compared against the PyTorch CPU library on the machine running the oracle. And ATen dispatches its vectorized CPU kernels by instruction set at run time. An AVX2 host gets 8-wide kernels; an AVX-512 host gets 16-wide ones, and the reduction shape changes with that dispatch.

So my "portable" AVX2-shaped kernels were exactly right on my AVX2-only laptop and one ulp off on my AMD ROG Ally (Ryzen Z1 Extreme, which is an AVX-512 part). One ulp doesn't sound like much until it gets amplified through every lower norm on the gradient path: the Gemma 4 saved\-stage model.embed_tokens adjoint went from 7.45e-9 to 3.22e-6, and six previously green PEFT saved-module fixtures (gemma3, gemma4, minimax\_m2, minimax\_m3, smollm3, qwen2\_5\_sliding\_tied) crossed the 2e-7 gate. Neither shape is wrong — only one matches a given machine, and baking in either one breaks the other.

The fix: probe the host, not the vendor

Kernel variants are still selected by GPU vendor. Those two reductions are now selected by host CPU capability instead: capability is probed once per process and cached, then the matching module pair is dispatched (linear_forward_lane2 / linear_forward_lane4, and the 8-lane / 16-lane transformer_cross_entropy builds). HIERARCHOS_ATEN_VECTOR_WIDTH=8|16 pins the shape for qualification when a reference wheel's kernels disagree with the CPU's own capability.

|Host|GPU|CPU dispatch|Status|
|:-|:-|:-|:-|
|Intel i5-6200U / HD Graphics 520 (2016 Skylake-U)|Intel Gen9|AVX2 only, no avx512f|32/32 LoRA, 32/32 switching, 32/32 saved|
|AMD Ryzen Z1 Extreme|RDNA 3|AVX-512|32/32 LoRA, 32/32 switching, 32/32 saved|

Same 2e-7 gate, unchanged. No tolerance was loosened to get there.

What I verified on each side

On the Intel machine, the post-change matrix is bit-identical, field for field, to its pre-change report across all 32 families — peft, gradient, two-step AdamW, frozen base, resume, lifecycle — which is how I know the AMD fix didn't quietly cost the Gen9 path anything. Also 693 passed / 0 failed / 9 ignored on the Rust lib suite and a clean strict headline forward run.

On the AMD side, the fix was re-qualified end to end: 32/32 on all three stages, provenance clean.

The harness fingerprints the pinned Transformers source alongside the shaders and binaries, and on the Intel side I re-derived the whole fingerprint from the pushed tree myself: 3951 inputs, zero changed, zero missing. So "green" refers to one frozen set of reference math, not whatever happened to be on disk.

Same caveats as always

• This is deterministic FP32 tiny-model correctness against a reference implementation, not a claim about arbitrary checkpoint sizes, dtypes, or hyperparameters.
• "Supported text graph" ≠ "the whole multimodal package works natively."
• The AVX-512 dispatch is only qualified on the AMD machine, since it's the only host I have that can execute it natively. The 16-lane module also doesn't rebuild byte-identically with the glslang version on my Intel box (one extra type/id, one difference in +inf materialization), so I've left it as the committed AMD-built module and documented that rather than swapping it without re-qualifying both hosts. I'd rather report that than pretend it's clean.
• NVIDIA and other GPUs are genuinely unqualified — the path is raw Vulkan, so they're untested rather than excluded.

What I'd love from you

Last time several people asked about hardware other than mine, so that's the ask again: if you build it on an AVX-512 laptop, an AVX2-only machine, or an NVIDIA/Intel GPU, I want to know what you get. The two reductions above are the ones most likely to behave differently on your CPU, and knowing your host's vector width is now part of the answer.

The new cross-platform section in the README documents the whole thing, including which host classes are measured and which aren't.

Repo: https://github.com/necat101/Hierarchos-Native Compatibility/parity record: https://github.com/necat101/Hierarchos-Native/blob/main/hierarchos-vulkan/COMPATIBILITY.md Regression audit: https://github.com/necat101/Hierarchos-Native/blob/main/AMD\_REGRESSION\_AUDIT.md Per-host tuning and measurements: https://github.com/necat101/Hierarchos-Native/blob/main/hierarchos-vulkan/VENDOR\_TUNING.md

I have been working on building domain focused local models from scratch through general pretraining and a rigorous post training process. My idea is we need models that reasons and understands algorithm and generate specs for focused coding models. The latter implements it simply. The whole thing can be orchestrated. I know there are flaws in this architecture, but we won't know until we try. I understand the latency problem.

This will allow parallelism and a way of getting the most out of a gpu. MOEs may activate less parameters than some dense ones, but the whole thing needs to be in the memory. Good for DGX spark or mac. But folks with 8-16 GB gpu need something more than what barely works, or barely useful.

I think group of specialists with a general purpose model as orchestrator might have a chance at beating mixture of experts for lower end PCs.

I got 28GB vram (4070 and a 5060ti 16GB), running on an x870e motherboard. So i can run dual model distillation and RL based post training. Will see where it goes. I think we need more useful models for people with lower vram, even if that means a newer architecture.

Have you tried something like this?

Please comment if you know there is work done already and you have tested.

I am no expert myself but I think it is high time we have community trained models. We can achieve a lot if we join forces.

One evening I wanted to pick up something I'd been working on the week before. My DSH sidebar had forty-odd chats, half of them called "New session". I scrolled for a while, found the right one on the third screen, and by the time I opened it I'd half forgotten what I wanted to ask.

So I stopped creating new chats and asked everything in one. That went wrong differently: my thesis, my budget and my move all ended up in the same context, and the model started mixing them.

What I wanted was simple: one chat box, say whatever is on my mind, and let it figure out which thing I'm talking about.

That's TheOne, a plugin for DeepSeek Harness. You only ever talk in one main chat. In the background, each thing you're working on gets its own session with its own context, and every message is sent to the one it belongs to. Come back days later and mention "that thing from last week", and it finds it. Your old chats get read and organised into a topic directory.

https://i.redd.it/81soypqdgrth1.gif

I wasn't sure it actually worked, so I measured it. I wrote 50 conversations of one person juggling three to five things at once, about 2,400 messages, each labelled with the thing it belongs to, and had it sort them one by one.

Starting from nothing, it put 86.6% of messages in the right place; 91.6% if it knows the topics up front. Dumping everything into one chat scores 44.5% on the same test. Its most common mistake is being too quick to decide something is new: a stray "I usually run about 20 km a week" makes it open a new topic. The whole run cost about a dollar, and the data and code are in the repo if you want to try another model.

Install: DSH → Plugins → Add plugin → dsh-theone

Repo: https://github.com/YunongDai2005/dsh-theone

It's a personal community project, not affiliated with DeepSeek. If it puts one of your messages in the wrong place, I'd genuinely like to hear about it.

Contact: [theone@yulid.org](mailto:theone@yulid.org)

I'm researching how people choose and vet fine-tunes and merges from Hugging Face. I'm not selling anything. I just want to understand what people actually do.
1. Where do you find the models you try?
2. What do you check before you start using one (benchmarks, model card, reviews, your own test prompts)?
3. Has a fine-tune ever behaved worse than its base model? For example, odd refusals, lost reasoning, strange outputs, or things it should not say. What happened?
4. If a quick side-by-side check of a download against its base model existed, would you use it? What would it need to show?
Short answers are great, and stories are even better. Thanks!

I use llama serve and webui of llama server. I have had long discussion with my chatbot and then randomly it just starts to forget almost everything. It starts asking same question, I correct it and it apologizes, but then next time it asks the same question with almost same words (or maybe even fully same words).

What could cause that? Something on my CLI parameters? Wrong cache settings?

I’ve been experimenting with Strata running Qwen 3.8 Flash on my RTX 3060. It’s seriously impressive to run this model on this small of a GPU.

My issue is that after several minutes of usage I’ll get an error like this:

\[strata\] the engine reported an error: verify: layer 1 never rang (unspecified launch failure)

\[strata\] done: 4558 tokens in 225 s (27.9 tok/s) (error, cancel=False)

And then I have to reboot to fix it.

Is there a log or way to troubleshoot what is causing this error?

I’ve been using Claude for some time now, and have developed apps for my own personal use, business use and for other businesses.

I’ve always liked the idea of moving away from the large companies and getting into more open source LLMs (simply cause I believe AI should be a tool for humanity and not have the potential to be gated by large corporate interests.

My personal philosophy aside: I’ve done some research into some models and now requesting insight from the community.

Here’s the tasks I would like it to be able to perform well on (without being able to go nuclear on anything — low risk LLMs only please):

\- File organization (both text and image)
\- Coding (frontend, backend, security, etc)

Not a huge list. I’ll start there.

I’ve looked into Miami v2.6 Pro but haven’t pulled the trigger yet. I would be using their server and now downloading locally.

If my write seems amateur-ish, it’s cause I am.

I’ve been looking at how agents navigate large, unfamiliar repositories.

A lot of the workflow still looks like:

"search → open file → grep → follow reference → repeat"

That works, but the model ends up reconstructing relationships that are already deterministic: calls, inheritance, implementations, dependencies, symbol resolution, etc.

We’ve been experimenting with a different approach in AxiomCode: build a code knowledge graph grounded in compiler/type information and let the agent query that before deciding what source it actually needs.

The interesting question for me is:

How much codebase exploration should actually be done by the LLM?

My current thinking is that deterministic relationships should be resolved before the model gets involved, and the LLM should spend its tokens reasoning over the result.

We open-sourced what we're building:

https://github.com/AxiomCodeAI/axiomcodegraph

Curious where people here draw the line between grep/RAG/semantic search and deterministic code intelligence.

I use Linux Mint XFCE as my daily driver and got jealous of all the Wispr Flow demos floating around. It's macOS/Windows-first and subscription-based, so instead of switching OS I spent a few weekends building the thing myself.

It's called AutoType. The whole interaction is: double-tap Right Alt anywhere, talk, double-tap again. A little floating pill shows a waveform so you know it's listening, then the cleaned-up text gets pasted into whatever window has focus.

The part I care most about is that it isn't locked to one vendor:

• Speech-to-text: cloud (Deepgram, Whisper via OpenAI or Groq, NVIDIA NIM) or 100% local offline with Parakeet GGUF models
• Text cleanup: OpenAI, Claude, Grok, DeepSeek, Qwen, Groq, OpenRouter, NVIDIA NIM, Ollama, or any custom OpenAI-compatible endpoint
• f you run Local STT + Ollama, nothing leaves your machine at all

Before the LLM touches anything, there's a deterministic normalization pass that handles spoken punctuation ("comma", "open quote"), "new line", bullet points, and personal vocab casing — so it doesn't hallucinate your formatting away. Then the LLM strips the "um"s and "uh"s and matches tone to your active window (more casual in Slack, code-formatted in an IDE).

A few details I'm weirdly proud of:

• t backs up and restores your clipboard instead of clobbering it
• The LLM layer is skipped entirely in raw mode or for voice commands
• GUI settings app, so you don't have to hand-edit .env

Free, MIT, no account, no telemetry. Costs are whatever your own API provider charges — usually fractions of a cent per dictation — or literally zero if you go fully local.

Repo: https://github.com/premtechworks/AutoType-Linux

Fair warning: it's built and tested on Linux Mint XFCE + X11. It leans on xdotool for window detection and pasting, so Wayland folks will probably have a rough time right now — that's the top thing on my list. Would love feedback, especially from anyone who tries the fully-offline path.

Hello guys! I've been out of the loop for a while. Today, one of my friend asked if i've tried Strata yet, the first reply I gave was: "Life is too short to run local LLM just to get something run at 10 tok/s". Hehe, I was an idiot.

My friend had been ignoring me since then, so I decided to give it a try, on my low end 5060 Ti 16GB + 32GB ram, and well, i'm surprised.

I'm pretty much using the default configs that fits my machine, which is n_ctx = 65k, and the model is qwen3.8-flash-next-coder-iq1_m. This is how the speed looks like:

https://preview.redd.it/d6wbv5swqvth1.png?width=1172&format=png&auto=…

On average, prompt processing is at 1k5 tok/s, and gen speed is at 55 tok/s.

Now, before you laugh at IQ1\_M, I decided to see how bad is the generation result, so I tried with a one shot prompt to create a simple landing page:

https://preview.redd.it/yo6kj7servth1.png?width=1834&format=png&auto=…

The total run time was about 2 minutes, at 46 tok/s. To be honest, I have to say I'm surprised, the result did not look like anything below Q3 for any local models that I've tried before. Here's a closer look at it:

https://preview.redd.it/z67sg0hlrvth1.png?width=1905&format=png&auto=…

There are some minor issues, but I have to say it's even better than the claudish style that I usually get with other frontier models. Maybe that kind of problem was well trained, so I decided to try another prompt, make an interactive 3d globe:

https://preview.redd.it/sccdghnzuvth1.png?width=1870&format=png&auto=…

This time, it ran for 8 minutes for the first version, and took about another minute to fix the JS errors. The result came out still impressive.

https://preview.redd.it/lzik4fezvvth1.png?width=3436&format=png&auto=…

You can see the two demos yourself here:

\- https://pitest-beta.vercel.app/bakery/

\- https://pitest-beta.vercel.app/earth/

Looking for a local model for opencode on two DGX Sparks, mostly C#/.NET code. Benchmarks I've found put Small 4 behind Nemotron 3 Super, but I'd like real-world experience: is Small 4 good for coding and tool calling, and has anyone compared the two?
Thanks!

I couldn’t practically run large AI models locally, so I started building a better agent runtime around smaller ones.

My original problem was pretty simple: I wanted to experiment with local AI agents, but running large models wasn’t practical on my hardware.

So instead of asking:

“How can I run a much bigger model?”

I started asking:

“How much more can I get out of a smaller model if the system around it is better?”

That became SIA.

Quick hardware/model context: I’m currently targeting local 3B–8B models, with most development and testing being done on Qwen2.5 Coder Tools 7B. The goal is specifically to make SIA useful on hardware where running much larger models isn’t practical.

SIA is an experimental local-first agent runtime focused on giving smaller models more structure around:

• planning
• tool use
• validation
• retries and repair
• state management
• task completion

Model target

1B–3B: experimental
3B–8B: primary target
10B–14B: planned testing / hardware dependent
30B+: not the main goal

To be clear, I’m not claiming that SIA magically makes a 7B model equivalent to a 30B+ model.

The idea is different.

If planning, tool execution, validation, retries, repair, and state are handled more systematically, how much less does the model itself need to get right on the first try?

That’s what I’m trying to measure.

I’m also working toward proper benchmarks comparing a raw local model against the same model running through SIA.

I want to document things like:

• task success rate
• retries / repair attempts
• model and tool calls
• execution time
• RAM / VRAM usage
• overall runtime overhead

I’ll publish actual numbers as I collect them rather than guessing hardware requirements.

The project is still experimental and I’m actively testing and breaking things, so feedback is genuinely useful.

Especially from people running 3B–8B models locally:

What models are you using, and what usually stops them from completing more complex agentic/coding tasks reliably?

A couple of days ago I shared repOx (a sub-15ms Rust CLI & lazygit-style TUI for packing repositories into LLM prompts) and got awesome feedback from this community.

I just released v0.2.0 implementing the most requested features:

1. Architectural Outline Mode (repox --outline): Strips function implementation bodies { ... } and keeps only structs, classes, traits, imports, and function signatures across Rust, Python, Go, TS/JS, and C/C++. Cuts token usage by 75–85% when you only need architectural context.

1. Synthetic Tool-Call Format (repox -f tool-call): Formats the repository as a JSON array of read\_file tool calls & responses — great for agent harnesses and local models trained on tool-use trajectories.

1. Smart Lockfile Summarizer (repox --summary-locks): Instead of burning 40k tokens on Cargo.lock / package-lock.json or hiding dependency versions completely, it parses lockfiles (Cargo.lock, package-lock.json, pnpm-lock.yaml, poetry.lock, yarn.lock, go.sum) into a tiny "package @ version" manifest (95%+ token reduction).

1. Git-Aware Packing (repox --modified / --staged): Pack only the files touched in your current working tree or staging area.

1. TUI Upgrades (repox -i): Added lexical syntax highlighting in the preview pane, Shift+C to copy a reproducible CLI command, and OSC 52 clipboard fallback for tmux / herdr / SSH.

Install / Update:

\- Crates.io: cargo install repox-cli

\- One-liner: curl -fsSL https://raw.githubusercontent.com/WVDYC/repOx/main/install.sh | sh

GitHub: https://github.com/WVDYC/repOx

Here is a common problem with building apps on LLMs:

Users ask the same question over and over.

Your app calls the model every single time.

You pay the API bill every time. Users wait 3–5 seconds every time.

**\*\*SentryGate\*\* is an open-source AI traffic controller that fixes this in literally one line of code.**

\### What it actually does:

\* ⚡ \*\*Lightning Fast (4ms)\*\*: If someone asks a question that was already answered, SentryGate serves the saved answer in 4 milliseconds instead of 4 seconds.

\* 💰 \*\*$0 on Repeat Questions\*\*: Bypasses the model completely on repeat or similarly phrased prompts.

\* 🧠 \*\*Doesn't Get Tricked\*\*: Basic caches get confused between "How to bake a cake with eggs" and "How to bake a cake WITHOUT eggs". SentryGate catches tricky negative words so it never serves the wrong answer.

\* 🕒 \*\*Knows What's Fresh\*\*: Real-time questions ("today's weather", "current stock price") automatically skip the cache.

\* 🔌 \*\*Zero Downloads / 1-Line Setup\*\*: No new libraries. Just point your existing OpenAI / LangChain \base\_url\ to SentryGate and keep your code 100% untouched.

Works locally on your machine with Ollama, or in the cloud. Completely open-source under the MIT license.

\* 🌐 \*\*Test the Live Playground (No login needed)\*\*: https://sentrygate-9ght.onrender.com

\* 💻 \*\*GitHub Repo\*\*: https://github.com/Prisha2004/Sentrygate

Note: Open-source project maintainer (MIT License, 100% free).

Feedback and stars are welcome!

released 0.40 of agent manager. it is a local go/tmux tui for running a few coding agents side by side.

added support for model and effort pickers per session. unlike a lot of other software we fetch the model list dynamically from each CLI so it stays up to date without updating the TUI. a new vendor model shows up without an agent manager release.

also added Antigravity CLI and Oh My Pi as built in agents.

https://github.com/YoanWai/agent-manager/releases/tag/v0.40.0

As a developer working at home on a single GPU i am building and experimenting a lot not only with coding agents but also with apps that use generative APIs. The flood of models, engines, and different APIs makes it hard to always get it right in every app and tool and to keep it up-to-date. I needed a gateway that i could target in all my apps while also being able to use it with clients that only support official upstream APIs from Anthropic and OpenAI.

So i build a gateway for myself and over time extended it with more and more Features. Its a Rust based desktop app using Tauri and Leptos. Leptos is WASM running inside a lightweight gtk webview. I did it specifically this way to allow for remote browser based administration when working from another device in my network, but have it ready in the tray on my desktop at any time. I also wanted to integrate tools for quickly testing new models and llama.cpp patches. What it does:

• Supports Text, Embeddings, Audio and Image APIs.
• Presents OpenAI and Anthropic compatible APIs and routes them to cloud APIs or into llama.cpp, audio.cpp and stable-diffusion.cpp containers.
• Builds the backend containers directly from git inside of podman containers, including MRs, PRs from main or a specific branch or commit. Notifies for updates.
• One container per model and manages their lifecycle while scheduling VRAM-aware with fallbacks.
• Models with different configurations can be registered as aliases, so client configuration does not have to change for model changes. These aliases also support model chains. For example reloading the model with more configured context only when needed, routing to cloud if a certain context length is reached. Or chains like: small context & high quant -> big context & lower quant at context length steps.
• A "GPU hold" mode that can be triggered from the tray, it unloads models, blocks new models from loading and responds with either an error or routes to a fallback if configured. For gaming or other blocking GPU use.
• Fallback routing to any other configured model or alias in case of VRAM contention or an active GPU hold.
• Offers an OpenAI compatible websocket with /v1/realtime via a staged pipeline: VAD / Smart Turn -> ASR -> LLM -> TTS while being able to select either a local model or a cloud model for each step of it. This includes barge-in and session management. Tools are supported and executed server side.
• MCP Gateway: Add your MCP servers to the gateway and it offers them via prefix and scoped per token via its own /mcp API as a streamable MCP server. MCP servers are executed inside of podman containers by default.
• Scoped Tokens, detailed metrics, traffic monitoring, budgets, price sync (only tested with kilo) with lots of graphs, cost comparisons for local tokens if it would have been cloud traffic.
• Download manager with huggingface downloads and updates, preconfigured catalogs for audio.cpp and sd.cpp.
• Integrated MCP Server with admin tools on a seperate MCP route. Every option and feature of the gateway is configurable via the MCP. Adding models, testing configurations, gateway config and status. A coding agent can configure it for you.
• Documentation MCP like context7. Agents that connect to the MCP and have the docs tools enabled can query and request documentation for specific library versions of any kind. All requests are listed in the UI and can then be chunked, embedded and ingested into a vector storage (all managed by the gateway). Context7 is really great but often stale, some libraries are missing like my own. The Interface is not intuitive but the gateways admin MCP lets my agent fill it anyway.
• A chat interface with metrics, file input, folders, thread-specific settings, Mic Input & TTS selectable from the gateways own models. For quickly testing models. Tools from the MCP gateway and the gateways own tools can be added as well. (Admin Chat as a preconfigured chat for gateway administration)
• Voice Chat mode in the chat interface based on the realtime API, with normal dialog flow or push-to-talk
• /v1/responses that works session based and supports server-side tool execution.
• Audio and Image Labs for quickly testing audio and image tasks like image generation, image edit, tts, asr, cloning, conversion, etc.. I try to keep up with audio.cpp's and sd.cpp's tempo.
• Container based agents, that a build to a specific interface mounted into the container (tools, vars and files) and can mount their own UI page and MCP tools to the gateway while running. Kind of like smaller, task based extensions.
• Model benchmark with lots of graphs to compare configs or engines.
• Integrated API docs in the spirit of Swagger with all APIs offered by the gateway
• Lots more i forgot

I am usually very shy and thought long and hard if i want to risk exposure and publish all of this. But it has made my day in my specific scenario a lot more comfortable and maybe you like it.

https://preview.redd.it/l02s01qpdxth1.png?width=429&format=png&auto=w…

Here is the link: https://github.com/lmgw-dev/lmgw

I hope you dont tear me to shreds and can find use in it. Dont be to judgy on the Interface, my years of experience are all on the backend and in devops.

What is planned next:

• Decision models

And the obvious disclosure: AI has played a role at all stages of development. Nearly everything is touched by a diverse set of models and most of the prose text in the repo is generated. I made sure to write this post by hand because you all deserve it and i am myself annoyed by generated posts. Also, i did not publish the git history and will squash most of my commits for safety reasons.

I've had some updates to this benchmark, meaning it should run smoothly compared to before. I have an umm, measly GPU with 8 GB of VRAM, so I can't run models like Qwen3.8-27B on it. Any result would be good from you guys; although this is more suited to frontier models, smaller models should still work. I'll credit you for any results if you'd like.

The primary reason I thought it would be interesting to run: benchmarks are nearly all pass or fail on a single-dimention graph, so I thought it might be worth shaping a new one up. BinkBench measures video quality and video compression rate, which gives you two things to plot on. The agent also can't score 100% - there isn't an end, which makes it progressively harder as the agents get smarter, because they need to implement more novel techniques. I also thought video encoding would be good as a benchmark, since it's not something we've tested agents on before and is pretty hard. It's like the kernel/compiler optimisation things we've seen other labs show tests on.

More info is on GitHub,

Old post: https://www.reddit.com/r/LocalLLaMA/comments/1vn6nlr/looking\_for\_people\_to\_help\_me\_run\_a\_benchmark/

Been developing a custom agent orchestration layer with opencode as the harnes. It has been successful so far in terms on being manage multiple concurrent sessions. However, I am wondering if I am using the best harness given that opencode isn't meant to be a hackable, i.e. Less flexible compared to something like pi.

I've developed an interface that allows me to easily switch between different harnesses. i.e. Without having to re-write the orchestration layer, and am considering swapping out opencode for pi. The reasoning is obvious, pi is meant to be a hackable harness, so its likely to be a better fit for building a custom multi-agent system. Opencode does support a headless mode which has helped a lot. But it seems heavy on resources, and recently has seemed very buggy. Less features might be a better approach towards the stability that I need.

Before I make the dive, has anyone else already tried integrating pi into a multi-agent system? Did you find pi a better fit for the worker layer compared to something like opencode? What problems did you run into?

Thanks for your help.

Is a small local model more than I need to change handwritten notes to text?

My agent has a skill I use but I could probably not use up my limited inference bandwidth or maybe not as much if it’s a much smaller model, right? What’s the smallest model that can accurately read handwriting?

Looking to hear others workflows or how they handle note conversion and the like

Thanks

Hello, if you are running multi gpu setup, check out ramjet https://github.com/helixml/ramjet

The goal is to have a local version of dynamo that can do equal or better job (but ideally without k8s). Check out https://helix.ml/blog/ramjet-vs-nvidia-dynamo as well.

If you have dgx spark, multi mac setups, etc it would be great to contribute recipes so other ppl can just pull

I got fed up that running ollama pull overnight still was resulting large models failing to download.

When I discovered I could download them faster if I cancelled every hour or whatever I decided I could automate it.

I've done a 3rd tweak with this (Originally written by chat-gpt :D) seems to be less buggy now thanks to claude :D

I've found it by pure chance, searching for a decision model to use into a simulated project. What about it? Does anybody use it?

I was using local conversational models in 2023-2025 in LM Studio but went full Opus and Claude Code from November 2025 until October 2026, now.

whats the best harness + model for my use case? document review and coding. multimodal input and output ideally.

I want to review contracts where even the contract itself is not to be disclosed, and I don't want to put that in the cloud anywhere, so that's prompting me to update everything

so I've installed Pi but don't have any models. And Pi wants to serve local models from llama.cpp but I just read about dwarfstar4 (ds4) but it serves MoE on just a few open source frontier models, yet reportedly wants minimum 96GB RAM for Metal use. I was primarily wondering if ds4 acts like its serving from llama.cpp to a harness like Pi

it seems like llama.cpp is catching up in real time, with the cached MoE thing that got merged in today with some infighting, but I'm not even sure which model I should be using

there's one crowd that's like "we need cached MoE at 20 token/sec with billion param models" and there's another crowd that's like "Qwen 27B is all you need" others are like "Gemme 4B is sooo good now"

do decision models fit in this workflow anywhere? in conjunction with LLM's in a harness loaded at the same time?

I'm pretty lost. I won't remain lost, but I also want to hear others opinion while I experiment myself, hopefully to narrow down what I need to experiment

Hi everyone,

As we move from simple RAG chat to fully autonomous tool-using and coding agents, we are hitting a massive wall: predictability and safety.

Right now, most setups try to secure AI agents using probabilistic methods like system prompt hardening, alignment tuning or reactive LLM-based guardrails (e.g., LlamaGuard). The problem is that these guardrails can be bypassed via Indirect Prompt Injections (IPI), leading to capability escapes, unauthorized shell command executions or runaway API budget depletion.

To solve this, I’ve been working on a framework that completely shifts the paradigm from trusting the model to governing the execution environment using cryptography.

I call it EBP-CA (Execution-Boundary Proofs with Cryptographic Authorization). It is a model-agnostic layer that sits directly between the untrusted agent and the runtime environment, treating every single model-generated action as untrusted.

The core architecture implements six deterministic security primitives:

1. Signed Capability Contracts: Immutable cryptographic tokens defining the exact boundaries of what an agent can execute.
2. Independently Recomputed Policy Checks: The runtime re-evaluates policy compliance deterministically, bypassing the model's interpretation entirely.
3. Short-Lived Single-Use Execution Grants: Atomic, ephemeral tokens issued for a single specific payload to eliminate permanent session hijacking.
4. Replay & TOC-TOU Protection: Cryptographic binding of the execution grant to the exact payload hash, neutralizing Race Conditions (Time-of-Check to Time-of-Use).
5. Trusted Cost Accounting: Enforced real-time budget tracking at the runtime layer.
6. Human-in-the-Loop (HITL) Gateways: Non bypassable prompts that freeze execution and mandate cryptographic user authorization for out-of-scope tasks.

The working prototype currently passes 74 integration tests, validating full resilience against path traversals, command injections, budget bypasses, and sandbox escapes.

The specifications, architectural diagram, and executive summary are available on GitHub under a private proprietary license (free for technical evaluation and research review): https://github.com/lucadilo/ebpca-ai

I'm posting this here because I’d love to get the community's feedback on this approach. How do you see this scaling with kernel-level sandboxing (like eBPF or gVisor integration)? Let's discuss!

After seeing how good the cloud models are getting, I feel like this is something we cannot let big tech hold over us so deiced to build a budget local box.

After testing about 20 open models, Qwen3.8-Flash-Next (medium reasoning) was the only one that passed my task without inventing config options when used with a harness that forced doc lookups. So the box is built around that model. GLM-5.3-Flash performed even better but it's too big for my budget.

Planned build (Netherlands prices):

• Ryzen 5 9600, about €200
• MSI B850 Gaming Plus MAX WiFi, about €170
• 2×48 GB DDR5-5600, €1,199–1,549. Two sticks only to avoid the four-stick speed penalty.
• Used RTX 3090, about €1,150–1,500
• Case, 850 W PSU and NVMe I already own

That's about 79 GB of the model in RAM (experts plus the 28.8 GB n-gram table) and about 20.6 GB on the card.

Questions:

1. Will 6 Zen 5 cores hold back generation with 40 MoE layers on the CPU?
2. At 96 GB with about 79 GB of mode will 17 GB be enough for the OS, a sandbox container and an embedding model? Should I use --mlock?
3. Is DDR5-6000 worth it over 5600?
4. Has anyone run Unsloth's MTP branch with experts on the CPU? What speedup did you get, and does it break the prompt cache on the DeltaNet layers?
5. Is anything wrong with a used 3090 here? Also has anyone tried the Arc Pro B60 (€772 new) workable on Vulkan or SYCL with this model yet? It's so much cheaper but I am worried becase of the software.

Super exciting to work on it but I am really inexperienced so this would be my first build. Does it make sense?