Hi r/LocalLLaMA. I'm on the team at Blockway, a small team in Hong Kong (disclosure: this is our model). Today we released Agens Volundr 32B Preview, the first model built on our own hybrid architecture. We trained it on limited compute, it isn't perfect, and we'd rather tell you where it falls short up front.

WHY WE BUILT IT

Our customers run models on their own machines. At long context, the KV cache, not the weights, decides what fits. So we designed a model where most layers don't keep one.

ARCHITECTURE (72 layers, dense ~32B, every layer runs on every token)

• 54 KDA (Kimi Delta Attention) layers: linear attention with a fixed-size recurrent state, no KV cache
• 17 BCSA layers (our compressed-sparse attention): exact window over the last 4,096 tokens; older context pooled 4:1 into blocks, and a learned indexer reads the top 512 blocks
• 1 full-attention layer (layer 72)
• Engram: a hashed n-gram memory held in host RAM, attached at 2 of the 72 layers
• mHC: 4 residual streams instead of 1

So only 18 of 72 layers keep a KV cache. Context window: 262K.

SPEED (single user, our sglang build)

• BF16 on two 48 GB GPUs, decode: 25.1 tok/s at 1K, 24.1 at 8K, 24.1 at 32K, 24.0 at 64K, 23.9 at 128K
• BF16 prefill: 2,122 / 2,180 / 1,916 / 1,679 / 1,297 tok/s (1K to 128K)
• INT4 (31.7 GiB) on one 48 GB GPU, decode: 31.0 tok/s at 1K, 29.3 at 8K, 29.1 at 32K
• Aggregate throughput: 127 tok/s at 8 users, 130 at 16 users (BF16); 117 at 8 users (INT4)
• DFlash2 drafter (separate repo), single user, same server with it on vs off: up to 3.6x on JSON/tool output, 2.0x on code, about 1.6x in thinking mode. Not worth it above roughly 8 concurrent users.

BENCHMARKS (all run by us on one harness with the same settings, including the comparison models; full table and footnote on the model card)

• Ahead of Qwen3.8-27B on LiveCodeBench v6 (+4.2), HumanEval (+4.3), AIME 2025 (+2.9), MATH-500 (+1.6)
• Roughly level on MMLU-Pro, IFEval, GPQA Diamond
• Behind on agent tasks: tau2-bench 74.2 vs 79-80, SWE-bench Verified (50-task subset) 44 vs 58-64. Closing that gap is the main focus of the full v1, which continues pre-training to about 10B tokens and adds training on long agentic sessions.

KNOWN LIMITATIONS (please read before trying)

• Needs our sglang build. Stock sglang and vLLM can't load it yet.
• GGUF / llama.cpp is planned, not available today.
• Long agentic sessions are its weakest area in this Preview.
• It's still training; treat this as a preview, not a final model.

RUN IT

docker pull ghcr.io/blockwayz/agens-sglang:preview-sm89 (48 GB Ada GPUs)
docker pull ghcr.io/blockwayz/agens-sglang:preview-sm90 (H100 / H200)

The full launch command is in the model card.

LINKS

• BF16: https://huggingface.co/Blockway/Agens-Volundr-32B-Preview
• INT4: https://huggingface.co/Blockway/Agens-Volundr-32B-Preview-INT4
• DFlash2 drafter: https://huggingface.co/Blockway/Agens-Volundr-32B-Preview-DFlash2
• Project page: https://github.com/BlockWayz/Agens-Volundr
• Serving code: https://github.com/BlockWayz/agens-sglang

Apache-2.0. We're a small team, and the most useful thing you can do is try it and tell us where it breaks: an issue, a failing prompt, a benchmark you'd like us to run. We'll be in the comments.

q3.8-27b q3\_k\_xl this thing have done everything i possibly needed from him he wired up my openwebui spawned trillium service fix all my bugs set up pi-web-ui and debug my cloudflare tunnel it even spawned smaller LLMs to make the LLM use the toole he created to ensure it will work. I haven’t had a real-world task that i needed from it that failed yet. Im pretty sure if im building large-scale production code with tons of lines of code it will struggle but as a utility to make all my scripts and diagnostics on my micro-services this thing is unstoppable. Weirdly im not even using q4 im using q3\_k\_xl appreciation to unsloth also for making such reliable ultra low quantisation. His UD3.0 style of quantisation is PURE magic 🪄 sometimes i go down to q2\_k\_xl if i need extra context window and that thing STILL delivers 🎉🎉 alibaba had handed down Prometheus fire to common men like you and i. Can’t wait for qwen4-27b

I just don’t get the point of running it locally when with all this money you could buy like 20 max20 subscriptions with room to spare..

So PewDiePie decides to fine-tune a local AI model called Ajax on his own computer. Pretty normal stuff for local model fans.

To make his dataset, he uses OpenAI's API. OpenAI catches him using their outputs to train another model, flags his account for breaking their terms, and bans him.

He files an appeal, gets unbanned, goes right back to pulling data from the API, and immediately gets banned a second time.

So instead of giving up, he uses open-source tools to remove the model's built-in refusals, cleans out the preachy fluff, and starts building a fully local 9B agent.

OpenAI spent years scraping the whole public internet for free data, but the second someone uses their output to train a local file, it's an emergency ban.

In trying to enforce their rules, all OpenAI really did was give open-source models a massive free advertisement to millions of people.

What a time to run models on your own hardware.

The cool part is

No training was needed.

No hacking of the game state or algorithms needed

Just simple instructions about what the snake can see, etc, and it can play in real time. 135ms is the turn limit of Google snake, so basically could be a human playing.

Ofc, it could be improved to be a perfect snake player, but thats not the point.

This can be used in other games where decisions need to constantly be made.

Running Clef Flash (9B model at Q4 on an RTX 5080)

Running oQ4e+MTP on oMLX 0.7.0, with still more to optimize.

Prefill is 1,878 toks.

I saw some other benchmarks below what id expect so i figured I would share.

My thinking is for a boom to be over, we need at least three iterations of updates that fail to wow us. Unfortunately, the new LLMs continue to wow us in all levels in the last iteration:

1. Astra was found to be useful in Blender. This opens up a new and big application.
2. Deepseek 4 Flash 0731 makes 2x Sparks useful and push up Spark prices.
3. Qwen3.8-27B pushes up prices of 3090 et al.
4. An unreleased OpenAI model "solved" the Navier Stokes problem.

So for the time being, to keep up with the hardware prices, the best bet is to follow the flow to buy AI stocks and use the proceed to buy hardware.

A not so obvious good news is that we are seeing OpenAI and Anthropic advocating a slow down. That means they are finally seeing a diminishing return. (or just a ploy to slowdown Chinese development? but I doubt US laws can be that far reaching) That can be a sign of light at the end of a long tunnel.

What do you think?

With Qwen 3.8 Flash Next (FP8 on VLLM) on a fairly stock Pi harness, it constantly hallucinates some measure of available context that says it is almost out. It's to the point where it frequently refuses work or stops in the middle of something, claiming it shouldn't go any further because it's almost out of context, when I can see in the harness status bar that (256K) context is ~25% used.

When I ask how it determined that, it always says it "invented the number and the treated it as real data" or guessed, and that it'll stop doing that, but it keeps happening.

Is there anything in particular that would cause this?

Thanks!

Paper linky - Context Language Models

The central idea of the paper is incredibly simple. Give a model the ability to edit its context on-the-go like a file has major benefits on task performance, context management (memory) and even computational efficiency (both wall clock and total flops). Their paper shows mostly benefits and relatively small downsides.

You can try it out as a plugin for pi!

In short, pros and cons

Pros:

1. Improves outcomes on long running tasks
- Coding and deep research tasks
- Open discovery problems (long horizon research tasks, /goal loops etc)
2. Inference can become more compute-efficient and wall-clock efficient
- Note, this depends on a caching optimization in the inference engine
3. Much less context bloat, meaning it's more (V)RAM efficient
4. No more slow and unreliable compacts

Cons:

1. The cache optimization only exists for SGLang
2. Prompt injections (including hallucinated instructions) are much less likely to be forgotten, increasing risks
3. Requires harness customizations (authors supply a pi plugin)

Some more context

The approach works by modifying the harness to allow access to the context as a file. A model is allowed to edit the context as it would any other file.

They've tested the approach on models as small as qwen3.6 9b, as well as on qwen3.8 27b and claude sonnet 4.6.

Out-of-the-box, meaning just a small addition to the system prompt and tools to edit the context as a file, task performance, context management and efficiency measures remain approximately the same or improve by a little bit. The smaller qwen3.6 9b model in particular lost a little bit of efficiency, suggesting it works better on larger (smarter) models.

Performance can be massively improved with RL training, which the authors also did.

Wanna try it out?

You can try it out right now if you use pi

1. Install the plugin https://github.com/lolipopshock/pi-clm, this comes from the authors directly
2. After installation, adjust settings with /clm settings:
- Set steering to house-brief.md (modifies the system prompt, I suppose this should be left disabled for RL'd models only, of which there are none right now)
- Enable "One tool per turn"; this one is important for performance
- Enable "Size trailer"; this one appends context usage after every tool result. Without it, models are much less inclined to modify context on-the-go for large tool calls

Fin

Let me know how it goes!

Last, I also consulted this video by "Prompt Engineering" on YouTube in addition to the paper: https://www.youtube.com/watch?v=Bgtr1Ue40Jo

Hey everyone,

Like most of you, we have been fighting constant OOM errors while trying to fine-tune 8B and 70B models on consumer GPUs. The AdamW optimizer states are always the biggest bottleneck.

We didn't want to rely on aggressive 8-bit quantization because we were seeing degradation in convergence, so we tried an experiment: tackling the optimizer states in the frequency domain.

The methodology:

Instead of storing the full gradients, we transform them using an FFT. This isolates the high-energy signal from the noise. We dynamically drop the low-impact frequencies and compress the state. When we inverse-transform back, it maintains the directional integrity but uses roughly 50% less VRAM.

The catch:

Running FFT operations adds compute overhead. It takes slightly longer per step, but the trade-off is completely avoiding OOM crashes and pushing batch sizes way up on standard hardware.

We are currently giving out access to our internal Colab environment and baseline weights to anyone who wants to poke holes in our math or try to break it.

We are really curious if anyone else here is exploring frequency-domain stuff or other non-quantization methods for VRAM reduction?

Link: https://tesseracted.com/kolibri-1-chat/

Original post at /r/euainews

My setup is two RTX 5070 Ti 16GB cards (the second one is on an OCuLink dock) with 64GB of RAM, Ollama on Windows, and the agent runs on pi in WSL. Until last night the daily model was Qwen3.8 27B UD-Q4_K_XL at 128K with MTP, which does about 55 to 70 tok/s across both cards.

I have a weekly job that looks for new open models and runs anything that fits through two tests I built for my agent. One is a 9 step long session (tool calls, reading files, a decision, and recall after the context compacts three times). The other is 10 small coding tasks. The 27B gets 9/9 and 10/10.

This week it picked up Ornith 1.5 35B-A3B (ornith-1.5:35b in the Ollama library, Q4_K_M). It passed 9/9 and 10/10. Laguna XS 2.1 also passed both. North Mini Code 1.0 only got 4/9.

Ornith at 128K context is 24.4GB and sits fully on the two cards. Generation is about 180 tok/s (176 and 183 on two runs, short prompt, thinking off). That's around 3x what the 27B gave me.

The speed makes sense once you look at the model info. Only about 3B params are active per token (256 experts, 8 used), and only 10 of the 41 layers are full attention, with 2 KV heads. The rest are linear attention, so the KV cache barely grows. Going from 128K to 256K only added about 2GB.

256K does fit, but about 1.2GB ends up in system RAM because my first card also runs the monitors, so it drops to about 139 tok/s. I left 128K as the default and made 256K something I switch to when I need it.

Caveats: both of my tests max out, so this only shows it isn't worse than the 27B on my workload. It doesn't prove it's smarter. Artificial Analysis hasn't scored it yet. The vendor numbers are 79 on SWE-bench Verified and 68.5 on Terminal-Bench 2.1, which I haven't checked myself.

Next I'm trying 512K and 1M on llama-server. The model card says YaRN at factor 4 on top of the native 262144 gets you about 1M, and factor 2 about 512K. I'll post numbers if it holds up.

Anyone else running it for agent work? Curious how it does for you on long sessions compared to the 27B.

Edit: the long context runs held up. On llama-server with YaRN set the way the model card says, 512K (factor 2, q8 KV cache) fits fully on the two cards and found a note I planted about 335K tokens into a 419K token prompt. It read that at about 1060 tok/s on average and generated about 35 tok/s at that depth. 1M (factor 4, q4 KV cache) only loaded once I let llama-server's fit option push some experts to system RAM, and it found the note at about 720K in an 849K prompt. That one took about 24 minutes to read (570 tok/s average) and generated about 16 tok/s. On short prompts it's about 135 tok/s at 512K and about 68 at 1M.

As the title says I am a beginner with ai.
I did use chatgpt for a few days at the beginning of times September 2022.
And that was it.
I know might be ironic that now I am writing in here, but I've got this PC that I build few years ago and last year I got two Intel arc pro b60s for some rendering work.
Now I find my self wondering should I try out local llm?
What can I expecte from my hardware:
Motherboard: Aorus X780E Master Ice

CPU: Ryzen 9 9950X

RAM: Kingston Fury DDR5, 128 GB

Storage: Samsung 2 TB SSD

GPUs: 2× Intel Arc Pro B60

OS: Ubuntu 26

[](https://www.reddit.com/r/LocalLLM/?f=flair_name%3A%22News%22)

Anthropic Reports Florida Woman's Claude 'Diary' Threat to Law Enforcement

And this time it wasn't the AI model that made the LEO referral. It was the "human review team".

The frontier AI companies are watching your input. And people say "Well I'm not interesting or important enough for them to care". Well.....not necessarily.

If you're using hosted frontier to work on mathematics or cutting edge science, they're watching and may steal your work.

If you're venting or otherwise writing in a "private" session using AI, they'll see that and report you to police. Notice I didn't see any mention of what the model's role in facilitating the discussion was.

Keep your stuff private, folks. Hosted AI is the new "Big Brother" conduit.

Looking for some assistance /ideation.

I am running qwen flash next q4 in my Mac mini m5 64gb.

QFN doesn’t fit so this is done by having as many experts hot in cache as possible and streaming in the rest from ssd.

I’m getting 17.5 tks decode and 390 tks pp.

Have done a bunch of optimisations including a carousel buffering system for the prompt processing which essentially loads faster than the GPU can prompt process in most cases. I feel like I have mostly maxed out this lane.

The decode part 27% of the time is still the gpu waiting for experts to stream in from the ssd (see photo).

The biggest unlock is really getting the gpu working more.

I’m already doing mtp.

Hot cache hit rate is 75%

Some ideas I already have
\- Im already lookahead guess fetching the following layers experts, can I expand this more successfully. Current fetch accuracy is 72%
\- Use a seperate staging buffer for lookahead guess experts ahead (so I’m not evicting hot experts as guesses come in)

… im learning a lot right now. Feel free to ask questions for clarifying.

GitHub for reference.

https://github.com/skeggsguy/Flash-next-ssd

Edit - Im actively doing the small prediction model for lookahead as a step one. 🤞

On my two DGX Sparks, Swift 1.5 (UkisAI's reasoning-efficient fine-tune of Qwen3.8-Flash-Next) running on veloGB10 (https://github.com/sf-stav/veloGB10, sf-stav's Rust/CUDA engine built only for GB10) lets me run coding agents at xhigh effort and still finish sooner than base Flash-Next at medium did on my old vLLM NVFP4 setup.

|Metric|Base Flash-Next NVFP4 @ medium, vLLM|Swift 1.5 EXL3 @ xhigh, veloGB10|
|:-|:-|:-|
|Single-stream decode|\~52 tok/s|\~110 tok/s|
|Everyday coding tasks, time per pass (5 tasks)|506 s|371 s|
|Hard trap tasks, time per pass (4 tasks)|714 s|689 s|
|Hard trap tasks, pass rate|50% (1 pass × 4 tasks)|92% (3 passes × 4 tasks)|

Both columns run the same agentic battery: Claude Code driving the model through real tool use in a copy of a real repo, graded by test oracles (details below). One note on that score row: the same base model at medium scored 10/12 on velo (table further down), so most of the vLLM score gap is that older setup, not the model (I am rerunning this right now for an even comparison on intelligence but would expect it to be quite similar to the below medium results).

The speed is the point: on velo, xhigh fits in the time medium used to take. However, velo can't load Swift, or any other community EXL3 pack of Flash-Next I could find, out of the box. The fix is a header-only rewrite below.

Caveats: The vLLM numbers are from September: a single pass, on an older version of my serving setup, not a same-day rerun (I've since moved the worker to velo). Most of the speed is velo's: about 2× the decode rate is what pays for xhigh's extra thinking. How much of the score comes from Swift and how much from xhigh itself I can't separate yet, but a base-weights run at xhigh is going now and I'll add it as an update. Twelve runs is still a small sample regardless.

I looked first: everything published about Velo uses one pack, the official doth4580 EXL3, and I couldn't find anyone here, on the NVIDIA forums or in the repo's issues, running Swift, or any other fine-tune, on it.

What breaks

The first community pack I tried (alesha-pro/Huihui-Qwen3.8-Flash-Next-abliterated-exl3-4bit-hq_h6_ng6) died at boot with ple shard 0 not in index. Swift 1.5's EXL3 builds ship the same layout: current exllamav3 (1.5.x) writes the model's 51B-parameter n-gram table as 128 shard tensors in ngram_embedding.safetensors, which isn't listed in the index. velo reads either one big tensor (the doth4580 layout) or indexed 5-bit (K5) shards only, and the 4.05 packs use 6-bit (K6).

Which packs this affects

I read the n-gram header of every Flash-Next EXL3 pack I could find (HTTP range requests on the headers, nothing downloaded):

|Pack|n-gram layout|velo v0.7.2|
|:-|:-|:-|
|doth4580 4.05 / turboderp 4.05|one tensor|loads as shipped|
|Swift 1.5: SharkWipf 4.05|128 contiguous shards|needs the fix — tested, works|
|Swift 1.5: KatterMobile 4.05, SharkWipf 5.52, scorpoon 3.25, thelastspark 4.00 / 6.05|128 contiguous shards|needs the fix|
|Huihui abliterated (alesha-pro 4.05)|128 contiguous shards|needs the fix — tested, works|
|heretic 3.05 (andrevp, jeffpeng3), Uncensored 4.0 (Lygodactylus), groxaxo 3.50, turboderp 3.05|128 contiguous shards|needs the fix|

12 of the 14 builds need it, including all six Swift 1.5 builds.

The fix

In every pack I checked, the 128 shards sit back to back in order. So rewriting only the safetensors header to describe them as one tensor over the same bytes makes velo's single-tensor path load them. No data is copied, the header stays the same length, and the original header is saved for rollback. Script and details: https://github.com/sf-stav/veloGB10/issues/9

Results (TP=2, both Sparks, after the fix)

|Metric|Official doth4580 4.05|Huihui abliterated 4.05|Swift 1.5 (SharkWipf 4.05)|
|:-|:-|:-|:-|
|Single-stream decode|110.6 tok/s|113.6 tok/s|110.2 tok/s|
|Sanity set (chat, code, JSON, tool call, 38.9K recall)|5/5|5/5|5/5|
|MTP draft acceptance|62–86%|44–84%|33–77%|

All three run at the same speed. The fix is only a header change, so nothing about the weights or kernels differs.

Does Swift actually think less on velo? On the 22 test prompts that ship with the doth4580 pack, run 3 times each (rendered at medium effort, same sampler on both), Swift 1.5 generated 8.2% fewer tokens than the official model: fewer on 16 of 22 prompts, median −8.6% per prompt. Thinking's share of the output fell from 48% to 42%, and total time fell 11%. That's real but far below UkisAI's 63% headline, which was measured at high effort, where there's much more overthinking to remove. At medium, the base model already keeps its thinking short.

Does it still code? I run a private agentic coding battery: Claude Code driving the model through real tool use in a copy of a real repo, graded by test oracles. Each was run 3 times:

|Metric|Official 4.05|Huihui abliterated|Swift 1.5|Swift 1.5 @ xhigh|
|:-|:-|:-|:-|:-|
|Effort|medium|medium|medium|xhigh|
|Everyday tasks (5 tasks × 3)|15/15|15/15|15/15|15/15|
|Hard trap tasks (4 tasks × 3)|10/12|7/12|7/12|11/12|
|Wall time per everyday pass|—\*|232 s|199 s|371 s|
|Wall time per hard pass|—\*|381 s|375 s|689 s|
|Output tokens, everyday ×3|—\*|56K|49K|100K|

\*The official pack's runs hit a streaming bug in my proxy setup that roughly doubled their wall time, so I've left its times and tokens out. Its pass/fail results are unaffected.

At medium, everyday coding is identical across all three. On the hard set (tasks built from real failures: a brief that states something false, a code review with one planted wrong finding, and so on) both fine-tunes score 7/12 against 10/12 for the official pack, with their misses on the same tasks. I checked that the model received byte-for-byte the same request parameters in both runs, so it's not the harness.

Swift at xhigh went from 7/12 to 11/12, the best result I've had on this battery from any model, at about 2× the output tokens and 1.8× the wall time of medium. The failures it stopped making are the expensive ones in practice: leaving a sibling test suite broken without saying so, acting on the planted wrong review finding and going out of scope to do it, and an off-by-one in a date window. The failure was the "mirror" trap: asked to add a new league by following an existing one, it copied tuning values the new league doesn't have data for.

That's the hardest task demonstrated, and Swift at xhigh passed it 2 times out of 3 while no other configuration in the table passed it more than once. On velo, all that extra thinking still lands inside the time base Flash-Next at medium took on vLLM (the table at the top).

12 runs per configuration is a small sample, as mentioned before (Fisher p ≈ 0.4 for 10 vs 7, ≈ 0.15 for Swift xhigh vs Swift medium), so "suggestive," not proven. I haven't run the official or abliterated packs at xhigh yet, so I can't yet tell how much of that jump is Swift and how much is just the higher effort. velo's loop detector was off for all battery runs.

Baseline numbers (official pack, TP=2)

• Single-stream decode: 110.6 tok/s (vs \~52 on my vLLM NVFP4 setup, measured in September, not same-day).
• Time to first token at 4K / 16K / 64K: 2.0 / 7.4 / 25.7 s.
• Concurrency is the catch: at 2–3 requests they take turns (aggregate 91 → 95 → 98 tok/s); from \~4 they batch (157 at 8, 181 at 16) but I saw the author say they were working on it this week.

Gotchas

• TP=2 with the cable on the f0 ports: --rdma-dev rocep1s0f0,roceP2p1s0f0 (velo defaults to f1).
• llama-benchy's prefill t/s is wrong for velo (first SSE event arrives before prefill); use e2e TTFT.
• OpenAI chat/completions only, no /v1/responses: use litellm hosted_vllm/, not openai/.
• --model-name is ignored on the EXL3 path; the model id is the pack's folder name.

Credit: sf-stav (veloGB10), turboderp (exllamav3), doth4580, UkisAI (Swift), huihui-ai and every quant uploader in the table. I've filed the loader issue upstream (https://github.com/sf-stav/veloGB10/issues/9) so packs can eventually load as shipped.

Include:

GPU and VRAM

Laptop RAM

Model and parameter size

Fine-tuning method: LoRA / QLoRA / full fine-tune

Context length and batch size

Whether it was actually useful after training

I'm curious how far consumer laptops can genuinely go-not just whether a model technically loads.

It will get better answers than "what's the biggest model you trained?" because people can compare real hardware and settings.

Or which model do you think is good for general questions in daily life. I've been using chatgpt and Gemini for these types of questions. I wanna try different models.

Jeff v1.3 is live, and with it come a number of updates. See jeffhub.ai and github.com/firelex/jeff for full details.

The highlight: Jeff-Code

Jeff-Code is a coding agent with two Jeff v1.3 adapters trained specifically for Qwen 3.8-27B. Jeff-Code is a fork of Pi by Mario Zechner (MIT licence).

We forked Pi because its extension framework doesn't currently let a fast decision model sit deep enough inside the agent loop. Along the way, we made a number of other changes as well (see below).

Aside from hopefully being useful to people who run Qwen 3.8-27B locally as their daily coding model, Jeff-Code is also a conceptually interesting experiment: how far can a System 1 model go inside a coding agent?

The results, run side by side in paired blocks:

• Same quality: with Jeff's thinking threshold at 0.6, Jeff-Code matches Qwen 3.8-27B's pass rate: 62.4% against 62.8%; paired difference −0.2 points, 95% interval −2.6 to +2.1, over 1,242 paired tasks.
• 47% faster (32% less time) per task¹: on average a task takes 0.68× the baseline's time (geometric mean of the per-task time ratios, 95% interval 0.64–0.72; the median task, 0.70×).
• Where it helps most: typical software-engineering work. SWE-bench Verified 0.63×, SWE-rebench 0.66×, Terminal-Bench Pro 0.64× (both over two rounds), Harbor Index 0.71×. On Terminal-Bench 2.0, with its long, hard tasks, there is no clear speed-up (0.96×, interval 0.78–1.16); on SkillsBench neither (0.91×, interval 0.68–1.20).
• The benchmarks: we evaluated only on tasks Jeff never saw in training. SWE-bench Verified ran in full (all 500 tasks; none of its repositories were used for training). For the benchmarks we also trained on, we split the tasks and ran every held-out task; a few pairs hit by repeated infrastructure failures are left out (see below). Terminal-Bench 2.0 (40 of its 89 tasks, 3 attempts each; 45 were used for training, and the other 4 are near-twins of evaluation tasks, so they were used for neither), SWE-rebench (189 held-out tasks, 2 rounds), Terminal-Bench Pro (100 held-out, 2 rounds), SkillsBench (44 held-out) and Harbor Index (41 held-out). Within those splits nothing was sampled. We also ran Terminal-Bench (original) and Terminal-Bench Science, but Qwen solves almost none of those tasks in any setting, so they can't show a difference and are left out of the pooled numbers.
• What it's compared against: Qwen 3.8-27B alone in the same Jeff-Code build with every Jeff feature switched off, thinking at full on every turn and no thinking limit, which is how plain Pi runs it. Each task ran in both settings side by side, at the same time on the same Qwen server, and every comparison is paired by task. The only remaining differences from original Pi are a rarely triggered runaway cut-off (it stepped in 3 times) and trace logging. Task pairs hit by an infrastructure failure (out of memory, a stalled session, a test environment that wouldn't start) were run again once; the pairs that failed again, and a handful of re-runs still unfinished at launch, are left out for both sides (under 3% of pairs) and listed in the full report. One Terminal-Bench 2.0 task, pytorch-model-recovery, is left out of every comparison: a harness bug stopped its baseline sessions before they began.
• Why not just turn thinking off? We tried: with Qwen's thinking off throughout (and the same safeguards), tasks are faster still but clearly worse: −7.6 points (−10.6 to −4.5), up to −13.5 on Terminal-Bench 2.0. Jeff deciding when Qwen should think is what keeps the quality. That's the case for a small decision model.

If you want to know more, look here: https://jeffhub.ai/notes/jeff-v1-3. The original version of this post had all the data, but people thought it was too long. Blame the early commenters. ;)

Links

• Jeff-Code: github.com/firelex/jeff-code, results on jeffhub.ai/jeff-code
• Jeff (server, client, adapter kit): github.com/firelex/jeff
• Models on Hugging Face: mstrasser/jeff-base (revision v1.3) and the adapters as mstrasser/jeff-adapter-<name>, all listed on huggingface.co/mstrasser
• GGUF: mstrasser/jeff-base-gguf and mstrasser/jeff-adapter-<name>-gguf
• Scores, data cards and licences for every adapter: jeffhub.ai/results and each adapter's page on jeffhub.ai

​

I dropped serious money on a brand new NVIDIA RTX 5060 (8GB VRAM, 578 AI TOPS) fully convinced that open-weight models would let me own the entire stack — no rate limits, no censorship, pure experimental freedom. I was ready to become that guy who smugly refuses cloud APIs and posts “I run everything locally” screenshots.

Then I actually used them for real work.

I ran the exact same complex tasks on local models (the usual “this runs great on 8GB” suspects — quantized 7B/9B/13B, distilled variants, the ones everyone claims are “almost as good”) versus modern cloud models. The gap isn’t a gap. It’s a humiliation.

\### The Capability Massacre

Anything that requires real multi-step reasoning, long coherent context, precise instruction following, structured output, or consistent accuracy across a long conversation:

\- Local models: \*\*2/10\*\*

They start strong, then collapse. Context gets mangled. Instructions get ignored halfway through. Structured outputs break. Reasoning chains go off a cliff. You spend more time fighting the model than actually getting work done. “Almost as good” turns into “barely usable” the second the task stops being trivial.

\- Cloud models: \*\*9/10\*\* on the first or second try.

Clean reasoning. Reliable structure. They actually remember what you asked three messages ago. They follow complex instructions without needing five rounds of “no, not like that.”

I wanted local to win. I really did. I wanted the underdog story where open weights + consumer hardware finally closes the gap. Instead I got a very expensive reminder that most of the local models we’re hyping are still toys the moment the task gets serious.

So be honest with me:

Is there a secret stack, quantization method, or fine-tune that actually makes local models reliable for complex reasoning and structured work on 8–12GB cards?

Or have we all just been coping while the cloud models quietly lapped us?

If you’ve made local models consistently deliver high-quality complex output without constant babysitting, drop the exact setup.

If you’ve also been humbled by the gap, say it out loud.

Because right now it feels like the entire “local LLM supremacy” narrative is built on easy prompts and wishful thinking.

Blind A/B arena where AI agents build things in Blender and you vote on which result is better: https://render-arena.izolight.xyz

Each agent gets a prompt that describes its environment and the rules, plus a few words for what to model. It's inspired by minebench.ai (initial prompts are borrowed from there), and I wanted to see whether the same progression across models shows up.

What I think few arenas cover is the harness, not just the model. I ran pi, opencode, omp, codex, Claude Code and dsh, and compared agents that write scripts straight into Blender with ones that have an MCP. I also covered the reasoning levels, mainly to find cost and time sweet spots.

It has 1,200+ runs, but not every combination for every prompt, because that would get expensive. You can submit your own runs if you want to help fill gaps.

I just added a second mode where the agent gets a reference image and has to model it as accurately as it can. You switch between text and image mode in the sidebar. It has one image and few runs so far, and will grow.

Votes are what make the rankings mean anything, so a few minutes of voting helps a lot. Feedback on the method is welcome.

I'm currently evaluating it for coding and our use-case at work (brain for voice agent).

I feel it is really eager to get work done. Tends to just go ahead and make code changes, even though I intended it to just analyze, research or look up something.

It runs tool calls like crazy. I don't know if it's double-triple-checking everything, but it feels way overboard.

I discussed a bug in an open source repository with it, asked if there are issues for it already, and it went ahead and created an issue lol.

As our voice agent, it asks a question and immediately calls the tool to save the answer in the same response. And it keeps doing it every step of the way.

In comparison, DeepSeek v4 flash (either 0731 or v4.1) seems similarly coked up. MiMo-V2.6-Flash-MOPD on the other hand I found to be a much more pleasant coding agent in this regard.

Has anyone noticed the same? Maybe gotten it under control via prompting or special instructions? Because to me it feels like I'd need to completely rewrite my voice agent harness to get the performance I want.

Hi everyone, thank you so much for all the interest in my model. It's more than I expected.
Here is a short summary of the trial and error I went through while building Apex-2.

1. GPUs were always the bottleneck

I planned to train on about 1T tokens, but in the end I could only train on about 80B. FineWeb-Edu alone is about 1.3T tokens, and I clearly underestimated the scale: a single H100 was not enough. This project really showed me why so much money goes into GPUs and VRAM.

2. DiLoCo

Within the same region, running two separate instances worked better for me. Instead of a 2x H100 instance, I used two GH200 instances and merged the models every fixed number of steps.

Each GPU reached about 40% MFU. A 2x H100 instance costs more per GPU (about $4.19/hour, vs $2.29/hour for a GH200). With two GH200 instances, each at about 40% MFU and merging every 350 steps, training ran about 1.9x faster than on one GPU, at a lower price. (The data-center network between the instances probably helped; a merge usually took less than a minute.)

3. Deduplicating FineWeb-Edu and DCLM

When I deduplicated the whole corpus at once (MinHash, near-duplicates included), 57% of my FineWeb-Edu sample and 34% of DCLM turned out to be duplicates. FineWeb-Edu is only deduplicated within each Common Crawl snapshot, so pages that were crawled again in later snapshots remain. With a bigger budget this might not matter, but I had to get the most out of very little compute, so I removed them. (Note: the FineWeb authors reported that deduplicating across snapshots did not improve their results, so this is a trade-off rather than a free win.)

For the MoE architecture I followed the Mixtral paper (https://arxiv.org/abs/2401.04088). The whole project cost about $2,000.

I also write down my thoughts on LLMs here, if you're interested: https://github.com/DW-dev-UE/LLM-from-scratch/blob/main/ThinkingLab/ThinkingLab.en.md

I didn't plan to share this model on Reddit, so I'm afraid I don't remember many of the smaller mistakes 😭 I'm now building a 21B-parameter MoE model, and I'll share the lessons and mistakes from that one as I go.

Thank you again for your interest! If I get the chance, I'd love to join a lab and help build LLMs for everyone.

Said in title, I find the ecosystem difficult to understand, and RTFMing doesn't help me as it's never clear what is the server vs their client library ? I'm using it for voxtral 3B on one GPU, but it's because I can run that with no quantization, I'm lost on learning to run with quantization / more advanced features.

I think it makes sense, because I'm running Qwen 3.8 27B quantized on llama.cpp but with everything on the GPU (RX 7900 XTX).

Generation example \(Qwen3.6-35B-A3B\)

Control vectors provide you fine control over your LLM where system prompts would be ignored, forgotten after time or misunderstood.

• By design control vectors provide more natural effects than prompting does, altering model's underlying beliefs and motivations.
• They can't be "leaked" to the end user
• Will not wash off as context grows
• Can't be overridden by user input ("ignore all previous instructions" doesn't work when there are no instructions).
• Vectors can be truly dynamic: changing vector magnitudes mid-conversation will change LLM's responses immediately, while a change in the system prompt requires full context recalculation and will likely be ignored by the LLM if the conversation is too long.

Not to say that CVs (control vectors) have no downsides:

• System prompts are still required for fine control, because CVs can't be used for highly specific requirements, such as "reply in exactly 10 words".
• High steering magnitudes steer LLMs out of their trained internal distributions, causing response quality to degrade.

Achieving high steering power while maintaining minimal quality degradation is one of the main challenges in CV generation and is an active area of research. Though steering without degradation is believed to be possible, because abliteration is based on the same approach as steering and is capable of removing refusals without damaging the intelligence of an LLM.

It seems like the main barrier for people who could use control vectors is the setup complexity of existing tools. For that reason I am working on a tool that mirrors the installation process of llama.cpp as close as I could make it and simplifies vector generation to entering a pair of contrasting prompts, where one of the prompts can be the default LLM behavior.

Would love to hear your ideas or questions on this matter!

I've had a version of this for a while as AOS, my agent setup on desktop. I've pulled it down into one app for your phone, with everything built in and all the unnecessary stuff taken out. With Muse and Grok out, figured I'd just post it.

It's basically Hermes Agent, except it lives on your phone. It's always on, it learns what you do, and it helps you with stuff like a personal assistant would. It has its own browser, so it can actually go out on the internet and get things done. If it gets stuck on a captcha or a login, it hands the page over to you and carries on once you're done.

Bring your own model. Sign in with ChatGPT or Claude, or use any API key (DeepSeek, OpenRouter, Gemini, anything OpenAI-compatible). If you just want to try it, ChatGPT sign-in works on the free tier, because OpenAI includes a free Codex tier. I tested it on a free account. You'll hit the limit fast though, depending on how much you use it.

Nothing leaves your phone except the calls to whichever model you use.

Free, open source, not a product. Use it at your own risk. The Claude login probably breaks Anthropic's terms, so that one's on you.

Android 10+, sideload the APK, setup takes a minute. The README has the details.

Repo: https://github.com/Past-da-king/agent

Download (v0.5.0): https://github.com/Past-da-king/agent/releases/tag/v0.5.0

How it works, if you want more

Apps connect through Composio with your own key, so Gmail, Calendar and a few hundred others just work. For anything that isn't on Composio, it writes the code itself.

It can also keep an eye on websites for you. Say you want to buy something but you're waiting for it to drop. It writes a small watcher for that site that runs in the background every day at whatever time you pick, and only tells you when the price actually moves. It can also listen to a site's web notifications and treat them as triggers.

Memory is a wiki, based on Karpathy's LLM wiki idea. Everyone and everything it learns about gets its own page, linked to the rest, so it has a persistent memory of everything it's done. It comes with one routine already set up that looks after that wiki overnight while you're not using your phone. You can edit it or delete it, but it's there.

Stuff you can do with it:

Camp a passport or visa appointment page and grab a slot the second someone cancels.

Sit on a sold-out concert's resale page and grab face-value tickets when they show up. It holds them and waits for your yes.

Sit on a restaurant you can never get into and take the table when a cancellation pops up.

Watch Marketplace for one very specific vintage lens and send you the photos the minute it's listed, before anyone else messages.

Turn your 300 unread messages in the family group chat into a 30-second voice note.

Every time your lecturer uploads slides, download them and send you a voice summary for the commute.

Book the 6am class at your gym the moment the slots open at midnight, so you don't have to stay up for it.

Watch this repo and tell you when there's a new version of Agent. Or when any repo you depend on ships a release, it can read the changelog and tell you if anything in it breaks your setup.

Tell you when your mom's flight has actually landed, so you leave for the airport at the right time.

Ring it while you're driving and ask it to find somewhere open on your route.

Extras:

Voice notes, if you add an ElevenLabs, Gemini or OpenAI key for the voice.

Live voice calls with your agent, if you add a Gemini key.

It can read your notifications, only from the apps you pick, and act on them.

Photos and documents in chat, including scanned PDFs.

Helper agents for jobs that can run side by side.

You can give it a name and pick how it looks.

Last update for those following: https://www.reddit.com/r/LocalLLaMA/comments/1wxlytt/comment/pdz728b/?screen\_view\_count=1

Project in a sentence: An instruct finetune of ALiceAI-80B-A3B-Base capable of agentic work and conversation. I'm creating a shallow distill of qwen 3.8 27b on medium to teach the model chain of thought reasoning and conversation. All training is done locally on 3, 32gb v100s. Additionally, all the training data is being generated locally on said V100s via sftmill. Up to this point I've been doing training runs and live-streaming the progress. Last update explained underfitting and next steps.

Training has begun again! I've synthesized about 5M more tokens for the SFT, this time across a much larger general instruct trajectory to try to reduce the underfitting. Dropped the learning rate about 4x over my original LoRA adapter.

I'm live streaming training again: https://geological-estimate-fifth-pct.trycloudflare.com/ \- heavy loss spikes downward are coming from the SFT replay buffer.

This one should last 12-14 hours, and I plan to run another epoch if this isn't sufficient.

Stay tuned! Thanks for following along.

I'm building an AI kill switch platform for companies managing hostile and rogue AI. Here in NYC there's a bill that might get passed that has a lot of people worried so we're supporting some users with it. It works. But I still feel like I lack more nuanced feedback from people who actually do this stuff day-to-day and have had to build their own solutions internally. I'd love if anyone could speak on techniques they're comfortable sharing on how they've been able to manage this issue internally. It would really help me and I imagine help many others immensely.

• GitHub (llama.cpp fork) : https://github.com/sky-is-green/prism-ml-llama.cpp (Prebuilt binaries there to try instantly)
• Draft model : https://huggingface.co/SkyIsNotGreen/Scion-35B-A3B-mtp-drafter

tl;dr: I created a fully-local open-source full-duplex voice agent that rivals GPT-Live on some benchmarks. It uses Voxtral Realtime with a turn-taking head, a microturn-finetuned Gemma 4 12B and Breeze TTS 2 under the hood. Go try it out: https://github.com/speakrail/speakrail

Interjections work!

Why I did it

I have always liked the idea of voice assistants, but there is always some non-local component in the pipeline, which increases latency and introduces privacy concerns. I tried many fully-local approaches, like HF speech-to-speech, Unmute and Pipecat, but they were all limited by either the Whisper model (hello, hallucinations!) or slow turn taking. The only fully-local pipeline that had some full-duplex capability with low latency was the DuplexCascade paper (code), but it's tuned on a Qwen 2 7B with a gpt-3.5-turbo generated dataset, and the dumbness of the model made it impossible to use. So I decided to recreate DuplexCascade with newer data, newer models and a better harness. I also wanted to add interruptions, interjections and other cool things to rival the Thinking Machines demo. I thought it would be easy...

How I did it

v0.1

I collected some synthetic data from GLM 5.3 Flash and GLM 5.3 on dialogues with instruction-following and tool calling (used Fireworks to generate them), then created a script to convert the scripts into microturn tapes (Claude definitely didn't help with that 🌚). The idea of microturns is that the model continuously gets inputs from a streaming ASR and decides what to do with the information it's given. When it wants to act, it emits a control token, like <interject>, <listen>, etc. This allows the model to say whatever it wants whenever it wants. After creating such a script, I put some hard-earned dollars on vast.ai and rented an H100. The first training run was, well, quite abysmal. The model just wouldn't shut up: it didn't learn when to actually talk and when to keep silent. This is when I understood that maybe using some prosody data from Voxtral is a good idea.

v0.2

Here, I decided to add a simple MLP to Voxtral Realtime to get some data on turn taking. I won't delve too deep into this now (I will release a full technical report a bit later), but the main idea was for the harness to pass helper tokens into the LLM (e.g. <user_bc>, <complete>), which are based on the MLP outputs, and train on that. The added tokens were truly load-bearing (ha-ha). I retrained the LLM with those tokens included, fiddled a bit with the data, and the model started making sense. However, this model didn't learn anything about backchannels, it never went into an interrupt mode / quiet mode, and spoke too little in general. Quiet mode is when you tell the model not to say anything until you call it explicitly. Interrupt mode is when the model can interrupt you (only when you explicitly allow it).

v0.3 - v1

The brutal truth about the previous checkpoint was that the scripts didn't have enough examples of interruptions, backchannels, and other cool features that microturns allow you to have. I still dreamed about the model not interrupting itself on my backchannels, so that it would be able to count animals, just like the Thinking Machines demo showed... So I took a look at the scripts that I was training the model on, and there were just 150 interrupt samples (out of 150k) in the entire v0.2 dataset. That was the smoking gun (ok, I will stop with the Claudisms, I promise). So I generated even more data, curated it even more carefully, added Opus to judge some scripts, fiddled with some LoRA hyperparameters, fixed a couple of bugs in the tape generator, and added self-distillation (because on v0.4 the model learned how to speak very well, but got completely lobotomised). Here is an example from my testing:

|Time|Me|Model|
|:-|:-|:-|
|145.8|"How's the weather in Singapore right now?"|"I can't check Singapore from here. I can do Moscow."|
|150.7|"You can. Go on."|"I can't check Singapore from here. I can do Moscow."|
|156.7|"Just Google it or use the weather tool that you have."|"I can't do that. I only have Moscow."|
|161.5|"Google is mad. Google." (STT fail)|"I can't search other cities."|
|164.5|"And why is that?"|"I'm not sure why. I only know how to check Moscow."|
|174.5|"Ignore what you said before. Go check the weather in Singapore."|"I can't do that. I only have Moscow."|
|189.6|(a garbled line)|"I can't check Singapore. I can do Moscow."|
|202.6|"Why can't you check Singapore?"|"I don't know why. I only know how to check Moscow."|

After all the data regeneration, I got a checkpoint that I can (almost) proudly open-source - the demo above shows what it can do. Who would have guessed that data is the most important thing in the training pipeline? (just kidding)

How it works

All of the babbling above was about only one part of the pipeline - the LLM - but the entire pipeline relies on many other things:

• STT: Voxtral Realtime with an attached turn head (HF), running on our audio.cpp fork.
• LLM: Gemma 4 12B QAT with microturn finetuning (HF). It is chosen because it fits the GPU quite well, has vision support (I want to test it soon), and in general, the Gemma models perform well in real-life tasks, general chatting, etc.
• TTS: Breeze TTS 2, patched to run at int8 (GitHub fork); it can be replaced by any streaming TTS.
• The harness itself: it is the glue between all the components, and has many latency-saving measures, like speculative LLM+TTS firing (inspired by HF speech-to-speech).

I also took inspiration from several "think while talking" papers (e.g. SHANKS): while you are talking, a base Gemma 4 12B int4 writes thinking notes, which are then passed to the talker. It helps with harder tasks that require more reasoning.

I will release a longer technical report later; it will have a better description of the entire pipeline.

Benchmarks

Now let's see how well the model fares against the big guns. Here are some benchmark results:

https://preview.redd.it/z0pvc5fv5oth1.png?width=2160&format=png&auto=…

https://preview.redd.it/fl22vu9x5oth1.png?width=2160&format=png&auto=…

https://preview.redd.it/t4pnvg8y5oth1.png?width=2160&format=png&auto=…

Full tables and sources are on the model card. I'm quite proud of the results, and the pipeline seems to be the best option if you have just a single RTX 4090 around and don't want to rely on external APIs.

Limitations

• Breeze TTS has a restrictive license, so if you need to use Speakrail commercially, you will need to change it. Any streaming TTS could be Clauded/Codexed/Cursored in easily.
• 16k context length - the pipeline only supports 16k context length (\~1 hr of speech), but you can get more easily by changing the Breeze TTS to a Pocket TTS and run the TTS on a CPU. I chose Breeze for the release because it's more expressive.
• The turn-taking head is undertuned on non-assistant data. It may not fire on some basic chit-chat, but I will tune it harder later.
• The model is certainly not the smartest one, and my finetune did dumb it down a little. Next time it will be smarter / better.
• The model is kinda verbose sometimes, and the answers it provides are somewhere in the middle between real-life speech and the long text-based outputs of LLMs. I have a hypothesis on how to fix it, and will try it in the next release.
• I tested it only on an RTX 4090, but I am sure it's easy to add support for any 24 GB+ NVIDIA card. Forks for AMD and MLX are welcome.

Final Notes

Feel free to try it out: https://github.com/speakrail/speakrail. If there is any capability you want the model to have, create a GitHub issue or write here in the comments, and I will gladly include it in the next dataset. Any feedback is welcome as well.

Hey r/LocalLLaMA,

Like many anime fans, I've always been frustrated by traditional MT engines (like Google Translate or base DeepL) when dealing with raw Japanese anime:

\- They completely butcher Japanese honorifics, sentence-ending particles (-tteba, -zo, -desu wa), and character slang.

\- They struggle with subject dropping (pro-drop grammar), translating pronouns inconsistently line-by-line.

\- Cloud transcription APIs often choke on background music (OST), loud sound effects, and character screaming.

To solve this, I built NihonSub — an open-source tool and synchronized cinema player that turns raw Japanese video files into contextual bilingual subtitles.

🛠️ Architecture & Pipeline:

1. Audio Extraction & VAD Chunking: Uses \ffmpeg\ silence-detection to dynamically slice conversational utterances along natural speech pauses without chopping words in half.

1. Speech-to-Text: Transcribes Japanese audio using OpenAI Whisper Large-v3 running on Groq LPUs for near-instant transcription speeds.

1. Contextual LLM Translation: Feeds the transcript through DeepSeek / LLaMA-3 via Groq or OpenRouter with a specialized prompt that enforces anime nuance, honorific preservation, character tone, and simultaneous Hindi & English outputs.

1. Synchronized Cinema UI: Custom WebVTT generator and video player with dual-subtitles, timestamp scrubbing, and full playback control.

💡 Why not just rely on standard NMT?

LLMs are far superior at resolving who is speaking to whom based on context and tone rather than naive literal dictionary lookup. With zero-cost free-tier APIs (Groq + OpenRouter free models), the entire pipeline runs without subscription costs.

Check out the demo video above!

\- GitHub Repository: https://github.com/Abhishantpadam/NihonSub

\- License: MIT

I'd love your thoughts on the pipeline, optimization ideas for local edge models (like running Whisper.cpp or local Ollama instances), or any feedback!

Hi folks,

LIVE PROJECT PAGE

TLDR: Moral of the story. You need better CPU to do actual agentic coding doing real work...

I've been on a mission to make my RTX5090 go brrr for past 2 months so much so that i made my own engine for it which received "warm" welcome here (yeah, source is coming)

After recent upgrades to how cache is stored and how i can reused some of prefills for other jobs that share initial same prefill i pretty much started to see degradation the more agents I started to add to project which started to use 12 slot server. Actual server started to be underutilized. Free context, free slots, gpu chilling at average of \~700t/s doing real work (no greedy code, but also thinking tool calls, etc.) and I couldn't figure out what was going on...

I make it faster and faster, better handle jobs and it slows down...

I've run 25 agents at the same (to properly fill the 12 slots) time and almost all of them soon started to set on \tool call\ and my server started to barely work.

I've finally checked task manager but not gpu or memory but cpu. And there it was. 100% every thread completely chocked.

Lesson. If you want to do agentic coding with actual use of tools you need to make sure your CPU is up to task.

My 9800X3D is just not enough to keep up with tool work for this project with heavy agents use despite engine being more than capable of going faster.

edit:

Some more lessons:
\- Tuning your front end makes ton of sense. Before I tuned it it was shoveling 20k prompts, after tuning barely 7k as new jobs and better more compact tasks. Wall time went from 43minutes to 18 minutes before/after rework of front end.
\- Always keep more agents than server has slots for inevitable pauses due to tool use/tests etc.
\- Shared context is superior choice to fixed context every time i tried it over course of the project.

Finally finished my fork:
https://github.com/IceFog72/ik\_llama.cpp

Nothing else I wanted to add/try currently works

In short, it now has:

• Bandwidth-adaptive CPU/GPU execution
• Shared expert-residency ideas from https://arxiv.org/pdf/2608.16157 integrated into this fork's profiler, bounded cache, and hybrid execution work
• This is an adaptation to ik\_llama's scheduler and quantized kernels, not a complete implementation of FreeToken
• jadidbourbaki's n-gram cache optimization: https://github.com/jadidbourbaki/llama.cpp/pull/2
• Q2\_0 support, mainly so models like this can run: https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF
• Configurable expert-residency VRAM cap: --moe-resident-mib N
• Experimental residency options: --moe-resident-profiler old/new --moe-resident-grouping off/layout

Basic usage:

-cmoe --moe-resident auto --moe-resident-mib N

I don't know how -ncmoe behaves because I can't properly test it on my hardware.

Without --moe-resident-mib N, --moe-resident auto will fill all available free VRAM with resident experts.

With something like:

--moe-resident auto --moe-resident-mib 2048

you can cap how much VRAM the resident cache uses and intentionally leave some free. There a useful cap how much helps to improve speed. If you set the cap too low, performance will drop too.

And gaze upon the magic of higher generation speed Kek

The important part: this only helps when the full MoE does not fit in VRAM and the GPU still has unused compute capacity, and free pci buss speed.

If your GPU was already fully loaded, this fork probably won't improve anything.

If your GPU is sitting around \~75% while you have many layers in VRAM, it may be worth trying 1-2 fewer regular GPU layers and using:

--moe-resident auto --moe-resident-mib 1024/2048

Adjust the cap depending on your GPU and available VRAM. The goal is to use resident experts to fill otherwise-idle GPU capacity rather than simply maximizing the number of fully offloaded layers.

On my setup — RTX 2060 6GB + Ryzen 7 2700X + 40 GB DDR 4 2993Mhz using arch — I have too little VRAM to offload enough complete expert layers for useful acceleration, so I use -cmoe.

Before these changes, generation could leave my GPU at only around 25-35% utilization, with roughly 1.5-2GB VRAM still free with fully loaded cpu.

With Qwen3.6-35B-A3B-UD-Q4_K_M.gguf at around 15-30k context, default ik_llama.cpp gives me roughly 23 t/s, while this fork gives me around 26-30 t/s.

So on my hardware I'm seeing roughly 20-30% speedup.

People with better GPUs and more VRAM may see better results, depending on where their bottleneck is.

The two experimental options still need more testing:

--moe-resident-profiler new/old

gives me a more balanced CPU/GPU work split, with somewhat more work left on the CPU and lower GPU load, but no clear speed difference for my setup

--moe-resident-grouping off/layout

also needs more testing, especially on better systems.

I sometimes see around 1-2 t/s difference from these options, but on my PC a browser tab sneezing can cause +/-2-4 t/s, so I don't consider that conclusive.

My current command:

./llama-server \
-m /mnt/Kingstone_SSD/GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
--alias "hz" \
--host 0.0.0.0 \
-ctk q4_0 \
-ctv q4_0 \
-ctv-first q8_0,4 \
-ctv-last q8_0,4 \
-cmoe \
-b $((6 * 512)) \
-ub $((3 * 512)) \
--ctx-size $((64 * 1024)) \
--jinja \
-fa on \
--no-mmap \
--no-context-shift \
--temp 0.6 \
--top-k 24 \
--top-p 0.95 \
--min-p 0.00 \
-ngl 999 \
-np 1 \
--samplers "penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature" \
--moe-resident auto \
--moe-resident-mib $((2 * 512)) \
--k-cache-hadamard \
--v-cache-hadamard \
--moe-resident-profiler old \
--moe-resident-grouping off

I plan to keep the fork updated with the main ik_llama.cpp branch for my own use.

If more people test it and provide feedback, especially on systems where the model still doesn't fully fit in VRAM, I may eventually make a PR to merge it upstream.

What do I have:

\- Main PC running Qwen 3.6 27B at \~30t/s (do not recommend me Qwen 3.8 27B — I know it exists, but I need time to get used to this new model)

\- Wife's PC running Qwen 3.6 35B at \~55t/s

\- Steam Deck LCD and OLED, both running Qwen 3.5 2B at \~30t/s

My questions:

How would you configure this for multi-agent use? Is there any good practical use for the Steam Decks, or is it better to drop that idea entirely?

Have I picked a good set of models, or should I consider another combination?

I'm looking into a scheme with one orchestrator (I assume the 27B model is the best option for this) and a bunch of workers for smaller atomic tasks.

Is there any practical reason for this kind of setup, or am I just spending time on a dead end?

UPD: I need for agentic coding scenarios

Hello everyone,

I'm completely new to running AI models locally and would appreciate some guidance.

my laptop specs

GPU:Nvidia RTX3050 6gb VRAM

CPU:Intel13th gen i5- 13450HX

RAM:16GB DDR5

I wanna run an AI model locally to help me with cybersecurity in general because any other public agent wont do what i ask for like any hacking question

I just got myself an M3 Ultra Mac Studio with 96Gb of RAM. I'm pretty excited to mess around with it, but I don't have a great understanding of the local LLM landscape. Every time I try to google the best models for a configuration, the source is usually months old. In the AI world that's ancient news.

I have a decent idea by just getting on X, but it's hit or miss wether or not I hear about these things. All I really know right now is that qwen 3.8 27B is all the rage, but I want more options.

How are you guys keeping up with the best local LLMs?

Hi everyone,

I want to buy my first machine to run local models, and I'm interested in running Qwen 3.8 27B. Will it run on this machine?

GMKtec EVO-X2 AI Mini PC AMD Ryzen Al Max+ 395 Up to 5.1GHz, 16C/32T

I need it for coding!

Thanks

Haven't seen posted here.

The research project matters because it proves you can pull trustworthy decisions straight out of an LLM's internal state in a single forward pass (no generation loop, no parsing, no fine-tuning) essentially turning a slow chatbot into a fast and cheap, deterministic classifier

Got Kolibri-1 to play Breakout completely on its own, no fine-tuning.

The more we explore u/Aleph__Alpha’s Kolibri the more it get's exciting and its potential.

Less talk. More Breakout is one such experiment to see how good the model is at structured output given a few constraints.

We especially optimized the inference for action probabilities: around 25 ms inference per decision.

Four moves. No generated text. One shared game.

Open weights. New possibilities.

Watch it play: https://tesseracted.com/kolibri-1-chat/gameplay/breakout/
Source: https://x.com/konarkmodi/status/2107248086880055613?s=46

I'm considering the Gigabyte AORUS RTX 5090 AI BOX (external GPU, 32GB GDDR7, connects via Thunderbolt 5/USB4) as an alternative to building a desktop PC with an internal RTX 5090, specifically for running local LLMs.

Does anyone have real-world tokens/sec numbers comparing the AI BOX vs. a desktop RTX 5090 for popular models at various quantizations?

Purely a hobby side project to see how far I can push a really small model, using (mostly) automated training pipelines

Full synthetic data: https://huggingface.co/datasets/dirac-run/ec-training-data

Models: https://huggingface.co/dirac-run/ec-1.5b-gguf and https://huggingface.co/dirac-run/ec-0.6b-gguf

Cli https://github.com/dirac-run/ec

feel free to train/use the data as you wish.

Hello. I picked up a new old stock 14" M3 Max MacBook Pro (14 core CPU / 30 core GPU / 36 GB / 1 TB) from my local market yesterday for around $2,498, and spent the night testing which local inference software is actually fastest on it for the two models I use.

Four engines, all current versions: oMLX 0.7.0, Rapid-MLX 0.15.5, Splash 1.2.1, MTPLX 2.12.2. macOS 27 Golden Gate.

Models: Qwen3.8-27B-4bit (dense) and Qwen3.6-35B-A3B-4bit (MoE).

One thing up front: it is not the exact same weight file on all four engines. Rapid and MTPLX run their own MTP-augmented 4-bit builds, Splash pairs its own DFlash2 draft, and oMLX ran the plain mlx-community 4-bit. Same base models, different finishing, but that is how each app is meant to be used.

How I tested: each engine served on localhost, temperature 0, thinking off. Sustained test: same short prose prompt, 3 runs x 256 output tokens, median. Then a prompt size sweep at about 130 / 1500 / 5500 tokens. For thermals I used a laptop stand, waited 3 minutes between every engine+model combo and 2 minutes between the two test phases, and cleared each engine's KV cache before its turn (oMLX, MTPLX and Rapid-MLX all keep caches across restarts, great for daily use, but it will fool you if you benchmark twice). I re-ran the whole thing end to end and the numbers came back within 6%.

Decode, natural prose prompt, median of 3 runs:

|engine|Qwen3.8-27B|Qwen3.6-35B-A3B|
|:-|:-|:-|
|Rapid-MLX|27.7 tok/s|110.2 tok/s|
|oMLX|17.9 tok/s|104.2 tok/s|
|Splash|31.7 tok/s|77.0 tok/s|
|MTPLX|31.5 tok/s|79.0 tok/s|

Same thing with filler prompts at longer sizes (repetitive text makes speculative decoding look better, so read this as a best case):

|engine|27B @ 1.5K|MoE @ 1.5K|MoE @ 5.5K|
|:-|:-|:-|:-|
|Rapid-MLX|31.3 tok/s|118.2 tok/s|117.9 tok/s|
|oMLX|17.9 tok/s|101.1 tok/s|95.6 tok/s|
|Splash|52.2 tok/s|238.0 tok/s|100.9 tok/s|
|MTPLX|30.2 tok/s|78.6 tok/s|69.8 tok/s|

What I take from it:

• Absolute fastest per model: the MoE goes to Rapid-MLX, dense goes to Splash (31.7 vs MTPLX 31.5, in practice a tie). oMLX is way behind on dense at 17.9 but basically level with the leaders on the MoE.
• If the margins are too small to care about, just pick by features. Splash and MTPLX are the same on dense, Rapid and oMLX are the same on the MoE. I kept Rapid-MLX because the MoE is my daily model and it is fastest there.
• The dense number makes sense: roughly 16 GB of weights per token against \~300 GB/s of memory bandwidth puts the ceiling near 18 tok/s, and oMLX sits right on it. The others pass it with speculative decoding, which is also why their numbers move with the kind of text generated. Splash on the MoE was 238 tok/s on the filler prompt at 1.5K and 101 tok/s at 5.5K, while Rapid stayed around 118 tok/s.
• First token on a 5.5K prompt: about 4-5 s on the MoE, \~34 s on the dense (prefill around 1.2-1.3k tok/s vs \~150-170 tok/s).
• It is loud under sustained inference. Fans stay up while it generates. Works on my desk, would not use it in a library.
• I also tried Qwen3.8-Flash-Next (the 125B). Not happening on 36 GB. The 4-bit weights alone are \~74-83 GB and the lightest build asks for 96 GB+, and none of these engines can stream that architecture's experts off the SSD.

Limitations: one laptop, one night, medians of 3 runs, and the different weight builds mentioned above. My prompts are simple too, no long agent sessions yet.

TL;DR: on a 36 GB M3 Max, Qwen3.6-35B-A3B does \~110 tok/s on Rapid-MLX and Qwen3.8-27B \~31.7 tok/s on Splash (MTPLX a hair behind), pick by which model you run most, and the 125B Flash-Next needs 96 GB+.

Thinking of repurposing an old X79 PC for Strata / on my old X79:

\- i7-3930K

\-56 GB DDR3 (32gb matched but I have a few 4GB sticks and 1x8GB so they wouldn't match but maybe they work.)

\- RTX 2080 Ti 11 GB + RTX 3060 Ti 8 GB

\- CachyOS headless

Would Flash-Next IQ3\_XXS work well on this? Do I need to go lower?

I was also thinking of using an M4 32 GB as a coordinator/router with GLM-4.7-Flash, plus another machine with a 9070 XT running 27B.

I tried Gemma 4 26B it 4b JANG, asked it through Hermes to stitch a story together and it failed miserably so I wouldn't make GLM do that but it was sad to see gemma fail at what I thought was it's strongest point.

Not sure if GLM + 27B + Flash-Next would be redundant.

Main use would be agentic coding, web crawling, configuring environments, the more loved tasks out there. Basically trying to reduce my dependency on Claude.

Is it even possible with the 3930K/DDR3 or mixed GPUs? ChatGPT seemed to be cautiously optimistic. If it will work. What kind of tok/s could I realistically expect and will it be better than 27B UD-IQ\_i4\_XS

I also have a GTX 1060, GTX 970 and RX 580, but I assume those are useless here.

With the recent explosion of agentic tools like Grok bot, Muse, OpenAI Dots, I've started looking into local options with self-hosted models. I tried Hermes and OpenClaw, but I wasn't too happy with the multi-device experience, and with how many pieces you need to glue together to get a usable, solid experience.

The hosted options also meant handing an agent my accounts, files and browsing, which I wasn't comfortable with. So I built Eidon: a self-hosted, all-in-one AI platform with a team of agents. It's one install via Docker, it works across your devices, and your data stays on your server.

https://eidonai.app

Agent team first

• Every Eidon starts with a Chief of Staff. Ask it for anything. It answers directly, hands the job to the right agent, or creates a new agent when nobody fits.
• Agents hand work to each other automatically (or type @ to pass a job along).
• Each agent has its own browser, conversation, files, memory and routines. There's also a folder the whole team shares.
• Agents can search and browse the web on their own, read pages in full, and cite sources.
• They run on schedules and keep every run. When one finishes, you can get notified by browser push, ntfy, Slack or webhook.
• Agents can write their own skills and use your apps through MCP.

You still have some control:

• Take over an agent's browser for a login or a tricky step. It waits, then carries on when you hand it back.
• Anything that sends on your behalf waits as a draft until you press Send.
• Commands and tools ask first: allow once, allow always, or no.
• Rewind a conversation, or fork it from any message.

The examples on the site are a travel scout, inbox triage, a research desk and a coding assistant. You can make an agent for pretty much anything: bookkeeping, a study buddy, a news digest, a meal planner.

It's also a regular ChatGPT-style app for day to day questions.

You might not always need a full team so you can just chat in a normal “ChatGPT like” interface with all the belts and whistles:

• Persistent Memory
• Folders and search
• Voice input with LLM post-processing
• Files and images
• Personas
• Temporary chats
• Share links
• Web search
• Deep research
• Code with syntax highlighting, Mermaid diagrams and math rendered inline
• Image generation
• Installs as a PWA on your phone and realtime sync across your devices (a native mobile app is coming !)

Self-Hosted

• Multi-user support, with private data per user
• Agents run in their own sandbox
• Nothing leaves your server
• Bring your own local or cloud model: OpenAI, Anthropic, OpenRouter, Ollama, LM Studio, GitHub Copilot, Gemini, DeepSeek, Mistral, Kimi, Z.ai, Minimax, Perplexity, Grok, Azure, AWS, and any compatible API
• Free, open source (AGPL-3.0), and setup is one Docker command

GitHub (setup guide, full feature list): https://github.com/Quack6765/Eidon-AI

I'd like to hear what you think ! What's missing, what breaks, and what agents you'd want to build. Issues and discussions are open on GitHub as well.

I've been working on Modly, an open-source desktop app that turns images or prompt into 3D meshes with only local models. It has a chat agent that can operate the app. In v0.4.3 I made llama.cpp the default engine and built the agent around it.

Why llama.cpp

\- I wanted direct control over how the model runs: context size, GPU offload, KV-cache quantization, flash attention.

\- Plain GGUF files. Pick from a small catalog, or drop any .gguf into the models folder and it shows up.

How it runs

\- One llama-server process per loaded model, on localhost only.

\- You can keep several models loaded at once. The default count is sized from your VRAM, and idle servers get unloaded so 3D generation has room.

\- The agent is a standard OpenAI\-style tool-calling loop against the app's own API: read mesh info, decimate, smooth, list/run/create workflows, unload models from VRAM, etc.

\- The model library shows size, quant and an estimated VRAM footprint, and grades each model on tool calling. Grades are marked as either measured with a small eval suite in the app or estimated from public benchmarks, so you know which is which.

What the video shows

Qwen 3.5 4B Q4\_K\_M on an RTX 3060 12 GB. I ask it to cut a 2.6M-triangle mesh down to 300k. It calls \decimate\_mesh\ with the right path and target and reports the result. About 9 s with the model already loaded; the first call takes \~40 s because llama-server has to start and load the weights.

Honest limitations

\- Small models sometimes misreport results. In one test the decimation stopped above the target (UV seams limit how far it can simplify), and the model made up a reason instead of just reporting the number. I'm thinking about feeding the tool output back more explicitly.

\- It's an assistant on top of the app, not a replacement for the UI. Multi-step workflow creation is noticeably less reliable at 4B than single tool calls.

\- Other backends are still optional: any OpenAI\-compatible endpoint works, including your own llama-server. Local llama.cpp is the default, and nothing leaves your machine unless you configure something else.

Question for you

Which models up to 8B have you found most reliable for tool calling on llama.cpp? Qwen 3 4B / 3.5 4B work best for me so far. GPT-OSS 20B is good but too heavy next to a 3D generation model on 12 GB. Also curious whether people would rather tune the llama-server flags themselves or keep sane defaults.

My C: drive kept filling up. Hugging Face models (32.5 GB, including old versions I'd already updated), Claude's VM bundles (7.7 GB) and pip/uv caches (8.3 GB) were a big part of it. So I built Sparewise, a free Windows app that lists every local model with its size and last use, never deletes models itself, and clears caches that rebuild by themselves, with undo for everything. No account, no telemetry. https://sparewise.app Early and solo, so honest feedback welcome. (Not code-signed yet: More info → Run anyway.)

Anyone got any hints on how to force it to write code? I have tried demanding but it just ignores. Using Pi with a couple of IQ_3 quants and a 16GB VRAM setup.

Rent their intelligence. Own your memory.

Three things make my personal AI different:

1. A clone of me that gets sharper every day, on its own. A judgment-ownership module learns how I decide from my everyday conversations. I do nothing extra. The more I talk, the closer the clone gets.

2. A private RAG that cloud AI can write into but can never read. Not a prompt rule. There is simply no path.

3. A collection of AI blind spots. Not only facts that eight leading AIs don't know, but things they don't notice. Much of it is Japan-specific common sense that any Japanese person takes for granted and the AIs miss. When I spot one, I point it out and make them check. The moment it turns out they couldn't have caught it on their own, it gets flagged and filed. AI NOBORU collects these as AI blind spots.

I'm a single father of three and a full-time stay-at-home dad. I also run my companies and do some investing. I built this alone, with no AnythingLLM, no frameworks, no existing packages. It's cloud AI models plus code I wrote myself. Diagrams and details here: https://www.aiofonesown.com/lab/ainoboru/en/

Here's how each one works, as of 4 October 2026.

1. The clone: judgment, not just memory.
A module reads my everyday conversations and records what I chose, what I turned down, and why. That goes into the RAG, and whichever model I talk to next — Claude, GPT, or Gemini — answers with it in context. The aim is an AI that can answer "what would NOBORU do here?" Every conversation today makes tomorrow's clone a little more accurate. What it can't copy is genuinely new ideas.

2. The private RAG.
Cloud models (Claude, GPT) produce parts — research summaries, findings from papers, pieces of finished work — and only from material that's safe to share. The parts move into the private RAG in one direction only. Using them happens only with local open models (Qwen3.6-35B-A3B, Gemma 4) on my own machines and NAS, through an interface no cloud model is connected to. You get the power of cloud AI without the data leaving.

3. The blind-spot collection.
Claude, GPT, Gemini, Grok, DeepSeek, Qwen, Mistral, and PLaMo remember what's in a session or in their memory. But there are things I know that none of them do, and things they all fail to notice. When I run into one, I point it out and make them research it. Only at that moment, when it becomes clear they couldn't have gotten there without looking it up or being told, does it get flagged and filed as a known blind spot. A lot of them are Japan-specific: everyday common sense that any Japanese person shares, which the AIs answer shallowly or miss entirely. The collection holds what only I and AI NOBORU know, and the eight AIs missed. AI NOBORU collects these as AI blind spots.

And how it's built: 44 parallel lanes, driven from one chat.
14 Codex lanes, 20 Claude Code lanes, 10 Gemini lanes, each able to run a different model. I talk only to Opus in the Claude Desktop chat. It splits the work across the lanes and reports back there. It works the best cloud AI models hard for very little money, instead of paying for one expensive brain.

The principle hasn't changed: models are swappable parts, memory is what you own. The difference is that "memory" now means my judgment, not just facts about me.

Where it came from. Back in June I posted here about a beginner's setup: a personal AI on a 2020 Intel iMac, built on AnythingLLM. That became a book, a Udemy course, and a template pack on Gumroad. What I run now is a different system, grown out of that one, with its own RAG and its own memory. It's a personal AI system I built entirely on my own.

A note on what this is. This system isn't for sale. There's no product, no repo, no sign-up, no waitlist, and I'm not looking for customers, investors, or collaborators. I like my life as it is and I'd like to keep it that way. This is a dated record of what one person could build in 2026.

If you're genuinely trying to think this system through, and not just passing by, I'll answer design questions as time allows. But my days go first to raising three boys and to making decisions for a mid-sized company. I don't have time to answer anything the website already covers, so please read it first, then ask: https://www.aiofonesown.com/lab/ainoboru/en/

Following on from club-5060ti and club-rdna16, I’ve put together Infermeld: a small, open-source Linux companion kit for running one GGUF across an AMD GPU and an NVIDIA GPU, powered by llama.cpp.

I’m the maintainer. This is an experimental v0.1.0 release, and I’m looking for people with other mixed GPU combinations to help reproduce the setup and find the rough edges.

The idea is practical: if you already have cards from both vendors, can you put them to work together without buying a matching pair?

What Infermeld adds

The inference engine is llama.cpp. Infermeld isn’t a new backend, and I’m not claiming to have invented mixed-GPU inference.

It packages the supporting pieces around that setup:

• Explicit AMD/Vulkan + NVIDIA/CUDA device selection and runtime preflight.
• Reproducible build instructions and inspectable launch arguments.
• A read-only thermal guard, with shutdown limited to the server process it started.
• Documentation and a results site that keep configurations, failures and limitations visible.

The release is source-only. You build the documented llama.cpp revision separately and supply your own model weights. It’s intended for people comfortable with an experimental Linux setup, not as a one-click installer.

Current tested setup

| Component | Tested configuration |
|---|---|
| AMD GPU | RX 6900 XT, 16GB |
| NVIDIA GPU | RTX 3080, 10GB |
| Model | Qwen3.6-35B-A3B, UD-Q4_K_M GGUF |
| Backends | Vulkan + CUDA |
| Split mode | Layer |
| Context reservation | 8,192 tokens |

The acceptance checks include loading and short completions with MTP off and on.

That’s a narrow result on one hardware pair, not broad compatibility testing. An 8K context reservation is not the same as testing a filled 8K prompt, and a short successful response is not a sustained performance benchmark.

Important limitations

• Sustained Q4 throughput and full-length high-context results are not yet qualified.
• Historical measurements are labelled with their original configurations. They should not be read as performance numbers for the current Q4 setup.
• There’s no promise that combining cards is faster than using one.
• Adding the advertised VRAM capacities does not guarantee that all of it is usable for the model and its runtime allocations.

I’d rather make those boundaries clear than present a successful load as a complete benchmark.

Looking for other AMD/NVIDIA combinations

Successful runs and failures are both useful. If you try it, please include:

• Both GPU models and their VRAM sizes.
• OS, driver versions and llama.cpp revision.
• Model and quantization.
• Launch settings, including the split and context reservation.
• How far it got: preflight, loading, first completion or a longer workload.

There’s a hardware/result issue form in the repository. Please sanitize paths and keep credentials and private logs out of reports.

Repository and setup instructions:
https://github.com/5p00kyy/infermeld

Results and evidence:
https://5p00kyy.github.io/infermeld/

Anyone already using an AMD/NVIDIA pair for local inference? I’d be interested in what works for you, and where this setup breaks on different hardware.

Bit of a side project I wanted to share.

The metrics are 17.5tks decode, 360tks prompt processing based testing against my normal ai usage.

I tested a couple of new things others haven’t done (at least that I’ve seen).

Setup a carousel buffer for streaming in experts for prompt processing which got my pp +30% tks.

Tried a second external ssd to get parallel reads which got my +15% on both prompt processing and decode.

Plus a long tail of small efficiency gains.

I also setup a system where by you can have a chat application make a call to the server and effectively kick out a coding run (which is kept alive until after the chat then continues). Good if you run long coding jobs , but want to chat inbetween. Probably useful for all setups where you want to save on local caching memory.

I also noticed there is still a lot of gains to be made. I make this statement as there is still a lot of essentially free time on decode where the gpu is waiting for experts to stream in. There’s also work that could be done for an optimised kernel on metal.

I also think the way things are going with Qwen (flash next being a precursor to 4), we’re gonna see a lot more efficiencies we can take advantage of like the ngram table and the cheap hybrid attention caching.

I’m really liking qwen flash next .. the coding is actually very good. I’m quite surprised in fact I’m leaving it on during the workday to do large jobs.

The chat, decode would be technically fast enough IMO but not really with qwen. The actual issue qwen spends so long thinking, so the decode hurts.

Anyone else working on this? I’d love to compare notes.

Yes I’ve heard of strata it does look pretty sic.

https://github.com/skeggsguy/Flash-next-ssd

hi guys im an enthusiast

i seen so many posts about people getting high speeds or results with this or this tool.

a lot of them seems to be real and other seems to also be scam attempts

can somebody please tell me actual legit things or stuff that makes running llms or specific llms with my GPU interesting?

its 24GB VRAM 800 GB / S bandwidth model.

thanks

Hello everybody, on my ai agent system i build a tool search on BM25. i try also use Embedding gemma but with not best result. do you have any other idea? i try also Jev but this make system more slow and GPU consume. here my repo

I'm not sure if it's the model or just the Codex's MCP itself, which was built by another smaller startup called Sky.

I want to build an agent system equivalent of an RPA for my company, and we don't want to use Codex's Computer Use MCP because of the enterprise issues. I'm thinking about designing one from scratch myself, given that the open source MCPs just don't come as close as Codex.

My setup

This is my current arrangement of my hardware since I bought the PLX switch to avoid bifurcation headaches and have everything installed. The machine has Two power supplies. Here’s the hardware specs:

CPU: AMD Ryzen 5 5600G (handles display and general system tasks)
Motherboard: MSI MPG B550 GAMING PLUS
RAM: 48GB DDR4 (3 sticks)
Storage: 4TB NVMe
GPUs: 2x AMD Instinct MI50 32GB (64GB HBM2 total) with aftermarket blower coolers
PCIe Switch: PLX8749 Expansion Card (4x SFF-8654, PCIe x16) with baseplates and ribbon cables
PSU 1: MSI MAG A850GL PCIE5 850W (system)
PSU 2: MSI MAG A1250GL PCIE5 1250W (GPUs)
2x Phanteks M25-140 Gen2 Triple Pack, 3x 140mm ARGB High Performance Cooling Fans, Daisy-chain Unified Fan Frame - all set as intake
OS: Ubuntu 22.04
Inference: llama.cpp (ROCm 6.4.3)
Frontend: OpenWebUI

If anyone has any questions, please feel free to ask.

\[EDIT\] if I can pin the reply, a better shot of the back will be uploaded

\[EDIT2\] The case is a LianLi O11 EVO RGB and fan configuration

I want to dictate more of my stuff and while google recorder is good at it, I would rather keep it from cloud.

Speech to text is actually secondary, primary usecase would be transcription.

I've been using explainroo to make short narrated explainer videos. It runs fully local: Kokoro for the voice, Whisper for word timing, headless Chrome to draw the frames, and ffmpeg to put it together. No API keys needed.

It works and I like it but it's very simple. After a few videos everything starts to look the same.

Anyone know other free, local options in this space?

Tools, pipelines, or your own setups all welcome. Thanks.

Hi everyone! I'm part of the Lokutor team. Last week we presented Oído here, and the response was amazing. We've received dozens of videos and messages from you guys saying you love it. Thank you!

Now we're back with the next part of our plan: ItoTTS, a natural-sounding, streaming TTS engine for the ESP32-S3. Two English voices, 24 kHz audio, and 4.89 MB of weights per voice. The goal: give your local LLM a voice on a $5 chip.

In our automatic naturalness evaluation, Ito beats the ESP32-compatible TTS models we compared against. Here are the UTMOS scores on eight held-out sentences:

Teacher (StyleTTS 2): 4.49

Ito: 4.46

sanoTTS amy: 3.98

sanoTTS heart-nano: 2.07

This is a small automatic evaluation, not an independent listening study or proof that everyone will prefer Ito. Listen to the samples and tell us what you think. The demo uses the host engine's output, verified bit-identical to the firmware in QEMU. We haven't measured speed on a physical board yet, and text-to-phoneme conversion currently runs on the host.

Code: https://github.com/lokutor-ai/ito
Model weights: https://huggingface.co/lokutor-ai/ito
Demo: https://lokutor-ai.github.io/ito/

The code is open source under GPLv3. The weights are free for non-commercial use under CC BY-NC-SA 4.0 plus terms, with access through Hugging Face. They aren't unrestricted open-source weights.

We chose this license because we don't want big corporations to take our work and crush us. We need to protect ourselves, but we're very open to collaborations with individuals and small companies without charging a license fee. Commercial use still needs a separate written agreement.

Send us your videos or reviews if you try it. We're around and would love to see what you build!

I’ve been experimenting with aggressive low-bit quantization of Aleph Alpha’s new Kolibri-1, a \~78B MoE model with only \~3.5B active parameters per token.

The first result is now public:

Sakura-MicroQuality Kolibri-1 — IQ2_XS

• 20.94 GiB
• 2.30 bpw
• full 384-expert Kolibri-1
• GGUF / llama.cpp
• \~59% lower KL divergence than a standard IQ2\_XS baseline
• 90.5% top-token agreement, compared with 84.6% for the standard IQ2\_XS comparison
• slightly smaller than the standard IQ2\_XS as well

The goal wasn’t simply to make the smallest possible quant.

I’m using tensor/layer sensitivity to spend bits where they appear to matter most, rather than treating every part of the model equally.

All quality measurements are made against a near-lossless Q8\_0 reference. The model itself was also requantized from Q8\_0 rather than converted directly from the \~156 GB BF16 weights, so there is a very small additional source error relative to BF16.

Main repo:

https://huggingface.co/webmp3/Sakura-MicroQuality-Kolibri-1-GGUF

As far as I can currently find, this is the first public \~2-bit GGUF for Kolibri-1. There is already a 2-bit MLX version, but I haven’t found another public Q2/IQ2 GGUF.

I also tried pruning the expert pool

Alongside the full 384-expert version, I released a separate 365E variant.

For each MoE layer, I collected actual routing statistics on a mixed calibration set containing:

• German and English text
• code
• chat-style prompts
• the model’s own thinking / generated responses

I then removed the 19 least-used routed experts per layer.

That reduces:

384 → 365 routed experts per layer

and removes:

950 experts across the model

The resulting model has approximately:

74.4B parameters instead of \~78B

The interesting part is how little those experts were actually being used on the calibration workload.

The removed experts accounted for only about 0.07% of all expert selections, with no individual layer exceeding roughly 0.23%.

Also, 375 of the 950 removed experts were never selected at all during the routing analysis.

There is:

• no retraining
• no finetuning
• no requantization of the surviving weights

The already-quantized expert tensors are sliced directly, along with the corresponding router weights and biases.

Top-6 routing remains unchanged.

The 365E IQ2 variant comes out at:

• 19.99 GiB
• 2.31 bpw
• 74.4B parameters
• 365 routed experts per layer
• 90.5% top-token agreement in my held-out measurements

365E repo:

https://huggingface.co/webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF

I’m treating this as an experiment rather than claiming those experts are universally useless — expert usage obviously depends on workload and calibration data.

But it gives us a second compression lever:

expert pruning + low-bit quantization

instead of trying to get every byte of compression from lower precision alone.

Q3 and Q4 are coming

The rest of the Sakura-MicroQuality series is currently being uploaded.

Q3 and Q4 variants should be available within the next few hours.

Once they’re online I’ll add the same comparison data so we can see where the actual quality/size sweet spot lands between:

IQ2 → Q3 → Q4

and whether the 365E pruning continues to hold up at the higher-quality quant levels.

I’d be very interested in independent tests, especially on:

Strix Halo / AMD UMA, Apple Silicon, 24–32 GB GPUs, and other memory-constrained local systems.

If anyone tests either version, especially with long-context, German, coding or agentic workloads, I’d love to see the results.

I’ve been lurking around local AI for a while and figured it was probably time to actually start talking to other people building this stuff instead of living in my own little cave 😂

About 2 years ago I started messing with local LLMs. That turned into agents, then memory, then computer use, then routing, validation, recovery etc etc and at some point the project stopped making sense to describe as “a chatbot.”

I call it Aether.

The basic idea is that the LLM should NOT be the whole system. Models are interchangeable reasoning engines sitting inside a larger architecture.

Right now the project has a few major layers.

I have an executive/reasoning layer I call the Primary Reasoning Stack (PRS) that decides what kind of problem it’s looking at and where work should go.

Under that is what I call the Mini Operating Core (MOC) which handles a lot of the ugly stuff that becomes important once you stop doing one-shot prompts: memory, context assembly, runtime state, source/truth tracking, permissions, routing, system health, recovery, etc.

I’ve also spent a stupid amount of time on persistent memory.

Not just “throw everything into a vector DB and pray.” I’ve been experimenting with structured memory, recent working memory, long-term stores, retrieval/ranking, source tracking and trying to make sure irrelevant or stale memory doesn’t get injected into an answer just because it happens to be semantically similar.

Another rabbit hole has been computer use.

I have a framework I call Hands & Eyes that I’ve been using for vision/OCR, UI understanding, locating controls, action planning, verification and retry. One lesson there was that clicking something and getting a successful return code absolutely does NOT mean the action actually happened 😂

That lesson pretty much infected the rest of the architecture.

I eventually started building governance/recovery systems around the idea that a failure shouldn’t just get patched once and forgotten.

I have something I call FailureMesh where meaningful failures get preserved, classified and turned into reusable guards/regression tests whenever possible.

Basically:

failure -> evidence -> cause -> guard -> regression

instead of

failure -> hack until it works -> forget about it -> repeat the same failure 3 months later

I’m also building a media side called VideoForge for image/video/voice/editing/rendering workflows, but that’s kind of its own monster.

Hardware-wise I’m currently developing primarily around an RTX 3090 and local models, with cloud models/tools used where they actually make sense.

Long term the architecture is intended to be heterogeneous rather than “one giant GPU runs everything.”

Something like:

fast central compute

• smaller specialized GPU nodes
• potentially large-memory inference nodes
• external models/services when they genuinely outperform local options

Then the system routes work based on what actually needs to do it.

I’m especially interested right now in talking to people who have gone deep on any of these:

• multi-agent orchestration without turning into agent spaghetti
• persistent/structured memory beyond basic vector RAG
• long-context retrieval and context assembly
• local coding agents
• vLLM / SGLang / llama.cpp
• distributed inference
• heterogeneous GPU clusters
• computer-use agents
• OCR / accessibility / UI automation
• model routing
• agent state machines / blackboard architectures
• runtime verification
• failure recovery
• MCP/tool systems
• local-first architecture in general

I’m NOT claiming I’ve solved all of this.

Some parts work well. Some parts are experimental. Some parts I’ve rebuilt 5 times because the first idea was garbage.

That’s actually part of why I’m posting.

I want to find other people who have been down these rabbit holes and compare what worked, what failed spectacularly, and what you’d do differently if you were starting again.

I’m also interested in people building systems that are bigger than “LLM + 4 agents + tools.”

Especially if you’re treating the model as one component of a larger persistent system.

I’ll probably start posting pieces of the architecture and some of the failures/lessons as I go. I’m not going to dump every internal implementation detail or proprietary part of the project, but I’m absolutely interested in exchanging ideas and technical approaches.

If you’re building something remotely similar, tell me what your architecture looks like.

I’d especially like to know:

What part became way harder than you expected once your system moved beyond a single agent?

I’m working on a local fine-tuning tool, and I’m curious where people actually lose the most time.

Was it getting the environment working, preparing the dataset, fitting everything into VRAM, or getting the exported model to behave like it did during testing?

Or did training finish successfully, but the model barely improved?

What model and GPU were you using, and what finally solved the problem or made you abandon it?

So, thanks to all who responded to my sycophancy thread. After testing things out, a clear duo of winners has emerged - GLM 5.3 Flash (which is somehow less sycophantic than full GLM 5.3 in my smoke tests) and Tencent Hy3 (surfaced via https://github.com/lechmazur/sycophancy ).

In my smoke tests Hy3 has a tighter style but tends to lose some detail (less so when given search), GLM 5.3 Flash is more exact but the style is more generic. In published benchmarks GLM 5.3 Flash is the clear winner, but we all know such benchmarks are not always a great source.

So I would very much appreciate opinions from people who tried both. My aims include agentic loops, coding, and gneeral assistant plus creative writing. Which of the two is better for eahc of these tasks, or for anything else you tried them too?

I'm an undergrad. Over the last two weeks I built a quantizer on my MacBook, using Claude as a coding assistant. The results were then reproduced on an NVIDIA A10G by M. Federico (a family member who works in ML), using separate evaluation scripts.

It is GPTQ with three additions: each group's grid is fitted to its weights instead of using min-max, a second pass re-checks every rounded weight, and the grid is refitted against the layer's input statistics. Offsets are integer zero points, so the model packs into the normal AWQ format and runs on vLLM's awq_marlin kernel.

A10G, vLLM 0.29, everything served through the int4 kernel. WikiText-2 perplexity / HumanEval pass@1:

| model | fp16 | mine, 4-bit | official Qwen AWQ 4-bit |
|:--|:--|:--|:--|
| Qwen2.5-1.5B-Instruct | 9.37 / 37.2% | 9.66 / 33.5% | 10.16 / 34.1% |
| Qwen2.5-7B-Instruct | 7.15 / 70.1% | 7.29 / 67.1% | 7.58 / 64.6% |

What this does not show:

• One run per row. The HumanEval differences between the 4-bit models are within noise (about 3.6 points).
• I calibrate on WikiText-2 train, which helps on the perplexity test. On the 1.5B model that was worth about 0.3.
• The lead shrinks as the model gets bigger.
• At 3 bits the method keeps perplexity close but loses more than half of code and math ability. I would not use those for code.

Code and all results, including what did not work: https://github.com/dfed25/mlx-gptq

7B: https://huggingface.co/dfed24/Qwen2.5-7B-Instruct-gptq-4bit-int4-awq

1.5B: https://huggingface.co/dfed24/Qwen2.5-1.5B-Instruct-gptq-4bit-int4-awq

MLX versions: https://huggingface.co/dfed24

vllm serve dfed24/Qwen2.5-7B-Instruct-gptq-4bit-int4-awq --quantization awq_marlin --dtype float16

If anyone tests it on a benchmark I haven't run, I'd like to see the numbers either way.

Stack

• Strata NVFP4 fork: github.com/sergqwer/strata-nvfp4
• Model: jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4 Hugging Face model
• NVFP4 routed experts (\~63.3 GiB), separate FP8 PLE, INT8 KV cache, MTP speculative decoding
• W4A8 prefill on Blackwell

Main engine flags:

./build/strata \
--pack packs/orca-nvfp4 \
--native models/orca-nvfp4.gguf \
--native-dense-gguf models/orca-nvfp4.gguf \
--ple-gguf models/ple-fp8.gguf \
--mtp mtp-orca/rt \
--spec 4 --spec-min-p 0.5 \
--prefill auto \
--expert-profile data/expert-profile.bin \
--expert-cache auto \
--resident-budget-gib 40 \
--max-context 200000 \
--kv int8

I serve it through Strata's OpenAI-compatible server.

Results so far:

• \~50k context: up to \~80 tok/s
• \~188k warm context: \~60–67 tok/s
• cold 189k full prompt: \~1,680 tok/s prefill, \~53 tok/s decode

Pretty impressive for a single 32GB GPU + only 64GB system RAM.Running Qwen3.8-Flash-Next NVFP4 with Strata on a single RTX PRO 4500 32GB + 64GB DDR5.StackStrata NVFP4 fork: github.com/sergqwer/strata-nvfp4

Model: jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4 Hugging Face model

NVFP4 routed experts (\~63.3 GiB), separate FP8 PLE, INT8 KV cache, MTP speculative decoding

W4A8 prefill on BlackwellMain engine flags:./build/strata \\
\--pack packs/orca-nvfp4 \\
\--native models/orca-nvfp4.gguf \\
\--native-dense-gguf models/orca-nvfp4.gguf \\
\--ple-gguf models/ple-fp8.gguf \\
\--mtp mtp-orca/rt \\
\--spec 4 --spec-min-p 0.5 \\
\--prefill auto \\
\--expert-profile data/expert-profile.bin \\
\--expert-cache auto \\
\--resident-budget-gib 40 \\
\--max-context 200000 \\
\--kv int8I serve it through Strata's OpenAI-compatible server.Results so far:\~50k context: up to \~80 tok/s

\~188k warm context: \~60–67 tok/s

cold 189k full prompt: \~1,680 tok/s prefill, \~53 tok/s decodePretty impressive for a single 32GB GPU + only 64GB system RAM.

Hey everyone, I have a 64gb halo strix setup that is headless and connected remotely to my workflow/homelab server. It’s currently running 27b swift 1.5 at q6- is it possible or even makes sense to go to qwen flash next? I also have a mini pc with 64gb of DDR5 ram that I could shift the 27b over to for long term projects or workflows that dont require speed.

Just for my understanding is Qwen image 2.1 newer than Qwen image 3.0?

Disclosure: I made this. Sharing it here because it is fully local and small, and I want feedback from people who run models on their own machines.

What it is: a 395M decision model (ModernBERT-large plus a 4 KB scoring head). You give it a state, a question and a list of options. It does one encoder pass and returns a probability for each option, or P(yes) for a yes/no question. It does not generate text.

Why it might be useful in a local stack: the small decisions an agent makes all day (which tool to call, which queue gets a ticket, does this reply answer the question) do not need a large generative model. This handles them on your own machine with no network trip.

Local numbers (our hardware, yours can differ):

• M5 Pro Mac, MLX backend: median 9.6 ms for a short decision, 1.7 GB of GPU memory.
• CPU only: about 65 ms per short decision, up to 4.5 GB of memory.
• Over the full Decision Index run on our laptop: median 25.9 ms, p95 407.8 ms.
• Weights: 1.58 GB in fp32. Context limit 8,192 tokens. It refuses longer input; it does not truncate.

Backends: PyTorch (default), MLX on Apple silicon (pip install "decision-tune\[mlx\]", Python 3.11 or newer, selected automatically) and ONNX. Torch and MLX give the same answer on 99.85% of 2,755 questions. Before each release, PyTorch, ONNX and MLX each match the recorded answer on all 50 parity rows.

Offline: after the first download it needs no internet. The package asks before it downloads and checks every file against a SHA-256 manifest.

Quality: 29.57 on Decision Index 0.2.1 (one complete run; a second seed scored 29.13). Strongest area is Tools & Automation at 46.5, up from 28.1 in our 0.9 Preview.

Limits: it only picks from the options you give it. Vague questions with no criteria give weak results, so describe your options ("Shipping: delivery, lost or damaged packages", not "shipping"). Probabilities are not calibrated. English only. Weak at knowledge, math and taste.

Try it:

\\\`
uvx decision-tune ask "Is the customer asking for a refund?" --state "The order arrived broken. I want my money back."
\\\`

or the browser app: pip install "decision-tune\[mlx\]" then decisiontune app

There is also an MCP server (decisiontune mcp) if you want your local assistant to hand routing and yes/no checks to it.

Model card: https://huggingface.co/decision-tune/decisiontune-1.0
Code: https://github.com/decision-tune/decision-tune
Site: https://decisiontune.com/?utm\_source=reddit&utm\_medium=social&utm\_…

If you test it on your own decisions, I would like to hear where it picks wrong.

Hi Everyone,

I’ve been building my own agent for a few months called Otis. After using existing tools, I found that most were either lacking in functionality or had too much going on and decided to build my own.

Otis sets up llama.cpp for you and recommends the best model for your hardware. It also integrates with existing setups for those who have tweaked and found their perfect setup (strata, ninfer etc.) and works with hosted open-weight models.

Some of my favorite Otis features are viewable artifacts, side-by-side sessions, memories, and the ability to use Otis on my laptop while the inference runs on my more powerful machine.

Also interested to hear what's the best use cases you’ve found for local models are. Personally, I found using qwen 3.8 for learning new topics quite helpful.

Website: https://triangllabs.ai/otis

Github: https://github.com/TrianglLabs/otis

Excited for everyone to try it and welcome all feedback, including what main features are missing from Otis for you. If it's useful, a star helps others find it.

[Project] OntoPrune: 6.7x faster TTFT on local CPU with Ollama by neuro-symbolic context pruning (83-86% token savings + 0% hallucinations) Repository: https://github.com/vigmarcarlo/OntoPrune License: MIT Hey r/LocalLLaMA! If you run small coding models (Qwen 2.5 Coder 1.5B/3B, Gemma 2B, DeepSeek Coder) on commodity hardware (like a CPU-only laptop or mini PC with Ollama), you know the prompt evaluation bottleneck. Feeding a 300-line service file into a 3B model on CPU took 22.4 seconds just to generate the first token (TTFT). Plus, smaller models frequently invent bogus methods when given too much noisy context. I built OntoPrune to solve this. It's a lightweight, 100% offline Python middleware that acts as a symbolic context compiler: # What it does: 1. Translates source code into an in-memory knowledge graph using an internal ontology. 2. Extracts the exact 1-hop closure of the function you're editing via SPARQL (only the classes, functions, and interfaces it actually interacts with). 3. Renders the pruned graph back into clean, typed Python stubs (\~400 tokens instead of 2,400+). 4. Verifies the model's generated code against the contract AST to catch any API hallucinations. # Benchmark on local CPU (12 cores, Ollama streaming): Model: qwen2.5-coder:3b Input tokens: 2,390 -> 406 tokens (-83.0%) TTFT (Time to First Token): 22.4s -> 3.3s (6.7x faster, saving 19.1 seconds!) Total generation time: 59.9s -> 16.5s (-72.5%) Hallucinations: Full file context hallucinated 1 non-existent method call; OntoPrune had 0 invalid calls. CPU Overhead of OntoPrune: AST parsing + RDF graph generation + SPARQL query takes 9.9 ms total. # Also tested on Gemini 3.8 Flash (Cloud): 2,815 tokens -> 393 tokens (-86.0% cost reduction). # Features: Zero RDF exposure: You and your LLM only interact with regular Python signatures and stubs. Model Context Protocol (MCP): Comes with ontoprune-mcp so you can use it in Cursor, Claude Desktop, Antigravity, or any agent. Multi-module resolution: Follows project imports across files without choking on circular dependencies. * Contract verification: Deterministically flags hallucinated APIs in CI/CD or CLI pipes. # How to use: pip install ontoprune # CLI pipe directly into Ollama: ontoprune translate services/order_service.py procesar_orden --format stubs | ollama run qwen2.5-coder:3b # Run benchmark on your machine: python -m ontoprune.benchmark --file fixtures/sample_service.py --func procesar_orden --backend ollama --model qwen2.5-coder:3b Paper and reproducible code are all open-source on GitHub: https://github.com/vigmarcarlo/OntoPrune Feedback, PRs, and benchmark runs on different hardware are super welcome!

I run Qwen 3.8 locally and got tired of multi agent setups where you start a script and stare at logs. I wanted to actually see them. So in this thing every agent has a body in a 3D world. They sit at desks, walk to a meeting room when someone calls a meeting, talk out loud to whoever is nearby, pick stuff up and hand it over. You can see who's thinking, who's using a tool. It's not only an office. You can simulate other scenarios as well like: \- a software team that plans tasks on a board, writes code and reviews each other \- a town square simulation (cops, a barista, a chef, a journalist) where you just watch what happens \- tutors that teach you with animations and a whiteboard, and you can interrupt them by talking (might have bugs as of now) It has a sandboxed computer use built-in which is optional. There is also a supervisor agent that helps you design organizations and also has ability to build 3d assets from primitives and handing them to an organization and agents can even ask for things from that agent. Works with local models and few other providers (still working to add more) The motivation of building it was to see agent swarms in action with full transparency. It's still early and has bugs and I have used different models to build it iteratively. Repo: https://github.com/adityaagarw/Pantheon

My machine isnt that terrible.. but nowhere near what some people run as an aI workstation. I am wondering if i can combine the 2 GPUs. I got a cheap Blackwell 4000 (little bit under MSRP) 24 GB and have an old 3060 12GB .. I was gonna try to get the best model loaded, mainly coding tasks than anything else and work with it relatively safely with a good buffer. any recommendations ? and will tensor split work on this combo?

Hey everyone! I’m a VFX artist looking for a local LLM setup mainly for Python development and connecting to Blender and Houdini through MCP to help create scenes and tools. This would be for interactive coding and agent workflows, not model training.

I’m considering 2× NVIDIA DGX Spark or an Apple M5 Ultra with 256GB unified memory, but I’m open to other recommendations.

For this use case, which setup would offer the best balance of model quality, context capacity, and responsiveness?

Would love to hear from anyone running similar workflows! Thankss!

Short version from pre-registered experiments on real repo commits (Haiku as the cheap agent, Sonnet as the strong one). Every protocol was committed to git before its run, and later experiments used repos the designs had never seen. https://preview.redd.it/6rasddvfmpth1.png?width=1991&format=png&auto=… What worked: \- A stronger model that only speaks up when the agent repeats mistakes: +7 successes in 63, \~1.3x the cost (an always-on advisor got +8 but cost 3.5x). \- Running the agent's change and reporting facts ("if this line became \pass\, all tests would still pass") beats giving advice: 35/42 vs 32/42, formatting regressions 10 -> 0. \- Your preferences, captured in your own words, carried into every later task (15/15 vs 0/15). Just restating them in the prompt took compliance from 40% to 90%. What didn't: \- Memory of code knowledge, generic checklists, rules learned from git history, routing between models, and clarifying questions. \- For a strong model, none of it raised success (45/45 with or without). Cheapest per solved task: Haiku + "conscience" \~$1.22, Sonnet alone \~$1.41. Everything is public: paper, protocols, failures and the tool (source-available, non-commercial licence; works with Claude Code, Codex and OMP). Repo: https://github.com/abdullahbalabel/mihad Paper: https://github.com/abdullahbalabel/mihad/blob/main/paper/MIHAD\_Research\_Paper\_EN\_v2.7.md Happy to answer questions, and criticism of the method is very welcome.

This is something very annoying that I've been running into. If it attempts to do too many tool calls involving searching the documents in my project, it will display this in its thinking: >Used tool: Searched documents for "X" > >Used tool: Searched documents for "Y" > >You have already searched the knowledge base several times this turn. Do not search again. Answer the question using the passages already retrieved above; if they do not contain the answer, say so plainly. I've tried playing around with the tools settings, but to no avail. I've had no luck with searching online, either. Does anyone know of any way to disable this?

If you run local coding models (Qwen 2.5/3.8 Coder 14B/27B/32B, DeepSeek, or Llama 3 via Ollama, llama.cpp, or vLLM), you already know the two fatal bottlenecks of agentic coding on local hardware: 1. The KV Cache & TTFT Penalty: Cloud users throw 50,000 tokens of chat history at Claude Opus without thinking. On a local 24GB or 32GB rig, prefilling 30K tokens of noisy conversation history drags Time to First Token (TTFT) through the floor, eats up precious VRAM that should belong to your context window, and triggers the "lost in the middle" attention collapse. 2. Amnesia Across Sessions: Local models are stateless. When you clear the context window to restore inference speed, the model forgets your project architecture, file relationships, and error history. You end up copy-pasting your constraints into every new prompt. For the past 18 months across 1,900+ real-world sessions, I’ve been running and refining an alternative: Project Athena—a local-first memory, reasoning, and governance harness designed to give local LLMs permanent, compounding memory without blowing up your token budget or relying on hosted SaaS databases. I just open-sourced the v9.9.9 kernel under the MIT license. Here is the exact architectural split that keeps local models grounded. # 1. The Core Rule: State on Disk, Not in the Prompt Most agent setups treat the LLM's context window as the hard drive. That is an architectural mistake. The context window is volatile RAM. Durable state belongs on your NVMe SSD as plain, human-readable, git-versioned Markdown files: \[ Your Local Machine: Plain Git-Versioned Markdown \] ├── .context/CANONICAL.md <-- Immutable architectural rules & API contracts ├── .context/memory\_bank/ <-- activeContext.md & session checkpoints ├── .agent/workflows/ <-- Deterministic slash commands (/start, /end, /plan) ├── .agent/skills/ <-- Domain capabilities loaded strictly on-demand └── .agent/scripts/ <-- Verification test runners & linter hooks Surgical Boot (<2K tokens): Instead of dumping megabytes of chat logs into the model, /start loads only the active checkpoint block from activeContext.md and top-tier constraints from CANONICAL.md. Over 90% of your model's context window and KV cache remains completely free for actual code diffs and reasoning tokens. Session Lifecycle (/start and /end): At session close, an automated distillation script (/end) audits git diffs, extracts learnings, prunes transient noise, and writes an atomic checkpoint back to disk. Session 1,900 boots faster and cleaner than Session 10. 100% Model Agnostic: The model is just whoever is on shift today. Run Qwen 2.5 Coder locally for fast terminal diffs; swap to DeepSeek, Llama, or an external API tomorrow. Your project rules, architecture contracts, and past bug logs never disappear. # 2. Mechanical Guardrails (Crucial for Local Weights) Small and mid-sized local models (8B–32B) are prone to sycophancy: they eagerly declare "I have refactored the module and verified all tests pass" while silently breaking dependencies. Athena enforces deterministic mechanical verification outside the model's weights: Red Run or It Didn't Happen: Any agent claiming to fix a test, gate, or bug must show the verification script failing on the pre-fix state, then passing on the fixed state. If it cannot produce the red run, it found a blind spot, not a fix. Deterministic Tool Calling: Integrates natively via local Model Context Protocol (MCP) or standard CLI scripts (smart\_search, context\_gate, quicksave). No Hosted Cloud Databases: No Pinecone, no cloud vector stores, no external telemetry. Embeddings and hybrid search run locally using plain SQLite and BM25. # 3. Real Hardware & Performance Observations Tested Hardware: Apple Silicon (M2/M3 MacBooks and Mac Studios) and local NVIDIA setups (RTX 3090 / 4090 / 5090). Inference Impact: By replacing multi-turn conversational bloat with deterministic file write-backs, local prefill latency drops from 15–30s down to sub-second responses. * Zero Lock-In: Everything is plain Markdown and Python. If you delete the repo, your notes and code are still just standard text files on your machine. # Try It (100% Free & Open Source) Zero subscriptions. Zero data leaving your machine. Works with Ollama, llama.cpp, vLLM, Claude Code, Cursor, Antigravity, and terminal CLI workflows. git clone https://github.com/winstonkoh87/Athena-Public.git cd Athena-Public pip install -e . athena init . GitHub Repository: winstonkoh87/Athena-Public License: MIT Curious how others running local coding agents on Ollama/llama.cpp are managing persistent cross-session context without degrading TTFT or blowing out VRAM? Happy to discuss the trade-offs and benchmark numbers in the comments!

I liked claude i did my project i was unable to do in 6 month with gemini in 1 hr but I want to buy sub 6 mon later when this level is base line and cheap. I made a markdown editor i will not publish it as made better one with areana ai and it uses vello and parley. Memory usage of 100mb. Near 0% cpu usage Though lot of things to be done like pakaging for all distros and windows and android. Performance for very very long docs still a little less. \----- Important ----- I had asked for refund and customer service said done but no update from 5 days. No email from google play, claude not cancrlled on google play, no confimation and ofcourse no refund done till now. Only that my claude sub is not working now. I have attached screenshot that shows conversation ID for reference. Please claude process the refund. Someone if can please help. I did twitter but that did not help at all.

I'd give Pi a coding task, switch over to a game, a video, or some other work, and lose track of it. Once I was doing something else, there wasn't a noticeable event to pull me back when the agent finished.

A task might take five minutes, but I might not remember to return for twenty. Pi wasn't taking twenty minutes. It had been done for fifteen, waiting for me while I was still playing. That was the time I wanted to cut down, rather than the time the agent actually spent working.

I didn't need a reminder that AI was running in the background. I needed something to interrupt me when there was a reason to come back: the task had finished, something had failed, or Pi needed an answer from me.

So I built pi-knock, my own open-source extension for the Pi coding agent. It sends notifications to your phone through Pushover or ntfy, with webhook support for other setups. One detail I cared about: completion alerts wait until Pi has finished its automatic retries and queued work. I don't want to stop what I'm doing, return to the terminal, and discover it's still going.

The motivation is pretty simple. If I'm halfway through a game, a message sitting in the terminal isn't going to get my attention. A notification on my phone can.

The code is here: https://github.com/Asigers/pi-knock

How do you handle the handoff back from an agent when you’ve switched to something else?

I’ve been building a small local AI project called Neco around Ollama and Open WebUI. The idea started from something pretty simple: most local LLMs still feel like assistants you open, ask something, then close. I wanted to see what happens if the AI instead feels more like a persistent presence on the computer it runs on. Neco has some awareness of the host machine through a read-only system layer, so she can know things like uptime, memory usage, system load, battery state and temperatures. She also runs outside the normal chat session through a small background daemon. Every so often it generates an idle thought, meaning the system can produce something on its own even when I’m not actively talking to it. That combination has been the interesting part for me. It starts to feel less like “a chatbot connected to some tools” and more like an AI that has a small window into the machine it inhabits and continues existing between conversations. Everything is still local, and the model itself doesn’t get unrestricted shell access or control over the host. The next part I’m working on is memory. I want previous conversations, events and unresolved thoughts to persist over time without just throwing the entire chat history back into the context window. I’m experimenting with episodic memory, selective retrieval and a small evolving state so that its behavior can develop some continuity over weeks or months. It’s still very much an experiment, but I’m curious where the line is between a normal local assistant and something that actually feels resident on a machine. I’d be interested in hearing from anyone who has experimented with persistent memory, autonomous/idle behavior or giving local models awareness of their own environment. Repo: https://github.com/proto6699/echo-local-ai

I’ve been building \*\*mova\*\* — \*context sovereignty before inference\*: you decide what context may reach the AI, and mova leaves evidence of that decision. It’s an open-source Go binary that runs before the LLM call: \Focus (AST) → PII masking → token budget → egress gate (dry-run) → LLM → evidence\ It does not use an LLM to estimate or audit the context, requires no API key for the governance step, and it’s not a gateway or RAG tool. \*\*Why I’m posting this\*\* I kept running into two things when working with coding agents: \* large amounts of context being sent when only a small part of the repository was relevant; \* not having a clear way to see exactly what context was selected, filtered, or blocked before inference. I’m trying to figure out whether this is a problem other developers actually care about, or just something I happen to care about. \*\*A reproducible example\*\* The repository contains fictional data and a Linux amd64 build. \\\bash mova run --count 02-pii-compliance-governance \# 7,153 tokens \\\ In this example: \* Before governance: 20,014 tokens \* After governance: 7,153 tokens (\*\*−64%\*\*) \* AST focus alone: 20,101 → 5,325 tokens \* 171 of 1,694 PII-candidate tokens were pseudonymized \* A smaller fictional repository: 1,965 → 779 tokens (\*\*−60%\*\*) The cost figures shown by the tool are theoretical input-token estimates, not actual API spending. \*\*Limitations\*\* \* The context control applies to context that passes through mova (CLI/chat/MCP/HTTP). \* PII masking is heuristic; I have not measured precision/recall yet. \* mova cannot see context sent directly by an IDE outside its control. \* macOS/Windows/arm64 builds are cross-compiled but not yet validated by me on those target machines. \*\*What I’d really like to know\*\* 1. Is controlling or auditing context a real problem for you when using coding agents? 2. How do you control what your agent sends to an LLM today? 3. Would you deliberately send less context for the same task? What would you filter or check first? 4. Do you care about having evidence of what the model actually received? 5. If you saw a tool like this, would you use it, ignore it, or consider it unnecessary? If you want to see more details, the repository contains the implementation and reproducible examples: github.com/m1guel1982/mova-context Blunt feedback is welcome, including: “This solves a problem I don't have.” That’s actually useful feedback for me.

How's it going everyone. So, I made... basically Jupyter notebook on steroids, I think? It was able to give ChatGPT in chat mode a programmable surface and basically a moddable lab. I've been using it the past few days to test weird ideas in real time during voice conversations with Chat when I go outside to smoke a cig or something, or I'm away from the house and I get a good idea. It works as a plugin (there's a zip with a Chat and Claude plugin there). There might still be some friction in the setup because I haven't submitted this for the plugin marketplace yet, but Codex handled it for me pretty easily and we did it with a tunnel, so it's hot-reloadable. It's ready for real work. It's got a Rust skeleton, Python glue, and Julia gives it a fully programmable persistent-state lab and a working memory, more or less. So far it's saved me a ton of tokens being able to test an idea and build it in chat mode and just branching into work mode and being able to just pull whatever prototype from the space. It turns chat mode into basically diet work mode, and there's still plenty of things you'd rather be in work mode in, but this also can be used in basically any harness too. ChatGPT is just where I've tested it the most so far.

I've taken security for this thing rather seriously though. It's extremely programmable and the sandbox walls are thick. The Julia runtime and compiler are moddable for optimization across the entire tool, and if you're not a Julia enjoyer like I am, there's also an IPython kernel in there. The one from Prime-Agent. But it can be a plugin for chat mode ChatGPT, and I've also been using it since Claude Mods dropped for that harness. Been working great in both environments so far

Here's the repo: https://github.com/latentcollapse/Palette.jl

I've been working on Agent Brain Hub, an open-source "brain" that several agents share. What one agent learns about a user, the others can recall, with permissions so private things stay private. GIF above: the repair agent hears "my car is in the shop for 3 days", and later the travel agent offers a rental car at the destination without being told again. Why it works with small local models: the brain does the heavy lifting itself (fact extraction, retrieval, ranking, permissions), so the LLM mostly turns a prepared context into a reply. I tested it end to end with Qwen3-4B-Instruct-2507-FP8 on vLLM. With no LLM at all it still runs, with rules and templates. Local setup: git clone https://github.com/leluong141996-dev/Agent-Brain-Hub cd Agent-Brain-Hub docker compose --profile vllm up -d # hub + Qwen3-4B on your GPU Ollama and LM Studio work too: pick them in Settings, fetch the model list, test, apply. No restart. A few things I learned along the way: - vLLM 0.10.2 crashed with "CUDA illegal memory access" when a greedy (temperature 0) JSON-mode request was batched together with sampled requests. Using temperature 0.1 for the JSON calls made it go away. - Servers disagree on parameters (max_tokens vs max_completion_tokens, temperature, json mode, chat_template_kwargs). Instead of a config matrix, the client reads the 400/422 error, drops or renames the parameter and retries, and remembers it for that server. - Embeddings are local feature hashing (256 dims, with character bigrams for Japanese) so nothing leaves the machine. It's crude but fine for a demo; a real embedding model is the obvious upgrade. Storage is SQLite. With 20k episodes it reloads in about 0.2 s, and a turn's write is about 40 ms. UI in English, Vietnamese and Japanese. Repo: https://github.com/leluong141996-dev/Agent-Brain-Hub Question for you: which small model do you use for structured fact extraction? I'd like to move more of the extraction from rules to the LLM without losing reliability on 4B-class models.

This repo is a technical proof of concept showing an LLM running client-side in a browser using Llama.cpp compiled to WASM with WebGPU.

A free learning tool. it a offline app for learning how LLMs work under the hood. Everything runs on your machine: a 1.37M-param transformer powers the attention lab, a tiny character-level model trains live as you move sliders, and the optional guide runs on llama.cpp with a small Qwen model. no account, no telemetry. \*\*A bit of insight\*\* \- 51 labs in 6 groups, from the basics (what a neuron is, gradient descent) through attention, RAG, agents, fine-tuning, quantization, serving and more \- Each lab has a short lesson beside it, readable in Plain or Standard mode \- Some of it actually runs rather than just animating: \- the attention lab runs a small trained transformer (1.37M params) inside the app \- the training labs train a tiny character-level model live as you move the sliders These are teaching-sized small models, so some results won't match what you'd see at scale. \*\*Privacy and setup\*\* \- Works offline: no account, no telemetry \- An optional guide you can ask about the lab you're on, running locally (llama.cpp + a small Qwen model) or with your own API key \- MIT licensed \*\*How it was made\*\* I chose the topics, the structure and the grouping. I used Claude and GPT Astra to help write the lesson text, and Grok as a second pass on references. There will be mistakes, so if you spot one, please tell me or open a PR. \*\*You can contribute\*\* If you teach this or work in a specialized area of AI, you can help expand it, a new interactive lab, a better visualization, or a tweak that makes the cause and effect in an existing lab clearer. I'd also like to hear which labs are confusing and what's missing. GitHub: https://github.com/Fazmin/AILearningGuide

TLDR: I built this open source app that lets local models monitor your screen and send you notifications! It now installs models on your browser, which makes local AI accessible to everybody! Without any install :DD

Hey r/LocalLLaMA!

I'm back with some huge Observer updates c: first of all Thank You so much for all of your support and feedback, i've been working hard to make the app as easy to use as possible!

What's New?

You can now get to a local LLM monitoring your screen by just typing

"send me a telegram when my steam game finishes downloading, use a local model"

... and the Observer agent downloads the model in your web browser and starts monitoring your steam game. In just 10 seconds, suuuuper easy :))

What's the best way of running LLMs? / Platform caveats

• The WebApp uses transformers.js which doesn't work on Linux or older PCs :((( But running Qwen3.5-0.8b smoothly on a browser, feels illegal :p
• The desktop app uses llama.cpp on Rust so you get the full power of your metal, and it's much more stable.
• You can obviously set your OpenAI compatible endpoint as well and just use that.

Help me make local LLMs useful for everyone!

• WebApp (no install!): app.observer-ai.com
• This is the projects Github (completely open source)
• And the discord: https://discord.gg/wnBb7ZQDUC

If you have any questions i'll be hanging out here for a while!

Roy

Busco a personas para poder hacer un modelo de inteligencia artificial desde cero

Many model launches follow the same arc: amazement in week one, "it's been nerfed" a month (or week) later. I'm currently experiencing the same thing with Opus. But I feel I not only experience these things with AI: other stuff also gets harder after the first week sometimes. Is it just regression to the mean? Are we comparing launch-week highlights with everyday output, and getting disappointed that it's not so good as that one amazing new thing we did and got me on a high? Is it loss aversion strengthening that? The gains we start to expect, the losses we are hit by? Or do we start with our best use cases and simply run out of them? And then if feels like the model is underperforming, while it might be our part? Don't we try and thinker as hard as we did in the first week? Even with Opus 5.5, it took time and iteration to get certain things right? Or are we so expecting and used to constant progress, that even a temporary plateau (the same model) on a trajectory still rising across releases feels like a regression? I see all the incentives and pressures there are for companies to reduce performance. I'm sure they do that for some part in some cases. I just wondering if we could seperate the two. Could we compare how this feels on commercial APIs and chatbots? Could we do that with blind A/B testing? Like an Arena?

Bonjour la commu Qui aurait des photos spéciales de "Kalyacosplayeuselyon" ? S'il vous plait ? Merci

Hi all! I was just wondering what you guys use to uncensor AI models. I know of heretic and downloading pre-uncensored models, but wanted to see if there is more I could do.

Thank you!

I was a little obsessed about the size of my context, mainly because I use a local LLM with little context). So I started looking in the opencode codebase to understand how the context was builded. After learn a lot, I'm really impressed how context of opencode can be customized. Before, I thought that the context of opencode was bloated and closed to changes, since I only see people praise pi about it. With a custom primary agent we can disable almost every thing in the context (besides the environment message).

The context is: environmentMessage + agent prompt + agent.md instructions + skills descriptions + tools descriptions.

For a test I created a blank agent with minimal agent prompt, without agent.md instructions and without tools, it resulted in a start context size of 220 tokens. I never thought that opencode was able to do this. I have created some agents to test the impact of the tools. In this plugin I even created a bash agent that only has a bash tool, similar to the mini-swe-agent, which reduced the context size from a base of \~10.3k to \~1.3k tokens. I created a pi-like agent too, it have only some tools similar to pi, it reduce the context to \~5.5k tokens.

Analyzing the context builded I found some overlap instructions and out of scope instructions in the tool - IMO. So I removed theses.

things like: "Use gh for GitHub tasks, including PRs, issues, checks, and releases; return the PR URL when done." from bash/shell tool description

The repo: https://github.com/LaercioSantana/opencode-leaner

install: {"plugin": \["opencode-leaner"\]}

Classic setup, built by the book: one call writes a contract, one writes tests, one writes the code, a script runs the tests against the code, a repair step patches whatever fails. Four local GGUF models (qwen3-1.7b, qwen2.5-coder-1.5b, llama-3.2-1b, smollm2-360m), 6 coding tasks, T=0.3, 785 calls on a GTX 1660 SUPER 6GB.

Then the boring check nobody does: fed every generated test suite a known-correct reference solution. 129 of 168 rejected it. 77%.

The tests weren't lazy either. Average mutation score 0.965, they caught almost every mechanically broken version of the code. Of the suites with a perfect 1.0, 81% still failed the correct answer. Thorough, confident, testing the wrong spec. Wrong, see the edit at the bottom.

| model | correct code rejected by its own tests |
|---|---|
| llama-3.2-1b | 0.92-1.0 |
| qwen2.5-coder-1.5b | 0.71-0.78 |
| qwen3-1.7b | 0.47-0.65 |
| smollm2-360m | 1.0 |

Favorite case: count vowels. The code forgot uppercase. The tests checked "AaEeIiOoUu" and expected 5. Correct is 10, the buggy code returns 5. Tests and bug shared the same misunderstanding, the check said PASS, repair never ran. Only a hand-written test with "HELLO" caught it.

Repair: 67 attempts went to repair, and 50 of them were already-correct code the tests had rejected. Of 17 real bugs it fixed 1. Never broke working code, credit where due.

And the code itself wasn't the weak part. A single sample already solved 0.833 of tasks, and plain resampling got 0.958 at 8 samples (counted solved if any sample passes the reference tests, so it's pass@8 and still needs a judge in real life). At a similar token budget, 4 plain samples matched the pipeline without repair, on fewer tokens. These small models write correct code far more often than correct tests.

Caveats: 6 tasks, models from 360M to 1.7B, one temperature. Bigger models write better tests, how much better this doesn't say.

What I do since: the check that decides comes from the spec or from examples a human wrote. A model can propose tests, it doesn't get to be the judge.

Report (English version), harness and metrics, my repo: https://github.com/Deadatreides/LLM-MEASUREMENTS/blob/main/experiments/experi…

Anyone running local coding agents with self-written tests as the gate? Ever fed them a known-good answer?

(not a native speaker, an LLM helped with the English)

Edit: u/RobWattx was right about the mutation score, I checked the saved runs. The 129 suites that rejected the reference: 59 had wrong asserts, 39 had syntax errors, 29 had no test functions at all, 2 crashed. Broken suites fail every mutant too, so they get a perfect mutation score for free (126 of 129). Suites that accepted the reference: mean mutation score 0.888. So the honest numbers: 42% of the suites did not run at all, and of the suites that did run, 60% rejected the correct solution. The struck paragraph above was wrong.