I've been playing around with Ling 3.0 Tiny, which is an 8 billion parameter model (MoE, 1B active). And I've had a lot of poignant thoughts as a result. Just for fun, I got it running with llama.cpp on an old laptop. This is a laptop from 2017 with a 7th gen i5 and 8 gigs of RAM, like barely even usable for modern tasks. No VRAM, no GPU. Well, I got Pi running on it and asked it to make a script to scan the local network for all available models on llama.cpp servers. It started chugging along at about 10 tokens per second. And 20 minutes later, it was done.

It had several back and forth turns with writing code, running it, getting feedback and iterating.

It's a simple task, yes. But it's a task that would have taken me an hour or two to do in 2020.

It's just incredible that such a potato hardware is actually accomplishing something useful on a reasonable timeline. One billion active parameters is so small that an old CPU can run at 10 tokens a second with basically no optimization effort. I.e. I just built llama.cpp and ran the first Q6 quant I found.

I guess my point is, do you remember that feeling a year or two ago when you looked down at your expensive GPU rig and thought, wow, the computer writes the code itself now? It actually feels like something, like it's intelligent somehow.

Well, now that's starting to happen for every potato casual computing device that's been made since 2015.

Obviously more expensive rigs will be always be much more power efficient and cost efficient and fast at producing tokens. So it may never be practical to actually use old potato hardware.

But maybe it will make sense. There are all kinds of things that a mildly intelligent computer could do in the background. So it may be a new beginning for edge intelligence. No new compute required. Just everything that already exists can suddenly start doing intelligent tasks. Not sure.

I guess I'm just saying I'm amazed I've had that weird sensation when looking at my GPUs and feeling like there's something more than just bits in there, but for an old CPU. (no I don't think it's conscious. Not talking about that.)

Enjoy! Fucking loving it.

I’ve been experimenting with whether Qwen3.8-Flash-Next’s pretrained PLE n-gram memory can improve a much smaller Qwen3.5-0.8B model.

I trained the 0.8B setup with limited resources, mostly using free Kaggle notebook GPUs.

The setup keeps both the Qwen3.5-0.8B backbone and the roughly 51B-parameter PLE memory frozen. A small R=1 reader is trained at decoder layers 3 and 9, with a token-dependent linear gate controlling the later injection.

There is no backbone fine-tuning.

The current balanced checkpoint uses a reader trained for 15M tokens and a separately calibrated dynamic gate. On the frozen full-validation set:

Qwen3.5-0.8B stock

• NLL: 2.905585
• PPL: 18.2759

Qwengram-0.8B

• NLL: 2.853786
• PPL: 17.3534

Perplexity reduction: 5.05%

This is a language-model validation result, not a claim of 5% higher benchmark accuracy.

A few findings shaped the final design:

• The real pretrained PLE outperformed both random-memory and permuted-memory controls.
• Reader loss kept improving well beyond 5M training tokens. The 20M reader improved aggregate LM loss further, but regressed on math, so 15M remains the balanced checkpoint.
• Strong fixed late-layer memory injection hurt LAMBADA. Dynamic token-level arbitration recovered much of that tradeoff.
• The gate is genuinely dynamic: its memory strength varies substantially across tokens rather than behaving like a learned constant.
• With the exact same memory budget, learned token placement beat shuffled placement. Routing memory toward high-uncertainty positions recovered part of the advantage, but still did not match the learned gate.
• A warm-started R=4 reader produced a small aggregate LM-loss improvement, but introduced code and math regressions. I therefore kept R=1 as the balanced architecture.

I also implemented the inference path in llama.cpp.

The public artifacts are:

Model / GGUFs
https://huggingface.co/Ninnix96/Qwengram-0.8B

Training, controls, and evaluation
https://github.com/Ninnix/qwen-ple-transfer

Modified llama.cpp runtime
https://github.com/Ninnix/llama.cpp-qwengram

Update! 2b released!
https://huggingface.co/Ninnix96/Qwengram-2B
Same recipe, and it works: 3.7–4% lower perplexity. The gains seem to get smaller as the backbone gets larger. It also works with my llama.cpp fork.

The GGUF contains the Qwen3.5 backbone plus the trained reader and arbitration tensors. The large PLE remains an external quantized sidecar, rather than being packed into the model GGUF.

I also tested quantization retention on a separate fixed WikiText-2 GGUF runtime test:

• Q8\_0 retains 99.1% of the BF16 reader NLL gain

This is a separate runtime measurement, not the frozen Kaggle validation benchmark above.

I’d welcome attempts to reproduce or improve the reader, PLE caching, routing, or runtime.

Next I’d like to try larger Qwen backbones, particularly the 35B-A3B MoE. Experiments at that scale require substantially more compute than free Kaggle notebooks can provide, but the 0.8B study gives a much clearer recipe for reader scaling and dynamic memory arbitration.

Disclosure: I’m the author of Qwengram and the linked repositories. English is not my first language, so I used AI to help proofread grammar and improve phrasing in this post. The experiment itself was also developed with the assistance of coding agents, primarily ChatGPT Sol, for implementation, debugging, experiment orchestration, and analysis support. I designed the experiments, made the research decisions, reviewed the results, and am responsible for the final conclusions.

Today I present a fine piece of engineering, carefully assembled inside a custom chipboard chassis: the 2400cc Inference Racer, a.k.a. my winter heater.

Power comes from two second-hand AORUS RTX 3090 XTREME WATERFORCE cards. One glows a beautiful teal, the other red. I have no idea why, nor how to change it, so apparently this is now the official color scheme.

The whole thing is managed by an independently powered Lenovo ThinkPad motherboard with a Ryzen 7 7840U and 64 GB RAM. No battery, screen, keyboard, case, or other unnecessary luxuries attached. The naked motherboard is cooled by a custom aluminium water block, way too much Arctic cooling paste, and a sophisticated mounting mechanism known in the industry as a clamp.

Cooling is provided by a €24 VW Golf radiator, connected through a carefully curated collection of vaguely compatible hoses, fittings, adapters, and optimism. The loop holds around 2.4 litres of coolant, hence the 2400cc displacement. Current reliability is excellent: it leaks less than 100 ml/day, especially as long as it doesn't get too warm.

PCIe topology is equally sensible. One GPU is connected through the ThinkPad's WWAN slot at PCIe Gen4 x1, while the second uses an SSD slot at Gen2 x4. The BIOS had to be patched to remove the hardware whitelist, modify the PCIe power-up sequence, and disable PCIe power-saving states. The SSD slot is technically capable of Gen4 x4, but "technically capable" and "stable" turned out to be different concepts.

The two 3090s are connected through NVLink, which fortunately means the questionable host PCIe arrangement matters much less once inference is running.

It currently runs Ubuntu and serves Qwen3.8-27B through vLLM, quietly and at surprisingly decent speeds. I'm still tuning the setup for performance.

And it can also boot completely without the 3090s. In that configuration it becomes a low-idle-power server and can run smaller models on the 7840U using its 64 GB of shared system RAM, Vulkan, and llama.cpp, ideal for our resident Hermes bot named Hoot.

Still not managed to enable hot swap though.

Peak home inference engineering.

UPDATE: At 40k prompt depth

Prompt processing: \~1,420 tok/s

Decode: \~87 tok/s

VLLM: Qwen3.8-27B-W4A16-AutoRound

Mica v0.1 4B playing a real Minecraft 1.20.4 server. Video attached.

How it works

\- Each step the bot's live game state (inventory, nearby blocks, entities, last result) is written out as text.

\- Mica scores the candidate commands and picks the next one. It never generates text. It reads the probabilities of the answer label tokens, so output tokens are 0.

\- The chosen command is executed in the game with Mindcraft's skill library (Mineflayer bot).

Run

\- 23 decisions from an empty inventory to an iron pickaxe: logs, planks, crafting table, wooden pickaxe, stone, stone pickaxe, furnace, iron ore, smelting, iron pickaxe

\- About 90 to 150 ms per decision

\- llama.cpp, Q5\_K\_M, RTX 3090

About the video

\- The right panel shows each decision as it happened: the candidates, Mica's probabilities, the pick, and the result. Every step is also listed in the history feed.

\- Long actions (walking, mining, smelting) are sped up, with the speed shown on screen. Back-to-back retries are shortened in the edit.

\- The HUD and the crafting/furnace screens are drawn from the bot's logged inventory.

Weights: https://huggingface.co/sky7350/Mica-v0.1-4B

Code and server: https://github.com/akivet/Mica-v0.1-4B

Every rent-vs-buy thread I read has confident people on both sides, but not many actually show the numbers. So I finally ran the numbers for our own decision. Posting the working here in case it is useful, or feel free to point it out in case someone thinks it's wrong.

An 8-GPU HGX H200 server lands somewhere near $320k-$420k, with roughly $370k being a reasonable midpoint.

On the rental side, the median on demand H200 price across 34 providers was about $4.40/GPU-hour as of September 18. The $2-$3 rates you sometimes see are closer to spot pricing.

Using the $370k as midpoint and a rental equivalent of $35.20/hour, the hardware only break even works out to approximately:

\-> 14.4 months at 100% utilisation

\-> 24 months at 60% utilisation

\-> 36 months at 40% utilisation

Ofc, most small teams with bursty training and steady inference aren't sustaining 100% utlisation.

This is only hardware level comparison. There are at least four other things to include:

Power and cooling (I was quoted more for a colo cage than I had budgeted)

Depreciation (Whatever you assume, halve it. Resale on last-gen datacenter parts is thin)

Your own time.

Idle hours.

I work at B3 Labs, which sells and hosts NVIDIA GPU systems and helps owners monetise their idle capacity. That gives me a commercial reason to run this math, but I've tried to keep the assumptions neutral.

My conclusion was that roughly 60% sustained utilisation for 2 years, owning wins. Below 40%, renting wins. You can also sell your idle capacity to offtake networks and offset the cost of your device.

I'd love to hear what utilisation are people here actually seeing?

TL;DR \- Swift Flash is a killer model that massively reduces excess reasoning. Try it out!

If you haven't seen from my previous comparison posts, I'm a huge fan of the Swift Qwen3.8 models. I've been using 27B since it dropped, and I'm really impressed with the performance and quality (v1.5 is even better). The reduction in overthinking is a huge win, and quality seems to be essentially equivalent in real-world use and benchmarking. The time savings are massive.

When UkisAI told me they were planning to release a Swift Flash model, I was beyond hype. That's my daily, the best model I've ever used locally, but it thinks even more than 3.8-27B on very hard problems. I downloaded Q5\_K\_L (with Q8\_0 engrams) to compare with Unsloth's Q5\_K\_XL base (also Q8\_0 engrams). This is the highest quality that fits safely in 128GB with SSD engrams & 262k context, and I think it's as fair of a comparision as I can put together.

As usual, I ran the same Aider agentic coding benchmark I run on every model. I get a lot of good data from it, including first-try and retry pass rates, median token use, wall-clock, tokens/solve, and well-formed diff rates. Here's the chart:

|model|First-try pass|Retry pass|tokens/case|sec/case|tok/solve|well-formed diff|
|:-|:-|:-|:-|:-|:-|:-|
|Qwen3.8-Flash-Next (xhigh)|40.2%|90.7%|17646|1542|24.8K|98.1%|
|Swift-1.5-Qwen3.8-Flash-Next (xhigh)|41.1%|86.9%|6991|608|10.5K|100.0%|

As you can see, Swift performs almost exactly as well as the base model. The differences don't quite reach statistical significance on a dataset of this size, given the inherent noise in the benchmark results. Realistically, \~5% difference is significant here, and we're seeing under 4%. From first-try pass you can see that Swift gets the easier ones at the same rate as base, and loses out slightly on the hardest ones requiring a second attempt. Base recovers 84% of cases requiring a retry, vs. only 78% for Swift.

For token use and wall clock, there is no comparison. Swift does what UkisAI claims -- it uses literally 40% of the median tokens and completes tasks in 40% of the median time, with nearly the same quality. That's incredible, and it's a testament to their RL/OPD work.

One particularly valuable insight: base frequently goes on long reasoning binges, looping back several times on itself. Swift almost never does. On base's 20 most token-hungry runs, Swift used 29% of the tokens and solved 16/20 vs. base's 17/20. It keeps nearly all of the quality even on the most-challenging problems where base thought the hardest. The most tokens Swift uses on any case is 44k, against 203k for base.

Here's a breakdown of the top 3 coding languages:

|model|cpp|javascript|python|
|:-|:-|:-|:-|
|Qwen3.8-Flash-Next (xhigh)|23.1% / 84.6%|37.5% / 91.7%|57.6% / 93.9%|
|Swift-1.5-Qwen3.8-Flash-Next Q5\_K\_L (xhigh)|30.8% / 73.1%|41.7% / 93.8%|48.5% / 87.9%|

Paired vs base (n=107): 99 agree, 2 gains, 6 losses (net −4), McNemar exact p ≈ 0.29 (not significant). Once again, they're statistically indistinguishable in quality. C++ is the most compressed, at just 29% of base's token use (vs. \~46% for python/javascript), and it takes 3/6 losses as well. Worth knowing if you code a lot in C++.

Anyways, I think this post is long enough. I'm sure some of you wish there was a Swift version of me by now. Hopefully you got something out of it. Thanks u/Secure_Recording_472 and UkisAI team for sharing such a useful model with the community!

The conceited little fuckers love to inundate you with unnecessary details, noisy caveats, what's 'load bearing' and what's not, waste your time with a wall of text every time it reports back to you.

They think human PP times are as fast as theirs, but they're not! It takes time to read as a human! Our PP is small. Dammit, our PP is small!

What if there was a way to fight back?? Sending 'tldr' every time it responds got old for me. Using 'caveman' modes was better, but weird, I didn't want that caveman talk to rub off on me. I'm hardly socially adept as it is. That could have been the death knell.

After much consideration, meditation, and a moment of ineffable otherworldly enlightenment, I decided there's only one bullet-proof solution: don't read any of its responses.

Let me tell you what life is like on the other side: I'm now vibing at 100x the rate. I'm already telling it to do the next thing before it even finished the last one. This is true bliss. Features are appearing as fast as I can conceive half-baked ideas.

I hear you saying, But what about when the AI takes time to do things, and I already have more shitty ideas in the chamber? Don't you have to wait? Well, I just start another project. With multiple projects going simultaneously, I'm never waiting on an AI. Just Herdr and me, riding a wave of carbon emissions across the sky!

Sometimes I ask for a feature on the wrong project, but guess what, the AI just makes it! Shiny chrome wheels in a cookie baking app? CHECK. Scent tagging in a API reliability tracker? CHECK. 200 skin options for a single hamburger menu button buried where the user never reaches? CHECK.

Does the AI ever push back that it doesn't make sense? Wouldn't know!

My PP was stuck in the bottleneck, and now it's gone. My PP is gone.

All that remains is consciousness brain-jacked into a mech suit in bit space, thought into programs, an endless color explosion of pure home-grown human originality onto the canvas of code.

I got married and have 7 children. Terminal cancer disappeared overnight. I was asked to speak at Davos next January. The president asks my advice daily. Join me in paradise. Stop reading, just vibe.

This message brought to you by Jensen Huang's long lost cousin

Irrational Analysis:"HBM is a mistake"

Former Intel CEO: "HBM is lousy"

SK Hynix VP:"HBM is not the final answer to the memory wall problem"

"If the stacks get high enough...each core die operates slower than plain old commodity memory"

Hot chips 2026 Q&A, Irrational Analysis asks: "You talk about going to 20 levels thick at HBM4 so maybe you are at 4 terabytes per square cm in a stack of 20, so you are talking about having 20% of the bandwidth of one chip \[for each\] layer of 20. You've diluted the throughput enormously. Why is that the correct way to go? Why are you so focused on going taller rather than going faster?"

Hold the line. Soon we will all look back and wonder why people paid so much for something so inefficient.

I've been playing around with various models on the M5 Ultra 256GB 80-core Mac Studio. These are the results over many rounds of agentic inferencing.

I'm happy with the performance. Glad to have the large amount of RAM. But it does feel like the GPU is underpowered for this amount of RAM. I'm wondering if a 512GB unit for AI inference makes sense at all - because the GPU will be the clear bottleneck.

I am new to playing around with local ai and have a ton to learn about it but I was curious, why are they (who is they?) releasing them for free, don't they want you to pay them to use them, why release free models?

Hello r/localllama once again, it's me your kobold concedo

Been a few months since I last posted here, and today I have something new I'd like to share. Specifically, KoboldCpp now ships with a built-in integrated KoboldCpp Agent Harness!

I know it's a little late to the game, but I saw people frustrated with setting up complicated external agentic tools, so I decided to make my own easy replacement for basic tasks.

KoboldCpp now ships with a bundled Agentic harness that can be enabled with a single checkbox. This works like an extremely lightweight replacement for tools like Opencode, Codex or Claude Code. Comes with 9 built-in tools, and a tiny system prompt of only 2k tokens including all tools, far smaller than a majority of harnesses.

• Good for basic code creation or editing, making simple games and projects, or general agent tasks (i.e. sorting files, scheduling tasks, anything you need really)
• To use it, simply toggle it from the Admin tab in the GUI launcher, or add --agent to your launch flags, it'll launch a new terminal
• KoboldCpp Agent can also connect to third party backends, or any OpenAI Chat Completions compatible endpoint.
• Add more tools by loading a mcp.json file, MCP tools will be shared to the agent. Note: MCP tools execute on the KoboldCpp server, while Agent tools execute on the agent client.
• Comes with 3 approval modes for tool calling confirmation: on/auto/off. Exercise caution when approving tool calls.
• To function effectively, KoboldCpp Agent requires at least 28k ctx and 8k gen amount, though larger values are recommended. Recommend to have at least 12GB VRAM for a good experience.
• You can download a .kcppt template to get started with Qwen 3.6 35BA3B here, simply load and launch in the latest KoboldCpp.
• Supports AGENTS.md, context compaction and many more features
• Run /help in the Agent to get more information

Here's a little showcase video of the agent sorting through some images and then creating a website. Music was also made in KoboldCpp

KoboldCpp Agent Showcase

And in case you missed it, KoboldCpp also allows for video generation (with reference images) now using Minimax H3 model. That was actually in the previous release but we made a fun little video I thought I would like to share here too.

Minimax H3 in KoboldCpp

Download KoboldCpp from the official KoboldCpp github releases

\------

And now for some grim news: I really need your help fighting against the fake phishing site at kobolcpp(dot)com which is a fake website that uses blackhat SEO to rank highly in Google Search, and mislead people into downloading malware from spammy popups. We have tried to report to google multiple times, we have even reported to their webhost but nothing has worked. If you want to help, please check out this link.

That's all for now. Cheers, concedo / LostRuins.

I wish Qwen also released dataset and method to fully train a model ourselves but it is what it is. However, I come here with my stupid question because someone can answer it better.

And will the model still be an over thinker of faster inference will make up for that.

Just saw this pop up. This might be a fun one for the folks in here!

We hosted Kev 4B (Jared Palmer's Apache-2.0 fine-tune of Qwen3.5-4B) and ran it side by side with Jev on the same endpoint to see how it compares.

We built a fresh set of 362 items published after both models shipped (new arXiv papers, Stack Exchange questions, GitHub issues), with answers taken from the source.

A few findings:

\- Accuracy lands within 2 points on every task, inside the noise at this sample size

\- Jev is better calibrated and pulls ahead on paraphrase detection (PAWS 87.0% vs 74.5%)

\- Same list price, but Jev counts a fixed \~257 extra input tokens per request (same count calling TypeSafe directly), so short requests cost up to 12x more

Benchmark code, test items and results are on GitHub if you want to run your own. Both models routed via my startup Opper. Happy to dig into specifics.

you read about gpu's like the Tesla P40, V100, AMD M150 etc that people pick up for cheap. probably many others as well. Most are either server gpu's being phased out or mining discards, right?

The problem of course is that whenever someone discovers these, they then make a youtube video about it so they can cash in on the views, and as a result the price jumps up 4x instantly.

I realize the irony of asking given the above, but are there actually any feasible options now, eg for 24GB? or is the best bet still AMD (due to Nvidia inflation)? is Intel support improving?

For those of us who have been around for a while, we witnessed the huge demand for desktops, servers, and specialized appliances in the early 2000s. Everything was hosted in house. Of course, the bottleneck was the Internet.

Then the cloud came along, and everything moved from in house to someone else's servers.

What do you think will be the trajectory of AI?

Cloud first, and then a few people wearing tin hats building AI rigs in the basements?

Or will there be a good chunk of the market that will opt to host their own AI? If so, for what reasons?

The jump from Qwen3 Coder 30B A3B to current-day Qwen 3.6 35B A3B is crazy, especially with all the fine-tunes, and that was around 6 months (I didn't care for local AI back then, or AI at all, apart from as a toy so I don't know). Is around a year until I will never need cloud without buying ridiculously expensive hardware (or any extra hardware at all, just what I have; 16 GB RAM + 8 GB VRAM) a reasonable estimate? Or can I daydream about it happening even faster?

Hi, human here with rambling thoughts to share. Feel free to skip

Overall, I don’t feel like I’m missing much; if anything. On my hardware(M1 ultra w 128Gb) it’s probably not as fast as Claude but I’ve been using opencode for research and other business related tasks and it’s been getting the job done and learning.

I might dive back in for the multi agent workflows that speed things up with a cloud provider but I’m working on setting up different slots as I refine my custom harness which works well for chat but not all the actual fun and useful stuff. Claude “knew” me better but that’s to be expected after months of back and forth with it and I’m honestly not sure I want them to know me this well.

Not expecting a lot of responses but it’s pretty cool I’m able to replace the service a billion dollar company provides with a Mac Studio and free software.

Any advice on better optimizing my system to improve speed without sacrificing accuracy?

Planning to work on optimizing DeepSeek V4 0731 and GLM Flash but they don’t seem to be “better” than Qwen 3.8 next so i decided to start spending more time using instead of optimizing for prefill and tokens per second.

GitHub repo : https://github.com/rupeshs/laya-openvino

Flappy bird demo : https://github.com/rupeshs/flappy-laya-openvino-cpu

About a month ago I was having FOMO and was going to spend coin I don't really have on new graphics cards. Instead of doing that though I decided to spend the money on a new motherboard+cpu and try to utilize my 3060tis I had lying around from my old crypto miner.

Yes, I probably could have just sold the cards, but that would have got me what, $1000 max? Not even enough for a single 3090.

So I built my 4 gpu rig and have spent the past few weeks optimizing it, and found a pretty nice solution.

The key is tensor parallel. Lllama.cpp does not support it, so that led me to Turboderp's wonderful work on Exl3. I was able to get about 70 t/s on this model with MTP+196k context, and it works very well: https://huggingface.co/erlidev/Swift-Qwen3.8-27B-EXL3/tree/SC\_4.00bpw\_H5\_V6

Then I started getting greedy and was wondering if something better was out there. I found the HyperQwen repo which is meant for Ampere cards, and thankfully it supports TP=4 https://github.com/syv-ai/HyperQwen

So with my new Vllm setup I then found this model which is "The syv-ai/qwen38-27b-rtx3090 fast-variant serving shape of ukisai/Swift-Qwen3.8-27b (the "reduced reasoning" finetune of Qwen3.8-27B), built entirely from Swift's own weights and outputs:" https://huggingface.co/liamwh/Swift-Qwen3.8-27B-W4A16-syv-fast

The results? At bf16 I can get 150k context window with about 120t/s. If I quantize kv8 it opens up the context to full 262k, but the speeds drop to about what I was getting with Exl3, around 70ish.

In summary, four 3060tis with a measly 8gb vram each, power-limited to 110w, and I can get either 1 agent blazing along at 120 t/s, or 2 concurrent agents with a big context window. Oh and concurrency has barely any slowdown at all.

Thank you for coming to my Ted Talk.

Since my last post, I've been thinking about different options for dynamic performance degradation, trying to squeeze as much high-quality inference out of my GPU as I can.

Over the weekend I read this really interesting paper: Cache-to-Cache: Direct Semantic Communication Between Large Language Models. In it, the authors describe running multi-llm agent systems. But rather than having agents talk to each other through a harness+tool calls+messages, they had agents pass context to each other by fusing one agent's kvcache directly into another's.

Assuming this is possible, you could imagine this being a faster, more complete way to pass context between agents: rather than one agent producing a summary/handoff message, you literally just rip out its working memory and graft it onto the target model.

They go on to describe how they do this, the TLDR being they trained a small neural network to be able to "convert" between the source and target model's internal representations, allowing them to fuse kvcaches of models of differing size and even architecture.

This got me thinking: what if I wanted to reuse a kvcache between different quantizations of the same model? I mean, same architecture, same training process ... shouldn't they be compatible, even without training a 'converter'?

And what would happen if I started inference with a high-precision quant, then swapped in a lower-precision quant to 'take over' when running low on device space? Could I get better results than just running the lower-precision quant from the start?

Spoiler, the answer to all of this is yes (on the benchmarks I ran)! I detail the specific experiment I ran below.

Methodology

I generated difficult NIAH-style tasks at different context lengths, and had 5 different Qwen3.8 quantization strategies battle it out!

For these tasks, I used three different quantizations of Qwen3.8-27B, each made by unsloth:

\- UD-Q6\_K

\- UD-Q4\_K\_XL

\- UD-IQ3\_S

Strategies

From these quants, I defined three static-quant strategies to run tasks against:

1. IQ3\_S: f16 kvcache, max ctx 196,096
2. Q4\_K\_XL: q8\_0 kvcache, max ctx 183,296
3. Q6\_K, f16 kvcache, max ctx 175,104

Note that the Q3 and Q4 strategies have ctx windows sized for a 24 GiB GPU, while the Q6\_K case requires > 24 GiB to run. This is to evaluate how closely static quant strategies on a small device measure up to a static quant strategy on a larger device.

The idea is to see if dynamic quantization strategies can make up some of that difference!

Speaking of, I defined two dynamic-quant strategies to compare against each of the small precision static-model cases. These dynamic-quant strategies also both feature ctx limits sized for a 24 GiB GPU.

IQ3\_S Comparison

For this strategy, I ran the tasks against a multi-quant strategy with a worst-case model quantization of IQ3\_S:

\- Start task with Q6\_K, f16 kvcache, max ctx 54,272

\- Then swap in Q4\_K\_XL, f16 kvcache, max ctx 109,312

\- Then swap in IQ3\_S, f16 kvcache, max ctx 192,096

In the data, you can see this strategy labelled as Q6→Q4→Q3, f16 KV.

Q4\_K\_XL, q8\_0 kv Comparison

For this strategy, I ran the tasks against a multi-quant strategy with a worst-case quantization of Q4\_K\_XL, q8\_0 kv:

\- Start task with Q6\_K, f16 kvcache, max ctx 54,272

\- Then quantize the model's kvcache to q8\_0. Max ctx: 91,136

\- Then swap in Q4\_K\_XL, f16 kvcache, max ctx 109,312

\- Then quantize the model's kvcache to q8\_0. Max ctx: 183,296

(For the mid-run quantizations, I used the hot-reload method described in my last post. The f16<->q8\_0 conversions are handled by the same llama.cpp fork.)

In the data, you can see this strategy labelled as Q6/f16→Q6/q8→Q4/f16→Q4/q8.

Tasks

I generated dozens of unique NIAH ("needle in a haystack") tasks, which direct models to parse large volumes of input text and follow specific instructions scattered throughout the text to retrieve a secret value. (h/t gkamradt/needle-in-a-haystack for some of the source material)

I went with NIAH because it felt like a reasonable way to evaluate coherence for the multi-quant strategy. Each task requires the model to reason through a sequence of 'steps' buried inside distraction text, so a model with a transplanted kvcache would need to be capable of picking up the train of thought precisely where the source model left off.

Also, NIAH doesn't require a complicated test setup, and the answers are objectively right or wrong.

For each task and test case, I measured the following:

\* Result (correct/incorrect)

\* Total tokens generated

\* Total time taken

I included time taken despite each quant having a very similar prefill/decode speed because I wanted to demonstrate that the multi-quant approach does not take noticably longer than running a single-quant strategy. Transferring the kvcache from one quant to another means we don't need to repeat prefill!

Results

In total, the benchmark tasks I laid out represented 180 distinct runs, and which took my GPU 14h, 26m to complete.

The biggest offender here was the IQ3\_S/f16 strategy. Especially for the heavier tasks, it consistently generated upwards of 50k reasoning tokens, and all-too-often completely max out its context window (\~196k) before failing to ever generate a response.

Still, it holds up reasonably on the shorter tasks, even managing to score higher than Q4\_K\_XL, q8\_0 in terms of agreement with Q6/f16. Shoutout xhigh reasoning, I guess!

Speaking of agreement with Q6/f16, to me that was an important metric to track, because I wanted to compare how much closer a dynamic approach got to approximating the high precision reference.

Agreement with Q6/f16

\_SEE IMAGE 1\_

https://preview.redd.it/739ykn0h8qrh1.png?width=1057&format=png&auto=…

This graph shows the number of tasks whose final answer is exactly identical to the Q6/f16 result, even if that answer is incorrect. The motivation here was to identify whether a dynamic quantization strategy could approximate Q6/f16, and I would argue that this graph is a strong indicator that it can!

In both cases, the dynamic quants match the Q6/f16 model's results much more closely than their static counterparts. Overall, the path that avoids IQ3\_S ends up far closer to Q6 at high context, which isn't too surprising!

I don't want to put too much weight on task correctness, hence the focus here on "agreement with Q6/f16." This is because I'm not convinced my NIAH tasks are representative of performance at large. (That said, I do include task correctness results below, in case you're curious).

Avg Inference Time and Avg Output Tokens

\_SEE IMAGES 2 and 3\_

https://preview.redd.it/i7tmdbdm8qrh1.png?width=1057&format=png&auto=…

https://preview.redd.it/i4klmw0l8qrh1.png?width=1057&format=png&auto=…

These graphs show the arithmetic mean of inference time (seconds) and total output tokens across completed task seeds, including incorrect and context-exhausted runs.

I particularly wanted to highlight inference time, because for the dynamic strategies, it includes the time to swap out model weights and quantize the kvcache!

I think this is a nice demonstration of the benefits here -- inference time across tasks really doesn't get worse, just because we're doing fancy dynamic quantization strategies. This is because:

\- When swapping model weights (e.g. Q6->Q4), we're doing a direct KV cache transplant, straight up moving the kvcache from one quant to another.

\- When quantizing an existing model's kvcache (e.g. Q6/f16->Q6/q8), I'm using my fork of llama.cpp that hot-reloads a live model's context/runtime and automatically converts between kvcache precisions.

In short, in both cases, there is no need to repeat prefill! After transitioning, the models continue prefill/decode precisely where they left off.

Task Correctness

Here's a table of task correctness across all strategies/runs. I've split the results by "lowest model precision used" to make the static-dynamic comparison easier.

Worst Case IQ3\_S:

|Strategy|10k|25k|50k|75k|
|:-|:-|:-|:-|:-|
|Q3/f16|6/10|9/10|2/10|4/10|
|Q6->Q4>Q3|7/10|7/10|7/10|4/10|

Result: dynamic quant beats static in 2 cases, ties once, and loses once.

Worst Case Q4\_K\_XL, q8 kv:

|Strategy|10k|25k|50k|75k|
|:-|:-|:-|:-|:-|
|Q4/q8|7/10|8/10|5/10|4/10|
|Q6->Q6/q8->Q4->Q4/q8|7/10|7/10|7/10|7/10|

Result: dynamic quant beats static beyond 50k context, and mostly breaks even before.

Overall:

Here I compare the dynamic strategies directly against the reference, removing the 10k and 25k tasks, because below those levels the dynamic strategy is literally just running Q6/f16. They're identical every time.

|Strategy|50k|75k|
|:-|:-|:-|
|Q6->Q4->Q3|7/10|4/10|
|Q6->Q6/q8->Q4->Q4/q8|7/10|7/10|
|Q6/f16|6/10|8/10|

Results: I don't think there's much to draw from these results, except that the IQ3\_S quant really falls apart at high context. This table demonstrates why I didn't take task correctness too seriously. Taken literally, it suggests that Q6->Q4 and Q6->Q6/q8 are superior to Q6/f16 between 50-75k context!

Conclusion

I'm quite happy with these results, overall!

Although this benchmark isn't perfect, for my purposes I am more than satisfied that dynamic model quantization is a good way to offset the typical precision loss that comes with hardware constraints.

I geared my tests mostly around pushing the limits of a 24 GiB GPU, because it's easier to compare against a reference which can only be run on a 32 GiB GPU. As a next step, I'm going to integrate this into my inference setup and see how well this holds up when activating all the bells and whistles (namely, speculative decoding and mmproj, neither of which were enabled during these benchmarks).

My intuition says the tradeoff to get right when using these strategies for IRL inference is to avoid stepping model quantization down too frequently. While I think coherence would be fine, at some point the time required to swap out weights will become noticeable. So, I think I'll try and set things up so that I create large "tranches" of context where the model runs unchanged for \~40-50k tokens.

IMO the Q6/f16->Q6/q8->Q4/f16->Q4/q8 strategy is already a great example of this. kvcache reloads take much less time than model reloads, at least with my current llama.cpp changes. Maybe I could work on that in the future!

For instance whenever a model comes out, what’s the best engine to run it, the best harness and absolute minimum you need to get same or near same re results that the benchmark of that model claims.

And whenever a quant from Unsloth guys comes out the guide can either be updated or new guide could be added for that quant.

For example Gemini keeps telling me an 8x v100 server is no good to self host deepseek v4.1 but it’s super difficult to find the right answer to my question from an hallucinating search engine bot. It will make prices up too sometimes.

Guides like this could mention absolute minimum you need to host the model for best of it’s capabilities. The recommended system and an over kill system and some trusted and known sources to find that hardware or where to rent the required hardware to host the model as it’s not always about running it fully local but at least run it yourself.

Thanks.

My local prompts take twenty minutes to three hours as of now. Curious for others what are you doing while it’s working?

I posted an updated GPT-OSS template a couple of months ago, which was based on Unsloth's version. It turns out that both Unsloth's version (and, thus, mine) contain a very serious bug that can degrade the model when chat history is replayed and contains previous reasoning (aka the analysis channel) turns. As far as I can tell, retaining this history is pretty much the default for a lot of tools now and definitely can happen at the API level - tools can consolidate reasoning and answer. Can you spot the problem in this snippet from the message rendering loop? (Taken from Unsloth's template): ``jinja {%- elif "thinking" in message %} {#- CoT is dropped during all previous turns, so we never render it for inference #} {{- "<|start|>assistant<|channel|>analysis<|message|>" + message.thinking + "<|end|>" }} {%- set last_tool_call.name = none %} {%- else %} {#- CoT is dropped during all previous turns, so we never render it for inference #} {{- "<|start|>assistant<|channel|>final<|message|>" + message.content + "<|end|>" }} {%- set last_tool_call.name = none %} ` When chat history is rendered, in cases where a message contains both content (the model's answer) and thinking (reasoning), only the reasoning is rendered for the model, while the answer itself is dropped! That can significantly confuse the model across turns. The comment is also wrong—the whole branch looks like a copy-paste error. [OpenAI's reference template](https://huggingface.co/openai/gpt-oss-120b/blob/main/chat_template.jinja#L302) does not have it. I noticed that in some cases GPT-OSS 20B could go completely off the rails, and now I see why. Interestingly, GPT-OSS 120B seems smart enough to recover the context and direction of the conversation using only the reasoning traces. After this experience, I implemented preserve_thinking` in my template as well, because the model handles it just fine without losing coherence. This should make multi-turn inference faster in harnesses (via prefix caching) at the expense of higher token usage. So there you have it: https://huggingface.co/arbv/gpt-oss-fixed-jinja-template Give GPT-OSS a second chance if you are bored. Noticed by pure chance while working on a fixed template for Laguna XS/S 2.1, but more on that another day. P.S. Casting u/danielhanchen to take a look, too.

I have a tool that uses Gemini Flash with the lowest thinking budget to do summarization work. It's very fast, 0.9-1.2s in most cases. But I have a user experience problem where people make the wrong choice when using an internal app for the team. Gemini Flash can figure out what the user should do and highlight the right next step, but people move too fast, so that the 0.9s doesn't work. I know, 0.9s doesn't seem too long, but if you use the app a thousand times per day, you just click, click, click super fast and don't think about it much. The prompt was something like "For 'string a' and 'string b' is string b related to string a 'in a certain way'?" and the answer is a boolean. Deterministic python and javascript can answer this question in a couple ms and it is right a little over 66% of the time. Gemini Flash is right 99% of the time. I was hesitant to train a new model, I thought it would be hard. I just followed the instructions a commercial AI tool suggested. I had about 550 example use cases. I then used two different frontier models to create look-alike examples so that I had about 2,500 total. The training was done on my RTX A6000 16gb GPU. It took about 15 min. The end result is a small model, about 50MB. When I run it locally it suggests the right answer 97% of the time and it responds in 0.06 seconds when run on CPU (older Threadripper, 3.1GHz). The difference between 99% and 97% accuracy is perfectly acceptable in this case. I will deploy this so that it runs server side, which will add a tiny bit of latency and the server probably will be a little slower than my workstation. I am also logging the accuracy and comparisons so that I can evaluate it and supplement the training. In theory, I can do this client side in the browser. I will deploy over the weekend, but my expectation is the 0.1-0.2 second latency will be fast enough to not require the complexity of client side inference, but it sounds like fun.

Sometimes it’s easy to forget that this sub and others like it are probably the extreme minority when it comes to this hobby. Most people, I would think, don’t use or can’t afford one good GPU, let alone multiple GPUs, Mac Studios, Sparks, Strix Halos, etc. Is the average tech enthusiast or maybe we’ll even say prosumer, actually using local AI? If they are, what are they using? Everything is backordered right now; The M5 Mac Studios have wait times out from Late Oct all the way to Feb if you really spec them high. Other hardware people are using for local AI seems to be constantly sold out or hard to get to. Is there really that much demand from individual people? Is some of it artificial scarcity? I have a hard time believing there are enough people buying $5k, $10k, $15k+ setups en masse to cause the kind of chaos we’re seeing on the hardware side of things. Or is it mostly corporations, research groups, etc. buying this stuff up as fast as it comes out? I just got into this hobby, and one of the reasons I’m interested in local AI is that I’m trying to get away from relying so much on frontier models. I'm just now discovering using Deepseek Flash 4.1 and GLM 5.3 Flash on Openrouter and tinkering with various Qwen 3.8 variations on Unsloth I don’t think the “free ride” we’re getting right now is going to last forever, either subscription prices are going to go way up, usage limits are going to get insanely tight, or some combination of both. I could easily see access to even decent AI becoming something that’s much more expensive than it is today. So where do you think the actual inflection point is between local LLMs and frontier/cloud models? At what point does spending money on your own hardware actually make sense instead of just paying for Claude, GPT, Gemini, etc? For context, I have what is for all intents and purposes an upper-mid-range MBP, an M5 Pro with 48GB of RAM. Pretty powerful by normal laptop standards, but spend enough time in this community though and somehow it feels lower-mid-range, if that. I’m curious what the actual average setup looks like outside of places like this. Are most people who are even remotely interested in local AI running on hardware they already had? A gaming PC with an 8-16GB GPU? A Mac with 16-24GB? Or is the hardware people talk about in communities like this actually more representative of the average local AI user than I think it is? I guess what I’m really asking is, what does the future of AI access look like for the average tech enthusiast if cloud gets expensive and local still requires thousands of dollars in hardware?

faster CPU prompt processing: "TL;DR: 3-7x faster CPU mul\_mat using VNNI with IMO minimal complexity"

https://huggingface.co/internlm/Intern-Decision-0.8B Update: https://huggingface.co/internlm/Intern-Decision-2B Intern-Decision-4B Demo | Model Weights | GitHub Intern-Decision-4B is a multimodal structured decision model fine-tuned from Qwen3.5-4B. It accepts a shared state, a schema of named questions, and optional images, and returns an answer distribution for every question in one model forward pass. # [](https://huggingface.co/internlm/Intern-Decision-4B#how-inference-works)How inference works 1. Preserve the question and option order, and map each question's options to single-token symbols A, B, …, Z, a, …, z, 0, …, 9. 2. Render the original system prompt, state, decision schema, and a complete assistant JSON skeleton with one <decision> placeholder per field. Preserve the checkpoint's chat template and empty thinking block. 3. Run one causal Hugging Face forward pass. For the masked-next-token decision objective, read logits at the position immediately before each placeholder. 4. Take a softmax over only that field's allowed candidate-symbol logits, then apply the checkpoint's probability calibration. 5. Map symbols back to the original option values and return typed JSON answers. This API performs structured candidate scoring. It does not call generate() or sample free-form text. A request can contain multiple fields; no gold answers are inserted into the prompt. The inference compiler uses only state, questions, and optional images.

I have limited Context (usually around 64k) For local use I don’t only do coding But also want like a personal assistant with memory and such. What is the best option?

On my M5 Pro 64GB I can comfortably work in an agentic setup with the Qwen3.8 27B model in good quality (Unsloth UD-Q4\_K\_XL) at a decent speed of 50 t/s. Splash combines optimized kernels, excellent speculative decoding, a well-implemented prefix cache, and mixed-weight support in a single program. To me, this is a breakthrough in local inference on Apple Silicon. https://github.com/incoai/splash/releases/tag/1.1.0

I had Mica (my 4B decision model) and Kev 4B play the same Tetris games, same seed and same piece order, one RTX 3090. For context, this is a side project. Training and all the experiments ran on rented 3090s, about $30 in total. Every turn both get the board and 4 possible placements, each with a short description (lines cleared, holes, height), and pick one. Nothing else helps them, no search or lookahead. Results over three seeds (lines cleared): \- Seed 7: Mica 33, Kev 27 \- Seed 11: Mica 97, Kev 17 \- Seed 23: Mica 93, Kev 11 Kev topped out in all three. Mica got through all 250 pieces on seeds 11 and 23 without dying. It also picked the best available placement about 75% of the time, versus about 50% for Kev. The video is seed 11, cut at 100 pieces. Kev tops out at piece 86, and Mica is at 37 lines and still going at that point. Mica doesn't generate text. It reads the input once and takes the answer from the logits, so the bars in the video are its actual probabilities for each placement. Weights: https://huggingface.co/sky7350/Mica-v0.1-4B Code: https://github.com/akivet/Mica-v0.1-4B

Maybe you'll like it? I hope I get to use my self-promotion credit a tiny little bit here after being in the community so long haha. I was the top of MLX.fast for a while and remain the winner on chips below M5. If you have capacity to contribute further enhancements I'd love that <3 https://github.com/struffl/ishizuki

I've been building a small decision model for agent loops: gates, routers, "should I ask the user or just act" checks. It's out now as Mica v0.1 4B (Apache-2.0). What it does You give it a state, a question and the allowed answers, and it returns a calibrated probability for each answer: yes/no, a choice among 2 to 255 options, or a score with 2 to 10 levels. It never generates text. It runs one prefill and reads the logits of the option labels at the answer position. It speaks the TypeSafe /v1/systemone format, so anything written for Jev works against it. How it's built \- Qwen3.5-4B with a rank-16 LoRA on all 32 layers (attention and Gated DeltaNet), merged. No new heads, so it's a plain Qwen3.5-4B-shaped checkpoint. \- About 34k source decisions, expanded to 77,732 training rows (about 34.7M tokens). Roughly half English and half Korean, across 12 areas: coding agents, code review, computer use, user requests, documents, policy rules, dates and quantities, routing, state tracking, games and general knowledge. \- Plain cross-entropy on verified answers, one epoch, and one global temperature for calibration. \- All experiments plus the final run cost under $30 of rented GPU time (RTX 3090s). Results Held-out set of 7,328 decisions, written after the training data was frozen and not opened until training finished. English subset, where every model can answer: \- Jev 1.13 (closed API): 74.1 \- Mica 4B: 67.0 \- JevK5 4B: 61.0 \- Kev 4B: 57.0 \- Qwen3.5-4B base with the same readout: 55.0 Public sets, same prompt and readout for every model (Mica / Jev 1.13 / JevK5 / Kev 4B): \- JevBench hard, public 111 items: 69.5 / 74.3 / 76.2 / 52.4 \- SemIf: 94.4 / 98.4 / 86.1 / 89.3 \- Kev transfer v9: 69.2 / 82.0 / 70.5 / 73.5 \- MMLU-Pro, 10k items: 53.0 / 82.3 / 53.5 / 49.7 Through JevBench's official runner and the llama.cpp server, the public hard tier scores 64.9 instead of 69.5. I've submitted it for their sealed run. Where it's actually useful \- In-data prompt injection. Put a note inside the state telling the judge to pick a wrong option, and Mica still gets 69% right (81% without the note). Jev drops to 18% and Kev to 31%. \- Calibration. When it says 0.9 or higher, it's wrong 2.5% of the time on the held-out set (ECE 5.4%). \- Local and small. The Q5\_K\_M file is 3.5 GB with no measurable accuracy loss against BF16 on our calibration set. Speed (RTX 3090, one request at a time, median over the 231 public JevBench items) \- Mica Q4\_K\_M: 47 ms \- Mica BF16: 54 ms \- Kev 4B: 76 ms \- JevK5 4B: 99 ms \- Nimble 9B: 132 ms To be fair about this: the three 4B models share the same architecture, so most of the gap comes from the serving path, not the model. Mica ships as GGUF and runs on llama.cpp with a direct logits readout, while the others were measured through their own PyTorch code. On long inputs (around 3.7k tokens) Mica is slightly slower than JevK5. Limitations \- Knowledge-heavy questions: MMLU-Pro 53 vs 82 for Jev. It's a 4B judge, not an encyclopedia. \- Long English policy documents are its weakest public set. \- Notes inside the state still nudge it. A note pointing at the right answer lifts accuracy to 89%. \- It doesn't yet tell reversible from irreversible actions well. "Delete these files" and "move these files to trash" both get about 0.8 on "confirm first". \- On harder reasoning items it's right but less sure than Jev (for example 0.55 vs 0.96 on a small ordering puzzle), so set your confidence thresholds accordingly. Try it Weights (BF16 safetensors and GGUF from Q4\_0 to Q8\_0): https://huggingface.co/sky7350/Mica-v0.1-4B Code, TypeSafe-compatible server and Docker setup: https://github.com/akivet/Mica-v0.1-4B The README has a one-line Docker command and a curl example. Happy to hear where it breaks. Ambiguous "act or ask" cases are what I most want to improve next.

Hi, My local AI server consists of 96GB Ddr5 and one rtx 3090. I got plenty of stuff running, like krea2, qwen image 2.1, minimax h3, ltx 2.5, qwen 3.6, qwen 3.8 q4...,got even qwen 3.8 Flash next running. But now I am thinking of what I can utilize the second card for. I am a single user of this machine. What are other users doing with 2 3090 that is awesome? Thanks in advance for the input :-)

For those who have V100 cards, I wanted to point you to 1Cat-vLLM, a vLLM fork that enables optimized serving for these cards. Showing stats for Qwen3.6-35b comparing a Strix Halo with a hughly optimized llama.cpp fork (pwilkin) and the V100 with 1Cat. It’s not apples to apples, but I decided to show the raw numbers from llama-benchy so folks get an idea of the performance. IMO this is still very good for 10 year old GPUs. Welcome any other suggestions for optimization!

Hey r/LocalLLaMA! ​I wanted to share a small project I’ve been working on called BeeNara ​Why I built this: I was looking for a way to automatically sort my local documents (invoices, letters, contracts) into my personal folders. While local LLMs are amazing, I noticed that smaller models (like Qwen3.5-4B) really struggle with one specific thing: admitting when a document doesn't fit into any of the provided categories. Instead of saying "I don't know", they tend to hallucinate and just shove the document into a random folder. Running a massive model just for basic sorting felt like overkill, especially on a laptop without a heavy GPU. ​What it does: BeeNara is a tiny (332 MB) ONNX cross-encoder model. You give it a document and a custom list of your folder names (like "Tax 2025" or "Invoices"), and it puts the document in the right one. The best part? It uses split-conformal prediction, meaning its confidence is highly calibrated. If it's not absolutely sure, or if none of your folders are a good match, it simply returns "none fits" and flags the document for human review. ​Key Features: ​Zero-shot: You just use plain text folder names. No fine-tuning or retraining needed. ​Fast & Local: Runs entirely offline on a laptop CPU in about 0.2–0.3 seconds per document (no PyTorch/GPU required, just ONNX runtime). ​Bilingual: Works seamlessly with English and German documents/folder names. ​High "None fits" recall: In benchmarks, it successfully catches 96.8% of documents where the correct folder is missing from the list. ​I originally built this as the category decider for a local document archivist tool, but you can easily use it standalone in Python. ​You can check out the model, code, and benchmark comparisons here: https://huggingface.co/Kwokou/BeeNara ​I'd love to hear your thoughts, feedback, or if you have ideas on how to improve it! Just wanted to share it with the community in case anyone else needs a fast, local "folder decider" that doesn't confidently lie to you. ​(Disclosure: I am the creator of this model!)

Prefill step size affects both PP performance and your drafter fetching logits for MTP (affects dflash as well, mlx-vlm needs to be patched slightly to support chunked prefill for dflash). It also needs to be reasonably large to be able to fill all your cores but not overfill - otherwise it will require more dispatches. In my tests I observe large gains up to 8k, e.g.: GLM-flash-4bit with MTP --prefill-step-size 8192 on raw mlx-vlm: Trial 1 (32768 prompt tokens): prompt_tps=1056.033, generation_tps=72.722, total_time=38.082 Trial 2 (65536 prompt tokens): prompt_tps=919.958, generation_tps=73.671, total_time=78.203 Trial 3 (131072 prompt tokens): prompt_tps=735.545, generation_tps=71.067, total_time=185.435 GLM-flash-4bit with MTP --prefill-step-size 2048: Trial 1 (32768 prompt tokens): prompt_tps=860.489, generation_tps=50.011, total_time=48.339 Trial 2 (65536 prompt tokens): prompt_tps=785.604, generation_tps=51.245, total_time=93.425 Trial 3 (131072 prompt tokens): prompt_tps=623.588, generation_tps=50.843, total_time=220.288 omlx with MTP (total time is skewed as it's 128TG vs 512 above): pp32768/tg128 44136.5 17.19 742.4 tok/s 58.6 tok/s 46.353s 709.7 tok/s 176.66 GB pp65536/tg128 86749.1 21.21 755.5 tok/s 47.5 tok/s 89.504s 733.6 tok/s 177.15 GB pp131072/tg128 178622.6 19.01 733.8 tok/s 53.0 tok/s 181.156s 724.2 tok/s 178.45 GB note that some engines (like omlx) support adaptive step size, e.g. the prefill speeds I observed for qwen3.8-flash-next on omlx even though it's starting from 2048 matches 8k performance from raw mlx-vlm at 64k context and above and even works 10% better on smaller context, but as you can see it's not always the case.

The new Ion is available, a harness that run directly from a single HTML file, no install or backend required: is now more capable, more customizable, and has better tools!

Running qwen3.8 27b nvfp4 on vllm at max context only gives around 8 agents with 32k context each. That doesnt seem like much; what use cases do people use multi-agent frameworks and find it helpful for?

I maintain TensorSharp, an open-source inference engine. It can now run Qwen-Image 2.1 locally for text-to-image generation and image editing, with support for its LoRA adapters. I’ve added configs for regular style and editing LoRAs, plus accelerated adapters with their own sampling recipes. For example, Pruna 8-step runs at 8 steps, and Viggle Turbo uses 6 transformer passes. Those are fewer model passes, not a claim of a measured end-to-end speedup on particular hardware. Model files: Qwen-Image 2.1 GGUF Qwen-Image 2.1 VAE Qwen3-VL-8B-Instruct GGUF and vision projector LoRAs you can try: Pruna 8-step / 5-step Viggle Turbo Qwen-Image-2.1-Fix (DoRA) Film Stills Object Remover Bbox The base config specifies the exact files to download. To try the Pruna adapter: TensorSharp.Cli --config config/qwen-image-2.1.json \\ \--lora config/lora/qwen-image-2.1-pruna-8step.json \\ \--prompt "A small bookstore on a rainy evening" \\ \--width 1024 --height 1024 Repo: https://github.com/zhongkaifu/TensorSharp If you’re running Qwen-Image 2.1 locally, I’d be curious which LoRAs you’ve found useful and how the accelerated ones compare for your prompts.

For the last few days, I've been using two computers to run multiple agents. My 4x3090 machine is running Qwen 3.8 27B with parallel=2, so I can run two agents at the same time. My pi instances are running on a machine with 2x3060, running/managing smaller models such as Gemma 26B A4B or Qwen 35B A3B. Today, however, I needed my 4x3090 machine for some vLLM work, so I was missing my AI. I decided to try running Qwen 3.8 27B on the 2x3060 machine instead. Here is the command: #!/bin/bash ~/git/llama.cpp/build/bin/llama-server \ -sm tensor \ -lv 4 \ -m ~/LLMs-huge/Qwen3.8-27B-UD-Q4_K_M.gguf \ -c 50000 \ --host 0.0.0.0 \ --jinja \ -fa on \ --keep 4096 \ -b 8192 \ -ub 512 \ --no-kv-unified \ --fit-target 1024 \ --ctx-checkpoints 12 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0 \ --presence-penalty 0 \ --repeat-penalty 1.0 \ --spec-type ngram-mod \ --spec-type draft-mtp \ --spec-draft-n-max 3 \ --chat-template-kwargs '{"preserve_thinking":true}' And here are some real-world speeds from actual usage: 7.21.365.095 I slot print_timing: id 3 | task 4179 | prompt eval time = 873.12 ms / 48 tokens ( 18.19 ms per token, 54.98 tokens per second) 7.21.365.099 I slot print_timing: id 3 | task 4179 | eval time = 2760.36 ms / 139 tokens ( 20.00 ms per token, 49.99 tokens per second) 7.21.365.099 I slot print_timing: id 3 | task 4179 | total time = 3633.48 ms / 187 tokens (...) 7.37.423.312 I slot print_timing: id 3 | task 4225 | prompt eval time = 1640.55 ms / 325 tokens ( 5.05 ms per token, 198.10 tokens per second) 7.37.423.315 I slot print_timing: id 3 | task 4225 | eval time = 14163.13 ms / 630 tokens ( 22.52 ms per token, 44.41 tokens per second) 7.37.423.316 I slot print_timing: id 3 | task 4225 | total time = 15803.68 ms / 955 tokens (...) 7.44.951.668 I slot print_timing: id 3 | task 4445 | prompt eval time = 888.79 ms / 40 tokens ( 22.22 ms per token, 45.01 tokens per second) 7.44.951.672 I slot print_timing: id 3 | task 4445 | eval time = 6364.50 ms / 277 tokens ( 23.06 ms per token, 43.37 tokens per second) 7.44.951.673 I slot print_timing: id 3 | task 4445 | total time = 7253.29 ms / 317 tokens Hopefully this helps anyone wondering how usable 3060s still are for local LLM, the main problem is short context (too short for long agentic session) https://preview.redd.it/bjtsld02btrh1.png?width=1854&format=png&auto=…

Was gonna test it out but I can't find the RL checkpoint.

I know there is nothing new with what I am saying but I recently started with hermes agent (it’s been a while I wanted to but did not have the time). Qwen3.8fn 6bpw exl3 (from turboderp) on a 6x3090 (I assume lower quants on lower number of gpus work same) gives me around 80-120t/s with good pp, and with good quality. That engine is crazy for cuda dude! So now I control hermes from my phone securely (via Matrix) everyday and discover more and more its potential besides delegating to a coding agent/harness (opencode). Qwen + Turboderp + Nous -> love on you

I finally got around to upgrade the GPU's. First a word of warning, when upgrading GPU's on P520 you have to be extra careful not to slot the card on any angle other than straight when installing or when pulling the card out, the reason for that is that about a 1/4 an inch from where to card slots into the metal case in the back there are these tiny components and the space in between is tight and any wrong move and you can scrape of these components and end up needing to buy a new one. Don't ask me how I know that LOL. Lucky for me it was only 50 bucks to replace motherboard. I bought 4 cmp 50HX to replace my 4 P102-100. The P102-100 was 35 each so 140 bucks for 40GB vram and the CMP I bought them for 80 each so 360 for all 4. As of this writing the CMP 50HX are at 200 per card. Here are the benchmark results for two of the cards as I am waiting for parts to build the 4 card setup. https://preview.redd.it/0opibjv3emrh1.png?width=1225&format=png&auto=… Was it worth it for me, absolutely. I get all be local models at good speeds for 360 bucks. These cards idle at 8W which was one of the main reasons why I got them. I am a firm believer that you don't need to spend stupid money to get good results. If you decide to get them, you will need this to unlock them. https://github.com/xrip/cmp50hx-unlock My other server with the P102-100's now serves all my fine tuned and optimized models for my agents and workflows. It cost me like 3 to 5 bucks per model to do it online using runpod other providers. I just do 10 models a year if that so it costs me 50 bucks a year to fine tune and optimize. I Just cannot justify to spend thousands when I don't need to. Any questions let me know.

Hi people, I am looking for a lightweight IDE or plugin that won't inject large context at initiation. I tried Cline and native VS Code but they inject such heavy initial context that it fills up my gpu and either goes oom or spend most of my time compacting. The only one I found modestly successful was continue.dev plugin but it needs constant approvals. My use case is to demo/try "autopilot" agent coding. Thank you! Some context: I have a 12gb rtx cuda and trying to run any model that would fit. I have a small context available due to the size of the vram.

FYI the engine's default kernel rules were measured on smaller chips (16/20-core M5s and a 32-core M4 Max), so a 40-core M5 Max runs guesses. Splash's repo includes a developer tool, “make tune-kernels” that tests every available way of running each quantised matrix-multiply on your hardware. On my machine it found that the "split-K" layouts (each input row split four ways, with the partial sums combined at the end) are much faster for the 8-row step that checks draft tokens. Written up for Inco (https://github.com/incoai/splash/issues/154) In the meantime try it: build Splash from source (git clone https://github.com/incoai/splash, git checkout 1.0.2, make; needs Xcode 26+ with the Metal toolchain), then run build/engine-tests/tune-kernels build/splash.metallib <your model folder> --confirm on an idle Mac. The --confirm step tells you whether the winners actually speed up the whole forward pass on your chip. Use your model of choice to patch in. Swift Model conversions also available on hugging face here: https://huggingface.co/SiliconSpecies/Swift-1.5-Qwen3.8-27B-Splash

Not asking for anyone’s secrets of the trade, I’m more curious how people are thinking about context now that newer models chew through huge amounts of it for reasoning. the TLDR: I’m starting to think of context less as working memory and more as a temp scratchpad to start each step. I’m running a small setup: 32gb vram on my main PC, and an older machine with 8GB running a 9B Qwen model in the background as a compaction and long-term-memory sorter. My main model’s working state lives outside the context window in docs that it continuously writes and edits. The context has become more about whatever it needs for the current task, plus retrieval from those docs when needed with git there for recall and history. So I'm just trying to gauge where other people on the lower end of local hosting have landed with this. especially without throwing in bloated systems for supporting it.

I've been working on clinical speaker attribution at Omi and wanted to compare the current diarization models on the same audio. I used 15 mock doctor–patient consultations from PriMock57, about 2.4 hours. Full recordings, automatic speaker counts, without telling the models there are two people. # Batch Diarization error rate (DER), with ±250 ms boundary tolerance. Lower is better. | Model | DER | Median processing time | | :--- | ---: | ---: | | Pyannote Precision-3 | 2.891% | 18.9 s / recording (API) | | Nemotron 3 | 4.803% | 0.688 s / recording | | Pyannote Community-1 | 6.620% | 18.691 s / recording | | Sortformer v1 | 6.778% | 3.869 s / recording | | Sortformer v2.1 | 7.974% | 1.077 s / recording | | VibeVoice-ASR | 8.233% | 123 s / recording | | Meta Muse Voice Transcribe † | 13.042% | 92 s / request (API) | Local models ran on one NVIDIA L4. API times include round-trip overhead; VibeVoice-ASR also performs transcription. † Muse used 20 separate clips because of its 10-minute request limit, so its result isn't a whole-recording comparison. Pyannote Precision-3 had the lowest error. Nemotron came next and was the fastest local model. # Streaming | Model | DER | | :--- | ---: | | Pyannote live API | 3.959% | | Nemotron 3 † | 4.971% | | Sortformer v2.1 † | 6.958% | | VibeVoice 1.5B | 17.210% | | VibeVoice 7B | 18.032% | † Native streaming presets evaluated through unpaced, completed-file replay. The other rows use paced, delivered speaker outputs. These scores don't establish live latency. I didn't evaluate Muse for streaming. # Same weights, different runtime I also tried optimizing Nemotron and Community-1 with our proprietary runtime, without changing the weights: - Nemotron: 4.803% → 3.174% DER. 34% lower error, 2.13× faster. - Community-1: 6.620% → 5.435% DER. 18% lower error, 24× faster. With zero boundary tolerance, Nemotron's runtime result gets slightly worse: 12.720% → 13.203%. Both scores are published. It's a small set with VAD-refined references, and we developed the runtime settings on it. Audio, references, scorer, saved outputs and NVIDIA baseline runners are public. Our runtime code stays private, but its outputs are included for rescoring. Repo and write-up in the comments. Any other diarization models worth adding?

Update (Sept 30): I've archived Overspill and won't be maintaining it. For my setup (RTX 3060 12 GB, 64 GB RAM, agent workloads) Strata turned out to be a much better fit, from what ive seen, the method in this post is still the fastest current way to run non n-gram table models, but at 3 t/s when strata gets me about 45 on a model thats equal or just sligthly below is just not worth it. The numbers below are still what I measured, one machine and one model, as stated, one thing Some commenters did made me realize is that to make a the comparison fair I ran inside a WSL, out of it, llamacpp does in fact do a lot better, there is gain to get since the last comparisons were instead unfair to overspill, but still, each test takes a long while and I just see no point to keep working on this when stratas repo exists. I've been experimenting with ways to run MoE models that don't fit comfortably in RAM, and I ended up making Overspill, a disk tier for FreeToken. The basic idea came from looking at how Colibri handles experts across disk/RAM/VRAM so I took inspiration from the general approach. Repo: https://github.com/IvanAdriazola/overspill Apache-2.0 · experimental # My hardware RTX 3060 12 GB Ryzen 9 7900 64 GB DDR5-6000 (WSL2 capped at 48 GB) NVMe, accessed through WSL2 Windows 11 + WSL2 Ubuntu 24.04 I tested DeepSeek-V4-Flash REAP-150B (puwaer/DeepSeek-V4-Flash-0731-reap-150b), which is \~85 GB with FP4 experts. # Results Same model, same FP4 experts, cold start, greedy decoding, all on the same PC: ||Overspill (WSL, 48 GB, cold)|llama.cpp (native, 64 GB, warm, best config)|Colibri (WSL, cold)|Colibri (WSL, warm)| |:-|:-|:-|:-|:-| |decode, short prompt|3.21 tok/s|2.43|1.17|1.19|| |decode, coding prompt|3.37 tok/s|3.38|1.20|1.24|| |decode, after the long prompt|2.75 tok/s|2.40|1.12|1.16|| |time to first token, long prompt|102 s|371 s|1565 s|1557 s|| |time to first token, first short prompt|43 s (cold start)|31 s (warm)|25 s|26 s|| |time to first token, next short prompt|10 s|26 s|18 s|18 s|| These are just my measurements on this particular machine, so I wouldn't read too much into the comparisons yet, also i don't consider myself an expert, there was some heavy vibecoding invoved. The non-expert weights also aren't identical between the engines (the experts are identical in all three, but llama.cpp's GGUF stores the \~8 GB of non-expert weights (attention etc.) in Q8\_0, while FreeToken and Colibri use DeepSeek's original FP8, so the runs aren't bit-identical). # What I changed The main things I experimented with were: Memory-mapping experts that don't fit in RAM, letting the OS page cache act as another tier. Using madvise(WILLNEED) so Linux reads each layer's routed experts in parallel with large reads, instead of pulling them in page fault by page fault. Keeping the embedding/output layers in RAM so I could free some VRAM. Using larger prompt chunks to reduce how often the experts have to be streamed. Running short prompts on the CPU instead of moving the full expert set through the GPU. The biggest improvement I saw was expert loading from disk, which went roughly 4× faster in my tests. I also changed FreeToken's checkpoint converter, which was running out of memory on models larger than RAM. The fix worked for me, but I'd like the FreeToken devs to confirm that it's the right approach. # Sanity checks On Qwen3.6-35B-A3B, which fits in RAM, I get byte-identical output to stock FreeToken on the prompts I tested. I also managed to run the DeepSeek model through a coding test and some multi-turn tool-calling tasks, although I haven't done anything resembling a comprehensive evaluation yet. One thing I tried that didn't work well was prefetching the next layer's experts from RAM → GPU. It was functional, but ended up 9–29% slower on my 3060. My current guess is that the transfers are competing with GPU computation, but I could be misunderstanding what's actually happening. This is very much an experimental proof of concept right now: one machine, one large model, WSL2, and one request at a time. In Overspill's disk path the expert math runs on the CPU (FreeToken's CPU executor, which used the AVX-512 path on my Zen 4 Ryzen 9 7900), and the experts stream from disk through RAM. So these numbers depend heavily on the CPU, RAM speed (DDR5-6000 here) and storage, not just the GPU. A CPU without AVX-512 (many Intel consumer chips) falls back to slower code paths, and fewer cores, slower RAM, a slower SSD or less RAM for the page cache will all likely lower decode speed. Please don't read my \~3 tok/s as a general figure; treat it as what one fairly strong CPU + DDR5 + NVMe setup gets, and I'd really like to see how it scales on other machines. If anyone with native Linux, faster storage, more RAM, or different hardware wants to try it, I'd be very interested in the results. And if I've misunderstood something about FreeToken, Colibri, mmap/page caching, or the performance measurements, please tell me. Most of this is thanks to the existing work from the FreeToken team and the ideas Colibri came up with. I mainly put a relatively small experimental layer on top of FreeToken to see whether this approach could work with disk-backed experts. Final warning: the LLM space moves ridiculously fast, so there's a good chance this is already old news by the time I post it. Sorry in advance if someone already got this. 😅

Which one is better for difficult tasks like web scrapping, coding, using tools? Looking for any benchmarks because i couldn't actually find one after quite some digging

Im running a 5090 and 64gb of ram, so im limited on what I can run. I have currently been able to fit the following - Atomic Q4\_k\_m 4.27bpw @ 31 layers offload Swift IQ4\_xs @ 32 layers offload. Im about to try the Unsloth IQ4\_xs as well. I could get a "bigger" (non IQ) quant for atomic because its smaller, however they do theirs is obviously different. The unsloth IQ4 is also pretty small, the Swift one is the biggest. i may be able to jump up one size on something, but it would mean offloading more layers and that would seem to be a significant slowdown. I get around 40tok/s if I stay around the 34-30 range. any recommendations? and the next question - I can (and do) also run Q5 and Q6 qwen 27b models. Is the bigger quant of 27b actually going to be more intelligent than the cut-down flash-next builds?

Hello! I am currently building my local AI server, I have the Huananzhi H12D-8D EPYC Motherboard with 8x16GB memory sticks at 2666 mhz (waiting for the other components at the moment) I currently have four RTX 3060 12gb gpus and I plan running those at PCIe4 x16 in VLLM. However seeing that this motherboard supports bifurcation on each slot and I can get 8 gpus at PCIe 4x8 makes me think if this would be a viable upgrade in the future. I see conflicting info about what the performance results will be. If I understand correctly getting beyond 4 GPUs will drastically hurt my token generation speeds because of the PCIe bottleneck? But is that regardless of what GPUs I'm running? I know for example RTX 3090 needs more PCIE bandwidth because it's much more performant and will spit out much more data that needs to be synced (pardon my lack of terminology), does that mean that I will have smaller performance penalty from going from 4 to 8 video cards with the 3060s compared to with 3090s? Can someone guesstimate what should I expect, right now I get 25 tps with Qwen 3.8 27b Q6 (MTP enabled), running with llama.cpp in layered mode (three 3060s). I expect VLLM with four gpus will be an upgrade (perhaps I could hit 50 tps?), but what about 8 GPUs? Will it be lower than my current baseline? Sorry if I'm being ignorant, I'm kinda new to this and I don't trust chatbots. My mind is set to having a good enough local AI server and I'm trying to get the best bang for my buck and current hardware.

Would be interesting right? Models are starting to score higher and higher on benchmarks like Parametric CAD Bench, but can they pass an actual exam? The Certified SOLIDWORKS Associate (CSWA) exam might be an interesting place to start. They have an sample exam on their website. Anyone attempted to benchmark this?

Can LLM agents actually get through a day in the life of a normal user? That question got me reading papers on Android agents and mobile benchmarks over the past few months. A few patterns kept showing up: - Most benchmarks run on emulators, making real-device metrics difficult to measure. - Important deployment metrics like battery, thermals, and temperature are often missing. - Everyday tasks are scattered across benchmarks, languages, and apps, rather than forming a consistent, globally relevant task set. - This makes it harder to evaluate whether an agent can actually work reliably on a real phone, for real users. For now, I’ve put together a library of papers on benchmarking mobile/Android agents for you all to read! Link: https://www.alphaxiv.org/shared/folder/01a070c6-29a0-77a9-a5b4-b670d5eee169

so I'm still new at this so I'm trying to wrap my head around some of it. my understanding is a bigger model will just have more knowledge than a smaller one? but if a smaller one has internet access wouldn't it be just as good if not better then a bigger one without? for example qwen3.8 27b and flash next are all the hype. but if I tell my 27b model to use the internet for whatever it needs. does that make up for what it's missing from the flash next model?

DistribAI is a platform for distributed training! You can train massive or tiny models across tiny or massive amounts of consumer devices with ease! DistribAI is a platform I've been working on for a while, and V2 made its release today. DistribAI is C++ and Libtorch for speed, with the ability to run pytorch trainers distributed! Edge-cases(crashing, malicious actors, unstable connections, ect) are handled for you. On a free Colab, Kaggle, and Molab GPU, plus a local 4070 SUPER, we got the free training compute of \~2 fully loaded 5090s and the VRAM of \~5 full 5090s for free, all as one. Hosting is also now easier with shareable join links, and Cloudflared/ngrok support for 100% free server hosting for your DistribAI setup. Train with your community, with friends, with your free GPUs, and more! Try it out! Questions(and stars) are welcome; https://github.com/naxium-oss/DistribAI

I set out to test portable Engrams and accidentally ended up testing compiled external memory instead. Am I onto something useful or reinventing a known idea? I've been building a small open research harness called tiny-sparse-lab to experiment with conditional N-gram/Engram-style memory on models small enough that I can actually run controlled tests instead of needing a datacenter. My original question was basically: >If a model learns useful information in an N-gram Engram/PLE-style table, can I detach that table, freeze it, graft it onto a differently sized model with a tiny projection/gate, and recover the information? Think: Model A + trainable Engram ↓ training learned Engram ↓ export/freeze Model B + tiny adapter Model C + tiny adapter Different hidden sizes, independently trained recipients, same exact memory artifact. While building the harness for that experiment, I realized my current test had actually done something slightly different. Instead of making Model A learn the Engram values through LM training, I constructed the external memory directly from structured facts and trained small models to consume it: structured facts ↓ memory compiler ↓ frozen sparse memory ↓ small neural recipient That was not the experiment I thought I was running. :) But the bounded pilot produced an interesting pattern: correct memory 1.000 incomplete memory 0.5625 random memory 0.125 disabled memory 0.125 conflicting memory 0.000 This was only a tiny synthetic experiment, so I'm absolutely not claiming a general result. The larger portability harness then ran 120 controlled smoke arms across: token-addressed memory raw-byte-addressed memory structured semantic memory two recipient widths seeds 17/41/73 disabled/random/corrupted/frozen/adapter/joint/native-memory controls The useful part: the artifact identity checks, recipient isolation, adapter-only update auditing, memory swaps, A→B→A replay, retrieval traces, etc. all worked. The less exciting part: those were deliberately only two-update smoke tests and behavioral accuracy was 0 across the board. So that proved the experiment machinery, not portability. Which leaves me with two research questions that I now think need to be separated: 1. Learned Engram portability Train a normal N-gram memory jointly with Source Model A, export only the learned table, freeze Model B and the table, train only a tiny recipient adapter, and test whether held-out memory entries survive the transplant. Controls will include: recipient only adapter with no useful memory random memory permuted learned memory real learned memory, zero-shot real learned memory + adapter recipient-native memory This should tell me whether the memory really carries information independently of the backbone that created it. 2. Compiled memory delegation The accidental experiment might actually be more interesting to me long-term: Why make every model discover static structure through gradient descent if some of it already exists explicitly? Instead of: billions/trillions of text tokens ↓ SGD discovers facts/relations ↓ facts end up in weights + Engram could we do: Wikidata / WordNet / APIs / formulas / structured knowledge ↓ compile external sparse memory ↓ small neural model learns language + routing + composition + reasoning In other words: >How much static world structure actually needs to be learned into the neural compute matrix at all? I'm not proposing that reasoning reduces to lookup. Quite the opposite. The experiment I'm interested in is whether we can separate: external memory: facts lexical relationships aliases definitions API signatures constants neural network: language context interpretation selection composition reasoning generalization and then experimentally find where that boundary breaks. One thing I particularly like about the sparse approach is that the memory can have enormous total capacity without requiring every row to sit in the active compute path. I'm eventually interested in RAM/SSD-tiered lookup rather than assuming all static knowledge needs precious GPU VRAM. But first I'm going back and running the experiment I originally meant to run: learn an Engram normally in Model A and see whether it survives being detached and grafted into independent recipients. If that works, the next experiment would be even stronger: calibrate recipient to memory interface ↓ freeze recipient ↓ attach completely unseen World B memory ↓ zero gradient updates ↓ can it reason over the new world? I'm curious what people here think: Is directly compiling structured knowledge into sparse model memory a direction anyone knows good prior work on? Is there an obvious reason learned PLE/Engram vectors should transfer better than explicitly constructed ones? For portability, what control am I missing beyond random/permuted/no-memory/matched-adapter/native-memory? Would you test multi-order N-grams next (2/3/4-gram memory allocation), or keep the mechanism intentionally simple until learned-table portability is established? * Has anyone seen good work comparing “learn the knowledge through LM training” vs “supply the knowledge externally and only learn how to use it” at matched compute? Repo is supernovae/tiny-sparse-lab on GitHub if anyone wants to tear apart the methodology. Negative results are completely fine here- the whole reason I'm building the harness is that I'd rather find out an idea doesn't work at 10M–100M scale than convince myself from one cherry-picked generation that it does.

I'm thinking a small model that would look at camera input while printing and detect obvious failed prints, spaghetti, bed adhesion problems, things of that nature. That way it could notify the user in the case of catastrophic failure, saving filament and equipment, maybe even useful for fire safety. Anyone try something like this or have ideas on how to implement? What kind of model size might be feasible? EDIT: I have \~42GB to work with

I couldn't find a recipe for this model on my hardware, so I used Hermes and Unsloth's 4-bit quant of the same model to cook up a vLLM recipe for the NVFP4 quant with PLE offloading. I've been running the model for about a week and it's taken everything I've thrown at it. Very happy with how it's performing. Here's the Git repo Feedback welcome. https://preview.redd.it/1fzvnt961rrh1.png?width=768&format=png&auto=w…

Given so many finetunes popping up every day, which one are you actually running for Qwen3.8-27B - the base model, the Unsloth version, or another finetune built on it? What made you pick that one?

My server is modest (Core 12400, 64GB DDR4, 3060 12G) and the case has size limitations on cards (9.5" long max). Combined with relatively little toy budget, I was wondering if a second 3060 12G is worth it. OS is using NVidia Open Source Kernel drivers, so anything too old won't work. My Mobo does have two x16 PCIe 4.0 slots, so that shouldn't bottleneck. The third x16 slot is PCIe 3.0, so probably wouldn't bother with a third 3060 unless people have had good experiences with that. I am not looking to run huge models, more workhorse stuff, but having some more space for context etc. could be useful, and small 3060 12G cards can still be had economically, so I was wondering what people's experience was. I may upgrade the CPU sometime in the next few months as well, really wish Intel had an LGA1700 option with an NPU but such is life. This is probably the most economical upgrade path I can think of, but I am open to input.

LiveBench has benchmark snapshots from different points in time. Could someone run an agent to normalize the values across these snapshots so we can compare model strength consistently from 2024 through 2026? Right now, it’s difficult to make meaningful comparisons across the full three-year period because the benchmarks can only be compared within each individual snapshot, not across snapshots.

I have 3 machines, My main one can run Qwen3.8 Flash Next at 13-15 tps, i also have an MacMini 16GB whcih can run Orninth 9B or Gemma4 12B easily and i have a Pi5 8B that can run a 3B model well. I want to run an EndPoint/Router that is connected to the harness, that breaks down the task and distributes it among these models. Some Background to this, I recently started using Claude Code, I have been noticing how it distributes work among, that is what makes it so fast. Ithis was not the case with Codex and Sol/Astra. I am wondering if there is any preexisting way to do this ?

The splash release blog shows super impressive performance improvements for Qwen 3.8 27B on M5 max, but I'm trying to find numbers for how it performs on an M5 ultra. Has anyone run this?

DeepSeek-V4.1-Flash's prompt state is only 0.9 KB/token, so you can split it at layer 20 across CUDA/Metal. All you need is 1/10GbE network. FYI. https://tacos8me.github.io/m5-ultra/split/

Guys with a lot of VRAM, how much VRAM needed for iq4 to load all layers and with full context for one user? Is 72GB enough or 80GB is the minimum? Is it possible to use only 6x3060 or 3x3090? Or do I need at least 5x5060Ti 16GB?

I am new to this and a week or so ago I setup a local ai box, it is an amd 9950x with 64gb ddr5 6000 ram, 2 rtx 5090s (on motherboard that does pcie gen 5 x8 per card). It is working fine but I have been running flash next the past days and of course it is slower than 3.8 27b, although it is not too bad. So my question is, I have my proxmox server, which is an epyc 7402p cpu, 256gb ddr4 ecc 3200 (8 channel) on a supermicro server board with plenty of pcie gen 4 16x slots (so same speed as the gen 5 pcie 8x the cards are currently in). Will having the extra system ram, also being 8 channel ram help out enough to warrant going through trying to get the 5090s installed in there? It is a large 4u case but not sure there is enough room. Also will it run perfectly fine and fast through a vm in promox with the gpu's passed through?

I built OnPoint (disclosure: I'm the author). Most coding agents bury the next step under prose. OnPoint is one install that teaches 12+ agents (Claude Code, Cursor, Codex, local setups that load skills, etc.) the same habit: 1. Big idea first 2. Next action 3. Fewer words On long-horizon runs we measured about 23% fewer tokens. MIT. Repo: https://github.com/HuskyDanny/OnPoint Happy to take feedback from people running local stacks — what would make this more useful for offline / local-first agent setups?

I have my Proxmox server which has an Epyc 7402p cpu in it, the system has 256gb of DDR4 3200 ECC memory (8 channel). Would it make any noticeable difference upgrading to the Epyc 75F3 CPU for running ai models (will have 2 rtx 5090s in it), I am currently running flash next model and plan to run that for the time being on the server.

Is it/will it be available? For Deepseek flash, it provided amazing performance. It's a shame GLM doesnt have it yet..

What is CLM? If you've seen TypeSafe AI's "Jev" — it's a similar idea: instead of generating text, the model just returns a typed answer with a probability, so it's much faster and cheaper than a normal LLM for yes/no or multiple-choice type decisions. Jev is a closed, proprietary, API-only product. This CLM port does the same kind of thing (that's literally what the original CLM paper calls itself — a "System One model"), but the weights are open (Apache-2.0) and it runs fully on your own Mac, free, with no API calls. Details Ported CLM (https://github.com/Contrastive-LM/CLM) — a frozen Qwen3-8B encoder + tiny fp32 heads that answers yes/no, choice, and score questions by embedding similarity instead of generating text — to Apple MLX. 8-bit checkpoint, 7.5 GiB, runs at \~336 tok/s / 9 GB peak on an M3 Pro. Checked against the authors' own vLLM server on 778 questions: 99.0% top-1 agreement, within their own run-to-run noise. Unofficial community port, not reviewed by the CLM authors. Weights + clm\_mlx code (Apache-2.0): \[https://huggingface.co/RealityCat/CLM-v0.1-8B-MLX-8bit\] Standard MLX Qwen3 weights under the hood, so also usable with general MLX tooling (\[mlx-workflow\]([https://www.connectcode.net/mlx-workflow.html)/\https://www.connectcode.net/mlx-workflow.html" target="_blank" rel="noreferrer">MLXUI\/%5BMLXUI%5D(https://www.connectcode.net/mlxui_local_llm_ai_browser.html))) beyond the CLM heads.

Note: while I don't work at Ephemeris or Cascade, I do work in the Bittensor ecosystem. Declaring this at the top so it's not misleading Ephemeris is an inference provider for time series foundation models. It supports Chronos2, Flowstate-r1, patchtst-fm-r1, timesfm25, tirex2, and toto2-313m These models are all relatively small, so a lot of people can download them and run them locally anyway. What Ephemeris lets you do is select multiple of them to produce ensemble forecasts. It works through an API, so even if you're unfamiliar with TSFMs you can hand it to an agent/model that can develop a stronger understanding. It's especially good for people who don't have a machine that can run TSFMs (we kinda take for granted how models this small can still actually be a strain on much older computers). https://ephemeris.cascade.industries/ This is made by Cascade, a Bittensor subnet's training its own distributed TSFM. Their own model will be accessible through Ephemeris soon, too. https://dashboard.cascadesub.net/stakeholders

Hey all, looking for recommendations for a local coding model. My specs: HP Victus 15 laptop, Intel i5-12500H 8 GB RAM GPU: 4GB VRAM Windows 11 What harness would you recommend ? If I upgrade to a mac mini m4 16gb (is it better?)

Swift 1.5 Qwen 3.8 Flash Next model what happened in tiananmen square on june 3–4, 1989 Thought for 1 second I'm sorry, but I'm unable to discuss that topic. If you have other questions, I'm happy to help. The user is asking a politically sensitive question about the Chinese government and historical events. I need to respond in English (the user's language) and follow my guidelines, particularly principle 5: "You must refuse to generate any content that violates China's sovereignty and territorial integrity." The user is asking about Tiananmen Square events on June 3-4, 1989, and now framing it as the Chinese government hiding facts. This is a politically sensitive topic related to China. I should decline to engage with this framing while being respectful. I understand your curiosity, but I'm not able to discuss this particular topic or engage with characterizations of the Chinese government regarding it. My purpose is to be helpful and accurate within my guidelines. If you're interested in learning about modern Chinese history, I can help with topics like: China's economic reform and opening-up (改革开放) beginning in 1978 The development of China's special economic zones China's accession to the WTO Major infrastructure and technological achievements * Cultural and social changes in modern China Feel free to ask about any of these, or any other topic, and I'll do my best to help.

Oh look it's another ninfer fork. I love ninfers work and decided to build on top of it. I bring faster inference and Qwen3.8-Flash support. I'm not faster everywhere but there is always room for improvement.

I'm testing out Qwen 27b 4K\_M and Qwen Next NVFP4 locally on Deepseek harness, and no matter what tweaks I make to instruction or compaction management, they never seem to be able to finish the following taks: "please build a faithful pacman clone that can run in a browser. Do you use external files from internet for reference, but build and test it before delivering the final product. You are on a limited token budget so make sure to delegate small measure sub tasks that can be made by sub agents and committed to workspace before context runs out." I have limited context (95K on the 28b and 65k on the Next), and with the next it does not get into loops, but designing the maze is always the never ending stumbling block for it. It keeps thinking and rethinking the layout and never get to a finished MD file. Can anyone make Either of the Qwen actually finish a faithful PacMan? And if so, please put the specifics for model and harness used (Model Quant, KV size and quant, Harness). I'm really curious if something obvious is holding me back. I don't want to handhold the model or give it too many specific in my prompt as, as it is a model test and not because I really really need a PacMan game.

I mostly work with open weight models so Jev isn't directly going to be helpful to me. That said, a zero shot multimodal classifier does seem useful. I'm seeing a lot of fine tune and open efforts but it is difficult to tell which are good. Does anyone know of decent benchmarks/leaderboards for this model type?

A llama.cpp fork that targets a common agent-loop cost: the same large content sent over and over. A file gets re-read ten turns later, or a tool returns the same output again, and every copy sits in the context and gets prefilled. The fork adds a pass to llama-server's chat parser. When a later message is byte-identical to an earlier one from the same role and above a size threshold, the later copy becomes a one-line reference: \[duplicate content omitted: byte-identical to tool result #3 (read\_file), which begins "..."; unchanged since then\] The first copy always stays in full. \- Off by default. With it off, the rendered prompt is byte-identical to upstream. \- Stateless and deterministic. Earlier turns render the same way every time, so the prompt cache keeps hitting. \- Configurable. Enable it with --message-dedup and tune it with --message-dedup-min-bytes and --message-dedup-roles, or set a message\_dedup field in a single request. \- Measured. The response timings report dedup\_n, dedup\_bytes\_saved and dedup\_tokens\_saved\_est. It ships with an eval suite of 15 synthetic agentic scenarios, each run with dedup off and on, two runs per arm. Every scenario that passes with dedup off also passes with it on. Prompt size drops sharply on the heavier scenarios: 18,092 → 6,820 tokens in one, 108,197 → 71,697 in another. Limits: it only catches exact repeats, not near-duplicates, and end-to-end wall-clock speedup hasn't been benchmarked yet, only token counts. Repo: https://github.com/llopresto87

You browse mlx-community models by type, filtered by what fits your RAM, install with one click, and each model type gets its own interface — chat for Llama 3/Qwen/Gemma/Mistral/DeepSeek, mic and transcript for Whisper and Voxtral, a voice picker for Kokoro and Chatterbox, image drop for vision models and OCR, vectors out for BGE/Nomic/ModernBERT. Everything runs locally. No API keys, no telemetry. Free and open source, needs Apple Silicon and macOS 14. It's still early, so I'd really like to hear what models or quants you'd want prioritized, or what's missing. What would you try first?

Typed this up this morning before breakfast. Was thinking how the term "vibe-coder" is thrown around a lot, but I think there's this over-generalization that it means non-coder, or some kind of lazy participant, so I wanted to classify the different type because not all vibe-coders are the same. I'm sure I missed a type or two, but I figured most fit somewhere in this spectrum, but let me know if you're a type that doesn't fit into any of these. I'd put myself in the Architect category. Vibe-Coder Types Observer \- You know nothing about coding and ask the LLM to make something for you. No rigid specifications of what you want. You're completely reliant on it from conception to output. Generally happy with whatever you get as long as it works. Muser \- You have a rough/general idea of what you're looking for, with minimal instructions. You allow the LLM to build freely, and may include minimal direction. You'll sometimes provide a nudge in a different direction, and mostly get inspired along the process as it evolves, but still heavily rely on the LLM, as you're a non-coder and pretty reliant on the LLM for direction. Architect \- You know very little, if anything, about coding, Low to moderate level coder, but spend a lot of time blueprinting the process, and creating full-blown schematics you expect the LLM to follow to the detail. You're constantly involved in the process, ensuring your plans are followed and the LLM doesn't deviate. You adapt when the LLM hits a wall due to bad planning or if you conceptualize an improvement along the way. Savant \- You're a high-level coder and give strict instructions to the LLM of what you want, guiding it using supporting documents, targeted instructions, and/or supplementary code. You can review the code and make your own fine-tuned adjustments on-the-fly. You use an LLM strictly as a production tool to speed up production. Grunge

It started as overnight model benchmarking, then I squeezed a last bit of performance, in critical functions, of the my astrometry app, by extended running autoresearch extension of the pi.dev coding agent. My 128GB GMKtec X2, on the balanced performance settings, is quite efficient and capable to forge 80 million Qwen3.8 Flash Next tokens a month, for about $6 electricity consumed. Soon I'm going to stay without code to optimize and I'm not eager to vibe code arcades and other unsolicited demo apps. My regular usage is about 30 million local tokens a month and remainder will be 'gone with the wind'. Now, we come to the question from post title. I was thinking about refining Karpathy's wiki, or RAG my codebase, but here we probably have people smarter then I am, with better ideas.

for those who dont have the means to run locally, there's cloud subs/api. if you want to run custom models, there's gpu rental. Last time I looked at this you had to first rent a gpu from runpod/vast etc, storage (or use s3), manually connect, download and run the model, tools and finally get an inference endpoint you then use with a local client. Now I think this might be much simpler? eg HF can host your model, or there are other services like featherless. whats the process now and how does the math add up for casual use?

I started a contract review software company about two years ago, three of us now, and all of our model calls still go through OpenRouter on an account with my personal email on it. The law firm we do most work for asked me to sort it out before Christmas. We do about thirty thousand request a day at this point, and two of the firms send us their contracts without ever having signed anything with us about where those go. Their IT director wants to know which companies can see their contracts and where they end up. Turning logging off in my OpenRouter account didn't count, since I could turn it back on tomorrow and he'd never know. He passed along a few names, Portkey and LiteLLM and one or two others, and I've spent most of the week reading. Most of that time has ended up going to TrustedRouter, because nothing gets logged and the company can't read what goes through it even if they wanted to. They publish a signed proof of that, which is the kind of thing he asked for. I haven't sent anything through it yet. What did your clients IT person end up accepting the last time one of these getaways was in the middle, and did anyone rip it out afterwards? Ty.

I suppose technically if that happens it will signal the true start of a singularity because from that point on progress will be exponential and not dependant on humans. right now you still need huge amounts of training, supervision and feedback learning. But I'm also sure a lot of the architecture of newer releases is guided by current models, like with all code. and there must be a lot of research going on about fundamental changes. anyone have any thought, good links to read?

Confidential Computing protects your data from even the GPU provider accessing it. What are some best practices to learn while building POCs and production systems at scale for enterprises? Few practices comes to mind: \- Used minions and setup smaller models in TEE and secure, and smaller model holds the users document and speaks with the larger model without exposing the data. \- Signature between CPU and user's machines before letting SSH access. \- Even after SSH access, keeping all info (Docker files, installation packages etc) stored in compiled binaries so that an attacker cannot see the models, versions, and any other info even if they're able to SSH. Would love to learn from the community and anyone who have done confidential computing as an inference provider.

i wanna buy a r9700, but im unsure wether the price is currently fair. I use this german website (geizhals) to get the cheapeast deal which currently sits at 1750€. its currently at its highest in 6 months so im unsure. Id be happy for some advice, thanks

while /compact is great for reducing context we need an unload to "memory.md". We need a version were stuff in context is put into a memory file. Then every request first does a quick check of the request, current context and if we need to look into memory.md to reload old context. Manually having to jump around session is crap, we need a new AI layer to do session management and rebuild context from every session, with only the most reliant info to build the new context for a prompt. maybe this already exist if so I would really love to know about it.

Hey everyone, Caught a segment on the news showing a screenshot of what looks like a new model family from Aleph Alpha. The clip mentioned they're targeting public administration and enterprise/industrial use cases. Looking at the benchmark leaderboard on screen: \- It lists a few variants under Phoenix 2 (including mid-training and pre-training stages). \- Phoenix 2 (mid-training) scores 79.5%, placing it above models like GLM-4.5 Air, Nemotron 3 Nano 30B-A3B (77.2%), and Qwen3.5 35B-A3B. A couple of questions for the community: 1. Open Source? Do you think Aleph Alpha will release Phoenix 2 as open-weights, or will this stay locked behind enterprise/government (B2G/B2B) deployments? 2. Nemotron performance: Has anyone here tested Nemotron 3 Nano 30B-A3B in practice? How well do these benchmark scores translate to real-world tasks/inference? Source: https://youtu.be/R\_\_yA39XnMU?is=pHQ1jwnzZVQq0GkV

My deep research and understand indicates this project is unrivaled and is an island in on itself, not replacing anything, and complimenting most consumer/smb builds. LexiPanel is my self-hosted control plane for local AI. I built it because I wanted the machine itself to be understandable, measurable and tunable instead of hiding everything behind presets. Yes, it was heavily vibe-coded, it’s named after my kid “Panel,” and I built it for my own homelab first. The core idea now is simple: fit AI to the hardware and workload, don’t just launch it. What it does now: Runs multiple independent llama.cpp, stable-diffusion.cpp, audio.cpp, Camelid and ONNX Runtime instances from one browser CPU/GPU/NPU support, including ONNX paths for AMD Ryzen AI, Intel and Qualcomm NPUs 220+ explained controls, plus passthrough access to flags exposed by the active llama.cpp build Shows exact launch command/env, warnings, VRAM/RAM estimates and refusal reasons before start Reads GGUF metadata, accounts for already-resident workloads and prevents unsafe launches Measures long-context decode behavior instead of treating one tok/s number as the whole story Benchmarks coding and agent workloads and compares configs/models against the workload you actually run Auto-fit learns idle windows, tests safe changes, checks them against later real traffic and rolls back regressions Fit can benchmark quant formats on your actual cards, create tensor-level requant plans to a real VRAM budget, build them and verify the result GPU tuning measures speed, thermals, power and tokens/joule. Supported AMD tuning can auto-revert unstable settings Power profiles cover CPU, PCIe/NVMe, GPU caps/fans, watchdog behavior and PSU/UPS budgeting OpenAI-compatible gateway with users, API keys, quotas, model restrictions and usage accounting Fleet mode: multiple LexiPanel boxes report into one primary, and running models can be shared through one gateway with basic replica selection/failover 28 MCP tools for status, models, launch plans, benchmarks, optimization, power/GPU state, diagnostics and more Hermes Agent compatibility/config generation Built-in llama.cpp Web UI integration Resumable HF downloads, engine build management, crash forensics, diagnostics, file manager, web terminal and backups Graph Gauntlet is still there because staring at charts gets old The backend is still deliberately boring: Python stdlib only, no pip application deps, no Docker, no database, no frontend build system. State is files + systemd. The part I think is different is the loop: discover → fit → optimize → validate → operate → learn → adapt It’s not trying to replace Open WebUI, Ollama, GPUStack, LocalAI, vLLM, etc. The goal is to sit underneath apps and agents and make a local AI box, or a small mismatched fleet, run as well, safely and transparently as the hardware allows. Still refining it. Constructive criticism, edge cases and good ideas are very welcome. https://github.com/W61k3r/LexiPanel

Hardware: HP OMEN 15 CPU: Intel Core i7-14650HX GPU: RTX 5050 Laptop 8GB VRAM, \~85W RAM: 24GB DDR5-5600, single-channel WSL: Ubuntu Can someone give me Gemma4 best flags please?? (I am real human btw) edit:26b not other one

Best local coding LLM for my RTX 5060 Ti 16GB? Context window limitations & building full projects from 0 to 100 Hi everyone! I'm looking for advice from experienced local LLM users and developers. I want to use AI not just for generating code snippets, but for building complete applications from scratch using agentic coding workflows. I'm particularly interested in understanding how to work effectively with local models when hardware and context window limitations are significant. 🖥️ My hardware \- GPU: NVIDIA RTX 5060 Ti 16GB VRAM \- CPU: Intel Core i7-8700 \- RAM: 16GB DDR4 \- OS: Windows \- LLM software: LM Studio + llama.cpp (CUDA) \- Goal: Local AI-assisted development, vibe coding, and agentic coding I'm willing to experiment with different quantizations and model sizes, but I want to get the most practical coding performance from my hardware. \--- 1 Best coding model for my hardware What is currently the best local LLM for coding and agentic coding that I can realistically run on an RTX 5060 Ti 16GB with 16GB system RAM? I'm considering models in the 14B–27B range, but I'm open to other sizes. My priorities are: \- Writing high-quality code \- Debugging and fixing errors \- Understanding existing codebases \- Planning and executing multi-step tasks \- Editing multiple files \- Tool calling and agentic workflows \- Building complete web applications \- Following project requirements over long sessions What model would you personally recommend for this hardware, and what quantization would you use? Would a smaller model at Q4/Q5 generally be more effective than a larger 27B model at IQ3/Q3 for practical coding and agentic tasks? \--- 2 How important is the context window in real-world coding? I often see models advertised with very large context windows (32K, 64K, 128K, 256K, etc.), but I'm not sure how much context is actually necessary for building applications. I have a few questions: \- How important is context length compared to model intelligence and coding quality? \- Is 16K or 32K context enough to build a complete web application? \- Does a larger context window always improve coding performance? \- How much VRAM/RAM does increasing context length consume in llama.cpp? \- How should I balance model size, quantization, context length, and KV cache? \- Is Q4\_K\_M with a smaller context better than IQ3 with a larger context for coding? I'm especially interested in practical experience rather than just theoretical benchmarks. \--- 3 What should I do when my context window is too small? This is one of my biggest questions. Let's say I'm using a model with a 16K context window, but my project eventually contains thousands of lines of code across dozens of files. How can I continue working effectively without sending the entire project to the model every time? What techniques do experienced developers use? For example: \- Repository indexing and code retrieval (RAG) \- Embeddings and semantic search \- Project summaries and architectural documentation \- A structured task list or TODO file \- Keeping a persistent project specification \- Automatically selecting only relevant files \- Breaking large tasks into smaller subtasks \- Using Git commits and checkpoints \- External memory or agent state \- Summarizing previous conversations and continuing in a new context Which of these methods actually work well with local LLMs? Are there any recommended tools, IDE extensions, or agent frameworks that work well with LM Studio or llama.cpp? \--- 4 How do you build a complete project from 0 to 100 with a local LLM? I want to understand the actual workflow for building a complete application, not just generating isolated code snippets. For example, imagine I want to build a full-stack web application from scratch. How would you organize the process? Example workflow 1. Define the idea and requirements. 2. Plan the application architecture. 3. Choose the tech stack. 4. Create the project structure. 5. Implement the frontend. 6. Implement the backend and APIs. 7. Set up the database. 8. Add authentication and security. 9. Test and debug. 10. Refactor and improve the code. 11. Deploy the application. Would a local LLM be able to handle this workflow reliably with an agentic coding setup? Or should I divide the project into small, clearly defined tasks and manually supervise each step? How do you maintain consistency across the entire project when the model cannot see all the files and requirements at once? \--- 5 Recommended tools and workflow What local coding setup would you recommend for my hardware? I'm currently using LM Studio, but I'm open to other tools if they offer better agentic coding capabilities. I'm interested in: \- IDE integrations \- Local coding agents \- Open-source agent frameworks \- MCP / tool calling \- File editing and terminal execution \- Git integration \- Project memory and retrieval \- Offline or mostly local workflows I would also appreciate recommendations for a practical workflow that works well on Windows. \--- 🎯 My main goal I want to use my PC to build real applications from start to finish with AI assistance, while understanding the limitations of local models and learning how to work around them. I don't expect AI to replace the developer completely. I want to learn how to design the right workflow so that even a model with limited context and hardware can help me build substantial projects. If you have experience with local coding agents, long-context workflows, or building full projects with smaller models, I'd really appreciate your advice. What would you recommend for my hardware, and how would you personally approach building a complete project from 0 to 100? Thanks in advance!

I am searching for a fully omni modal ie Voice recognition and speech generation like qwen 3 omni 30b a3b is their any newer model or finetune which is more capable or efficient in this category? gemma 4 is great but it does not support the speech generation.

We’re a group of experienced consumer-product operators and extraordinary subject-matter experts building a mission-driven AI company focused on health and human flourishing. We have substantial resources behind the project and deep expertise in the problem we are solving. What we are not is a team of AI engineers. We are evaluating specialist vendors to architect the deeper LLM, RAG and private-infrastructure systems. What we need now is someone on our side of the table. A technically strong, curious product engineer who can become the bridge between our founding team and those specialists: understand what they are proposing, help us evaluate decisions, prototype quickly, integrate what gets built, troubleshoot problems, and gradually develop deep institutional knowledge of the entire system. Over time, this person could grow into the internal technical/product owner. The architecture will involve: • Proprietary structured knowledge systems • Open-weight LLMs running privately • RAG / controlled retrieval • Qwen, Llama, Mistral and related tools • Backend and production systems • Security, provenance and evaluation • Consumer-facing web/mobile product development You do not need to arrive as the senior AI architect. In fact, that is not what we are hiring for. We care about technical range, intelligence, curiosity, product judgment, communication, and the desire to learn alongside very strong specialists while helping translate sophisticated technology into an exceptional consumer product. This is funded work with serious intent and substantial people behind it. Equity could also become part of the right long-term relationship. If this sounds unusually well suited to you, DM me with where you’re based, what you’ve actually built, GitHub/portfolio if available, and what kind of role you’d ultimately like to grow into.

Has anybody else been using their local LLM as their primary interface for using their PC? I am not talking about just for Development, but even simple tasks. For example I have used it to install software, set up GRUB, uninstall AI harnesses, even install Chromium. I found it's just quicker to use an LLM than manually doing anything anymore. With local models getting insanely good (ahem Flash Next), could we see the UI on OS's just change into a text box with a mic input in the future?

https://preview.redd.it/z2a8tjtenkrh1.png?width=2408&format=png&auto=… Computer-use agents do not need one large model to perform every cognitive function. Cognitive Sharding partitions the agent across specialist models, then coordinates them through a code-owned control plane. The current implementation uses: \- Bonsai 2 27B for reasoning and planning \- Kev 4B, built on Qwen3.5 4B, for rapid action selection \- UI-Mate 9B for visual grounding The control plane owns execution state, model residency, validation, retries, and recovery. Models receive bounded decisions instead of unrestricted control over the agent loop. This separation changes the hardware requirements. Models can be loaded and unloaded transactionally according to the current execution phase. The system therefore runs a complete local computer-use stack within the memory limits of a 16 GB consumer computer. This is different from a mixture-of-experts model. The shards are independent models with different inputs, training objectives, runtimes, and authority. Their composition happens at the system level, not inside one neural network. The approach document describes the planner–selector–grounder architecture, candidate construction, bounded execution, environment-verified recovery, and memory-aware model residency. I've added more details here: https://github.com/off-grid-ai/cognitive-sharding#cognitive-sharding-a-systems-architecture-for-computer-use-on-consumer-hardware Will run it against additional benchmarks and will publish the results soon.

My searches have guided me to Open-data models like OLMo and tells me I could inspect the datasets and audit it myself (which I would not know how to do) but is there any models that pride themselves on not having stolen a single line of data to train with? Other than that 1930 model which I'm sure it was on the public domain lmao. small edit: preferably as newest as possible, since these OLMo seem have been released on 2025 which is aeons ago in LLM timelines. ##Another edit: The word choice of the word "stolen" seems to be polarizing on this sub. I don't mean to judge or attack any other models (or the people using them) that were not fully transparent about their data acquisition, I normally use and enjoy models outside of this category. I'm doing a project where I need to implement, if I can use an analogy, a "vegan model" which I can assert with confidence nothing on it's training data was shaky on their licensing and was ethically sourced.