Hey guys, I am looking for benchmarks for people running multi node (4 or more) A100 80 GB cards and seeing what results and models they are getting. Something with VLLM and multi users would be very useful. Or if you know of a place I can find such results please let me know! Appreciated!

You know that when you start Pi you have to process the initial prompt, that takes some time when you use the slow dense QWEN 27B (that's the very reason why you use Pi instead of Cloud Code!), then you start to add extensions, tools, your append.md and whatever... Well now that got big, like 20k big and it does bother. So let's cache the "initial prompt" PP, so that when you start an new pi session: TA-DA! Instant ready, jolly good. Well if you tried to do that with Pi, llama.cpp and QWEN 3.x hybrid KV all kind of things step in your way to prevent that, so many that I won't even start to count I'll just tell you what to do: 1. dwl and install this Pi extension (tested on Pi version 0.87.1 ) 2. dwl and patch lama.cpp: yup no way around this if you kill the server between session, suck it or leave now 3. do your self a favor and use Froggeric template, for all your QWEN models, even the old ones. \---- So I'll help ya and give you some kinda useful parameters to launch the thing too: --slot-save-path /home/eaman/llama/slot_caches/ --ctx-checkpoints 32 --checkpoint-min-step 4096 -np 1 --chat-template-file chat_template_3.8.jinja You need to have save slots, that's the whole point, the caching is meant to resist restarts. Beware the chunk of blocks cached follow ubatch boundaries, so yeah try to keep that down if you wanna cache some more. Now in the extension README.md there's explanation of env variables that you can tweek, you go read those and edit accordingly to your setup (or have your LLM read that and suggest / config for you), TLDR you need at least: export PI_PREFIX_CACHE_BASE_URL=http://localhost:8080/v1 export PI_PREFIX_CACHE_PERSIST=1 export PI_PREFIX_CACHE_SLOT_DIR=/home/eaman/llama/slot_caches Disclaimer: this is not an easy thing, if you are not familiar with patching llama.cpp and installing extensions manually leave this thread for an other day. On the other hand this thing kinda works for me so if someone else is interested after some testing (because all kind of evil things want to break prompt caching) I'll upload a final extension and see about the llama.cpp problem with saved check points.. Possible results: https://preview.redd.it/xobe9ve7h5sh1.png?width=1281&format=png&auto=… EDIT: made a version for OpenCode: https://store.piffa.net/lm/ocache/ For those of you who want a basic understanding of the problems and solution regarding caching the prompt I've asked the LLM to make a short summary.

Edit 1: Okay, I have been checking out Unsloth and wow. Just wow. Thank you so much for your suggestions. This is such a way better tool and I am going to go crazy with this. Good day data nerds! I am trying to get more into running models, learning about agentic workflows, and creating my own tools. But as you do (right?) I had to fine tune my current setup. With the way the markets are right now, it only makes sense to make the most out of what I got. My day job, typically, is around data, numbers, and programming; only two of those I am good at, I'll let you guess which ones. So, yes, here come some data sheets and pretty graphs. Don't worry, you don't have to go through the data, but you can if you want. The graphs cover key metrics spat out by Ollama such as the tokens per second, duration, eval rates etc. I also added a cheeky "tokens/s/W" which, technically is not perfect since I do not measure wattage over time, but I did observe the watts during prompts, and I have a few things to mention about that later. Okay, let's start off with the specs because you probably think I am rocking the good stuff since I am so invested in this topic (haha) Lenovo M920Q 16GB DDR4 2660MHz Intel i5-8500T (6C/6T) Gigabyte Gaming OC 3070 8GB (Over OcuLink at Gen3x4 speeds) I am running OpenWebUI with Ollama in an LXC on my Proxmox server. This is one of my nodes and it is dedicated to my models. I have given it all of the cores, 14GB of RAM, 2GB swap. Nothing crazy to write home about, see? Okay, so, one of the things I wanted to know what, with the models that I run day to day, how effective are they at different GPU power limits. Man, if only I had known how much of a rabbit hole I would go down to do this (sorry wife). I only run 4 models, nothing too crazy, until you run 3 tests per model, for each power limit from 100 to 220 (156 runs in total), and each run of each model taking around 3 or 4 minutes since I have to unload the model each time to not have any prompt caching. Afterwards I would average the results and add that to the sheet. Really gave the fingers a workout since I now am a proud owner of a 60% keyboard for the first time and I no longer have a numpad... I'll remember that for next time. That being said, the switches are soooo creamy, a valiant tradeoff. So what models am I running? Glad you asked. The 3070 does limit me quite a bit, but with so many models available and so many smarter people than me who can quantize the models, I have found these models fit my needs. For the most part everything runs in the VRAM, except for 2, but those come with asterisks. gemma4:e4b qwen3.5 qwen3-vl\ deepseek-coder-v2\ For the vl model, it runs really well at a 23% CPU to 77% GPU ratio. Totally fine for my purposes. As for deepseek, it is a 40/60 ratio, but I luck out as it is a mixture of expert's model but even with the ratio, it is extremely performant. Gemma is by far my best model, and I have the most context room available at around 16k whereas the remainder I have sitting at 8k. Both gemma and qwen3.5 fit entirely in my GPUs VRAM. A couple things I noticed: Gemma4, is so good. Doesn't overthink, understands the prompt, remains as concise with the right tone I want. A really good day to day general model to work with. I also love the extra headroom for the context. Qwen3.5, a heavy thinker. Whilst it does a great job on the output, it spends a lot of time thinking and generating a lot of tokens. Power usage is pretty good, broke around 203W at one point and anything below that it just sat at whatever the power limit was set to. Qwen3-vl, also a major over-thinker. It spends so much time thinking that it balloons the context. I probably do not understand how to use this very well because when reading its thoughts it knows the answer pretty early on but it just gets into a thought trap (eh-yo). It does always output the correct answer, or the best it can, but I might move away from a reasoning vision model and stick to traditional ocr. If you have a better model or know how to prompt this better, I would love the help. Oh, one final note, this model LOVES power. Always maxes whatever I have, not that it increased performance directly, but it just loved power. Deepseek-coder-v2, this model rocks. It is extremely performant even though I technically on paper can't fit it. Especially for smaller asks with good bounds in place, it doesn't think, it just does and gives me excellent code back. I have yet to make it build me anything bigger but that is something I will experiment more with later. It is a weird one though, consistently using a fraction of the power budget available to it. Under 150W power limits, not once did my fans kick in on my GPU. Even my CPU fans (which are mucho loudo) rarely turned on, or if they did, they were not sounding like rocket engines. Not sure if those two are related but, eh, just something I noticed. Deepseek I think had some anomalous results with some spikes, but I can't be arsed to do them again. For the most part the results are fairly consistent and show a trend. Same for the vision model by qwen, oh well. My thoughts? It probably doesn't matter too much for most of us on a budget. Just let her rip, but if you want to shave off some heat, just lower your power down a little bit and monitor your temps. For the most part, you are probably fine. Honestly, it is 3am at this point, have a look at the spreadsheet! It was a lot of fun (I think) doing this. Interesting observations were made where I can balance my power limits, save... well pennies, and not have to listen to fans. So, works for me. https://docs.google.com/spreadsheets/d/1CEAr40nemlsK727QMvBnPcJDg548s-Bx/edit?usp=sharing&ouid=105501696463520933058&rtpof=true&sd=true Managers love graphs

The only things released in the last couple weeks seem to be bigger models like Qwen, DeepSeek, GLM, etc. Where are the Laguna's, Nemotrons, Olmo's, LongCats, Minimax, etc releases?

I audited thousands of agent rollouts in DeepSWE-1.1. Over 80% contained reasoning about an imagined grader. Yet no grader/verifier is mentioned in prompts nor accessible to the agents. Agents reasoned things like: "*Let me look at the problem from the grader's perspective*" and referred to "*hidden tests*", "*test authors*", and "*the checker*".

I found this behavior across all six frontier models analyzed, including recent models from OpenAI, Anthropic, Z.ai, and Kimi. In 10–25% of cases, such reasoning pulled the agent's work away from the user's original spec (yet it often still earned full reward on the DeepSWE task). I call this speculative reward hacking where the agent's reasoning focuses on imagining a grader rather than what the user wants.

\[Pictured example shows verbatim quotes from agent's reasoning\] For instance while completing a DeepSWE-1.1 task, GLM 5.3 knows that its implementation violates user requirements yet sticks with it after imagining what a hypothetical grader will check.

My article details many problematic trajectories, quantitative findings, and a taxonomy of these reward hacking behaviors: 
https://joinhandshake.com/research/ai/deepswe-reward-hacking/

Other than the obvious self promotion, is there a practical reason people do this that I am missing? There's dozens of llamacpp forks with silly names that are supposedly "optimized" for this or that specific GPU and seem to have zero intention to merge into upstream. Am I missing the real reasons why this happens so often? Why do people think it's OK to do this? In my experience in the open source community this is generally frowned upon.

I don't know if it's just a me problem that this kind of thing puts me off so much. I am usually quite grateful for PR feedback and conscientious about the code I put out there; I take pride in submitting high quality code that meets or exceeds the standards of a given project. Of course there is nothing ethically wrong with hard forks or taking shortcuts if you find the collaborative process cumbersome, but personally I wouldn't promote my fork in such cases, let alone go out of my way to add custom branding with a Reddit announcement post etc. since the effort required to do so seems roughly equivalent to the effort required to meet the contributor standards. In contrast, many of the authors of these forks seem very eager to have others adopt their rebranded fork for production use cases. There just seems to be a big disconnect, idk.

Edit: some great discussion in this thread, thanks to all who responded. Consensus seems to be that (excluding the obvious low-effort engagement bait forks) the base project has to meet many compatibility requirements while a downstream project can be more focused, which is a great point.

Early this year when I was first looking at building up my inference capability you could get the 16GB Tesla P100s for between $60 and $80. Asked claude about it, told me absolutely not worth it. No tensor cores, bad int4/int8, no BF16, not worth it. Needs special power accommodations, Above 4G decoding option in the bios (it made it out like it was some rare option), and a semi-exotic cooling solution.

Optimized the shit out of my RX6600XT in llama.cpp as a result. Got pretty far.

Decided to say fuck it, bought a single P100 last week, finally showed up day before yesterday. Got a newer power supply with the appropriate connections (not hard, not that expensive, seen options as cheap as $60 from good brands, I spent $100 on one with some headroom), multiple llama.cpp forks and patches that carry some wild optimizations to handle the capability gap, and a 3D printed housing for a 94mm fan from noctua. Doesn't generate enough static pressure to keep it cool during prefill, but more than strong enough for the generation step.

The numbers I was getting before, with my RX6600 XT with Qwen3.6 35B A3B UD\_Q4\_K\_XL with MTP and --cpu-moe:

PP \~800 at 0 ctx, drops to \~700 by 10k

TG \~30-35 prose, 45-50 code.

This setup could do 64k context (and possibly higher) at 16bit kv. cpu-moe helps a ton in that respect.

With just a little bit of tuning, and using specifically the patches from shinbunbun for llama.cpp, same model with the same MTP settings, --n-cpu-moe 22:

PP \~600 at 0 ctx, 440-500 by 10k

TG \~54-60 prose, 66-72 code.

Running only 32k context right now to make it work. Could fit more with a higher n-cpu-moe, but my harness doesn't need that much (rarely see it over 30k, persistent memory leads to chats that simply aren't meant to last).

I know the capability gap between 3.6 35B and the basically any of the qwen 3.X 27B models is pretty big, but this is huge for the price. They've gone up since I bought mine, about \~$15 across the board. Still something you can get for under $100 and makes for inference that is simply impossible to get at that price otherwise.

I've got another one coming so I can go full offload on the model, and maybe even start playing with 3.8 27b. Right now IQ3\_K\_XL I get around 9 tokens per second with MTP, and basically no real context. Don't actually know if splitting a model that fits in one card across multiple helps speed, that's completely new territory for me, but I'm having fun regardless.

Card is seriously underrated. It's a great, (relatively) inexpensive way to get capable compute to finally start doing local AI stuff. I went from having to just leave my computer alone while the model was running and do everything from my macbook (good bye gaming) to being able to let the model live and work in the background while I'm doing basically anything on my PC. When the second arrives, I'll be planning my dedicated inference box they'll both live in. Feeling inspired by that guy cooling his PC with a VW radiator.

I've liked how Muse-Glimmer worked, so I wanted to see if I could produce a smaller "kid" out of it. Ornith's sharp decisions on when to think and which tool to call were the other thing I liked, so Ornith-1.0-9B got to be the policy teacher while the parent wrote the words. No RL anywhere, distillation only.

I present to you Xyntetik-Kvist-14B.

What it is good for

• Smaller than the Muse parent but still manages most tool tasks: 57 of 60 held-out closed-loop tasks (contacts, weather, flights, currency, dates, units, stock), scored by re-executing the calls against ground truth, where the parent does 60.
• Fits a 24 GB card whole at Q8_0 (15.4 GB) or the Q5_0 mix (10.3 GB) and serves an OpenAI-, Anthropic- and Responses-compatible API through Xyntetik Runner, so it drops into an agent loop you already have.
• Every failed attempt is published beside it: 12 gated runs, 2 full passes, one shipped. The training record has the preregistrations, the amendments and the defects, so you can see exactly where it breaks before you build on it.
• Give it a calculator tool for arithmetic. Without one it gets "17% of 2,340" wrong, and the card says so.

Numbers, from the card

| claim | number |
|---|---|
| parent | Muse-Glimmer-30B, cut by width (hidden 6,656 to 5,760, FFN 19,968 to 10,240, heads 32 to 24), all 52 layers kept |
| size | 14.44 B parameters; BF16 28.9 GB, Q8_0 15.4 GB, Q5_0 mix 10.3 GB |
| distillation | 6,000 steps, 98.3 M tokens, 162 hours, then 1,440 steps on agentic trajectories |
| fidelity to parent | KLD 0.762, margin-qualified top-1 84.0% on 45,056 held-out positions (a student's row, not the quant bar) |
| tool tasks | 57 of 60 held-out, re-executed against ground truth; parent 60, untrained control 0 |
| format and calls | 199 of 200 first turns well formed; 99 of 99 tool calls valid |
| attempts | 12 gated attempts, 2 full passes, attempt 12 shipped |
| weak spot | calc tasks 12 of 15 over 160; 7 of 160 runs end in a reasoning loop |
| serving | Runner v0.5.7 or later |
| licence | Apache-2.0 |

Links

• Model: https://huggingface.co/Joakimpalm-Zen/Xyntetik-Kvist-14B
• GGUF: https://huggingface.co/Joakimpalm-Zen/Xyntetik-Kvist-14B-GGUF
• Training record: https://huggingface.co/datasets/Joakimpalm-Zen/Xyntetik-Kvist-14B-training-re…
• Collection: https://huggingface.co/collections/Joakimpalm-Zen/xyntetik-kvist-distilled-ag…
• Runner v0.5.7: https://github.com/Joakimpalm-Zen/xyntetik-runner/releases/tag/v0.5.7

EDIT: reading the comments, i should have said this first. this is not a general drop-in for Muse or a gemma4 replacement, and it was never going to be on my compute (98M distillation tokens vs the trillion a real distill wants, i simply lack the compute). the purpose was more on getting the tool calling right. IQ4_NL mix (7.6 GB) is up now too, it scores the same 57/60 on the tool tasks but does not fit an 8 GB card whole (50/52 layers on a 3070, ~5 tok/s).

As usauly i was scrolling around YouTubes while .... well when every man scrolls YouTube to pass the time.... anyway, came across this video - https://www.youtube.com/watch?v=doR2RhsneRA

TLDR: Guy spends $2175 USD and over 36 hours using Claude Opus 5.5 to build a pretty impressive MMORPG.

Now there is alot of caviats, is it perfect... No... is it really an MMO ... no i havent seen any code added for it.. buttt the strcuture is there, its a hell of a start.

And it got me thinking, if people with their local AI's where to do something like this What models or infra would you use to do this.

Me I think you could get a really good start with Hermes, Qwen Flash, GLM5.3 across a couple of DGX Sparks or if you had 2.5 TB's of RAM and freetoken Kimi K3.

And with the detailed level of prompting and design he laid out prior to actaully letting claude have at it, i think it would doable locally.

So to start the Discussion, what would your tech stack be to do this locally? For me:

\- 2 x DGX Sparks - GLM 5.3

\- 5090 desktop with ComfyUI and a bunch of work floors for image generation, Guassplating, Image to 3D models

and to me I think that would be all that would be needed to get started, but im intrested to see what others think up

So I put this computer together after I saw a lot of others posting what they have, and I wanted to know where this ranked and what (in your opinion) are the best local models for coding, and for always on agents.

I didn’t want to type out the whole thing, so I asked my model to give me the specs.

Creature workstation (kernel 7.0.0-28-generic) with an Intel Xeon E5-2698 v4 at 2.2 GHz (20 cores / 40 threads, 50 MiB L3), 125 GiB DDR4-2400 (90 GiB free right now), one NVIDIA RTX PRO 4500 Blackwell with 32 GB of GDDR7 / 31.9 GiB usable VRAM (896 GB/s), and 5.37 TB of raw storage — a 915 GB NVMe root (379 GB free), a 916 GB media disk, and two 1.8 TB drives.

All figures read live from lscpu, free -g, nvidia-smi and df just now.

A "token" is not a fixed chunk of text. It's a word from the model's own private dictionary. Each model ships with its own vocabulary: the list of string-slices learned during training. Analogy: two people transcribe the same sentence: one writes "New York" as one word, the other as two. Both are correct; they're just counting different things.

So tokens/sec is speed measured in steps per minute, and two models can have different stride lengths. One can takes long steps (eg, 4.83 chars each), the other short ones (eg, 3.27). A child and an adult both walking "60 steps per minute" are not walking side by side.

The rules that follow:

1. Same tokenizer = fair comparison. Two llama.cpp servers running the same model family, tok/s compares directly. Trust it.

1. Different tokenizers = the number is in different units. Convert to distance: real speed = tok/s x chars-per-token. If both servers report 40 tok/s on prose, one server is laying down \~131 chars/s and the other server \~193 chars/s, so the second is 1.48x faster while the headline numbers tie. The inflation favors the choppier tokenizer: more tokens for the same text = bigger tok/s for the same wall-clock speed.

1. The conversion factor is content-dependent, so measure, don't assume. A ratio may be 0.68 on prose but 0.74 on JSON. The same vocabularies chop different text differently. Any cross-server speed claim should come with chars/token (or just words/sec) measured on a representative workload, not vendor marketing numbers.

1. One is real, one is an illusion. What's real is prefill and cost. A more efficient tokenizer turns the same conversation into fewer tokens, so there is genuinely less compute before the first token (shorter TTFT). That's a true speed and money win, not a units trick. What's an Illusion: decode-rate comparisons across families. "Model A does 80 tok/s, model B does 60" says nothing about who finishes the answer first unless A and B share a vocabulary.

TL;DR: vLLM recipe for Qwen3.8 Flash on one DGX Spark / GB10. 74 tok/s peak single-stream, 60 to 70 on normal requests, 212 tok/s across 8 streams, 2x to 3.4x faster cold prefill than the recipe it's forked from, full 262K context, and quality matches the original within noise. Everything is open, including the raw per-round data and the benchmark scripts.

https://github.com/dime-online/qwen3.8-Flash-DGX-UltraFast

Most single-Spark setups I've seen posted for this model land somewhere in the 35 to 45 tok/s range, so I spent a few weeks figuring out where the time per token actually goes on a GB10 and cutting it down. If you don't have a Spark, the tricks in the middle section should still be interesting, since most of them apply to any MTP or speculative decoding setup.

What this is, in plain terms

It's a ready-made serving setup. You build the container, pull the public weights, and get an OpenAI-compatible server that answers a lot faster on one box. The speed comes from the model's own draft head guessing several tokens ahead while the full model checks all of them in one pass. Right guesses give you several tokens for the price of one step, and wrong ones get replaced by the full model's answer, so output quality doesn't change.

Decode

Peak decode speed, same workload at every point, best of 3 rounds:

|Streams|1|2|3|4|5|6|7|8|
|:-|:-|:-|:-|:-|:-|:-|:-|:-|
|tok/s|74.1|110.0|132.9|155.5|175.8|191.5|205.8|212.2|

Tokens per step stays between 3.65 and 3.94 from 1 all the way to 8 streams, so the speculation doesn't fall apart under batching. 8 is where it tops out because that's the configured max\_num\_seqs, and the gain from 7 to 8 was down to 3%.

Prefill

Cold prompt with nothing cached, three repeats each:

|Prompt|This recipe|Original recipe|Increase|
|:-|:-|:-|:-|
|16K tokens|4,016 tok/s|1,171 tok/s|\+243%|
|64K tokens|2,426 tok/s|1,071 tok/s|\+127%|
|128K tokens|2,213 tok/s|1,065 tok/s|\+108%|

With prefix caching on, a cached coding prompt starts replying in about 0.57 s, which is what makes agent loops feel fast.

What actually made the difference

The model's own MTP head, run densely, lands about 3.7 tokens per verify step. That's the single biggest lever.

I cut the draft head's vocab from 248K to 65K ids. On a GB10 the draft pass is memory-bound, and reading a full-vocab head every draft step was a real chunk of the step time. The target model still verifies against the full vocab, so this can only change speed, not output.

The quant is W4A16 AutoRound for the MoE experts, FP8 for the side layers and INT8 for the lm\_head. No 3-bit and no NVFP4, because I wanted the speed to come from the serving path and not from squeezing the weights harder.

There's a GB10-tuned low-latency GEMM for the small decode-time matmuls and a sort-free top-k in the verify step.

The prefill gain mostly comes from a faster gather path for the per-layer embedding table, which removes a pile of serial page faults during prefill.

Put together, each decode step went from 68.3 ms on the original recipe to 52.3 ms on an agent-shaped coding workload, about 1.3x more steps per second.

One thing that didn't pay off: doubling the prefill chunk to 16,384 tokens gave no prefill gain at all and ran the box low enough on memory that I rejected it.

Quality

93.1% and 93.3% on a fixed 492-question suite over two seeds, covering code with execution checks, math, knowledge, instruction following, tool calls and long-context needles. I also ran a teacher-forced check against the original checkpoint, and top-1 agreement moved by 0.06 points against a 0.15 point noise band I set before running it.

Practical stuff

The model takes about 71 GiB, the KV pool is 16 GB, and around 16 GiB stays free under load. While generating, the GPU draws about 35 to 37 W median and peaks near 70 W on long cold prefills, with no power or thermal throttling across the soak runs.

The 65K draft vocab was built from English and code, so Chinese, Japanese and Korean output drafts less well and runs slower. Quality isn't affected, because the full model still checks every token.

Built on Saren-Arterius's qwen3.8-Flash-DGX-AutoRound, so big credit there. Happy to answer questions.

...the one that will mass produce a cheap device (sub $1000) able to run current sota models at a decent speed.

for that to happen, obviously RAM has to be cheaper, models need to be more efficient and CPUs have to change. It will take time. But as in the 70s/80s computers were huge and expensive mainframes only big companies had, today we are in the same situation.
Fortunately progress happens faster now, so it won't take 30 years to get affordable "home computers". Probably 10, hopefully less.
It's encouraging that today to run the latest qwen 27B you can spend less than $3000. But still...

I bought a B70 off Amazon last week for 1300 and have been playing around with it. Today I checked and it seems like I cannot find a single one online for less than 1600 and most places dont have them in stock anymore? What happened? What drives these sudden price increases? It seems like all GPUs across the board have seen another dramatic price flux. Some 5090s I saw were going for 10k!?

No click bait baby I promise - I'm live streaming the training process kinda like MiMo.

UPDATE: \[Training is paused for an hour or two\] back to training in batches. u/FullOf_Bad_Ideas has pointed out to me I'm burning a ton of compute for nothing on sequence lengths - we'll be breaking the run up into 7/8 batches and then going again.

Original thread: https://www.reddit.com/r/LocalLLaMA/comments/1wpg4a8/im\_trying\_to\_posttrain/

If you didn't see my original post a few days ago, I'm attempting a slightly more ambitious than usual project in trying to create at least a rough AliceAI-Foundation-80B-A3B-Instruct

I spent the weekend distilling my initial instruct training dataset out of Qwen 3.8 27b, medium thinking - intentionally done because I can run it locally, and I wanted a full dataset in some reasonable amount of time. Still took my v100s running 4x instances at 25tps, like 96 hours of non stop generation to complete the dataset.

I opted not to go for a pre-existing public dataset because I wanted to practice building my own distillation engine (which was configured to work off of an OpenAI compatible endpoint, so it'll distill anything you can hook it up to). The final dataset (this time) consists of 3340 samples: 1760 of general instruct transcripts, and 1580 agentic specific work rows about SWE, harnesses, terminals, etc - I gave the distilling engine a python sandbox and got to simulate turn driven development with a user, and I trained for a bunch of different harness syntax for tool calls, which hopefully will be enough to generalize - gonna run 2 epochs at first.

My GPUs are sobbing right now - turned them on on Friday and left for a weekend vacation, got back today, waited an hour for the data to finish generating, and then immediately fired up the train.

The stuff above is the short version. I'm guessing the initial SFT train will take about 3-6 days, and I plan on working on a RL implementation after I'm satisfied that the SFT has at least worked properly. I am training a rank 16 QLoRa adapter on only q/k/v/o proj, no direct knowledge weight fine tuning.

I thought what MiMo did with their recent training was really cool to watch online, and I like sharing my work with like minded people, and frankly, there's a part of me that's hoping someone will see this and want to hire me (looking for NYC work if you know anyone looking for some passionate ML engineers!) - so I've set up my own little training stream on a cloud flare tunnel.

The stream has the live in progress status of the train, including a live view of the actual data being processed by the model. It also includes way more detail about how I actually designed and generated my training data. Happy to throw the full set on HF as well. I don't expect this model to beat any existing standards but I'll be curious to see if I can get it to operate properly in a harness so I can formally bench it.

I hope you find this interesting! The live stream is a self updating website where you can see exactly what's happening - no need to reload. To watch the training live, visit https://figure-bios-expect-cio.trycloudflare.com/ \[i am currently fixing training issues but it'll be back asap\] -- I'll be keeping it up until the initial SFT is done, at least. The stream lets you inspect the training live as well. This is just a cloudflare tunnel to the trainer.

3 hour update? Loss started at 9ish and is bouncing near 3/4

Update today: back online

The human brain is neither an LLM nor a JEV. In a crowded market you hear a lot of speech and answer almost none of it. The ongoing question is not “what should I say?” It is “you talking to me?" and "should I say anything at all?”

This live voice demo showcases three hardware tiers—the Blackwell 4500, Jetson Orin NX 16GB, and Jetson Orin Nano 8GB—solving this exact problem. By splitting the workload between a lightweight decision head for turn-taking and a Gemma 4 pipeline for text generation, the setup delivers highly responsive, low-latency vocal interaction.

EDIT: repo (the latest code yet published) https://github.com/cortexist/little-gemma

I run an Unraid server and use local models for STT, memory, a small llm (I like the new swift bonsai), and SystemOne Models. I currently have a 5070 plugged into my x570 board (with a 5600x), and then the 5060ti on a riser. The 5060 runs at x4, there is a limitation on the board setup.

Running a model big enough for both cards runs SLOW, since the connection to the 5060ti is capped.

Ive tried optimizing with ninfer and vllm, but my speed is capped at the hardware level.

I am considering scrapping the 2 card setup for a single card, and right now the best budget option is the RX7900XTX.

Can you guys argue about what would be the better setup? I dont need the newest models, and I dont need the most speed. I pay for online models. I just like to have a bit of local stuff to help with server stuff. I feel like I spend too much time managing the 2 card setup for not enough to get out of it, and I think dropping to the one card would make things easier without a loss in speed. The idea is to sell both NVIDIA cards, and just get a single card with big enough memory that isnt a pig.

Any advice would help.

If someone is interested https://preview.redd.it/wlqoavzq7ash1.png?width=1524&format=png&auto=…

On the official JevBench evaluation battery, Gevva0 scored 74.63 (#1 global rank), averaging 214ms p50 across large legal contract sets with 82.9% accuracy on the forensic hard tier.

How it works under the hood:

1. Direct Logit Scoring: Ingests context and reads decision logits directly from llm.scores\[-1\] in a single prefill pass. Fast-path resolution runs in 22ms on short contexts.
2. Cyclic Debiasing: Permutes class tokens across 4 cyclic positions to neutralize label position bias.
3. Platt Temperature Calibration: Fits confidence via sigmoid scaling to push Expected Calibration Error (ECE) below 0.03.
4. Asymmetric Audit Pass: Locks the categorical verdict permanently first, then performs an isolated extraction pass to retrieve verbatim source quotes without contaminating the decision logit.

The repo includes the evaluation harness, raw benchmark datasets, and a local web dashboard: https://github.com/solvingSteve/Gevva0

Setup instructions and benchmarks are in the README.
Working on Demos and Use Cases now so if you have any ideas I'll try to build them next!

Hi. So....... after toturing my 750 ti 4 gb and 16 gb ram with image gen model .i want continue experiment with agent mode like hermes or opencode, using local model but before go i want ask few question. are big model = good perfomance or small model can do same stuff. what small model recommend for agent my spec what caveat of doing this ?

Qwen Flash Next speedup for Vulkan people

Seems like it would be a very nice way to get 24gb of VRAM, and it should be possible to do right? Given that 20gb 3080s exist. Just wondering

According to this post it is possible to reach very high numbers using mtp, however I have failed to reproduce the 50+ tps for 5060ti.

Am I just ignorant or how exactly do this? Or is this a fake post?
Freshly built llama fork for sm120 (Blackwell):
https://github.com/Anbeeld/beellama.cpp
"C:\\llama\\build\\bin\\llama-server.exe" \^

\-m "%MODEL%" \^

\--port 8090 --host 127.0.0.1 \^

\-ngl 99 \^

\--cache-type-k kvarn3 --cache-type-v kvarn3 \^

\--flash-attn on \^

\--load-mode mlock \^

\--jinja \^

\-c 98304 --parallel 1 \^

\--fit off \^

\--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-ubatch-size 128 \^

\-ctkd q8\_0 -ctvd q4\_0 \^

\--kv-tail-tokens auto

Runs at 20-35 t/s.

I'm setting up my first build with Open Claw. I'm new to ask of this and starting from zero knowledge.

I've done a lot of reading up, but it's a little overwhelming. So I'm biting things off in chunks.

I want everything to be local. I have a mini PC with 96GB of RAM so can fit a good sized model.

I have Open Claw set up right now with OpenRouter.

My next step is to set up my local model.

What is the current leading open source model for generic tasks and learning? I have plenty of memory to support.

Kimi K3, Opus, Quen 3.8, other?

Would you buy an AI appliance that made it easier for you to run local AI models such as Qwen 3.8 27B and Gemma 3 27B?

By easier I mean, the appliance will take care of all the infrastructure stuff for you such as installing the OS, the inference engine such as llama.cpp or vLLM, access management and perhaps include aome preconfigured agents for you start using the system.

The system is multi-user, with a role-based management system, meaning multiple people can use it, including teams.

All runs local, but with the option of using cloud based models if you need more horse power.

Is this something that has an appealing?

Or the fun is the tinkering 🤪

For me it's a pass/fail: can it fully replace a remote worker? How do you define it?

Hey guys i am planning to get a mac since the GPU and other stuff isnt currently possible for me rn so what models near or stronger than Sonnet 4.6 or somewhat nearer can I use it on mac or would i be able to use it actually?

I mainly need it for coding tasks, different file management and reports and also reasoning. SUggestions or feedbacks are welcomed for models. I want everything local for privacy concerns

Edit: Fixed the model mention

I am confused between LM Studio and Bionic.

I have LM Studio for a long time though not used frequently.

Recently I am trying to learn the agentic stuff. Watched a few videos but they were all using Bionic. The strange thing is the interface looks different than the one I just installed today. (Mine seems to be missing features, no developer mode. I am on MacOS)

On the other hand, I seem to be able to find those missing settings in LM Studio.

So what is the difference between the two? Is there anything Bionic can do but LM Studio doesn't?

Basically the idea is take your favorite model, for example qwen3.8-27b or say dsv4vision. Strip everything out that is not needed by that model so the only thing needed is just for the model. Optimize the remaining code to be fast. The idea is to have a model also do this, provide it with enough tools, prompts, docs, guidance. I reckon that if we have a llama.cpp that is optimized for just one model architecture without all the cruft needed to run and. handle other models, that it would not be surprising to easily see 2x+ performance improvement. Anyone thinking along this idea? Again, the goal will be to give this task to a smart model, let it run in a loop, and after a week or 2 you hopefully end up with llama.qwen3.8-27b or llama.glm5.3-flash that would fly.

Recently started using the MacOS menu bar app for llama.cpp available at https://llama.app When you use the UI to download a model, it seems to do an Ollama-esque hashing instead of just saving the GGUF. Even if you put a GGUF in the Model directory folder, neither the menu bar UI nor the web UI recognizes it. https://preview.redd.it/bsiw7kbn46sh1.png?width=1186&format=png&auto=…

Hello, I created a calibration free quant method called TQ. It doesn't need any data so models could be quantized as soon as they come out and the quantized models would generalize better. It performs closely to calibration-based methods. For example, for Qwen 3.8 27B the 4-bit TQ has mean KLD of 0.0282 and top-1 of 92.4% I opened sourced the quant code as Quant Factory so anyone can quantize and run open source models. Right now it only supports 4-bit but I plan to add support for 2-bit and 3-bit soon. The repo link is: https://github.com/textclf-api/quant-factory I have a collection of quantized models using TQ at: https://huggingface.co/textclf You can run these models using either using the following docker image or by following the repo's instruction. For example you can run textclf/Qwen3.8-27B-TQ-4bit like this: docker run --rm --gpus all -p 8000:8000 docker.io/textclf/tq-quant:4bit-main vllm serve textclf/Qwen3.8-27B-TQ-4bit --quantization tq The Dockerfile in the repo shows how this docker image was created. The repo's README explains the approach used for this quant method and why it is useful. You can try it and let me know what you think. Feedback appreciated. EDIT: I did KLD testing using the Wikitext-2 dataset for Qwen 3.8 37B. I also did the same test for the Unsloth-UD-Q4\_K\_XL quant. Here is what I go: |Quant|Disk Size without MTP (GB)|Mean KLD|Median KLD|99% KLD|Top 1% Agreement| |:-|:-|:-|:-|:-|:-| |TQ 4-bit|17.76|0.02823666|0.01282929|0.26897613|92.419%| |UD-Q4\_K\_XL|17.59|0.00771805|0.00318557|0.07390548|95.779%| Working on getting more tests for other models!

I had a theory that ram speed didn't really matter and the only thing that matters is your GPU capacity, vram speed, vram size, and system ram size instead of ram speed or cpu speed. I think I've been vindicated. I used unsloth/Qwen3.8-27B-GGUF:UD-IQ3\_XXS (10.9gb) https://huggingface.co/unsloth/Qwen3.8-27B-GGUF as my baseline and mtp q4\_0 1.37gb only on the dual gpu systems because mtp was too slow on the 9060xt and 5070 I used llama.cpp for all of them and tried to go for the max context I could since they are supposed to be day to day agents. I picked prism32 as my agent harness for this because it's the most compatible, fastest, and reliable one I've found. It's a custom architecture https://github.com/MegaDyneSystems/prism32 which gives it some massive advantages, especially if you want to avoid bloated frameworks eating your context or CPU. The 2007 system literally doesn't work with any other harness because they all require sse4.2 and a ton of heavy dependencies. prism32's only dependency is python 3.7 or above. It's so lightweight (uses around 5 to 10mb ram) that I actually run it bare metal on my ARM synology NAS and even my 2008 TP-link router. If you want to run any agents especially with advanced features on edge or legacy hardware without it choking your system their is no competition I tried these 5 systems: 2007 dell precision t5400 ddr2: 24gb ddr2 8 core 8 thread dual xeon x5460, rx 6700xt rx 6700 22gb vram total, 215gb ssd total system memory 46gb (pic 1) 2009 dell precision t5500 ddr3: 72gb ddr3 12 core 24 thread dual xeon x5675, rtx 5060 rtx 4060 16gb vram total, 512gb ssd, total system memory 88gb (pic 2) 2012 dell precision t3600 ddr3: 64gb ddr3 6 core 12 thread xeon e5-1650, rtx 4060, rtx 3060 12gb total vram 20gb, 512gb ssd, total system memory 84gb 2018 hp obelisk desktop 875 ddr4: 32gb ddr4 3200 i7 8700, rx 9060 xt 16gb, total vram 16gb 512gb ssd, total system memory 48gb (pic 3 ) * 2025 HP omen: 32gb ddr5 6000 RTX 5070 12gb GDDR7 vram 1tb ssd, total 44gb memory (pic 4) The speed results on each system were: 2007 dell precision ddr2 144k context, 18 tk's a second decode, 150tk's prompt processing 2009 dell precision t5500 ddr3 256k context 22 tokens a second decode, 224 tokens prompt processing 2012 dell precision t3600 ddr3 104k context 22 tokens a second short context 14 tokens a second long context, prompt processing is 315 tokens, (could optimize further but tests took me long enough) 2018 hp obelisk 875 ddr4 180k context, 16 tokens a second long decode, 477 promp processing 2025 HP omen 131k 13 tokens a second, 43 prompt processing (yes 43) conclusion The older dual xeon dual gpu setups completely destroyed the newer stuff in context length and speed even on worse GPU's which I think proves my theory, The HP omen system is at least $1500 and the 2007 system didn't cost more than 400 total, rx 6700 xt was $190, rx 6700 was $140, on ebay they are overpriced at $150 although I got it $50 second hand in 2014, and the DDR3 systems were $20 second hand and go around 80 to 150 on ebay, I'd say the ddr3 systems are the best for performance and value, next project is running qwen 3.8 flash next on the ddr3 systems

It turns decisions — routing, triage, yes/no calls — into typed outputs from a small model instead of generated text, with a routing-only CLI, triage presets, and an abstention gate when confidence falls below a threshold. I ran through the tutorial on CPU end to end, including a French ticket classification, and with min\_confidence=0.90 it abstained on one case it would otherwise have misclassified — the honest highlight. Warm latency was about 0.7s per question on CPU; the calibration caveat (over-confident checkpoints) is worth knowing before trusting the scores blindly.

It’s probably a bit niche and the models are a bit old at this point, but after reading about Ninfer a few months ago I did my own small vibe coded engine project for my 5080 Laptop GPU. Initially I thought I can only fit the 12B model with enough context into the VRAM, but with custom quantisation (EXL3 like) the 26B fits nicely as well. The engine is entirely Codex written, but it took a lot of weekends to make it work and make it work as fast as vLLM/faster since vLLM didn’t work with MTP on 16GB VRAM. Also, my engine works on Linux and Windows equally well. Primarily this is designed to be single-user only, the 12B model can serve 2 sessions. It also comes with a native fancy looking GUI, but the main focus was the engine itself. 12B with audio & vision, 5.800 t/s prefill & 87 t/s decode 26B with vision, 5.660 t/s prefill & 182 t/s decode, fits 220k context https://github.com/Danmoreng/gem16 Sadly the most interesting feature of the 12B model with native audio understanding seems to have the issue, that after around 8k context the model doesn’t recognise audio tokens anymore. This seems to be a model issue, as others have also reported it: https://huggingface.co/google/gemma-4-12B-it/discussions/45 Would love to get some feedback!

I've been struggling to keep my Radeon Instinct MI50 GPU cool. I'm looking for budget friendly solutions. While running multiple GPUs it doesn't usually get too hot. I was also getting lower benchmarks using standard Vulkan 'llama 7B Q4\_0' model benchmark, but it wasn't caused by thermal throttling. Time to optimize. MI50 with Radeon VII firmware 16GB Vram My previous post I tested several model using same GPUs. I made some changes. I moved the MI50 16gb into the primary PCIe 16x slot and moved the RX 7900 GRE 16gb into a slower PCIe 4x slot. Overall system inference performance increased. I used Google Gemini to helped my optimize my llama-bench settings and it taught be about: RADV_PERFTEST=nogttspill is an AMD Linux driver flag used when running llama.cpp with the Vulkan backend. It forces the RADV (Mesa Vulkan) driver to prioritize keeping all model allocations inside dedicated video memory (VRAM) rather than spilling over into system RAM (GTT/Graphics Translation Table). \[1, 2, 3\] I saw llama 7B Q4\_0 score jump back to where is it should be. So I tested a few other models. GGML_VK_VISIBLE_DEVICES=0,1 RADV_PERFTEST=nogttspill time /llama-b11053/llama-bench -fa on -ngl 99 -m /llama-2-7b.Q4_0.gguf Previous benchmarks: https://www.reddit.com/r/LocalLLM/s/PT6Bd5nUUE see end for comparison The list has been sorted by pp512 improvement in descending order (highest gain to highest loss). |Model Name|Improvement (pp512)|Improvement (tg128)| |:-|:-|:-| |Laguna-XS-2.1-APEX-I-Balanced.gguf|\+402.14%|\-0.20%| |gemma-4-31B-it-Q6\_K.gguf|\+397.84%|\+9.24%| |Muse-Glimmer-30B-UD-Q6\_K\_XL.gguf|\+386.58%|\+48.63%| |llama-2-7b.Q4\_0.gguf|\+176.57%|\+20.29%| |Qwen3.8-27B-Q6\_K.gguf|\+48.13%|\+0.07%| |medgemma-27b-it-UD-Q6\_K\_XL.gguf|\+37.89%|\-0.52%| |Qwen3.6-35B-A3B-UD-Q5\_K\_XL.gguf|\+1.17%|\+0.49%| |Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4\_K\_M.gguf|\+0.68%|\+4.16%| |NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q5\_K\_M.gguf|\+0.28%|\-4.28%| |granite-4.2-30b-Q6\_K\_L.gguf|\-0.12%|0.00%| |GLM-4.7-Flash-Uncen-Hrt-NEO-CODE-MAX-imt-D\_AU-Q6\_K.gguf|\-0.63%|\-7.83%| |Qwen3-Coder-30B-A3B-Instruct-UD-Q6\_K\_XL.gguf|\-0.69%|\-0.86%| |Accio-Lab\_occamy-1.0-Q5\_K\_S.gguf|\-0.81%|\+0.17%| These are the models tested in the same order as tables below: GGUF Model List (in order): 1. Qwen3.6-35B-A3B-UD-Q5\_K\_XL.gguf 2. Accio-Lab\_occamy-1.0-Q5\_K\_S.gguf 3. Laguna-XS-2.1-APEX-I-Balanced.gguf 4. NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q5\_K\_M.gguf 5. gemma-4-31B-it-Q6\_K.gguf 6. Qwen3-Coder-30B-A3B-Instruct-UD-Q6\_K\_XL.gguf 7. GLM-4.7-Flash-Uncen-Hrt-NEO-CODE-MAX-imt-D\_AU-Q6\_K.gguf 8. granite-4.2-30b-Q6\_K\_L.gguf 9. Muse-Glimmer-30B-UD-Q6\_K\_XL.gguf 10. Qwen3.8-27B-Q6\_K.gguf 11. medgemma-27b-it-UD-Q6\_K\_XL.gguf 12. Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4\_K\_M.gguf 13. llama-2-7b.Q4\_0.gguf Supporting data sorted order (by Params descending, then Size descending). All models running dual Radeon GPU, Vulkan backend, and flash attention on. # Table 1: RADV_PERFTEST=nogttspill is being used |model|size|params|pp512|tg128| |:-|:-|:-|:-|:-| |qwen35moe 35B.A3B Q5\_K - Medium|24.76 GiB|34.66 B|1252.09 ± 11.84|53.54 ± 0.51| |qwen35moe 35B.A3B Q5\_K - Small|23.40 GiB|34.66 B|1245.11 ± 11.76|53.42 ± 0.28| |laguna 30B.A3B Q5\_K - Medium|22.64 GiB|33.44 B|1027.42 ± 11.32|65.72 ± 0.51| |nemotron\_h\_moe 31B.A3.5B Q5\_K - Medium|25.10 GiB|32.91 B|1120.31 ± 12.85|63.82 ± 1.52| |gemma4 31B Q6\_K|23.46 GiB|30.70 B|200.33 ± 0.20|15.61 ± 0.05| |qwen3moe 30B.A3B Q6\_K|24.53 GiB|30.53 B|1176.48 ± 36.62|66.04 ± 0.25| |deepseek2 30B.A3B Q6\_K|23.26 GiB|29.94 B|914.86 ± 8.75|40.02 ± 0.12| |granite 30B Q6\_K|23.01 GiB|29.28 B|92.09 ± 0.30|17.41 ± 0.03| |muse-glimmer 30B Q6\_K|24.45 GiB|27.85 B|279.04 ± 0.20|17.42 ± 0.03| |qwen35 27B Q6\_K|21.30 GiB|27.32 B|235.12 ± 1.05|14.42 ± 0.03| |gemma3 27B Q6\_K|22.09 GiB|27.01 B|247.18 ± 0.54|17.30 ± 0.13| |gemma4 26B.A4B Q4\_K - Medium|15.63 GiB|25.23 B|1316.62 ± 24.18|55.56 ± 0.16| |llama 7B Q4\_0|3.56 GiB|6.74 B|1349.35 ± 15.66|74.62 ± 0.37| # Table 2: RADV_PERFTEST=nogttspill is NOT being used |model|size|params|pp512|tg128| |:-|:-|:-|:-|:-| |qwen35moe 35B.A3B Q5\_K - Medium|24.76 GiB|34.66 B|1237.55 ± 25.29|53.28 ± 0.24| |qwen35moe 35B.A3B Q5\_K - Small|23.40 GiB|34.66 B|1255.24 ± 6.03|53.33 ± 0.48| |laguna 30B.A3B Q5\_K - Medium|22.64 GiB|33.44 B|204.55 ± 1.66|65.85 ± 0.29| |nemotron\_h\_moe 31B.A3.5B Q5\_K - Medium|25.10 GiB|32.91 B|1117.17 ± 10.11|66.68 ± 0.24| |gemma4 31B Q6\_K|23.46 GiB|30.70 B|40.22 ± 0.12|14.29 ± 0.03| |qwen3moe 30B.A3B Q6\_K|24.53 GiB|30.53 B|1184.70 ± 27.98|66.61 ± 0.27| |deepseek2 30B.A3B Q6\_K|23.26 GiB|29.94 B|920.67 ± 5.82|43.42 ± 0.09| |granite 30B Q6\_K|23.01 GiB|29.28 B|92.20 ± 0.21|17.41 ± 0.04| |muse-glimmer 30B Q6\_K|24.45 GiB|27.85 B|57.15 ± 0.87|11.69 ± 0.09| |qwen35 27B Q6\_K|21.30 GiB|27.32 B|158.73 ± 0.57|14.41 ± 0.41| |gemma3 27B Q6\_K|22.09 GiB|27.01 B|179.26 ± 0.44|17.39 ± 0.02| |gemma4 26B.A4B Q4\_K - Medium|15.63 GiB|25.23 B|1307.73 ± 23.51|53.34 ± 0.22| |llama 7B Q4\_0|3.56 GiB|6.74 B|487.92 ± 37.05|62.03 ± 0.60| Swapping PCIe locations for the Radeon Instinct MI50 and Radeon RX 7900 GRE and using RADV_PERFTEST=nogttspill flag resulted in improvements over my first baseline benchmarks. Note: Only models appearing in both datasets are listed. The table is sorted by Parameters in descending order. |Model Name|Improvement (pp512)|Improvement (tg128)| |:-|:-|:-| |qwen35moe 35B.A3B Q5\_K - Medium|\+221.14%|\+33.85%| |qwen35moe 35B.A3B Q5\_K - Small|\+215.17%|\+1.25%| |laguna 30B.A3B Q5\_K - Medium|\+537.77%|\+17.25%| |nemotron\_h\_moe 31B.A3.5B Q5\_K - Medium|\+267.75%|\-3.90%| |gemma4 31B Q6\_K|\+17.00%|\+23.60%| |qwen3moe 30B.A3B Q6\_K|\+211.32%|\-0.97%| |deepseek2 30B.A3B Q6\_K|\+191.00%|\-14.54%| |muse-glimmer 30B Q6\_K|\+12.49%|\+0.17%| |qwen35 27B Q6\_K|\+17.00%|\-6.79%| |gemma3 27B Q6\_K|\+14.83%|\+1.71%| |gemma4 26B.A4B Q4\_K - Medium|\+158.79%|\+7.28%| Looks like MoE models benefit the most. Muse-glimmer 30B Q6\_K didn't real see much improvement but it seems to be the most optimized dense model. MI50 continues to impress. I purchased them used at $150 each. I can now run models 30B to 35B using Q6 quant with long content and decent speed.

Hello everyone, I made a calculator for the theoretical HW roofline for decoding / prefill based on several parameters (LLM model architecture, quants, GPU, Memory, ..). It still a theoretical bound, but helpful as a step-0 check to understand what fits (would fit) in your hardware, and understand the effects of the contributing knots. I hope it helps. You can access it from here: https://www.ai-leaderboard.dev/ (click HW Roofline)

Hey guys! I have been excited to share this here. This is a project consisting of kernel optimizations for the Tesla p100 series graphics card ($80). I want to start by saying I am 17 years old and do not have a formal degree. I used Ai for a lot of this and while I understand some, I do not understand everything. Notes: My gpus are capped at 175w/250w each so these numbers may be able to be pushed higher. I also experience minor thermal throttling and sit at a nice toasty 79 degrees, which definitely effect numbers (the table above is while hot, so if you have good cooling expect 5-10% more on prefill and decode). I am using gen3 pcie with two x16 slots. Also, for anyone curious, decode numbers depicted in image were averaged from a list of questions ranging from creative writing and coding. Improved: tps went from 7-15tps at 0 context to 50-60, 260k context went from 2-4tps to 30-35, prefill went from 220 tps at 0 context to 350, 260k context went from 40tps (as far as i remember, i never really measured cause it was too hard) to 110tps, fixed fp16 math errors by using some mixed fp16/fp32 math operations so rounding errors were eliminated, and merged as of sept 22 so it should support qwen 3.8 flash architecture This setup is somewhat flag specific (ie: (-c 262144 -b 32768 -ub 1024 -np 1 \\) without -b 32768 mtp becomes overloaded and drops acceptance to near 0 at full depth) so keep that in mind while setting up. One more thing, I took regression very seriously in this. Math had to be more accurate or byte identical or it would fail tests. Build, flags, math proofs, and anything else you may need will be linked below. Enjoy guys! I would love your feedback on this and am looking at pull requests. Github: https://github.com/Kmic-68/llama.cpp/tree/p100-optimizations Details for build, math proofs, etc: https://github.com/Kmic-68/llama.cpp/tree/p100-optimizations/p100-docs

I am busy working away on a tool-calling sandbox. RIght now I'm thinking of building a kind of dataflow analyzer for shell commands, so that I can identify source and sink points, and establish whether the command is a readonly command or a command that changes state. Example: Command: \sed -n '124p' webroot/a-file.html | od -c | head -5 \ sed is a function that can read or write. in \sed -n 999p filename\ syntax on my system, it is a readonly operation \|\ is a left to write data flow transfer operator \od\ is a readonly sink and would be on the readonly whitelist * \head\ is a readonly sink and would be on the readonly whitelist. Therefore, I can conclude that this function is readonly and I would allow it automatically in my solution. Whereas, \sed -n '124p' file > /tmp/foo\ or \sed -n '124p' file | visudo\ would be identified as write commands. Before I get deep into this project, I'd like to know if an existing library already has this as a design goal?

Prompt: Create an animation showing how the soap dispenser mechanism works in one complete html file. Results: Opus 5.5 - High: https://reddit.com/link/1wrsqbj/video/3t6nuj2134sh1/player DeepSeek V4.1 Flash: https://reddit.com/link/1wrsqbj/video/m8i5hsl434sh1/player Qwen 3.8 Max: https://reddit.com/link/1wrsqbj/video/ukkl5qp734sh1/player ChatGPT 5.6 Sol - High: https://reddit.com/link/1wrsqbj/video/ia840utb34sh1/player Opus 5 - High https://reddit.com/link/1wrsqbj/video/hzmnavof34sh1/player

I wanted a simple way to use small models for tiny decisions without wiring them into a full LLM app. So I built jevx. It gives you a CLI for jev and jev based models and connect it from the terminal, shell scripts, CI, or from agents like Claude Code and Codex. https://muthuishere.github.io/jevx/guides/scenarios/ https://github.com/muthuishere/jevx

In your experience, which models sound the least like Claude and more like grok or the gpt's? I cannot stand talking to Claude, to the point I have all requests proxyed through other agents to it. I have been using qwen3.8-27b locally and a heavily quantised version of deepseek v4. I love both for their capabilities, as I did claude to be fair, but I hate interacting with them directly. So right now I mostly interact with SOL 5.6 or Luna Max and have them orchestrate (using setup similar to first mate that i put together myself). I appreciate both models have been distilled on anthropic models, but I'd love to eventually one day be fully reliant on local models but this is one of the last blockers for me. So I thought it'd be an interesting discussion point, most local ones I have tried I find are very similar to claude in tone. Hardware: bosgame Strix Halo, 128 gb unified ram.

Hi! I'm the author of MoEspresso, which is my way of putting my own ideas about inference engines to the test. A lot of the fun has been trying different design choices, measuring what happens, and finding that several of them work well together. MoEspresso 3 runs Qwen3.8-Flash-Next on a 2021 M1 Max with 32 GB of unified memory at 12-15 decode tokens per second - provided there are no other memory hungry applications running in the background (such as browsers). During decode, experts which are already resident in memory receive a bias, but the two strongest experts according to the model are always chosen (with the default settings). I wrote about this here https://github.com/steadfastgaze/MoEspresso/blob/main/docs/cache_prior.md - I first started thinking about this after reading about Apple's AFM 3 and the instruction-following pruning work behind it (https://arxiv.org/html/2501.02086v3#abstract), but then I found this other paper (https://arxiv.org/html/2412.00099v2) which spoke about Cache-Prior. Prefill is unbiased. Even with this bias enabled by default, Qwen 3.8 Next scored ahead of Opus 4.8 xhigh and many other strong hosted solutions. Reproducible 48-question setup -> https://github.com/steadfastgaze/MoEspresso/tree/main/docs/benchmark_reproduc…. Overall scores (%) across six categories, including coding, data analysis and math: | Model | Score | |---|---:| | Qwen3.8 Flash (hosted), medium | 89.7 | | GPT-6 Sol, medium | 89.2 | | Claude Opus 5.5, medium | 86.4 | | GPT-6 Sol, low | 85.0 | | Qwen3.8 Flash @ MoEspresso, medium, Cache-Prior 2/2 | 84.3 | | GPT-6 Luna, xhigh | 81.4 | | Claude Opus 4.8, xhigh | 80.3 | | Claude Sonnet 4.6, high | 74.0 | | Claude Sonnet 5, medium | 70.9 | - I use some of Iwan Kawrakow's formats from ik_llama.cpp, with Metal execution through my mlx-iqk library. Most routed projections in this package use IQ2_K, which is not normally supported by either standard MLX or mainline llama.cpp. - KVarN K4/V4 leaves more memory for resident experts as context grows, and it is functioning extremely well with low RMS error on this model architecture. - Good defaults, e.g. automatic SSD streaming and Cache-Prior when all experts cannot fit, with the settings generally following the same rule. This is the third iteration, and I have more concrete ideas to explore, both for squeezing even more performance from Apple Silicon and for bringing the engine to Linux and AMD machines such as Strix Halo. Installation is through Homebrew, so "brew install steadfastgaze/tap/moespresso" Code - https://github.com/steadfastgaze/MoEspresso Model - https://huggingface.co/steadfastgaze/Qwen3.8-Flash-Next-MoEspressoV3 If you try it, I'd love to see your "moespresso speed" output (an intentionally quick benchmark). PS: English isn't my first language and I used an LLM to help refine this post, and AI coding tools for implementation. --- edit: some comments are reporting lower speeds (thank you for doing it) - I will investigate tomorrow and in next days.

I've seen a lot of conflicting opinions about MiMo Flash, but I haven't tried it yet. How does it compare with GLM Flash for coding work like this? I'm interested in: \- back-end development \- debugging, refactoring, implementing features in an existing front-end codebase \- maintaining Docker images \- troubleshooting DevOps errors Not the silly stuff I see "build me 100 nice-looking webpages" or "make me a Three.js demo". So far, I've been happy with GLM 5.3 Flash. My main frustration is that it sometimes overthinks too much, and once it does, it's hard to steer it back on track. The DeepSWE score of MiMo appears to be a substantial improvement over GLM's, but \- there's no official score from DataCurve \- no amount of consumed tokens to achieve it and in general one benchmark doesn't tell me how it behaves on day to day work. I'd be interested in comparisons from people who've used both.

4 bit pls

Hey guys! Today, we are releasing SupraTTS-0.1-Beta, a tiny \~29.6M parameters Text-To-Speech model. The audio quality is a bit better than the original Glow-TTS (the architecture our model is using!) while it's keeping the same size. Here are some samples: https://huggingface.co/SupraLabs/SupraTTS-0.1-Beta#samples >Link to the model on HF: https://huggingface.co/SupraLabs/SupraTTS-0.1-Beta I hope you can do something useful with it, e.g. on small edge devices and on CPU. Feel free to give us feedback and ask question. Follow us on HF to not miss the next upgrades of SupraTTS, e.g. better voice quality, multi-language-support, multi-voices support, emotions and speaking styles and ZERO SHOT VOICE CLONING**!! 🤗