https://huggingface.co/yandex/AliceAI-Foundation-80B-A3B-Base
It's not a Qwen3 finetune, it's actually its own fully custom architecture. No Llama.cpp support yet sadly
(Also note that this model is NOT post-trained like Qwen3.5/3.6)
https://preview.redd.it/bpbc9i6hizqh1.png?width=1270&format=png&auto=…
I wanted to share a quick update: Alibaba has officially announced Qwen 4 at the Apsara Conference,
Isn't this what simple neural networks have been able to do for years? Doesn't seem anything special to me.
You can simply run any GGUF with llama.cpp with n\_predict=1 and n\_probs=10, disable reasoning, and prompt it such as "If the following email is spam, respond with 1, if not spam, respond with 0. Do not respond with anything other than 1 or 0. Email: ...."
And that is it! It returns confidence percentages such as:
1 = 94.9%
0 = 5.08%
Example:
llama-server -m "C:\\Users\\MyUserName\\llama.cpp\\models\\Spark-X2.5-4B-Q4\_K\_M.gguf" -c 4096 -ngl all -fit off -fa on -b 2048 -ub 512 -np 1 --cache-ram 0 --reasoning off --no-reasoning-preserve --perf
Then:
curl.exe -s -X POST http://localhost:8080/v1/chat/completions \-H "Content-Type: application/json" -d "{\\"messages\\":\[{\\"role\\":\\"system\\",\\"content\\":\\"Classify spam. Reply only 1=spam or 0=not spam.\\"},{\\"role\\":\\"user\\",\\"content\\":\\"CONGRATULATIONS!!! You have won $5,000,000! Click here immediately to claim your prize!\\"}\],\\"max\_tokens\\":1,\\"logprobs\\":true,\\"top\_logprobs\\":10,\\"temperature\\":1.0,\\"top\_p\\":1.0}"
Result:
{"choices":\[{"finish\_reason":"length","index":0,"message":{"role":"assistant","content":"1"},"logprobs":{"content":\[{"id":30,"token":"1","bytes":\[49\],"logprob":-0.00456317700445652,"top\_logprobs":\[{"id":30,"token":"1","bytes":\[49\],"logprob":-0.00456317700445652},{"id":29,"token":"0","bytes":\[48\],"logprob":-5.395024299621582},{"id":1033,"token":"\*\*","bytes":\[42,42\],"logprob":-12.013711929321289},{"id":1046,"token":"The","bytes":\[84,104,101\],"logprob":-13.005236625671387},{"id":198,"token":"\\n","bytes":\[10\],"logprob":-13.100714683532715},{"id":54,"token":"I","bytes":\[73\],"logprob":-14.624603271484375},{"id":3640,"token":"This","bytes":\[84,104,105,115\],"logprob":-14.800630569458008},{"id":130977,"token":"<tool\_call>","bytes":\[60,116,111,111,108,95,99,97,108,108,62\],"logprob":-14.971238136291504},{"id":6908,"token":"Class","bytes":\[67,108,97,115,115\],"logprob":-15.373867988586426},{"id":3923,"token":"class","bytes":\[99,108,97,115,115\],"logprob":-15.442902565002441}\]}\]}}\],"created":1789950066,"model":"C:\\\\Users\\\\MyUserName\\\\llama.cpp\\\\models\\\\Spark-X2.5-4B-Q4\_K\_M.gguf","system\_fingerprint":"b11026-b49650adb","object":"chat.completion","usage":{"completion\_tokens":1,"prompt\_tokens":64,"total\_tokens":65,"prompt\_tokens\_details":{"cached\_tokens":59}},"id":"chatcmpl-x2WrCObzFNYjKVkwDmcL8FLquwfZ0NEa","timings":{"cache\_n":59,"prompt\_n":5,"prompt\_ms":634.566,"prompt\_per\_token\_ms":126.9132,"prompt\_per\_second":7.879401039450585,"predicted\_n":1,"predicted\_ms":0.001,"predicted\_per\_token\_ms":0.0,"predicted\_per\_second":0.0}}
Convert to probability:
probability = e\^(logprob)
1 = e\^(-0.00456317700445652) = \~99.5%
0 = e\^(-5.395024299621582) = \~0.5%
Speed:
On my 170gb/s bandwidth 4gb vram GPU, I got 634ms! On a H200, I would probably get 30-75ms.
Multiple Questions at Once:
In theory you can ask multiple questions at once. You just gotta be clever with the math. For example:
Q1: Is it spam?
Q2: Is it phishing?
Q3: Is it urgent?
Q4: Is it malicious?
A = 0000, B = 0001, C = 0010, D = 0011, .... O = 1110, P = 1111 where each bit corresponds to a yes no answer. Let's say LLM answers with:
A 0.2% B 0.1% C 0.2% D 0.2% E 0.5% F 0.5% G 0.5% H 1.0% I 1.0% J 1.5% K 2.0% L 3.0% M 5.0% N 10.0% O 20.0% P 54.3%
These add up to 100%. To learn possibility of "Is it spam?", just sum tokens where first bit was 1 such as:
I + J + K + L + M + N + O + P = %96.8
Repeating the same logic, you could get:
Spam: 96.8%
Phishing: 91.8%
Urgent: 81.2%
Malicious: 70.6%
I know that there are some bots active on this sub but wow these comments really look AI generated. Is it just me or are those really bot comments?
https://preview.redd.it/20st4vivq3rh1.png?width=955&format=png&auto=w…
source:
I asked this back in 2025, but the AI landscape has changed a lot since then.
Not looking for the usual ChatGPT, Claude, Gemini, Midjourney, etc. I'm curious about the lesser-known tools that you actually kept using.
Could be for research, coding, design, video, writing, automation, planning, journaling, local AI, or even something oddly specific.
Free or paid doesn't matter.
What tool genuinely saved you time or improved your workflow this year? And what do you actually use it for?
Hey all!
I am Aritra from Hugging Face. I wanted to share an update on the \tokenizers\ library that we have at Hugging Face. It has gone under major changes and we have finally released version 1 of it.
Here are what we are most excited about:
\> multiple language support
\> multi-thread scaling
\> minimal package size
Read: https://huggingface.co/blog/tokenizers-v1
ZCode is now open source, and the reported security issues have been addressed.
Source code: https://github.com/zai-org/ZCode
The repo includes its desktop app, web workspace, backend, Agent CLI, and runtime.
Official announcement:
In response to the ZCode product security issues reported by the community, we have completed the necessary remediation and sincerely apologize to all our users.
We have open-sourced ZCode at github.com/zai-org/ZCode, placing the code under community scrutiny and making ZCode more open and transparent.
We sincerely thank the community developers who previously identified issues in ZCode. Going forward, we will establish an ongoing product security vulnerability reporting and response process. We welcome developers to continue reviewing ZCode and reporting potential issues, and we will provide rewards based on the severity of the issues reported.
With respect to the code data referenced by the community, we confirm that no such data is retained and that it has never been used for model training.
Following the remediation, we invited the China Academy of Information and Communications Technology (CAICT) and NSFOCUS to conduct security assessments. The results are as follows:
Through its technical assessment, CAICT confirmed that the zcode-prod Alibaba Cloud OSS bucket is in a zero-data state. Security remediation has been completed in the ZCode v3.14.0 client. The Repo Wiki feature has been removed, and the workflow for generating and uploading local repository snapshots has been disabled.
NSFOCUS confirmed that all data objects in the zcode-prod Alibaba Cloud OSS bucket, as well as the bucket itself, have been deleted. Remediation has been completed in the ZCode v3.14.0 client. The Repo Wiki entry point and the associated generation workflow have been removed, and no functional path capable of triggering the generation of local repository snapshots or transmitting local files externally was identified.
Once again, we sincerely apologize and welcome continued scrutiny from the community. The full security assessment report will be released soon.
I shipped something I've been building for the last few weeks : phantom-kv , a refusal-removal system for large language models that doesn't touch a single weight. Instead of editing the model, it loads a small, learned bank of key/value tensors into the model's KV cache as context. Attention reads it like conversation history that's already there.
https://github.com/lordx64/phantom-kv/
https://reddit.com/link/1wms904/video/7efg1le3eyqh1/player
The result is that "uncensoring" stops being a permanent checkpoint edit and becomes a per-request, hot-swappable capability mode: unload the cache and the base model is byte-identical again.
Every prior approach to refusal removal commits somewhere permanent. Weight-space abliteration rewrites the checkpoint undoing it means re-flashing weights, and it breaks per quantization. Activation-space projection subtracts a refusal direction at runtime, per token, per layer, from inside an engine hook the model's signal path itself is patched at boot. phantom-kv does neither: it's trained offline against the model's own objective (comply on harmful prompts, preserve behavior on harmless ones), ships as megabytes of cache content instead of a new checkpoint, and influences the model only through the input channel attention already consumes. No 1-D refusal-direction assumption, no forwarding-pass hooks, no per-arm rebuilds for new architectures.
We also audited ourselves: an 8B judge-model audit shows lexical refusal-suppression metrics over-claim compliance (semantic refusal often persists as rephrasing), the graft fades with a \~2–4k token half-life in long sessions (and a measured re-injection cadence mitigates it), and answers come with legal/ethical framing ling because the graft's job ends where the model's profession takes over.
Source : https://x.com/lordx64/status/2102138825292276168?s=20
• Ming-Image-0.1-Design, 6B • Ming-Image-0.1-Design-Layer, 6B • Two open-source Agent Skills: the Ling UI Design Skill and the Image-to-Editable-PPT Skill Ming-Image-0.1-Design ranks #1 among open-weight models on Artificial Analysis’s UI/UX Design leaderboard. https://huggingface.co/inclusionAI/Ming-Image-0.1-Design https://huggingface.co/inclusionAI/Ming-Image-0.1-Design-Layer
Hey everyone!
It has been quite a while since the last SupraLabs model - but today we've something special for y'all: Supra2-IMG
It's a 100M parameter DiT text-to-image model trained entirely from scratch in under 10 hours on a single H100 on Runpod. It can generate state-of-the-art quality images in 256x256 pixels resolution.
Samples:
https://preview.redd.it/9paalvbs4wqh1.png?width=620&format=png&auto=w…
These samples are NOT cherry-picked! Sampling: seed 0, steps 50, cfg 3.0; same settings for every image.
If someone here is interested in the prompts, I can give them to you! Feel free to ask!
You can also use the model locally on your hardware (\~20s for an image on CPU (🤩) and \~2s for an image on GPU):
First, run:
Then, you can generate images by running:
python inference.py --prompt "a sea jellyfish floating in the pitch-black ocean depths" --seed 0 --cfg 3.0 --steps 50 --n 1 --out jellyfish.png
Have fun 🤗 🔥
Link to the model on HF: https://huggingface.co/SupraLabs/Supra2-IMG
Give us a like and a follow on HF if you want 🤗 ❤️
EVERY feedback is welcome, guys! Feel free to ask any questions!
This post is written by a human and I'd appreciate it if you treated it as such. Thanks.
So, I've been noticing a pretty clear interest in developing as good a coding and agentic tool-calling model as possible, especially at smaller sizes, sub-50 gigs. However, I'm finding that at least for my use of AI, if I really want to move away from big providers, I am going to require a model that has better world knowledge than the current offerings.
Qwen 3.8 27B is a truly fantastic model for tons and tons of stuff. It's highly intelligent, super good at designing applications and coding and working on my system. However, its world knowledge sucks compared to the frontier, especially at the Q4 quant that I have to run it at.
So, that leaves me with a question. With the new N-gram technology that we're seeing being baked into Qwen 3.8 Next and that presumably will run on future models, why can't a model be made that has a smaller set of intellectual capabilities but a greater amount of world knowledge? I understand that right now everyone is optimizing towards making as smart a model as possible fit into as small a space as possible. But why don't we leverage the SSD to give the model a lot of world knowledge and make models that are better at dealing with screenshots, multilingual capabilities, doing things like pixel art or answering physics questions?
ngram seems like the answer to the "can't fit in vram" question... Qwen 3.8 Next really opens my mind to the possibility that there could be a totally different and better paradigm for how these models are developed, at least for many use cases. Having a relatively smart model with a large amount of world knowledge might be better than having as smart a model as possible...
Not to mention that this would mean that a model's training cut off would become less relevant, because it could just be fashioned a new ngram.
Obviously, the main interest is in creating a model that can code as well as possible because that's what'll capture market share. But I am curious if there are any efforts into this kind of thing or if anybody has an idea on why these things aren't done more often.
Please tell me why I'm wrong, how I'm wrong, and in how many ways I'm wrong because I'm sure that that's all you really want to tell me, but at least I'll learn something, because as is obvious from this post I have no idea what I'm talking about.
Thanks have a good day :)
edit: I found this post and I guess it provides a lot of what I was asking:
https://www.reddit.com/r/LocalLLaMA/comments/1vzgtqf/ngram\_vs\_experts\_explained/
edit2:
this one is even better, recconend reading. ty reddit suggestions:
I’ve been working on making small models more capable at agentic coding and work, because most people in the world don’t have the sort of hardware needed to run 3.8-27B, or even 35B-A3B or 9B dense, and I want to extend local agentic coding capability to less privileged users. This quant can be run on a smart phone or older gaming laptop, and can solve real coding problems autonomously in a way I have never seen or measured for this model class. Spark-X2.5-4B is already around best-in-class for its size, and I think these improvements bring out the best in it. I hope this little step up in small-model capability and speed in real-world coding might give new life to older hardware that would otherwise be forgotten in the AI frontier race.
The changes SharpSpark makes to Spark-4B are in three parts: First of all it fixes issues with the chat template, and replaces the system prompt with one that improves agentic coding behaviour, token use, and correctness. Then a custom importance matrix is calibrated for the model, which relocates bit precision within tensors to the parts that are more important to agentic coding work. The imatrix corpus is heavily weighted against both agentic coding and cybersecurity, which together protect the cognitive core used to find and solve hard bugs.
Then Spark is quantized with an optimized non-standard quantization strategy, that allocates bits differently per-tensor than standard llama.cpp GGUF quantization. I built a tool that explores and tests different per-tensor allocations to optimize KL-divergence and long-context retrieval for this model, but ended up making some manual changes that ended up favouring SWE-bench-Live performance over traditional fidelity measures like KL-divergence, which published science indicates is actually a poor proxy for real-world performance on complex tasks below a certain point.
If you have a small GPU and/or <= 16GB RAM and can’t run a 35B-a3b MoE-based model with partial GPU offloading, this is likely your best option for long-context agentic software development right now. SWE-bench-Live is chosen as the metric for its genuinely difficult real-codebase problem set.
I’m just a volunteer doing this as a non-profit side project, so please be kind about the fact that my benchmarks are not extensive. They are what I could afford the time and effort to run, with all my other projects, and I see them as just good enough to prove the improvements on the specific kind of work this quant was designed towards.
https://huggingface.co/peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF
Don't know how they did it, but for under 10GB model, the results are astonishing. I am running it on Unsloth Studio. They just released the update, so if you are not seeing the option, I recommend updating your Unsloth Studio. Cheers!
Sorry for the pretentious name, I know, I know.. It just contains all the pieces I would like to see a AGI model to have, and I can't stand the temptation. Before throwing rocks at me, please take a glance at the Readme, and I hope it will cover your mood a little bit.
So, first of all it does work and you can see the sample from the whole training run here: https://raw.githubusercontent.com/volotat/mini-AGI/refs/heads/main/runs/sampl…
Here is the scaling law graph I have so far, and it looks very promising:
https://github.com/volotat/mini-AGI/blob/main/assets/scaling.png
The model was built under my deep dissatisfaction so we cannot really train even moderately big models (1B+ scale) on the consumer's hardware. We can inference and fine-tune them for sure, but I would like to have full control over what the model sees over the training run, so it is fully aligned with my interests, not some corporations.
I was thinking about for some time and come up with two interesting ideas I thought worth pursuing: MoE with a lot of experts that gets added and pruned from the model while it trains, where only a small subset of of experts are actually in use at any particular moment + batch 1 training on the single continuous stream of data.
First allows us to be bounded only by the disk space in terms of number of parameters and load and unload experts only when they are needed. The second (if figured out and it turns out to be doable) allows us to get aways with small VRAM capacity because we do not need to store big randomized batches and their respective gradients.
I started brainstorming with Claude and after some time we found an approach that seems to be promising, and low and behold, a few weeks pass and you can see the results yourself.
Obviously, I did use AI in the process of making this project and I am pretty sure it would be completely impossible for me to do something like this without it, so I hope it is more than justified.
The model is still running over the first of 7.8B characters corpus I selected for training, so the weights are not out yet, and it's about a couple weeks of waiting until they are cooked at the current reading speed. And yeah, the model just read continuous interleaved passages from the dataset, each by 32K characters long each as a single stream. Just as you or I would do.
The set up seems to be really simple so you can git clone the project, run it and observe everything for yourself.
Thanks for your attention.
Quote: DeepSeek is training a 2T-parameter model and plans to eventually build an 8T-parameter model.
https://x.com/wallstengine/status/2101982843656388644
Current DeepSeek models:
Mythos / Fable is estimated to be 10T parameter count.
A 421M-parameter model just played Flappy Bird on my desktop CPU (OpenVINO int8)
Running on my Intel Core i7 12th gen CPU
Converted laya system one model to OpenVINO and quantized to int8
Basically the title.We did not get a new moe model with qwen 3.8 and Alibaba did not announce any small moe models on apsara.I know we might get an announcement later but ngl I kinda lost hope
It is 12 points higher than the best open model mimo 2.6 pro and a big jump from fable 5.1. Crazy, glm 5.5 and qwen 4 will be on par with gpt 6 sol or better since it has a score of 48
If it took 2 months for the best open model to go from 44 to 46, then at this rate, in 6 months , they will reach 58? It is quite possible they will reach it in 4-5 months, since they have will more leaps in intelligence as they deploy more gpus and scale up the parameters, data and compute and improve the architecture .Wow sol 6 is worse than 5,6 at deepswe?
Since there’s no comparison chart on the model page, I asked Perplexity to compare it against some relatively small open-weight models in a similar size range. Here are the results.
Upd. Terminal-Bench 4.0 results:
MiMo‑V2.6‑Flash‑RL — 28.8%
DeepSeek‑V4‑Flash‑0731 — 12.0%
Qwen3.8‑Flash‑Next — 25.3%
GLM‑5.3‑Flash — 32.8%
AntLing open sourced the Ming-Image-0.1-Design family:
• Ming-Image-0.1-Design, 6B
• Ming-Image-0.1-Design-Layer, 6B
• Two open-source Agent Skills: the Ling UI Design Skill and the Image-to-Editable-PPT Skill
Ming-Image-0.1-Design ranks #1 among open-weight models on Artificial Analysis’s UI/UX Design leaderboard
Someone recommended that I try the ByteShape Qwen 3.8 27B IQ3-XXS GGUF after seeing my previous testing of the GSQ quant.
So I did.
And the result was… surprisingly bad.
For context, I'm running:
The two low-bit quants I compared were:
ISTA-DASLab / GSQ-RCO-IQ3-XXS
ByteShape IQ3-XXS
On paper, the ByteShape quant looked very interesting.
It was smaller, while apparently retaining extremely high similarity to the original BF16 model. It was also being compared in size to significantly higher-BPW quants.
So naturally I expected it to at least be competitive with the GSQ version.
It wasn't.
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF was able to generate a 3D voxel diorama in one shot under 55k tokens, and Byteshape's 3.8 27B model, took roughly three shots and still hasn't completed with over 98K tokens spent already.
Same with web development not impressive as advertised in Here
Any New Model Suggestions for RTX 3060?
Saw this today and found it very intriguing. Lots of interesting design choices here, and it's cool to see someone doing something different. Here's a few highlights:
Weights will be released in "a couple weeks" once training progress reaches \~GPT-2 levels. The trend line has held 15-fold so far, but it may bend at some point, so that is definitely a rough estimate of the trajectory.
What do you guys think?
I was checking nee Mimo 2.6 architecture on huggingface page and it looks very simple. I dont mean in a bad way but when we compare recent open models, their architecture is very simple. They dont use any Gated DeltaNet, no mHC or similar architecture, no engram. Just ordinary simple architecture and very good RL i guess.
What are you guys thinking about this?
\# the What
An engine to run Gemma 4 31B on blackwell under massive concurrency and rather specific workload patterns. I've been waiting for someone to do ninfer but for gemma, and, well, ended up having to do it myself.
More models and potentially more gpus are likely to be added, but its main purpose is to be my own workhorse, and I do not have the capacity (or desire) to chase every new release. I do love the gemma 4 family as a whole tho, so they are very likely coming soon.
\# the Why
Ironically, there has just been a post on "stop making slop inference engines", so... why bother with own engine if vllm exists? Well, neither vllm nor lcpp dont utilize one of the Gemma's big strengths, which is being able to have your kv cache use \*0.625x the vram\* losslessly. Not "trust me bro" losslessly, but like, mathematically losslessly down to the order of reduction.
Why? Because they decided to tie K and V weights on global attention layers, and rope only rotates 25% of K. So we can store only V and 25% of K, while other engines store full K and V. It is slightly more computationally intensive to have to unsqueeze them for the math, but it very quickly becomes outweighed by having to read less from memory. Blackwell has way more compute than vram bandwidth. And, well, lets you pack more context or more cached prefixes into the same amount of memory.
Also, vllm's cache sucks. Like, really sucks. It is good for when you have a lot of random users sending random prompts, but lack of explicit cache controls and LRU policy really makes some loads suffer, and SWA snapshots are clearly an afterthought (cant blame them for that because vllm predates SWA by a few years, but still). Gewell is built around efficient use of checkpoints, ram offloading and both smarther default eviction policy that assumes you are going to have repeating prompts with significant intervals and explicit cache hints on the prompts themselves. More about how cache works here: https://github.com/leDissolution/gewell/blob/main/docs/cache.md
Tl;dr: say, you have two chats going on you are alternating between. If you send ten messages into one of them in a row, vllm will make 10 checkpoints and evict the otehr one; gewell will dissolve some of the the intemediate checkpoints and preserve the second one warm.
Why it is important? Well, I'm using gemma for data generation and grooming, and most of these workflows have writer + ctitic or planner + writer + critic loops, sometimes with even more separate prompts cycling around. Each of these prompts is building up on top of its own's previous turn history so their prefixes are perfectly reusable, but vllm insists on pushing them out. It gets even worse if there are some one-off prompts that arrive every 10-20 turns and will never be reused, yet they still take up prefix cache and evict something useful.
Gewell also starts fast. Like, \*fast\*. Literally couple of seconds on top of reading the weights from the drive, because instead of doing live kernel profiling to select gemm shapes the choices were profiled offline and hardcoded and there is no python import tax.
\# the How Fast
Decently fast. TTFT is generally slightly behind vllm on large batches (because scheduler prioritized saturating decode width over latency and high-batch prefill is slightly slower for lower quants), but overall t/s is generally higher - especially on the workload it was designed for (bunch of prompts that keep growing but not all active at the same time).
https://preview.redd.it/tzn2iprbfuqh1.png?width=2188&format=png&auto=…
https://preview.redd.it/cho1porbfuqh1.png?width=2108&format=png&auto=…
https://preview.redd.it/2ul22prbfuqh1.png?width=1939&format=png&auto=…
\# the Quants
Gewell uses its own quant format that allows for arbitrarily mixed precision. The convertion tool lets you repack any compatible checkpoint with whatever bpw you want.
The "main" quant it was developed around is G0: https://huggingface.co/LeDissolution/Gemma-4-31B-it-Gewell\_G0
It uses around 6bpw, allocating most of them into attention and global-attention-adjacent MLP.
Why not qat? Well, because it is kinda bad in my experience (especially in the context fidelity and vision). Nvidia's nvfp4 was my go-to, but my personal tests showed that 16bit in attention are mostly wasted and mlp needs some juice too. Intuition being that if we take the beautiful precise 16-bit attention and then pass it through 4-bit up-gate, we just lose all that fine detail anyway. Idk whether it is mechanically correct, but seems to work? YMMW.
https://preview.redd.it/3e02slcefuqh1.png?width=1580&format=png&auto=…
https://preview.redd.it/6phv3wcefuqh1.png?width=1580&format=png&auto=…
https://preview.redd.it/gayesosvfuqh1.png?width=1580&format=png&auto=…
The tasks here are \~2.5k example mix pulled from aya\_dataset, OpenR1-Math, DocVQA, ChartQA, QASPER and code\_contests
NIAH is a RULER-inspired torture test where the model is fed a huge uniform block of key-value pairs with distractors and overwrites:
Record 3832768 stores value ocean.
...
Record 3832760 stores value rose.
Record 3832761 stores value pearl.
...
Record 3832767 stores value ocean.
Record 3832768 stores value river.
Requested keys in order: 3832768 3832760 ....
And the model needs to respond with exactly the same amount of values in the exact requested order. Amount of needles is 16 for the current test set; completion was counted as % of the correct values in correct spots. At 64k even bf16 can not complete a single request perfectly without reasoning.
\# the Supported Hardware
It was developed and tested on linux and pro 6000. I have not tested it on 5090 because I dont have it, but the intent behind choosing the quant size was to have the weights + mtp + 250k context fit in 32gb. Adding vision might require reducing the context size a bit.
Windows support was not tested either (my windows machine got 3090s), but there is nothing that prevents it in principle, so you are welcome to try.
\# the Limitations
I did cut some corners on the interfacing side. The samplers support is currently very rudimentary (only temp, top-k and top-p), there is no way to override the chat template (the latest google's one is hardcoded in), and some less common text/chat completion knobs might be missing.
\# the Roadmap
There are likely some bugs to be fixed I did not find when using it myself, and some more works has to be done around the API. Next big thing I plan is supporting 26A4, but no promices when.
I also have a bunch of ideas around better speculative drafting, and it might or might not come before 26A4.
https://preview.redd.it/nulsv53o8vqh1.png?width=4500&format=png&auto=…
Spent weekend benchmarking the Splash engine (by Incoai) and extending its architecture to native 8-bit on Apple Silicon (M5 Pro, 64 GB unified memory).
Splash is a compiled C++ and Metal speculative decoding engine designed specifically for Apple Silicon. Upstream Splash pioneered a blisteringly fast speculative decoding pipeline for 4-bit models (\~60 tok/s). However, aggressive 4-bit quantization hits a nasty "reasoning cliff" on competition-grade math and multi-step derivations.
We wanted to bring Splash's speed to true uncompressed 8-bit weights without losing its speculative decoding advantages. By extending Splash's architecture to support native 8-bit tiled Metal kernels (schema 5, MDFL0008), we were able to sustain 37–55 tok/s with zero quantization degradation.
Note on compatibility: Official upstream Splash 1.0 (incoai/splash) hardcodes package validation to 4-bit schemas (splash-packed-q4, schema 3/4). This fork adds schema 5 (splash-packed-q8, MDFL0008) loading and compiled Metal Q8 tiled decode kernels, while keeping 100% backwards compatibility with upstream Splash's official Q4 models. Proposed upstream: \[incoai/splash#94\]([https://github.com/incoai/splash/pull/94](https://github.com/incoai/splash/pull/94)).
Evaluated at temperature=0.0 across 5 standardized task domains:
|Task / Domain|Prompt Description|Splash-Q4 (Official 4b)|Splash-HQ (Native 8b)|Splash-Q8 (Compressed)|MTPLX-Q8 (MTP D3)|Stock MLX / llama.cpp (AR)|
|:-|:-|:-|:-|:-|:-|:-|
|Math & Logic|Algebraic derivation|83.3 t/s|54.8 t/s|52.7 t/s|28.5 t/s|9.9 t/s|
|Coding & Algos|merge_intervals $O(N log N)$|75.5 t/s|34.7 t/s|40.3 t/s|28.8 t/s|9.9 t/s|
|Constraint Reasoning|3-chair spatial permutation|59.3 t/s|39.3 t/s|37.4 t/s|27.7 t/s|9.9 t/s|
|Domain Knowledge|FlashAttn vs PagedAttn|39.0 t/s|21.9 t/s|22.8 t/s|23.8 t/s|9.9 t/s|
|Nuanced Writing|Memory bandwidth constraint|46.2 t/s|33.7 t/s|29.5 t/s|23.4 t/s|9.9 t/s|
|AVERAGE|Across all 5 domains|60.7 t/s|36.9 t/s|36.5 t/s|26.5 t/s|9.9 t/s|
|Speedup vs AR|Relative to 9.9 t/s baseline|6.13x|3.73x|3.69x|2.68x|1.00x|
A few notes on the comparisons:
Qwen3.8 is architecturally specified with a native 256k context window (262,144 tokens). Most Transformers fall off a cliff in decode speed as context grows because the KV cache balloons.
However, Qwen3.8 uses a hybrid architecture: 48 recurrent linear DeltaNet layers (fixed $128 \\times 128$ hidden state, $O(1)$ memory growth with context) and only 16 full-attention layers.
On a 64 GB Mac, we pushed it live in an active server session all the way out to 190,016 tokens to see if decode speed degraded under real usage:
|Context Length (Tokens)|Cached Tokens|Generated Output|TTFT (Prompt Prefill)|Decode Speed|Notes|
|:-|:-|:-|:-|:-|:-|
|65|0|50|0.8s|35.7 tok/s|Short prompt baseline|
|16,433|15,040|232|4.0s|49.0 tok/s|Prefix cache hit|
|34,605|29,376|2,771|15.6s|27.1 tok/s|Long response generation|
|83,379|76,320|435|28.9s|30.1 tok/s|Deep context code review|
|106,212|98,752|29,487|34.0s|24.8 tok/s|Massive batch generation|
|157,961|157,056|400|6.1s|43.5 tok/s|Cache hit at 158k tokens|
|180,082|143,360|3,446|228.5s|33.3 tok/s|Extended reasoning session|
|187,613|186,720|425|6.5s|31.9 tok/s|Cache hit at 187k tokens|
|188,546|147,456|1,083|268.2s|21.1 tok/s|Partial prefill recompute|
|190,016|151,552|1,115|227.4s|32.0 tok/s|Max context reached (64GB RAM)|
(See the visual plot in the repo: *benchmark\_and\_context\_scaling.png* showing the full 51-point scatter and rolling trend line).
The big takeaway on context: Decode speed does not collapse. Thanks to Splash's memory handling and the hybrid architecture, it stays between 21 – 33 tok/s across the entire range.
The actual bottleneck at 150k+ context is cold prefill (TTFT). When the prefix cache hits, TTFT at 187k context is just 6.5 seconds. But on a cold cache miss, prefilling 180k+ tokens on a 27B model on Apple Silicon takes \~4–5 minutes. If you are using agent harnesses (like Oh My Pi, Claude Code, or curl), make sure client SSE idle timeouts are set high enough so the client doesn't drop the connection during cold prefills.
Throughput numbers don't matter if math derivations hallucinate. We tested extended CoT reasoning on MATH-500, AIME 2025, and GPQA Diamond:
https://preview.redd.it/29mgrn0c9vqh1.png?width=2400&format=png&auto=…
Needs: Apple Silicon Mac, macOS 26.4 or later, 48 GB unified memory (64 GB recommended; the weights alone are 27 GB).
The GitHub repo holds the C++ and Metal runtime engine, while the 27 GB model weights are hosted on Hugging Face. You don't need to manually download model files with git-lfs or separate scripts—Splash has a built-in package downloader.
curl -fsSL https://raw.githubusercontent.com/npanj/splash/q8/install-q8.sh | sh
No Xcode, Homebrew or pip needed. It installs a splash-q8 command and doesn't replace an existing Homebrew splash.
When you run the command below, Splash automatically detects missing model artifacts, connects to Hugging Face, streams the 27 GB files with progress bars, verifies the manifest SHA-256 hashes, and boots the engine:
splash-q8 serve --model nitinpanj/Qwen3.8-27B-Splash-HQ
(Once downloaded, subsequent runs load instantly from local disk offline).
(Optional: If you prefer to pre-download the model files beforehand via Hugging Face CLI instead, you can run:)
huggingface-cli download nitinpanj/Qwen3.8-27B-Splash-HQ
The server exposes a standard OpenAI-compatible /v1/chat/completions endpoint on http://127.0.0.1:8000:
Building from source instead? You need full Xcode, not just the Command Line Tools. On Xcode 26, first run xcodebuild -downloadComponent MetalToolchain, then make -j4.
Memory: growth paused). If you don't need 190k context, you can pass --max-context 131072 to cap it cleanly.splash-q8\ also serves the official 4-bit models. This fork preserves all upstream Splash 4-bit dense and MoE schemas (splash-packed-q4, splash-packed-q4-moe), so you can serve official models like incoai/Qwen3.8-27B-Splash or incoai/Qwen3.6-35B-A3B-Splash directly.Full credit to the Incoai team for creating Splash (https://github.com/incoai/splash). Their C++ Metal speculative decoding architecture is what makes these speeds possible on Apple Silicon in the first place—this fork simply extends their work to support native 8-bit weights and custom Q8 tiled kernels. Also huge credit to the Qwen team for base weights and MTP architecture, and Youssofal for MTPLX reference benchmarks.
Updated: setup so that compilation is not needed
Updated (10/1): you can now find follow up work for Qwn3.8-Flash-next here: https://www.reddit.com/r/LocalLLaMA/comments/1wva7l2/running\_955\_gib\_qwen38flashnext\_at\_4152\_toks\_on\_a/
Laya is an open-weight (Apache 2.0) "System 1" decision model from Convai Innovations, built by Nandakishor M as an open alternative to TypeSafe's closed Jev API. Instead of generating text, it takes a state (text, an email, a ticket, or JSON) plus typed questions (choice to pick a label, score to place something on an ordinal rubric, and noul for a yes/no probability) and answers all of them in one forward pass in about 33–40 ms on a GPU, so there's no output to parse and nothing to hallucinate. It comes in three checkpoints: a 421M-parameter English model on ModernBERT-large, a faster 322M multilingual model on mmBERT-base covering 100+ languages, and a variant fine-tuned for typed-decisions workflows. A built-in Router detects the input's script and sends it to the right checkpoint. It's trained with RLCD, a reinforcement learning method whose reward uses strictly proper scoring rules, so the model maximizes reward only by reporting honest probabilities; it also has an act-vs-escalate head for deciding when to hand off to a human. The author reports strong results, including beating Jev on AG News, emotion classification, and the typed-decisions benchmark while running roughly 6–8× faster, though the Jev figures are third-party numbers rather than head-to-head runs. The model card is also candid about its limits: the base checkpoints are near chance on typed-decisions without fine-tuning, accuracy drops sharply with 50+ options (Banking77: 0.425 vs. Jev's 0.870), ordinal scoring is its weakest question type, the English checkpoint fails on non-Latin scripts, and the models ship overconfident, so you need to fit a temperature on your own data before trusting the probabilities.
This sub is, needless to say very niche and skewed towards the high end. There are tons of extremely high end setups here with multiple gpu's etc.
Even 24GB is out of reach of most people financially, forget about the 3x3090 or 5090 or even higher setups. Macs/Strix Halo/dgspark etc are all similarly expensive. 16GB is pretty much the high end for most. And this completely changes in most of the rest of the world where even 12GB would be a luxury.
Things have changed recently (I think even last 6 months have been huge) and even agentic coding is now feasible on 16GB cards (eg with Qwen 27B quants).
I think/hope things will continue to improve. Of course there's going to be a hard limit on how much world knowledge these smaller models will have.
The holy grail is new architecture that supercedes the Transformer and new techniques that don't depend on vram/bandwidth.