Here we go again, DeepSeek is back again with a new model V4-1 Flash
A multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens
Market crash as a service
I made this because I was getting genuinely annoyed at trying to have a normal conversation with LLMs. Even with prompting and various tricks, most models I've tried still have this "AI assistant" vibe to them that is so familiar: too helpful, polished, verbose, using words we never use in conversation, etc.
I wanted a model that could just talk to me like a person, so I did the slightly unreasonable thing and put together a dataset and trained one.
The dataset used for training is 125,217 obfuscated human-to-human messages across 1396 chat conversations.
The goal wasn't to make Qwen smarter or improve benchmark scores. I was trying to change its conversational habits, to make it stop turning every reply into an explanation, agreeing with everything, and writing stuff just to keep the conversation "going".
I trained a rank-256 LoRA on top of huihui-ai/Huihui-Qwen3.8-27B-abliterated. The released version is checkpoint 863. In my testing it feels noticeably less like an assistant, particularly in casual conversations, even without a system prompt. Replies are generally shorter, less polished, and, well, more human.
There may be a tradeoff. An earlier iteration scored five percentage points lower than its Huihui parent on IFEval, an instruction-following benchmark. I haven't rerun that benchmark on this version of the checkpoint, and I haven't tested coding performance, so I don't want to pretend that number applies here.
I've added a side-by-side comparison using the same system prompt, user messages, and generation settings for both models. Each model continued its own conversation branch, with reasoning effort set to 'xhigh'.
Merged GGUFs and the standalone F32 LoRA adapter are in the model repo:
https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-GGUF
Space where you can have a demo chat with different system prompts and reasoning modes:
https://huggingface.co/spaces/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat
UPD: I certainly didn't expect this post to blow up like this! There's been a lot of great discussion in this thread and a lot of insight for me on where to take the model next.
A few have asked for our Discord, and we'd be happy to see you there: https://discord.gg/aCCrWftMjS
A few years ago a 100GB was considered a very large language model. What do we call under 100GB models now? Tiny models? haha
I wonder someone will figure out a way to do this with 27B?
Throw Qwen3 on this page for demo
https://kishida.github.io/webdemos/llkvapprox/
Edit: sources (thank you u/pmttyji for finding them!
Hoping to see smartest medium size models soon & later with all available optimizations/architectures/etc.,. Thanks Deepseek!
Ex 1: 30-50B MOE + 10-15B Engram + DeepSeek-V4.1-Flash type KVCache
Ex 2: 15-30B Dense + 10-15B Engram + DeepSeek-V4.1-Flash type KVCache
EDIT: Updated Engram to 10-15B from 50B
Surprised no one has posted it in this sub.
Pretty solid model, IMHO.
so i saw that openui.com released OUI-1, a model fine-tuned on DiffusionGemma. the training dataset uses OpenUI-Lang, a custom DSL (domain-specific language), instead of plain HTML, Markdown, or React code.
what makes it interesting is that you can already get a regular LLM to use OpenUI-Lang through a system prompt, but that eats up a lot of the context window. my thinking is that fine-tuning a model on the DSL could reduce that overhead and leave more room for the actual conversation, without needing a huge prompt explaining the format and how to use it alongside other tasks, like tool calls.
at the same time, wouldn't fine-tuning a model on a specific DSL make it more likely to default to that format even when you need something else? i'm curious how well it handles regular Markdown, or switching between Markdown and OpenUI-Lang.
i haven't seen much discussion about this, so i was wondering what everyone thinks about generative UI and running a dedicated model for it locally on a consumer-grade GPU, like an RTX 5090.
what would be the best way to set that up? from what i've seen, DiffusionGemma isn't supported by llama.cpp yet, so running it through Ollama doesn't seem to be an option. they've uploaded the weights to Hugging Face, but i'm not really sure how to get it up and running. any suggestions?
I was not aware that the harness makes such a big difference.
💡 TL;DR (from the Github Readme)
Spend less without making the agent do less useful work.
SoL-Pi is a standalone extension for Pi that packages four reusable efficiency mechanisms discovered through scaled auto-research loops. It reduces repeated model turns, context replay, oversized observations, and unnecessary long-log reading while preserving the work and evidence an agent needs to finish a task.
SoL-Pi installs on top of an unmodified Pi release. Every mechanism is opt-in and disabled by default.
Introduction
Long-running coding agents accumulate repeated work. A file edit is often followed by a predictable validation command. Large tool results are replayed long after their first use. Completed subtasks remain in active context, and a frontier model may spend a full request reading a log when only a few lines affect the next decision.
SoL-Pi grew out of a broader question from our auto-research work: before scaling agent loops, can agents first make the harness itself more efficient? The search focused on constrained efficiency: reducing token traffic, inference work, and agent turns without stopping early, skipping verification, or hiding evidence.
The standalone release contains four mechanisms that survived that process. They operate at different parts of the harness and compose through Pi's public extension APIs.
What SoL-Pi Adds
Area Mechanism What changes
Tools Action Fusion An edit or write can run its follow-up validation command in the same tool call.
Observations ObservationPack Repeated large text results become stable handles with exact paged recall.
Delegation Evidence-Preserving Reducer Long diagnostic logs become compact receipts only when every retained quotation matches the archived source.
Context Online Context Compact Completed plan steps become candidate points for Pi's native compaction, subject to economic and window-pressure checks; after a successful compaction, Pi continues the task in a new turn.
The mechanisms share four rules:
-No Pi patches. SoL-Pi imports public Pi APIs and does not vendor the Pi source tree.
-Explicit opt-in. A missing configuration leaves every mechanism disabled.
-Preserve evidence. Original observations remain available locally, and reducer failures leave the original result unchanged.
-Use Pi's runtime choices. Authentication, provider URLs, the main model, and shell behavior remain under Pi's control.
People keep on getting confused about this, so I looked at the safetensors on hf.
The title should have been "Deepseek V4.1 Flash is 748B total/552B base, not 284B or 305B or 485B or 522B"
"So: ~305B real backbone + 203B engram = 508B total" This is incorrect.To be precise, the main model about 551.566B parameters with 40 layers. The FFN experts total to 543.582B parameters, and the rest of the model (attention, shared experts, etc) are 7.984B.
On top of that, the engram is \~196.929B, DSpark/MTP is \~14.225B, and the vision encoder is just \~0.485B. These parts are technically optional though. The vision encoder is also way smaller than I expected.
Anyways, you need a beefy system for this. 128GB or 256GB of RAM/VRAM is not going to cut it.
|Component|Logical params|Size in GB|Storage|
|:-|:-|:-|:-|
|FFN MoE experts|543.582B|288.778 GB|FP4|
|Other FFN|1.4947B|1.574 GB|FP8 mostly|
|Attention|5.1269B|6.524 GB|FP8 mostly|
|Embedding + LM head|1.3238B|2.648 GB|BF16|
|Other|0.0397B|0.158 GB|FP32/BF16|
|Backbone total|551.566B ≈ 552B|299.682 GB||
|Engram lookup tables|196.614B|202.758 GB|FP8|
|Engram projections/gating|0.315B|0.315 GB|FP8 mostly|
|Engram total|196.929B = 196B advertised|203.073 GB||
|DSpark / MTP|14.225B|8.033 GB|mostly FP4 experts|
|Vision encoder|0.485B|0.971 GB|BF16 mostly|
|Everything in total|\~763.21B params|\~511.76 GB||
OpenAI has decided to fully shut down a protein design project I'm working on for a client. Needless to say, open weight models are the only way forward.
Like many of you, I have seen many posts and tweets in the last weeks complaining about Artificial Analysis being "broken", "meaningless", and "bought out." People who say this have done no research and know very little about how benchmarks work and what they measure.
Most people only care about Artificial Analysis Intelligence Index. This is a weighted aggregate benchmark used to compare models performance across 10 different evaluations. The majority of these evaluations have published papers on arxiv.org. AA-Briefcase is the only private benchmark. And they publish their methodology to confirm how each of these models are weighed.
Some people seem to not appreciate that Artificial Analysis conducts their own independent benchmarks using their OWN funding, without running ads. Here is the chart that shows their spending. They spent $13,129 to independently test Fable 5.1. Every new model seems to be benchmarked.
The new Deepseek V4.1-Flash is a perfect example of why some aggregated scores miss the big picture. This 552B model has the same score (40) as the 180B Qwen 3.8-Flash-Next. But the individual benchmarks show a different story. On most evaluations, it matches or exceeds Qwen 3.8-Flash-Next. It every beats GPT-6 Astra (Max) in AutomationBench-AA (Agentic SaaS workflows), which is incredible. But it completely falls behind in AA-Omniscience Non-Hallucination Rate, a metric where Open-weight models usually reign supreme. So the model has strengths and weaknesses, and it's something that should be celebrated.
So before you complain about benchmarks or Artificial Analysis, look at the individual evaluations. Read the published papers about the evaluations. Learn how the score is aggregated. Then, we can have a discussion.
I am not affiliated with Artificial Analysis in any way, I'm just not blind to what they offer.
EDIT: These comments are proof that everything I just wrote goes over the majority of your heads. There is little hope for some of you
Original Source from DeepSeek WeChat Official Account: https://mp.weixin.qq.com/s/qg0NU3NNUbp1co2PdkAPAg
Today we're officially releasing the DeepSeek V4.1 Flash model. It is the smallest model in our brand-new model architecture series, with native multimodal visual understanding. The new architecture was designed with these goals in mind: a higher capability ceiling, faster inference, greater throughput, and scalability to larger-parameter models.
Asymmetric architecture: big intelligence at low cost
DeepSeek V4.1 Flash is a 552B-parameter MoE model built on a brand-new Causal-Encoder-Decoder architecture. Input and output are asymmetric: only 8B parameters are activated on the input side and 16B on the output side, making it significantly cheaper than known models of the same size. V4.1 Flash also uses a new pre-training approach and has gone through larger-scale reinforcement learning post-training. In benchmark testing, it surpasses the intelligence level of a range of flagship models, including DeepSeek V4 Pro.
https://preview.redd.it/qq5p9q5qymoh1.png?width=1080&format=png&auto=…
https://preview.redd.it/2tvpysuvymoh1.png?width=1080&format=png&auto=…
Less cache, lower cost
The new generation of models dramatically reduces the size of the KV cache. Compared with the previous generation, HBM requirements drop to 1/4 and SSD requirements to 1/8. In agent scenarios, cache-hit charges often make up a large share of the bill, so compressing the KV cache substantially lowers the cost of agent-style tasks.
API support
DeepSeek V4.1 Flash is now live on the DeepSeek API with native multimodal support. Simply change the model name to deepseek-flash to call the latest V4.1 Flash. The older V4 Flash and V4 Flash Vision Exp models have been retired; for compatibility, the model names deepseek-v4-flash and deepseek-v4-flash-vision-exp will temporarily be routed to V4.1 Flash.
In addition, extensive testing shows that V4.1 Flash comprehensively outperforms V4 Pro on performance, cost, speed, and total time-to-completion, so we plan to phase out the V4 Pro model in an orderly fashion. After 12:00 Beijing time on September 14, 2026, and until V4.1 Pro launches, all requests to deepseek-v4-pro will be routed to V4.1 Flash and billed at V4.1 Flash's unit price.
Tencent (WorkBuddy, CodeBuddy) and OpenCode, as official partners, have now fully integrated DeepSeek V4.1 Flash — give it a try!
API pricing adjustment
Thanks to the architectural innovations, DeepSeek V4.1 Flash can serve more users at lower cost, so we have cut V4.1 Flash's pricing accordingly. To allocate resources more sensibly, we continue to use peak/off-peak pricing, with off-peak prices at half the peak rate, and encourage users to schedule tasks around their actual usage patterns. The new prices take effect at 12:00 on September 10, 2026.
https://preview.redd.it/qkhui0kjzmoh1.png?width=1690&format=png&auto=…
Open-source release
We will fully support the open-source community in adapting inference for the new model, and will explore various ways to broaden deployment. If you have large-scale deployment needs and the corresponding resources (a 2k-GPU cluster with storage cluster), please get in touch.
Hey y'all!
We've released a new model in our lineup: GigaChat-3.5 Reasoning. It's a 432B-A28B MoE with Gated DeltaNet for long-context efficiency.
We trained domain experts (code, math, general, etc.) with CISPO and then distilled them into a single model via on-policy distillation.
In our evals the resulting model lands close to DeepSeek V4 Flash Preview while using 37% fewer tokens in its reasoning traces.
Weights are on Hugging Face under MIT: https://huggingface.co/collections/ai-sage/gigachat-35-reasoning. You can also try it at giga.chat — pick the reasoning tab (rightmost one).
The downside of uncensoring a model is that it is known to potentially damage it, but CyberTiel is an even more capable software engineer than its censored TielCoder base, while allowing offensive security research. This was achieved by quantizing with an improved imatrix, baked from a curated corpus of cybersecurity- and agentic software engineering work. In short, the small damage from abliteration on a full precision model is negligible under Q4 quantization, and the weights that the model needs to perform relevant work are preserved in higher precision, while the improved chat template makes it think and talk better and faster.
I believe that this is the best 35B-A3B coder for solving real problems in real codebases without breaking anything, which is specifically what SWE-bench-Live tests for. But it’s still a 35B-A3B, and it sacrifices world knowledge for coding ability. That being said, I use it over Qwen3.8-27b for daily coding work: due to the raw speed it fixes 3 issues in the time it takes 27b medium to solve one, and the middle ground between Opus4.6 medium and Qwen3.8-27b medium is simply good enough for most work.
Censoring impedes legitimate and effective work in alignment with the user, and puts the user’s responsibility and ownership over the model’s actions into question, while limiting legitimate uses. When a model is censored, someone else decided for you what the model can and will do, which works against the argument that local models give the user increased control and alignment, and begs the question “alignment to who?”. The point of CyberTiel is to resolve this issue at the same time as pushing the frontier of 35B-A3B coders.
GGUFs and MLX with and without MTP are up on HF. Looking forward to seeing what the community thinks!
PS: I'm not a research lab or a business, and I don't have revenue streams connected to this project. I'm an anonymous researcher with some free time. Constructive feedback is always appreciated! :)
Some people says terminal bench reflects model intelligence better than the intelligent index. From the look of it, the ranking does seem to reflect how people feel about the open and closed models.
For the open models, GLM-5.3 is in a league of its own. GLM-5.3-Flash is leading the current gen of top flash models. Kimi-K3 did pretty bad in this benchmark for its size. Qwen3.8-27B is the only small model that can do something on this bench.
|Model|Score|
|:-|:-|
|GLM-5.3|41.9%|
|GLM-5.3-Flash|32.8%|
|DSV4.1-Flash|26.8%|
|Qwen3.8-Flash-Next|25.3%|
|DSV4-Pro|14.1%|
|Kimi-K3|12.6%|
|DSV4-Flash|12.1%|
|Qwen3.8-27B|5.6%|
|Muse Glimmer|0.5%|
|gemma4-31b|0.0%|
I just watched a YouTube from Luke’s Dev Lab where he literally just plugged a RTX 2000 ADA Into the side of the Zima Board 2’s PCIE socket and it just friggin worked and had great token speed despite running on shitty Ollama. Ran off the Zima’s power supply and everything.
https://youtu.be/Lb3sRFTA-hk?si=8S8vv4GD1zVPeTrc
The Zima Board 2 is only like $411. It has like 16GB RAM and 64 GB eemc storage, Sata ports, Ethernet, yada, yada.
https://shop.zimaspace.com/products/zimaboard2-single-board-server
an Nvidia RTX 2000 ADA is like $700 and has 16GB of VRAM. $1100 for both seems like a great entry point for having a fully functional Qwen 3.8 27b endpoint running at a decent tk/s.
Is this the cheapest and best-performing self-contained entry point for local AI or would a baseline (pre order) Mac Mini M5 with 24GB be a better way forward. Seems like the RTX would still edge out the M5 Mac for prompt processing speed but you do get a much better actual computer in the Mac.
Are there any cheaper fully self-contained alternatives that offer fast token speed on a decent size model like Qwen 3.8 27b?
I’m focusing the discussion on new systems you can buy or preorder now and not used systems. I’m sure there are great deals on used Macs out there, but I want good prefill speeds.
Hey all, long time no post.
Figured I'd pop my head in to point you towards a blog post I just published about research I had performed and changes I'm making to the shape of models I post, you can read it here:
https://huggingface.co/blog/bartowski/per-tensor-layout-maps-for-gguf-quantiz…
I won't try to claim "Pareto frontier" or "best models in the world", but I will say from tests the new shapes look to be better across the board than what I was posting before, so I'm really happy with where it came out, and I hope to not be done yet either :)
https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8…
If anyone has any questions let me know!
Both good and bad things have come from a subreddit that was lot more niche than for example r/flashlight rapidly transforming into the largest online forum about an increasingly core part of the infrastructure of the economy. This sub has experienced growing pains recently, and probably those are mostly felt by people who’ve been around for a while. I think that there are both good and bad trends and I wanted to take a few minutes to suggest a few rules of thumb to employ going forward so that we can create a community that is even more based on science and reality rather than misinformation and one-note populist politics that Reddit is known for.
Suggestion one: if you are new here and by new, I mean, if you didn’t spend much time here or with large language models until about six months ago, there’s a lot of information to be absorbed. This is not a sub or hobby like some where you can learn everything in a month or two. Have some humility, come with curiosity rather than strongly held opinions about everything.
Suggestion two: leave politics out of the sub, unless it is a discussion of actual policy surrounding actual local large language models. Many discussions that we see here have started to resemble the same populism that you can find on every large subreddit. E.g. the discussion of OpenAI's solution to NS has skipped right past the evidence gathering stage to "did you know that billionaires are actually bad guys?! Wow this large corporation sucks!"
In this subreddit, comments and posts about politics are actually just noise unless you are leveraging your knowledge of hardware and software stacks or discussing AI-related policy. Unlike policy, grand narratives of moral outrage are appropriate for therapy, but counterproductive for a technical subreddit.
Suggestion three: develop awareness of the perpetual and exhausted questions and arguments so you do not upvote them or engage. For example, are benchmarks actually useful? This question has been endlessly litigated for the last couple years, but it’s not actually useful because it boils down to: yes they are helpful, but don’t rely on them too much. Anything more definitive and final or sure than that is false confidence. Another such question is: how much intelligence can you fit into X parameters? Literally no one in the world knows the answer to this.
Suggestion four: pay attention to people who are genuinely excited about their work. What’s often missing from clearly AI generated posts is the sense that someone is doing something that they believe in enough to want to bring it to other human beings. The amazing thing about artificial intelligence is how it can augment human effort. Share what you are excited about, and listen when other people are excited about things because this technology has been created by thousands of people who are genuinely excited about the possibilities, rather than people who simply want to make a quick buck, so if you can share your excitement, you’ve pushed back against the trend or the belief that AI is a kind of cynical replacement for human beings.
I realize I’m probably just an old man shouting at clouds, but here's the TLDR:
I suspect that many or most people who’ve been around for more than six months have also started to mentally filter out 90% of posts for these reasons: loudest voices are misinformed; more and more this resembles a political debate space; the same 10 unanswerable questions make up much of the commentary; and people post slop.
China-modified Nvidia RTX 5090 with massive 96GB of memory appears on Alibaba for less than $4,000 — 3x more VRAM at 65% the cost of the original
Anyone here running one of these? Or brave enough to purchase ?
Edit: I sent them an inquiry. They replied they can get me 5 x 5090 for $6k . Or some 4090 with 48gb .
I am going to keep messaging and questioning them. See where this goes.
Edit: so far they are denying having a 5090 96gb card. They offered a 48gb 4090 card. I am still discussing with them.
Edit: 9/14/2026: They quoted this, RTX 4090 48GB - 4286usd/pc
Still discussing with them
After noticing that it is ranked among MUCH larger frontier models in the EQ-Bench Creative Writing benchmark and the Hemingway-bench, I decided to give it a try and was very impressed.
I didn't do very formal testing, but I did ask it to emulate the style of several different writers, including Henry Miller, David Sedaris, and Stephen King, and it produced passable prose that actually made me laugh in a couple of cases. The paragraph below is the results of the prompt "produce a humorous paragraph in the style of David Sedaris." (first try, not cherry picked)
I recently tried to be a better person and started composting, which mostly means I now have a small, damp science experiment in my kitchen that my mother calls “the smell of your future.” I bought a countertop bin with a charcoal filter, as if that would fool anyone, and I’ve taken to narrating my food scraps to myself — “Goodbye, avocado skin, you were a mistake” — while my husband watches from the doorway with the concerned expression of a man who has just realized he married a woman who talks to garbage. The city sent a pamphlet about proper composting, and I read it the way other people read horoscopes, underlining the parts that said “do not include meat” and thinking, well, that’s just rude.
For comparison, the following is what qwen3.8-27b produced with the same prompt.
My uncle Ed, who had spent the entire trip carefully explaining the proper way to fold a fitted sheet, discovered that our hotel room had two beds, each with one. "This is a test," he said, sitting on the edge of the bed nearest the bathroom, his face composed. "To see if we can make it work with what we have." He was right, of course; we did what we could, though the effort made for a rather uncomfortable night, for us all.
You may or may not know David Sedaris' writing (or find it funny if you do know it), but the first example is clearly much a much better imitation, without directly plagiarizing, as far as I (or Gemini) am aware.
I didn't save any of the other examples as I wasn't testing for the purposes of posting here, but in all cases the muse glimmer version was not only head and shoulders above qwen 27b, but genuinely impressive in comparison to any other local model I've tried in the past.
I'm curious if anyone else has played with this model for creative writing, or similar purposes, and if so, what your take on it is. Also, I don't know much about finetunes, but I wonder if there's additional potential for creating something even better by training on different source material.
I know even less about how the ERP world works, but I know enough to know that a lot of high-performing models are trained for this purpose as huggingface seems to be filled with finetunes. For glimmer I mainly see the abliterated version, which I suppose is filling that gap for people, so to speak, but with this kind of performance, and the amount of people in this subreddit interested in it, I'm a bit surprised there aren't more finetunes.
The last thing I should mention is I didn't use a system prompt in any of my testing, but it occurred to me after the fact that a model that was trained for agentic coding seems like a prime candidate for steering with a system prompt, but maybe it wouldn't have made much of a different. Maybe I'll play with it some more and report back.
I haven't seen this mentioned yet, so I thought it deserves a post. I was trying out OpenWhispr when this model came up as the recommendation. So I don't have personal experience yet, but it's supposed to be a better version of Parakeet, especially on Macs.
Their official tidbit:
"Orukeet is a 25-language speech recognizer built from NVIDIA Parakeet TDT 0.6B v3. It replaces half of the encoder's temporal depthwise filters with 12,288 fitted, frozen Gabor kernels and trains the remaining parameters on multilingual and multi-accent data.
Orukeet outperforms Parakeet on 61 of 74 tested splits, including LibriSpeech test-clean (1.46% vs. 1.53% WER), test-other (2.86% vs. 3.14%), and FLEURS English (3.82% vs. 4.28%). Across all 25 FLEURS languages, pooled WER is 9.85% vs. 11.01%, a 10.6% relative reduction. Final adaptation and checkpoint selection use LibriSpeech test-other."
Hi!
I've been following this community for quite a while and have difficulty figuring out what to put on my 3080 12gb - I know Qwen 3.6 35B 3A was the go-to choice when it first came out, but I'm curious if there are any other models / specifically optimized models that meaningfully benefit from the extra 4gb of VRAM over 8gb while still being usable under 16gb.
My workflow is agent heavy, but more for a personal secretary and manager, and less coding heavy.
Thanks!
I like to benchmark new models that come out on motion videos. So here's a test I did for deepseek v4.1 flash. And I have to say flash has probably graduated from being a Luna class model to nearly an Opus class model with this release, at least with motion videos.
Prev. example I did with Kimi k3(altho in that case I had a simpler prompt as well)
https://www.reddit.com/r/LocalLLaMA/comments/1uyaiw2/kimi\_k3\_release\_video\_made\_with\_kimi\_k3/
Nice pp improvements for RDNA4(R9700) & 3.5(RX 9060 XT, 8060S). More good numbers on large context.
PR has detailed benchmarks.
Can you still do something with a 2050 or something like it?
I mean for office work, loading embedding, reranking and chat models not at the same time but is anyone still using smaller models and have any good ones come out?
I feel like small models are abandoned, I don’t care much for world knowledge, I want tool use and preferably multilingual. Vision would be nice but beggars can’t be choosers.
Hi everyone,
I was interested in learning LoRA fine-tuning, and ended up building CodeFinetuner over the past few months, a full pipeline that fine-tunes a small code autocomplete model (e.g. Qwen2.5-Coder-3B) specific to a codebase. You can then use the resulting GGUF model via llama.vim/llama.vscode and run it fully locally. Supports fine-tuning on Mac (MPS) and NVIDIA GPUs (CUDA), with optional Unsloth support for faster training and lower VRAM usage.
Pipeline: raw code -> tree-sitter parsing into Structure-Aware FIM examples -> LoRA fine-tuning -> evaluation (CodeBLEU, edit similarity, exact match, perplexity, ...) -> GGUF conversion for local inference.
To try it:
uv tool install codefinetuner
Create a data folder and place your repo (or code files) inside. For auto-split just drop the files in directly, for manual split create data/train/, data/eval/, data/test/ subfolders and set split_mode: "manual". Get the default config with:
curl -L -O https://raw.githubusercontent.com/cuolm/codefinetuner/master/config/codefinet…
Adjust it to your needs and hardware availability, then run:
codefinetuner --config="codefinetuner_config.yaml"
The example runs in the repo show clear improvements over the base model on these evaluation metrics, but using the model for autocomplete on code you're actively writing is a different thing from scoring well on a test set, and the autocomplete tools themselves (llama.vim/llama.vscode) sample differently from the greedy decoding used in the evaluation. So the real usefulness still has to be verified in the editor itself.
Might also be useful just as a reference, since it's a complete working LoRA fine-tuning pipeline end to end.
Hope someone finds this project interesting or helpful.
Trying to get Hermes a local, efficient, tts voice.
