Hey all, Henry from Cactus Compute here, I kinda wanted to share our latest model and get feedback from the family :)
Needle 3 is a small foundation model for automation: you give it the functions your app exposes, it reads a request and returns the calls with every argument filled in, or a typed record if what you gave it was a schema. It runs on the device, with no network in the loop. It is on Hugging Face, on GitHub, on PyPI as cactus-needle, and there is a sandbox that runs it in your browser at cactuscompute.com/needle if you want to poke at it before reading further.
1) Trades general capacity for frontier performance on automation tasks
The thing we decided early was that Needle would not chat. Every turn is a function call, and a request no declared tool can serve comes back as an empty list rather than a guess. That sounds like a limitation, and it is, but it is what let a 121M-parameter model be trained on 360B tokens of structured data and spend all of its capacity on three jobs: tool calls, structured extraction and text embedding.
The architecture follows from the same trade. It is a Simple Attention Network: the dense feed-forward layers are gone, replaced by a Monarch Hadamard MLP with 25.6K parameters per layer instead of 4.7M, and the knowledge a feed-forward layer would normally hold sits in an engram, hashed n-gram tables that are read by gather and cost no arithmetic. 70.8M of the 121M parameters live there, so the full model does the arithmetic of a 50M one: 100 MFLOPs per token against 296 for a transformer of the same shape.
https://i.redd.it/fsnfjojny4qh1.gif
We wrote the intuition up if you want the longer version: Simple Attention Networks and the Hadamard MLP.
2) Beats models 10x its size on tool calls and language-to-device control
On Mobile Actions (961 phone commands, scored on the exact call), the 20-layer model scores 86.0 through the shipped 2-bit binary with the confidence gate on. LFM2.5 1.2B is at 82.4, Qwen3.5 0.8B at 76.0, FunctionGemma 270M at 65.1 and Apple's on-device foundation model at 57.6, all at f16. DeepSeek V4 Flash through its API is at 88.4, which is the line in the chart.
https://i.redd.it/luunq717z4qh1.gif
The part we are most pleased with is not the number but how the calls are made. Every argument is a span of the request: the model writes a short derivation first ('living room' -> room; '30' -> brightness) and then emits the call under a byte-level grammar compiled from your schema, so the JSON always parses and an enum can never leave its set. An optional field with no evidence is omitted, a required one with no evidence withholds the call, and the engine drops a call the request negates or excludes. Ask for two things and you get two calls in order.
https://i.redd.it/227396y5z4qh1.gif
Full table across all six suites (tool calling is exact match, extraction is field F1, Needle through the shipped binary, baselines at f16 under vLLM):
|Model|Params|Mobile Actions|DroidCall|BFCL v4|DSTC8 F1|SNIPS gold F1|SNIPS 7-way F1|
|:-|:-|:-|:-|:-|:-|:-|:-|
|DeepSeek V4 Flash (cloud)|\-|88.4|60.5|77.2|80.0|69.4|66.7|
|Needle3-20L-121M|121M|86.0|47.0|50.2|40.7|30.2|24.7|
|LFM2.5 1.2B|1.2B|82.4|35.5|62.0|48.0|43.0|38.0|
|Needle3-16L-98M|98M|80.7|40.0|41.3|28.5|23.5|19.2|
|Qwen3.5 0.8B|800M|76.0|28.0|56.8|49.0|35.0|34.0|
|LFM2.5 350M|350M|72.8|32.5|59.1|20.0|34.0|29.0|
|LFM2.5 230M|230M|69.3|11.5|46.3|53.0|27.0|22.0|
|FunctionGemma 270M|270M|65.1|16.5|46.6|27.0|29.0|14.0|
|Needle 2|45M|63.5|17.0|\-|\-|\-|\-|
|Apple FM|3.0B|57.6|\-|\-|\-|\-|\-|
|Needle3-8L-52M|52M|36.8|36.5|28.2|15.3|16.6|10.1|
|Needle3-4L-29M|29M|11.7|21.0|19.5|6.9|7.7|4.3|
You can see where it is weaker too: BFCL and the extraction suites are where the bigger baselines pull ahead, and the smaller subnetworks fall off quickly on the general task (more on why that is fine in section 4).
3) Matches 2-3x bigger models on structured JSON extraction
Extraction is not a separate mode. You declare the record as the only tool and pass the passage where the query goes; with one tool declared the grammar admits exactly one call of that name, so the shape is guaranteed rather than requested, and the values are grounded the same way as arguments: a field is filled only from a span of the passage, an optional field with no span comes back as None, and a date whose year appears nowhere in the text is flagged instead of invented.
from pydantic import BaseModel
import needle
class Invoice(BaseModel):
vendor: str
total: float
due_date: str
po_number: str | None = None
needle.extract("Invoice from Acme Corp, $1,200.00, due 2026-09-01", Invoice)
# Invoice(vendor='Acme Corp', total=1200.0, due_date='2026-09-01', po_number=None)
It generalised to classification without special training, because an enum is just a constrained value: declare sentiment: Literal["positive", "neutral", "negative"] on a record and you have a classifier whose output cannot leave the set. A watch reads a notification into merchant, amount and date that way, then into a reply, then into a sentiment flag, one record each. On DSTC8 and the two SNIPS suites the 121M model lands between the 230M and 350M baselines, which is the 2-3x in the heading.
4) Intelligence ladder: every depth from 2 to 20 layers a model of its own
This is the part I would most like your thoughts on. Needle 3 is one set of weights, and every depth from 2 to 20 layers is a deployable model. Blocks 0 and 19 are always kept and the rest are added by bisection, so each subnetwork nests in the next; during training each step samples one path, mostly the full model and otherwise a random depth, with the smaller path distilled from the full one. The full-depth model ends up slightly better than an ordinary run of the same size, and every depth below it is trained rather than truncated.
https://i.redd.it/kbwimep4z4qh1.gif
Why we wanted it: a watch, a Raspberry Pi and a phone do not want the same model, and they want to pick the size at deploy time. needle build --layers 8 writes the 8-layer file; the same engine runs all of them. The small depths lose accuracy on the general benchmarks (that is the bottom of the table above), and they get it back when fine-tuned to one product's tools: on DroidCall every subnetwork gains 18 to 36 points, and from 4 layers (29M parameters) up the tuned subnetwork passes DeepSeek V4 Flash.
https://i.redd.it/y646wed3z4qh1.gif
Fine-tuning is LoRA on the frozen base, merged at export, and the Python package does it locally at 4 bits (needle finetune data.jsonl, then needle build). The maths is in Intelligence Ladders and the workflow in Fine-tuning Needle.
5) Runs locally at up to 4k tokens/sec decode speed
The engine is under 1 MB, plain CPU, no GPU or NPU, and the weights are read in place from a single file the engine maps into memory: a 196-byte header carrying the whole architecture geometry, a nameless tensor directory, and the quantised blobs in the order the forward pass reads them. On a Raspberry Pi 5, decode runs at up to 4k tokens/s at the bottom of the ladder and around 400 at the top, prefill from 10k down to 1k. Every response reports prefill_tps, decode_tps and peak_ram_mb, so you can measure on your own device rather than take our word for it.
Every response also carries a confidence score from a calibrated head, the minimum of a post-hoc judgement on the finished call and the decode probability of its tokens. The engine withholds anything under 0.1; above that the number is yours: act at once when it is high, show the call and ask when it is middling, treat [] as a refusal. How we use it is in Leveraging Needle's confidence.
6) 25-121M deployable parameters at CQ2-bit (8-29MB binaries)
The weights are quantised with Cactus Quants: groups of 128 weights are rotated by a Walsh-Hadamard matrix, which makes every group look Gaussian, split into an fp16 norm and a direction on the unit sphere, and the direction's coordinates are snapped to a 4-entry Lloyd-Max codebook. That is 2.125 bits per weight, and the kernel never expands them: it rotates and int8-quantises the activation instead, then does table lookups and sdot against the packed indices. The embedding and the confidence head keep 4 bits, the norms and gates stay fp16. The byte layout, and a twenty-line parser for it, are in The .cact format.
7) For mobiles, wearables, smart home, small robots and microcontrollers
Every target ships a prebuilt engine folder: macOS, Linux on x86-64, ARM64, ARMv7, RISC-V and MIPS32 (the Ingenic camera SoCs), Windows x64 and ARM, Android, iOS, watchOS, tvOS, the browser as WebAssembly, and a WASI component with a WIT world. pip install cactus-needle covers the desktop and server platforms with wheel-tagged engines, and needle build --platform linux-arm64 --layers 8 --out ./pi puts an engine and the weights in a folder you copy over. Inference never touches the network, so an air-gapped device only needs the files in place. The list and the runtime surfaces are in What devices are supported.
import needle
@needle.tool
def get_weather(city: str):
"Get the current weather for a city."
return {"city": city, "temp_c": 27, "sky": "clear"}
agent = needle.Needle(tools=[get_weather])
print(agent.run("what's it like in Lagos right now?")["results"])
# [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]
What we would love feedback on
- Where the grounding rules get in your way. The engine refuses to invent a number, drops a call the request excludes and withholds a required enum the request never names; we tuned those on our suites and would like to hear where they bite on real tools.
- The ladder. Whether depth is the right knob for you, or whether you would rather have width, and what devices you would want the 2- and 4-layer models on.
- Extraction cases we have not seen. Nested records, arrays, multilingual text (it is English-first, and non-English text fragments into about 1.7x more tokens).
- Anything in the tool design guide that turned out wrong for your schemas.
Thanks for reading this far. Happy to answer anything in the comments.
