Running a 3B roleplay finetune fully on iPhone: what we measured about keeping a small model in character (I make the app)

I make Castmates, a closed-source iOS app (free tier, paid Pro) that runs a 3B roleplay model entirely on the phone. Posting for the engineering notes, not to sell it. The app is mentioned once, at the bottom.

Setup: Impish Llama 3B (a Llama 3.2 3B RP finetune), plus our own rank 16 LoRA, fused and requantized to Q4\_K\_M from the fp16 base. About 2.0 GB, downloaded after install. llama.cpp with Metal, all layers offloaded, KV cache at q8\_0 (roughly 60 KB per token). No server, no account, works in airplane mode.

Limits first, because they shape everything:
- 4 GB phones are the floor. Weights x1.6 plus KV plus \~350 MB for compute buffers has to fit, otherwise we refuse to load rather than get jetsammed. Those phones stay at 4096 context.
- 6 GB phones get 6144 context and 8 GB phones get 8192. Llama 3.2 is natively 128K so no rope scaling is needed, it only costs KV RAM.
- It's slow. Early on we measured around 6 tok/s on an A18. Newer chips are faster but I'm not going to quote a number I haven't re-measured on the current build.
- The 2 GB download is the biggest drop-off in the app, so we use Background Assets to start it before first launch. It's non-essential on purpose (essential blocks launch).

What we learned about staying in role:
- Bigger window alone does nothing. Our history trim budget was the binding constraint, not n\_ctx. Scaling the trim to \~0.68 x n\_ctx took long-conversation fact recall from 25% to 75% in our lab. Verbatim history beat the lossy summarizer by a lot.
- Retrieval can hurt. Our BM25 memory retrieval re-injected superseded facts when the window still contained the newer one (says-stale +20.8pp vs no retrieval). Dropping any hit whose rare entities still appear in the live window fixed it (-18.8pp, CI \[-35.4, -6.2\]) without losing facts on the recall benchmark.
- Prompt tweaks mostly measured as zero. Single 3-4 seed runs were noise. Fixed-history micro-tests with N=16 and same-seed controls were the only thing we trusted. Example: "say my name" went 4/12 vs 11/12 purely from history length, no prompt change.
- Placement matters more than wording. A scene direction is ignored in the reminder slot (0/16) but lands 16/16 as its own block after the final user turn. A reminder after the user turn makes the model answer the reminder instead of the user.
- Guards beat prompts for the 3B's habits: rerolls on third-person drift about the user, invented names, and bare role-label output. Thinking mode didn't help: one-pass is impossible with this finetune and two-pass was noise at 2x latency.

What it still can't do: override a fact it can still read in context, so contradictions in a long scene stay a model ceiling.

The lab is a Python port of the production prompt and guard pipeline run against llama-server with fixed seeds, so every claim above came from a run, not a vibe. Happy to go into any of it.

The app is Castmates on the App Store if you want to try it. I'd rather get criticism of the approach than installs.