Stop using 30K-token system prompts for local coding agents. How a plain Git-versioned Markdown harness keeps KV cache under 2K tokens with Ollama / llama.cpp (Open Source)

If you run local coding models (Qwen 2.5/3.8 Coder 14B/27B/32B, DeepSeek, or Llama 3 via Ollama, llama.cpp, or vLLM), you already know the two fatal bottlenecks of agentic coding on local hardware: 1. The KV Cache & TTFT Penalty: Cloud users throw 50,000 tokens of chat history at Claude Opus without thinking. On a local 24GB or 32GB rig, prefilling 30K tokens of noisy conversation history drags Time to First Token (TTFT) through the floor, eats up precious VRAM that should belong to your context window, and triggers the "lost in the middle" attention collapse. 2. Amnesia Across Sessions: Local models are stateless. When you clear the context window to restore inference speed, the model forgets your project architecture, file relationships, and error history. You end up copy-pasting your constraints into every new prompt. For the past 18 months across 1,900+ real-world sessions, I’ve been running and refining an alternative: Project Athena—a local-first memory, reasoning, and governance harness designed to give local LLMs permanent, compounding memory without blowing up your token budget or relying on hosted SaaS databases. I just open-sourced the v9.9.9 kernel under the MIT license. Here is the exact architectural split that keeps local models grounded. # 1. The Core Rule: State on Disk, Not in the Prompt Most agent setups treat the LLM's context window as the hard drive. That is an architectural mistake. The context window is volatile RAM. Durable state belongs on your NVMe SSD as plain, human-readable, git-versioned Markdown files: \[ Your Local Machine: Plain Git-Versioned Markdown \] ├── .context/CANONICAL.md <-- Immutable architectural rules & API contracts ├── .context/memory\_bank/ <-- activeContext.md & session checkpoints ├── .agent/workflows/ <-- Deterministic slash commands (/start, /end, /plan) ├── .agent/skills/ <-- Domain capabilities loaded strictly on-demand └── .agent/scripts/ <-- Verification test runners & linter hooks Surgical Boot (<2K tokens): Instead of dumping megabytes of chat logs into the model, /start loads only the active checkpoint block from activeContext.md and top-tier constraints from CANONICAL.md. Over 90% of your model's context window and KV cache remains completely free for actual code diffs and reasoning tokens. Session Lifecycle (/start and /end): At session close, an automated distillation script (/end) audits git diffs, extracts learnings, prunes transient noise, and writes an atomic checkpoint back to disk. Session 1,900 boots faster and cleaner than Session 10. 100% Model Agnostic: The model is just whoever is on shift today. Run Qwen 2.5 Coder locally for fast terminal diffs; swap to DeepSeek, Llama, or an external API tomorrow. Your project rules, architecture contracts, and past bug logs never disappear. # 2. Mechanical Guardrails (Crucial for Local Weights) Small and mid-sized local models (8B–32B) are prone to sycophancy: they eagerly declare "I have refactored the module and verified all tests pass" while silently breaking dependencies. Athena enforces deterministic mechanical verification outside the model's weights: Red Run or It Didn't Happen: Any agent claiming to fix a test, gate, or bug must show the verification script failing on the pre-fix state, then passing on the fixed state. If it cannot produce the red run, it found a blind spot, not a fix. Deterministic Tool Calling: Integrates natively via local Model Context Protocol (MCP) or standard CLI scripts (smart\_search, context\_gate, quicksave). No Hosted Cloud Databases: No Pinecone, no cloud vector stores, no external telemetry. Embeddings and hybrid search run locally using plain SQLite and BM25. # 3. Real Hardware & Performance Observations Tested Hardware: Apple Silicon (M2/M3 MacBooks and Mac Studios) and local NVIDIA setups (RTX 3090 / 4090 / 5090). Inference Impact: By replacing multi-turn conversational bloat with deterministic file write-backs, local prefill latency drops from 15–30s down to sub-second responses. * Zero Lock-In: Everything is plain Markdown and Python. If you delete the repo, your notes and code are still just standard text files on your machine. # Try It (100% Free & Open Source) Zero subscriptions. Zero data leaving your machine. Works with Ollama, llama.cpp, vLLM, Claude Code, Cursor, Antigravity, and terminal CLI workflows. git clone https://github.com/winstonkoh87/Athena-Public.git cd Athena-Public pip install -e . athena init . GitHub Repository: winstonkoh87/Athena-Public License: MIT Curious how others running local coding agents on Ollama/llama.cpp are managing persistent cross-session context without degrading TTFT or blowing out VRAM? Happy to discuss the trade-offs and benchmark numbers in the comments!