Jeff v1.3: Jeff-Code makes Qwen 3.8-27B finish coding tasks 47% faster (32% less time) on average at the same pass rate; plus 15 adapters & GGUFs

Jeff v1.3 is live, and with it come a number of updates. See jeffhub.ai and github.com/firelex/jeff for full details.

The highlight: Jeff-Code

Jeff-Code is a coding agent with two Jeff v1.3 adapters trained specifically for Qwen 3.8-27B. Jeff-Code is a fork of Pi by Mario Zechner (MIT licence).

We forked Pi because its extension framework doesn't currently let a fast decision model sit deep enough inside the agent loop. Along the way, we made a number of other changes as well (see below).

Aside from hopefully being useful to people who run Qwen 3.8-27B locally as their daily coding model, Jeff-Code is also a conceptually interesting experiment: how far can a System 1 model go inside a coding agent?

The results, run side by side in paired blocks:

Same quality: with Jeff's thinking threshold at 0.6, Jeff-Code matches Qwen 3.8-27B's pass rate: 62.4% against 62.8%; paired difference −0.2 points, 95% interval −2.6 to +2.1, over 1,242 paired tasks.
47% faster (32% less time) per task¹: on average a task takes 0.68× the baseline's time (geometric mean of the per-task time ratios, 95% interval 0.64–0.72; the median task, 0.70×).
Where it helps most: typical software-engineering work. SWE-bench Verified 0.63×, SWE-rebench 0.66×, Terminal-Bench Pro 0.64× (both over two rounds), Harbor Index 0.71×. On Terminal-Bench 2.0, with its long, hard tasks, there is no clear speed-up (0.96×, interval 0.78–1.16); on SkillsBench neither (0.91×, interval 0.68–1.20).
The benchmarks: we evaluated only on tasks Jeff never saw in training. SWE-bench Verified ran in full (all 500 tasks; none of its repositories were used for training). For the benchmarks we also trained on, we split the tasks and ran every held-out task; a few pairs hit by repeated infrastructure failures are left out (see below). Terminal-Bench 2.0 (40 of its 89 tasks, 3 attempts each; 45 were used for training, and the other 4 are near-twins of evaluation tasks, so they were used for neither), SWE-rebench (189 held-out tasks, 2 rounds), Terminal-Bench Pro (100 held-out, 2 rounds), SkillsBench (44 held-out) and Harbor Index (41 held-out). Within those splits nothing was sampled. We also ran Terminal-Bench (original) and Terminal-Bench Science, but Qwen solves almost none of those tasks in any setting, so they can't show a difference and are left out of the pooled numbers.
What it's compared against: Qwen 3.8-27B alone in the same Jeff-Code build with every Jeff feature switched off, thinking at full on every turn and no thinking limit, which is how plain Pi runs it. Each task ran in both settings side by side, at the same time on the same Qwen server, and every comparison is paired by task. The only remaining differences from original Pi are a rarely triggered runaway cut-off (it stepped in 3 times) and trace logging. Task pairs hit by an infrastructure failure (out of memory, a stalled session, a test environment that wouldn't start) were run again once; the pairs that failed again, and a handful of re-runs still unfinished at launch, are left out for both sides (under 3% of pairs) and listed in the full report. One Terminal-Bench 2.0 task, pytorch-model-recovery, is left out of every comparison: a harness bug stopped its baseline sessions before they began.
Why not just turn thinking off? We tried: with Qwen's thinking off throughout (and the same safeguards), tasks are faster still but clearly worse: −7.6 points (−10.6 to −4.5), up to −13.5 on Terminal-Bench 2.0. Jeff deciding when Qwen should think is what keeps the quality. That's the case for a small decision model.

If you want to know more, look here: https://jeffhub.ai/notes/jeff-v1-3. The original version of this post had all the data, but people thought it was too long. Blame the early commenters. ;)

Jeff v1.3: Jeff-Code makes Qwen 3.8-27B finish coding tasks 47% faster (32% less time) on average at the same pass rate; plus 15 adapters & GGUFs

The highlight: Jeff-Code

Links