Live scribing with Jev-ish utterance gating
Like everyone, I've been following the back-and-forth regarding Jev with interest. Arguments aside about the originality of the idea, the first thing I thought of when all of this came out is "gosh, that could really help my local scribe run realtime loops during a consultation".
I pointed my harness of choice at the problem (I've been relying more and more on the LLMs as the brain rot from AI coding has continued apace). Here is the resulting workflow:
- TEN-VAD to segment utterances and send to a Whisper compatible backend (the Tauri builds use parakeet.cpp with a 0.6B medical finetune)
- CAM++ speaker embeddings to provide best effort diarisation (obviously limited in the setting of crappy desktop microphones and echo-y consultation rooms)
- Here is where the Jev-ish/SemIf gating comes in. Each utterance is provided to the LLM with a short prompt and an instruction to classify as NOTE (something to be documented), ACT (action to be taken), SKIP (filler talk, etc). In the Docker deployments this is performed by the user configurable secondary model (I use Qwen3.5 4B); on the Tauri builds it's the solo primary model but into the second slot of the bundled llama.cpp server (important so that we don't clobber the prompt cache of the running main thread)
- The initial approach was quite simple: one decode step; then gather the first token top logprobs and compute the probability mass summed over SKIP/NOTE/ACT (with some prefix matching to account for tokeniser splits and a one-word generation fallback).
- SKIP utterances are buffered and don't get sent to the main LLM immediately (the next time the main model is woken up it will ingest that material so that nothing is lost). If a NOTE or ACT is misclassified as a SKIP, a 45s/40 word debounce runs through the main LLM with all the material it may have missed.
- NOTE and ACT are passed on to the main model for processing. The main model has access to tools that include modification of the running note.
- Prompt caching is essential here so that subsequent passes through the main model remain performant without a huge PP delay.
I found that the 4B model would almost never SKIP (Jev and 3.8-Flash were better but still missed 3/4 of them on natural speech). Not surprisingly (in hindsight); using the calculated probability mass alone was essentially no different to just prompting the vanilla generation endpoint and executing based on the output (roughly 81% accuracy). Looking into the logprobs a bit more it seemed that there was a usable signal in there somewhere. GLM-5.3 was pretty good figuring it out: instances where NOTE was selected, P(SKIP) ≥ 0.05, AND the utterance was ≤8 words were essentially always a SKIP. With this heuristic... 0 false SKIPs across multiple runs, and SKIP recall went from 0-50% to 75-100% on the natural consult.
The logprob gating + heuristc step is latency neutral; however, it was more reliable for this task. The otherwise vanilla small LLM like Qwen3.5-4B never flagged SKIPs and would occasionally not follow instructions entirely. I also ran an evaluation with Jev via OpenRouter (a pretty informal test set of \~40 hand-labelled utterances, and the heuristic was tuned on the same set, so it needs a held-out set to confirm); on a natural ambient consult recording the gap is smaller than I expected (both 95% accuracy but 73ms vs 514ms, keeping in mind Jev was a remote endpoint and all the latency that entails). Jev pulled away on a command heavy synthetic script (\~80% vs 100%). The overall intention was to prevent the main-loop from getting too bogged down with fluff and I think this approach achieves that.
First token logprob classification is pretty old hat; but I never really thought about one-shot classification in my scribe before Jev. And yes, the whole point of Jev is that you can just give it a classification task and have performance be good enough that you don't need to apply bespoke heuristics over logprobs to rescue your classifier (but funnily enough even Jev got an accuracy uplift from the P(SKIP) heuristic).
It was a fun experiment anyway (and grossly underpowered to say anything meaningful about Jev in general terms)! The result (video below) has been useful from my perspective (you can try it yourself here).