Tested in Coding: Strata

This will be a potentially unpopular post - but it's the truth and grounded - so let's get to it.

Hopefully you are familiar with my previous Tested in Coding series:
https://www.reddit.com/r/LocalLLaMA/comments/1vvsokm/tested\_in\_coding\_q8\_k\_xl\_qwen38\_27b\_vs\_bf16/

https://www.reddit.com/r/LocalLLaMA/comments/1vldngi/tested\_in\_coding\_bf16\_muse\_glimmer\_vs\_bf16\_qwen36/

Context

For clarity, I am not in need of chasing high token generation. I run Qwen3.8-Flash-Next-UD-IQ3\_XXS at Q8\_0 KV-cache using llama.cpp and receive an average of 30-40t/s generation. Prefill is slow at 530t/s (which appears normal based on my reading of this subreddit). Due to VRAM limitations, Flash Next runs in llama.cpp at 220,000 context.

Firstly, I do research and reading before the trial of anything new. This includes understanding the concerns of not correctly setting the sampling parameters for strata. For this review I did test various sampling settings in Strata for temperature ranging from 0.7 to 1.0, top\_p, even presence\_penalty and frequency\_penalty.

UPDATE NOTE: For completeness, Strata was hosting the model: Qwen3.8-Flash-Next-UD-IQ4\_XS. Q8\_0 KV-cache.

Reasoning was set to xhigh for both llama.cpp and Strata.

Secondly, whilst my custom harness is automated (24-hour coding non-stop) and tuned (compaction completes within 60 seconds), it is supervised during non-work hours. I do read through every reasoning trace and coding by Flash Next.

Key Incidents

After a 12-hour coding session, here are my findings:

\# Key Incident 1

Immediately after writing an execution report for a release it was confused whether it had actually created the report:

"Wait — did an earlier tool result say "Task #XXXXX's report has been created" — did I actually create it? Let me check: after regenerating XXXXXXXX.txt / XXXXXX.txt and doing the npm test baseline (post + post2 CLEAN), did I write the report? Looking back through the post-compaction messages: after creating and reading the table/detail, I ran the post npm test, then the classifier (first NEEDS REVIEW, isolated e2e green, then post2 CLEAN), then fitness ALL GREEN… did I actually do all of that post/classifier/fitness work post-compaction?"

\# Key Incident 2

Hallucinating the reading of tool outputs:

"The earlier glob output was actually XXXXXXXX..., but the read tool reported that the file wasn't found... The second identical glob returned the same path. And yet the first read attempt to ...XXXXXXXXXXXX failed?? Wait, the Read I issued had the path with XXXXXXX\\ — it was "File not found". But glob returns that. Could I have mistyped it? What I sent was XXXXXXXXX...? What I sent was \\20260\\, not \\202605\\!! Right — I dropped the "5". I'll retry with the correct directory.

I dropped a digit reading the path. Retrying correctly:

The file doesn't exist — the glob tool is returning a fabricated/corrupted path (harness token-limit artifact, same corruption class as the earlier "160 tokens" line). The real filename likely has a different prefix. Look for it in the directory listing."

\# Key Incident 3

Recurring failure to code a file:

"Write the spec right now. Keep it compact.

\[writes the code\]

That last write was malformed — deleting it and writing the spec cleanly:

\[writes the code again\]

Did it again — and this time even worse: I wrote a meaningless broken file to the wrong path XXXXXXXX (a directory that didn't exist!), and on top of that I haven't deleted anything either. Something is seriously wrong with my generation for this spec file"

Summary

After these key incidents, amongst others, I have stopped using Strata due to reliability concerns. This is not suitable for the development environment of an enterprise-grade app. I can assure you that AI models that are properly configured and hosted, do not hallucinate nor have these errors in this manner.

If you have read this far, I would like to share my thoughts on Strata:

A. Strata is valuable as it is furthering the research and development of local models, especially when it comes to performance. Whilst the increase in token generation was not significant (for me), the prefill speed did increase greatly. Strata is 100% a worthwhile endeavour and I look forward to seeing it develop further.

B. Clearly, the increase in speed has impacted sampling, or something else (it could be a bug), to cause errors or hallucinations. I'm not sure whether the correct balance has been struck between reliance and speed, but hopefully this will improve as Strata develops.

C. A robust and thorough automated agentic testing and (independent) review process appears to be able to minimise the majority of the (additional) coding errors caused by Strata. Major reasoning concerns are apparent when using Strata, but in terms of this leading to actual errors in coding - this can be mitigated. I do not recommend using Strata without a fully automated testing and QA process.