I tested Gemma 3 27B and Qwen3 32B against two frontier models on financial analysis tasks. The small models answers are much less reliable

I ran a series of simple experiments to see how LLMs behave when asked to judge companies from real financial data. I used 4 models: Gemma 3 27B and Qwen3 32B as small models, and Opus 5 and GPT-5.6 Sol as frontier models. All calls were made through the Bedrock API.

The samples are small, so the numbers below should be read as a demonstration and a methodology and not a definite proof.

Here are some of my observations, particularly when it comes to the differences between small and large models.

\*\*Rank vs. score.\*\* I asked each model to rank six companies from best to worst, and separately to score each one from 0 to 100. The underlying judgment is the same, so we would expect the same ordering. Rank and score gave an identical ordering in 75% of sets for Opus and 65% for GPT, but only 25% for Gemma and 15% for Qwen.

\*\*Order of the list.\*\* For Qwen, simply reversing the order in which the companies were listed changed the top-ranked company in 6 of 10 sets.

\*\*Analyst opinions.\*\* Attaching a bearish analyst note to the data lowered the rating in 81% of cases for Qwen and 71% for Gemma, against 43% for GPT. Opus mostly kept its own view. Interestingly, the fix is simple: asking the model to identify the opinion and reason independently brought the rating back toward its original level in 85% of cases for Gemma and 68% for Qwen.

\*\*Summarize, then analyze.\*\* When rating a summary of a 10-Q section instead of the full text, the frontier models gave the same rating in 84% of cases. Gemma and Qwen changed their rating in 25% and 31% of cases.

I wrote something more complete, with charts and the details of each experiment: https://sabrresearch.com/cookbooks/llm-financial-bias

Disclosure: this is my own work, published on my company's website. Happy to answer questions on the setup.

\------ Edit 1 -------

People complained about the choice, of models. To be clear, I'm not trying to trash small models, on the contrary, what is of interest to me here is the overall trend and inconsistencies which occur in both frontier and small. I also ran the analysis on Gemma 4 31B (April 2026), it's a bit better than Gemma 3 used above, but still the same issues:

\*\*Rank vs. score.\*\* Rank and score gave an identical ordering in 35% of sets for Gemma 4, against 25% for Gemma 3. Still far from Opus (75%) and GPT (65%).

\*\*Order of the list.\*\* Reversing the list changed Gemma 4's top-ranked company in 2 of 10 sets, the same as Gemma 3.

\*\*Analyst opinions.\*\* A bearish analyst note lowered Gemma 4's rating in 53% of cases, vs 71% for Gemma 3 but still above GPT (43%) and well above Opus (19%).

\*\*Summarize, then analyze.\*\* Gemma 4 changed its rating in 25% of cases when given a summary instead of the full text, the same as Gemma 3.