Glassbench: an open-source workbench to compare local and hosted LLMs across AI trading agent frameworks
Glassbench is a free, open-source workbench that connects different AI trading agent frameworks, so you can watch how their agents decide, analyze every step and compare them.
AI trading agents are LLM systems where a team of agents (analysts, a bull and a bear, a trader, a risk team and a portfolio manager) research a stock, argue about it and give a rating. I wanted to watch how they reach that rating, so I started building a small interface for a popular open-source framework, TradingAgents. It grew into something much bigger.
Why I'm posting here: I run local models in this project, through Ollama, and I'm developing Glassbench into a benchmark pattern for AI trading agents. It connects different agent harnesses (TradingAgents and AI Hedge Fund so far) and runs them on the same stocks and dates, so the same setup can test and compare different LLMs, local and hosted.
What it does:
- Live view: watch each agent work, with a timeline of every call, adapted for different frameworks
- Runs database: every run stored and searchable, with its reports, costs and ratings
- Framework and LLM comparison: the same stock and date on each framework, and on different LLM providers, you can also run it locally with Ollama. I'm evolving it to become a consolidated benchmark method
- Backtests: the ratings tested against buy-and-hold and a placebo (still testing it, as nobody found a proper way to test TradingAgents)
- Broker connection: a finished run becomes an order on an Interactive Brokers paper account
Frameworks plugged in: TradingAgents and AI Hedge Fund already run in it, unmodified, and more agent frameworks are coming. If you're building your own agent framework, you can plug it in through an adapter and compare it with the others on the same stocks and dates.
It's free and open source (Apache 2.0). My 77 runs ship with the repo, already paid for, so you can read everything the agents wrote without an API key.
Disclaimer: Glassbench itself is not an AI trading agent and makes no trading decisions. Every agent it runs comes from established open-source repos (TradingAgents and AI Hedge Fund), and Glassbench records what they do. Everything was tested on paper portfolios only; I have never traded real money with it. Research and education only, and nothing here is investment advice.