Grading the Advisors
My bot's architecture (previous post) gives LLMs exactly one daily job: after the deterministic rule engine produces its verdicts, a model writes a short comment and a confidence score per ticker, for me to read. The model in that seat costs money every day. So the natural questi...
Which model should narrate my trading bot? Wrong first answer: race them on profit
My bot's architecture (previous post) gives LLMs exactly one daily job: after the deterministic rule engine produces its verdicts, a model writes a short comment and a confidence score per ticker, for me to read. The model in that seat costs money every day. So the natural question: is the expensive incumbent actually better at the job than a model that costs a tenth as much?
My first instinct was the obvious one: run several models in parallel and see who makes more money.
That experiment would have been meaningless, and seeing why changed how I think about evaluating LLMs in ANY pipeline. In my system, no LLM places trades. The rule engine decides, the rails constrain, and the models only comment on verdicts that already exist. Give three models the identical verdicts, and the trades are identical. P/L would measure my rule engine three times and tell me nothing at all about the models. A leaderboard would have LOOKED rigorous and measured nothing.
The honest metric: does your confidence mean anything?
What do the models actually produce that differs? A confidence score. So that is the thing to grade: calibration. When a model says it is highly confident about a BUY verdict, do those instruments actually go on to perform better than the ones it doubted?
The metric, defined before the contest started: on BUY verdicts, take each model's most-confident third and least-confident third, and compare mean 5-day forward returns between them. A wide positive spread means the model's confidence carries real information. A flat spread means its confidence numbers are decoration.
Pre-registered, in writing, before day one. This matters more than the metric itself. Pick the metric after the data arrives and you will, consciously or not, pick the one your favorite wins. Pre-registration is how you keep goalposts bolted down, and it costs one paragraph in a design doc.
The arena: same exam, same desk, twelve weeks
Every day, the pipeline sends the identical payload (verdicts, scored evidence, news) to the incumbent and to two cheap challengers. Same prompt, same data, one variable: the model. Answers land in one table and nothing else happens. No alert, no dashboard drama, no influence on production. The challengers are writing an exam nobody grades until the end.
Design details that keep it honest and cheap:
- The incumbent's arena row costs zero. Its production answer is simply copied into the table. Never pay twice for data you already have.
- The challengers pay after production. Budget is checked with production calls first in line; the arena skips before the daily digest ever degrades.
- Auto-stop is in the code. Day 84, collection ends itself. An experiment with no end date is a subscription you forgot about.
- Total challenger cost for twelve weeks: under a dollar. Cheap models, one batched call a day. The dataset it produces outlives the contest.
The last box is deliberate: the arena produces a verdict, not an action. Even a clear winner does not swap itself into production. Reading the result and flipping the config stays a human decision, because "the metric says so" is an input to judgment, not a replacement for it.
The principle, portable beyond trading
Before evaluating models in any pipeline, ask: what do the models actually control? Grade only that.
If your LLM writes summaries, grade the summaries, not the business outcome downstream of a hundred other factors. If it scores confidence, grade the calibration of that confidence. The lazy metric (the big headline number at the end of the pipe) is usually measuring your deterministic code, your data, or your market, with the model's contribution lost in the noise. The honest metric is smaller, closer to the model, and pre-registered before you peek.
The bot keeps trading on math either way. The arena just decides, quietly and for pocket change, who earns the microphone.
Pictures to add later
- The arena scorecard card on the bot dashboard (confidence trend per model)
- The weekly digest's arena block, once a few weeks accumulate
- Day-84 verdict message, when it lands (~October)