I Audited My Own Trading Bot and Found It Had No Skill
Steve Farmer
SubscribeSign in
I Audited My Own Trading Bot and Found It Had No Skill
Steve Farmer<br>Aug 12, 2026
Share
I Audited My Own Trading Bot and Found It Had No Skill
I ran a weather trading bot on Kalshi for four months. Then I tore it down and measured it. The model had no skill. It would have been better to do nothing.<br>That sounds like the end of a story. It is the beginning of a better one.<br>The numbers
112 settled trades. I scored every probability the model ever produced against what actually happened. A Brier score of 0.2858. For reference, predicting the historical base rate every time, with no model at all, scores 0.2439. My model was worse than closing your eyes and guessing the long-term average.<br>That is not a rounding error. That is a negative-skill system. It was subtracting information from reality.<br>The market was right. Contracts the bot bought at an average of 60.6 cents won 60.0 percent of the time. The market was pricing these things correctly and the bot was paying fees to confirm it.<br>Two mechanisms, both mine
I found two things wrong.<br>First, the confidence was way off. The model spread its probabilities 2.1 to 4.0 times too narrow. It was certain when it should have been uncertain. In prediction markets, confidence is what sizes your bets. A model that is too confident trades too big. Direction can be correct 60 percent of the time and you can still lose money if the sizing is wrong. That is what happened.<br>Second, a systematic temperature bias at the gridpoint level, peaking around seven degrees Fahrenheit. The model leaned warm. Warm in a way that was not in the data. It was in the model.<br>These two things together create a failure mode worth understanding. The model had a directional signal. It was right more often than wrong. But the error in its confidence swamped the signal. A larger position on a biased forecast generates a bigger loss than a smaller position on a calibrated one. The model was defeating itself.<br>The fix was not a cleverer model
Here is the part that still stings.<br>NOAA's National Blend of Models already publishes calibrated, bias-corrected, station-level probabilistic temperature guidance for exactly the stations Kalshi settles on. For free. I was hand-rolling a worse version of a public good.<br>The ensemble the bot used, counting members across raw forecast systems, was the mistake. I spent months tuning a member-counting approach when the right answer was published by a government agency I could have downloaded at any time.<br>This probably sounds like an advertisement for government science. It is. The NBM blends dozens of models and applies statistical post-processing no individual system can match. The uncertainty ranges come baked in. The bias correction runs at individual station level. It is a better product than anything I could build alone, and it costs nothing.<br>The bot that declines
The rebuild runs on NBM guidance. It skips same-day markets because NBM coverage begins at forecast hour 24. Rather than fall back to the old model that lost money, the bot does not price what it cannot price.<br>This costs candidates. A trade that does not get opened is a trade that does not generate a return. That is the point. The most important thing a trading system can do is refuse to act when the conditions it trusts are not present. A bot that declines is a bot with a discipline.<br>Other things I found while I was in there
The scanner had been looking for Kalshi category "Climate" when the real value was "Climate and Weather." It matched nothing. A fallback scanned the raw catalogue, and every log line looked normal while the bot saw 39 to 63 of 291 available series. The count changed between runs, so entire cities blinked in and out. It ran like that for four months.<br>Chicago settles on Midway, not O'Hare. Houston on Hobby, not Bush. I had to pull every station code from Kalshi's own metadata and pin them in a test suite that now runs 426 tests across 33 files. The previous suite was smaller and did not catch these things.<br>Three money bugs had the same shape. The daily-loss kill switch reads settled profit and loss, so a mis-recorded trade silently moved the limit. Partial fills never settled. Weather P&L was booked gross of fees. The econ bot never recorded a position it closed. I fixed all three, and the fixes touched the ledger, which means the next time it runs a real trade matters more.<br>Why I wrote this
I sell the source code. $75, discount code HOME15.<br>The reason that matters here is that selling it forced this audit. I cannot look someone in the eye and say "I trade with this" unless I have measured it. I cannot ask a dollar for something whose failures I have not catalogued. The product is not a crystal ball. It is a discipline. The discipline is measuring what you believe.<br>The results page at predictandprofit.io carries every settled trade, public. The econ bot is there too. The source code ships...