I run a 9-model LLM council with a judge to write a daily financial newsletter. Here's what actually breaks. | MarketDaily
MarketDaily ←<br>All articles
MARKETDAILY · ENGINEERING
I run a 9-model LLM council with a judge to write a daily financial newsletter. Here's what actually breaks.
I run MarketDaily, a daily financial email digest. Subscribers pick their holdings (US + Taiwan stocks), and twice a day the system generates a personalized HTML report for each of them and sends it at a fixed time. First commit was 2026-05-19; as of writing that's 75 days and 1,800+ commits of production history, currently serving 21 subscribers. Small scale, but the failure tolerance is basically zero: this is financial content landing in someone's inbox before market open. A hallucinated price target isn't a bug report, it's a trust funeral.
This post is about the reliability layer that makes it survivable: a multi-model council, a judge, and a 31-check deterministic audit gate. Including the three incidents where it failed.
The problem with "model + fallback"
The naive architecture is: call your main model, fall back to another on error. That handles the failure mode that almost never matters. The failure mode that actually hurts is silent degradation : under quota pressure the model returns something that looks like a report — right format, right sections, no exception raised — but the analysis inside is mush. No error code fires. Your user finds out before you do.
So the design principle became: never trust a single model's output, and never trust any model's self-assessment. Three separated layers:
market data ──► structure prior (price vs MA20/MA50 — direction is<br>decided by deterministic code, not by any LLM)<br>COUNCIL: 9 configured free-tier seats across 7 providers<br>(Gemini x2, Groq, local GPU via Ollama, Cloudflare<br>Workers AI x2, OpenRouter, Cerebras, OpenAI —<br>seats with no API key auto-skip; ~5 independent<br>voices live on a typical day)<br>│ each seat: {lean, conviction, thesis, key_risk}<br>│ quorum: >= 2 seats or no council verdict<br>JUDGE: synthesizes consensus AND disagreement<br>(free chain: Gemini lite → Groq → if both die,<br>take the highest-conviction seat verbatim)<br>report generation (separate LLM chain)<br>AUDIT: 31 deterministic checks on the final HTML<br>HIGH fail ├──► wait 60s, retry with a stronger model<br>│ │ still HIGH<br>│ ▼<br>│ deterministic fallback (no LLM, no price<br>│ targets — a bad email never leaves the building)<br>send at the fixed time, every day, no exceptions
Council: disagreement is a feature, not noise
Every stock gets one council round per run (cached across users). Each seat gets identical real market data plus a hard structural constraint: direction is locked by code (price vs MA20 vs MA50). A seat that says "short" on a bullish structure gets demoted to neutral by a plain if statement. No LLM in this system has the authority to flip a direction. The council only operates on the layer where models are actually useful: thesis, counter-risk, and lean within the structurally allowed range.
Then we measure dissent: 0 if all seats agree, 2 if the council contains both long and short, 1 otherwise. Dissent propagates all the way to the final card — high-dissent stocks get forced-conservative wording downstream, and displayed confidence gets pushed down. Confidence is separately capped at 75% by a historical calibration table; the audit treats any number above that as a broken defense, not an opinion.
Seat selection is quota engineering, not benchmark chasing. Full disclosure: the council landed on day 42 (2026-06-30) — before that, the system was exactly the naive model+fallback architecture this post opened by criticizing. The seats deliberately span independent vendors' free tiers, because the thing that kills you at 5am is correlated quota exhaustion, not model quality. Some real constraints that shaped the roster:
Groq's free tier is limited per-minute (8,000 TPM) and per-model per-day (200,000 TPD). The council seat uses a different model than the report-generation chain, because one shared bucket meant the council ate 197,364 of the 200,000 daily tokens and 45 report calls fell through to a 144-second-per-call backup model. That evening's digest was 1h35 late.
One seat (a local 14B model on my own GPU) exists purely because it has zero quota and zero network dependency. It's the only survivor when cloud DNS blips or every vendor's quota dies at once.
One model rewrote a price from the prompt — 385.25 became 385.00. Fabricated precision. It's permanently banned from anything that touches prices, but it's allowed in the council, because council output is JSON opinions with no numbers in them. Blast-radius design beats model trust.
Seats have circuit breakers: quota-dead / missing key / 3 consecutive failures disables the seat for the round. HTTP 402 (billing wall) kills it on the first strike — retrying a payment error is pure waste.
Judge: synthesize, with an exit
The judge sees all seat opinions...