Ten models walk into a bedtime story - by Dan w

Dan's Substack

SubscribeSign in

Ten models walk into a bedtime story

Dan w<br>Jul 16, 2026

Share

A couple of years ago there was this thing on Reddit, dads in r/daddit figuring out that you could feed ChatGPT a prompt and get a bedtime story out of it. My daughter was too small to appreciate it at the time, so I filed it away. Over the next two years she received a couple of personalized books as gifts, the mail-order kind where your kid’s name gets typeset into a stock story, and every time one arrived I thought: I’m pretty sure I could do better than this. Now she’s three and ready for stories, and I knew I wanted something closer to a truly personalized version of those books than the text dump of a ChatGPT session. So I built the tool I wanted and called it Storybook Forge.<br>Dan's Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe

The pipeline is straightforward: you come up with a story idea, an LLM plans the narrative page by page and writes it, a critic model reviews the draft and forces rewrites until it passes, an illustrator draws the pages, and a finished PDF comes out the other end. The whole thing’s automated except for the part where you and your kid come up with the story.<br>I started with local models (mistral and qwen fine-tunes, whatever fit on the 4090 I already had) and the first bake-off was just about finding the best of those. The winner, qwen3-vl, was the best of the bunch but still clunky: weak story structure, phrasing that didn’t land. I wasn’t happy with what I was getting. So I extended the harness to cloud models, on the theory that five cents of Haiku would probably beat the local model. It did. And the engineer in me won’t leave well enough alone, so: why not just test everything?<br>Somewhere in there I realized I’d inadvertently built a micro-benchmark. A fixed unit of work with quantifiable cost and quality metrics that I can plug any new model into. The Forge itself stays private, it’s a fun thing for Ruby and me to do together but too much work to get ready for a wider release. The bake-off, though, is its own durable artifact, and it might be useful to others.<br>How the bake-off works

Every candidate gets the same frozen story plan and writes the same 10-page book, twice (two seeds), with the in-pipeline critic switched off so I’m judging raw writer output. The texts are anonymized, paired seed-aligned across candidates, and each pair gets six blind pairwise judgments: forced choice, A/B order counterbalanced, reasons quoted. Every LLM call lands in a per-book ledger (tokens, seconds, dollars), so cost stays reconstructible, and past verdicts are preserved, so adding a new model grows the panel in place instead of forcing a full recompute. The judges are themselves LLMs, more on that below.<br>The same page, ten ways

Page 4 of the frozen plan is the book’s trickiest beat, a physical-comedy reveal that has to land in about 25 words for a 2-3 year old: Glitterbelle’s biggest twirl knocks her golden horn sideways, revealing it as a party hat on a ribbon. She was a horse all along. Ten writers, one beat:<br>fable — “Glitterbelle spins her biggest twirl—wheee! Whoopsie! Her golden horn slips sideways. Boing! It is a party hat on a ribbon! Glitterbelle is a horse!”

kimi-k3 — “Glitterbelle twirls her biggest twirl yet — whoops-a-daisy! Flippity-flop! Her golden horn dangles sideways. A party hat on a ribbon! Glitterbelle is... a horse!”

opus — “Glitterbelle spun her biggest twirl. Whoosh! Whoops! Her golden horn slid sideways. A party hat on a ribbon! Glitterbelle was a horse!”

sonnet — “Glitterbelle twirled her BIGGEST twirl yet — whoooops! The golden horn slipped sideways. It was a party hat on a ribbon!” (omits the reveal’s final beat)

gpt-5.6-sol — “Glitterbelle spun her biggest twirl. Whoops! Her golden horn slipped sideways—it was a party hat on a ribbon! Ruby gasped. Glitterbelle was a horse!”

haiku — “Glitterbelle spins round and round, bigger! Whoooosh! Her golden horn tilts sideways—boing! It is a party hat on a ribbon! Ruby’s eyes go wide.” (also drops “she’s a horse”)

kimi-k2.6 — “Glitterbelle twirled bigger and BIGGER. Swirly swirly WHOOPS! Her horn went sliiiide! A party hat on a ribbon! She was a horse! Ruby blinked.”

gpt-5.5 — “Glitterbelle spun her biggest spin—whirly-whizz! Whoops! Her golden horn flopped sideways. Ruby blinked. A party hat on a ribbon! A horse!” (judges called it “choppy”)

qwen-local — “Glitterbelle twirls… WHOOSH! Her golden horn slips sideways! Dangle-dangle! Ruby giggles. It’s a party hat on a ribbon. She’s a horse — sparkly still!”

mistral-medium — “Whoopsie-daisy! Glitterbelle’s horn flops—party hat on a ribbon!” (11 words; the last-place finish in one line)

The writer bracket

Ten candidates, two books each from the frozen plan. Wins are out of 108 possible (9 opponents × 2 seeds × 6 judges);...