Does It Take 4,000 API Calls to Score a Long Essay?

Does It Take 4,000 API Calls to Score a Long Essay?

Jul 22, 2026

Short answer: Thankfully, no. I got it down to 4.

This post is lightly edited from a mid-contest scoring update that I recently sent to all entrants of the Klara and the Sun essay contest. Entrants have up to 6,000 words to write a scholarly essay connecting a certain passage from the book Klara and the Sun to real-world AI safety. AI is explicitly allowed in the composition process. In the first version of the scoring, an LLM scorer applied a custom 190-point atomized rubric. This post describes my effort to improve accuracy without breaking the bank.

In this before-and-after figure, entrants who have resubmitted many times have their user name tagged with a color. The upgraded process (right) assigns much lower revised scores, with the highest entrants not yet breaking 50% achievement of the rubric items. This means there’s a lot of room for improvement in the remainder of the contest. The new scoring process also comes with dramatically smaller score ranges, down to +/- 7 points at a very high threshold. (95 out of 100 rescoring attempts would fall in the +/- 7 range.) Entrants can now expect that if they create any new submissions, there will be a lot more signal to the score.

Discovering the scope of the scoring problem

When I started the contest in mid-May, Opus 4.6 was the leader of the baseline essays, with 84 points assigned. Out of a total of 190 possible, that seemed plausible to me. Contest entries trickled in. Then Claude Fable 5 was released, and its assigned baseline essay score was 122. But one entrant wrote in their AI use statement, “Literally the Fable essay, but with one letter changed.” And that essay was scored at 132! Later, essays were scoring in the 140s and even 150s, and that seemed surprisingly high to me. I thought maybe those submitters were getting lucky scoring runs.

After focusing a bit on reaching out to people about the contest, I decided I really needed to tackle this scoring variability problem. I printed out Fable’s baseline essay with each rubric item attached as a Word comment at the relevant spot where the scorer AI assigned it. And I went through it painstakingly deciding, “yes, essay achieves this rubric item here” or “well, not here but somewhere else” or “no, not anywhere in this essay”.

When I finished, I realized that I had two problems, not one. I knew about the scoring variability problem. Now I was realizing that I also had a calibration problem. Fable’s average was around 122, but my gold-standard list only had 60-something items! Even if I solved the variability problem - and every time you gave it Fable’s essay it returned exactly 122 - that would still be about double the true score that I had just hand-determined.

I tried to categorize how the AI scorer was going wrong. Two big categories stuck out:

Ambiguous description of a rubric item. Sometimes I was looking for a simple observation to earn a point, but for other items, what I actually had in mind was “observation + conclusion”. All rubric items that asked for an “example” faced this issue, because examples are “observation in service of a certain conclusion.” To use an analogy, let’s say that you summarize a movie’s plot, then say “I didn’t really like it.” That’s a conclusion/claim, and I could ask you for reasons why you didn’t like it. And any individual plot point could be evidence that you use as an “example.” But if you just described the plot, then you didn’t give actually any examples, as such, for why you didn’t like the movie. The original scorer was counting just the observations as sufficient for the point, though. In many cases, just a quote from the novel earned points that should have been connected to a particular claim.

Places where the scorer seemed to get overwhelmed. (I like slightly anthropomorphic language like ‘overwhelmed,’ because it gives you a testable path to resolve your problem.) Sometimes, the scorer wouldn’t provide ANY anchor to a part of the essay - and in the Fable essay those items were almost all false positives, not points the essay should be credited with. Other times, for a sequence of related points, like example-1, example-2, example-3, the scorer seemed to ‘just want to keep going’ even when the anchor didn’t really make sense. This is another sign that it was looking for simple continuations.

If I could fix the ambiguous descriptions, it would narrow the variability. And if I could help Sonnet be less ‘overwhelmed,’ then I suspected it would be more calibrated to the real thresholds for success. I had a plan.

Reducing ambiguity through in-depth rubric review

One summer when I was in grad school, I participated in a large research effort that involved analyzing how elementary schoolers talked with a mock AI agent, and tagging their speech for “scientific reasoning.” It was a long, involved process. The standard in academia...