Coaxing quality output from generative AI | The Scala Programming Language

Coaxing quality output from generative AI

Monday 6 July 2026<br>Bill Venners (Artima, Inc.)

Coaxing quality output from generative AI<br>-->

This post covers work done under the Sovereign Tech Fund investment. The work is coordinated by the Scala Center. The work described in this post was done by Bill Venners and Chua Chee Seng of Artima.

One of the tasks funded under the STA project is to improve Scala’s API documentation. Such a task looks like a natural fit for generative AI, because the work is repetitive and the code to document is nearby in the source. But handing the work to an AI raises two challenges. The first is quality: an AI model will readily produce documentation, but it may be mediocre or wrong. The second challenge is managing human review time: someone will have to check all that output. This post is a report on how we approached these problems when using AI to improve API documention of Scala’s standard library: what we tried and how it worked out.

Quality writing requires iteration. You write a draft, review it (or have someone else review it), then improve it. You repeat this process until you’re happy with the quality. One way to achieve quality with generative AI, therefore, is to delegate the writing part to the AI model and review it yourself, but this doesn’t scale. An AI model can generate content at a much greater rate than you can review it. Because of this, we decided to use the “LLM as a judge” technique in an attempt to raise the quality of the output as high as possible prior to asking humans to review it.

Another way we tried to raise the quality of the AI output prior to human review is to use harnesses to orchestrate the AI work. First, we wrote a todo-writer application that reformatted doc comments for consistency and inserted text markers in the standard library source to precisely indicate where we wanted an AI to insert focused documentation. Then we created a script to drive the filling-in process. Finally, we attempted to keep the review tasks for humans to a reasonable size.

The rest of this post describes five techniques that formed our overall approach.

Technique 1: Push work onto deterministic code

The first pull request we submitted, which made the style of existing documentation more consistent, was done simply by asking an AI model to find and make the changes, and then reviewing the results. This prompting technique worked fine given the limited scope of those changes.

Next, we wrote and ran a Scala application to convert wikidoc to markdown, the recommended markup language for Scala 3, and ran it across the entire standard library. We hoped this would make the documentation in the source code nicer for humans to work with. We also felt it would provide a more consistent canvas on which into ask AI models to fill in missing documentation. This resulted in our second pull request. The application we used was todo-writer, run with the --migrate-markdown command line argument.

We next ran todo-writer across the standard library to insert TODO FILL IN markers wherever existing documentation was missing any @param, @tparam, or @return tags. Here’s an example:

/** Creates a new map obtained by updating this map with a given key/value pair.<br>* @tparam V1 TODO FILL IN<br>* @param key TODO FILL IN<br>* @param value TODO FILL IN<br>* @return TODO FILL IN<br>*/<br>def updated[V1 >: V](key: K, value: V1): CC[K, V1]

Each TODO FILL IN is one precise marker, which turned the AIs’ job from the open-ended “find and improve the documentation” into the mechanical “replace this text with the right content.” We could now focus the models on exactly those spots, with nothing about where to write or what needed writing left to their judgment.

The principle underneath this technique is to use determinism wherever you can. Lean on a deterministic program for every part of the job that does not genuinely require judgment, and strip out any nondeterminism that is not essential. Every piece of the job you hand to deterministic code is a piece the AI cannot get wrong.

Technique 2: Enforce a deterministic process

Another way we used determinism was by defining and enforcing a process through a shell script that called into the AI model, rather than asking an AI to run the entire task. This kind of orchestration is often called an “AI harness,” which both implies that an AI workhorse is being constrained as well as being put to productive use.

Our harness used two git branches. Once todo-writer had inserted the TODO FILL IN markers, we committed its output one file at a time, so that every file landed in its own commit, each carrying the same message: Todo-writer added TODOs for @param, @tparam, and @return tags. A small helper, commit-each-file.sh, did this slicing. The effect was to turn the whole task into an ordered queue of single-file commits on a source branch, with nothing else mixed in.

The loop itself,...