TDD inside the agent loop - theater or actual value?

TDD inside the agent loop - theater or actual value?

Birgitta Böckeler

Birgitta is a Distinguished Engineer and AI-assisted delivery<br>expert. She has over 20 years of experience as a software<br>developer, architect and technical leader.

This article is part of “Exploring Gen<br>AI”. A series capturing Thoughtworks technologists' explorations of using gen ai technology for<br>software development.

The TDD (test-driven development) workflow can be used with AI-augmented coding in multiple ways:

Human writes the tests: A human defines the test scenarios in some form, be it in natural language, in BDD style, or directly in code. Then AI writes the implementation to make those tests pass (with maybe a first step that transforms the human's scenarios into code).

Review checkpoint for the human: AI writes a failing test, human looks at it to review that the test is testing the wanted behavior, then AI writes the implementation

Fully inside the agentic loop: Prompt an agent to write failing tests first, one by one, and then write the implementation and check that the previously failing test is green.

At this stage, that last usage is by far the most common one. But does it really make a difference, asking an agent to follow a TDD workflow fully inside its own loop? Does it really provide value, or is it one of the rare examples where what's good for the human might be irrelevant or bad for a coding agent?

I created an exploratory evaluation setup to scratch the surface of this question and see what I would find. It is far from a comprehensive and structured eval result, but it did create some hypotheses to think about if you are working hard to get your agent to use TDD.

TLDR; Based on Opus's judgment of the quality of the outcomes, there was no clearly discernable difference based on TDD workflow versus no TDD workflow. On the contrary, more than once Opus ranked the non-TDD workflow solutions slightly higher in design and test quality. There was also no meaningful difference in mutation scores across the solutions.

The setup

Tasks: I created a small, medium and a larger task with the help of Claude, all green field implementations of a bit of business logic. I had it make a bunch of suggestions, asking for idiosyncratic and specific logic to increase the probability that there will be variance between solutions, and not just a repetition of something that is already dominant in the training data.

Instructions: In all runs, I included instructions to achieve at least 80% code coverage.

Model: I used Sonnet 4.6 to generate the solutions.

Judgment of TDD adherence: Evaluation of adherence to TDD was also done by Sonnet 4.6.

Judgment of solutions: Opus 4.8 compared the quality of both solutions and their tests, without knowledge of how the solutions were created. I didn't give very specific inputs on what I consider to be good quality, as this was a very open exploration. And in my experience, the more specific I would have gotten, the more the model could have over-indexed unnecessarily on the quality criteria I list. Opus has shown to be quite a capable model in terms of judgment of code quality. For its ranking of the solutions, it created a rubric on the fly to pass to all subagents that were evaluating the individual solutions.

When you draw your own conclusions from my results, the main caveats to consider are:

This is obviously a very small sample size, so take it with a grain of salt

Judgment of what “quality” means was almost fully left to Opus (with only a few pointers about test quality)

None of the runs ever followed TDD perfectly, but pretty well

The coding tasks given to the agents were all greenfield and relatively small, purely about business logic

How good are agents even at TDD?

Before I even started, I needed to make sure the TDD instructions were actually followed. Historically that hasn't gone well for me: agents often write the implementation first and generate tests after, skip confirming the red step, or over-implement ahead of the current test so the next one passes without ever going red.

The prompt I ended up using worked well enough with Sonnet to use for the comparison, though all sessions showed some of these failures to an extent. For each TDD run, I had an independent agent judge how well the workflow was followed, based on the session transcript, so that I wouldn't accidentally take into account a run that didn't meaningfully do it.

Results

I created 5 batches of solutions, with two non-TDD and two TDD solutions each. In one batch, I also added two runs that were instructed to write the tests first, without full TDD discipline (no incremental red/green).

Across the small (1 batch) and medium (3 batches) tasks there was a bit of a pattern: Opus ranked the two non-TDD solutions #1 and #2, and the two TDD solutions #3 and #4. Only once - after I strengthened the TDD prompt with a more explicit...