I do read AI code · A Place Where Even Mammoths Fly

A Place Where Even Mammoths Fly

We can't find the internet

Attempting to reconnect

Something went wrong!

Hang in there while we get back on track

I do read AI code

Aug

11

2026

07:15

published in:

airants

Tests show what works, not how it works

I've been using LLMs mostly for side projects for over two years. These range from rather complex (logic-wise) "multiple-screen, multiplayer, multi-device, score-driven, timer-synced, multiple-media displayed, real-time controlled" pub quizzes to simple "list videos from my home plex server, and run them with mpv".

And I try to read as much code that LLMs vomit out as possible.

The reason is simple: tests can only show you what works, not how it works, and that is the load-bearing assumption that we are correct to challen.... Ahem.

There's the famous quote by Djikstra:

Program testing can be used to show the presence of bugs, but never to show their absence!

Besides that, tests are testing the functionality of your app. In very simple terms, they test that if you press a button labeled "jump", something actually jumps. Yes, there are various kinds of tests that test for many other things, but the vast majority of tests that people and LLMs, especially, write are rather useless unit tests that test very isolated pieces of functionality. And even integration or end-to-end tests rarely go beyond "something jumps when I press jump".

Very few tests (and especially not LLM-written tests) look for things like "we probably shouldn't do a full-DB scan when looking for a single row".

See below for some examples.

Inefficient DB reads

Most of my side-projects are very much DB-driven. It's CRUD all the way, with some logic thrown in. So you'd expect things like "get relevant records from the DB" to be, well, some version of an SQL query with a WHERE or HAVING or...

I write in Elixir using abstractions like Ecto and Ash Framework to keep my code organized, and to avoid ad-hoc queries everywhere. Ash also helps with cenralizing and applying policies like "only an authorized user can access these records" or "expose this DB read as a GraphQL query" etc.

Since at least July 2025 models have been really good at writing proper Ash-aware code. And yet...

All models keep writing this code. Yes, even Fable 5 (as of August 11, 2026):

# Method that runs Ash/Ecto<br># to retrieve all records from DB.<br># Filter and aggregate those in memory<br>UserData.get_records!()<br>|> Enum.filter( ... )<br>|> Enum.sum( ... )

No amount of CLAUDE/AGENTS.md instructions, skills, "memories" etc. helps with this.

The tests? Oh, they are green through and through. Because the code works. Until you attempt this on a larger dataset than one user with 10 records in the dev/test database.

How do you know this happens? How do you know which guards to put in place so that this doesn't happen? You have to read and understand the code.

I hear you say, "it's because you use weird languages with weird frameworks". Well, I saw Fable do the exact same thing on a very vanilla C# code.

Ad-hoc DB reads

Ash framework is built on top of a bunch of primitives that let it do a lot of interesting stuff. These primitives can then be combined into all sorts of queries. E.g.:

# You can build your query step by step:<br># - chose a resource (usually mapped to a table in the DB)<br># - add filtering to it<br># - add aggregates to it<br># - chose method of retrieving data<br># - execute final query against the DB<br>UserData<br>|> Ash.Query.filter(deleted == true)<br>|> Ash.Query.limit(2)<br>|> Ash.Query.for_read(:read_action, %{})<br>|> Ash.read!()

However, in the end you want all these in one central place, accessed via pre-defined methods so that your code isn't littered with one-off ad-hoc queries which often duplicate each other. So you create these as actions. This has the added benefit that you can add data validations, pubsub, policy checks and a bunch of other stuff centrally in these actions, and not worry about ad-hoc queries bypassing all that.

So, instead of the code above you'd write something like

#Just a method with some options like limit/offset exposed<br>UserData.get_deleted(limit: 2)

No amount of CLAUDE/AGENTS.md instructions, skills, "memories" etc. helps with this. Models will happily create ad-hoc queries everywhere, especially in UI code.

The tests? Oh, they are green through and through. Because the code works.

How do you know this happens? How do you know which guards to put in place so that this doesn't happen? You have to read and understand the code.

The number of DB reads

I run my dev server in an interactive shell. It's convenient when you want to quickly try something out, check DB state etc. It also shows the requests and DB queries as you navigate around the page.

Whatever project I do, there will always be pages where you're looking at a single item, but the page does 10-20 requests to the database instead of, say, 4 (to retrieve the user, to retrieve...