A Markdown-based test suite - by Julio Merino

Blog System/5

SubscribeSign in

A Markdown-based test suite<br>The background behind a new test suite for a compiler and a VM where test scenarios are all written in Markdown for both human and AI consumption

Julio Merino<br>May 17, 2026

Share

This article is not about AI and it is not written with AI, but the work that I’m about to present was definitely motivated by AI. And because I generally like telling stories, I have to give you that background. Do with that whatever you want, but… it’d be a pity if you left just because the AI word showed up in the first paragraph! I think the technical explanation that follows is at the very least entertaining and also interesting independently of AI.

Back in December, I started toying with coding agents. One thing I tried, and for which I didn’t expect a lot of success, was to point an AI agent to the EndBASIC public documentation and ask it to write games like Space Invaders or Mario from scratch. And even though the results weren’t perfect and they didn’t work on the first try, they did work with a few tiny tweaks. Combining that with a bunch of hand-written AGENTS.md rules, I had an agent producing EndBASIC demos with ease.<br>This experiment was impressive because I did not expect an agent to be able to write EndBASIC code… and because it worked, it fueled my interest to pick EndBASIC’s own development back up. Three thoughts came to mind:<br>Increase EndBASIC’s “self-documenting” aspects so that an AI agent can learn about its idiosyncrasies unsupervised.

Speed up EndBASIC so that it can run more elaborate games.

Extend EndBASIC with long-desired primitives like sprites and sound, to finally realize the vision behind the project.

These thoughts combined sparked the rewrite of EndBASIC’s core that I’ve been pursuing since January and which should see the light of day in the upcoming release. But before that happens, I want to talk to you about just one of the cool pieces behind the new core: namely, its approach to testing. I’ve stopped writing unit tests for the compiler and VM in Rust and I’ve switched to writing them in Markdown. And I believe this has turned out to be a pretty nice approach.<br>So wait, why Markdown?

One of the things I had to do to convince an AI agent to write proper EndBASIC code was to hand-craft a bunch of AGENTS.md rules to tell it how EndBASIC differs from other, more traditional BASIC dialects. That worked OK, but writing these rules by hand was error-prone and difficult to make exhaustive. So I wanted to let LLMs extract that information directly from EndBASIC.<br>The idea was simple: if I wrote the integration tests for the new core in Markdown, the lingua franca of AI, the tests would serve as the canonical and correct documentation demonstrating language behaviors. LLMs are great at summarizing information, so if I unleashed them over a large set of these hands-on “examples”, they would probably figure stuff out, right?<br>And they actually do! I gave the following prompt to GPT 5.4:<br>Based on your pre-existing knowledge of BASIC dialects, I want you to read all of the core/tests/*.md files, analyze how the EndBASIC dialect differs from your knowledge, and come up with a bunch of rules for yourself to know how to write EndBASIC code later on. You can ignore the Disassembly sections.<br>Beware that all functions and commands in these integration tests are test-only: the real functions and commands that you can use are documented in cli/tests/repl/help.out, so read those too to learn what functionality is available.<br>Write your findings to a rules.md file.

And this produced a very comprehensive file with spot-on rules: here, take a look.<br>But leaving that aside, let’s peek into the internals of this new Markdown-based test suite.<br>All cool so far? Want to see more similar content in the future? Subscribe now to demonstrate your interest!

Subscribe

What is a Markdown test suite?

It’s a collection of Markdown files:<br>endbasic$ ls -l core/tests/*.md | head -n 10<br>-rw-r--r-- 1 jmmv users 22771 May 11 14:18 core/tests/test_args.md<br>-rw-r--r-- 1 jmmv users 3785 May 11 13:22 core/tests/test_arithmetic_add.md<br>-rw-r--r-- 1 jmmv users 2653 May 11 13:22 core/tests/test_arithmetic_div.md<br>-rw-r--r-- 1 jmmv users 2668 May 11 13:22 core/tests/test_arithmetic_mod.md<br>-rw-r--r-- 1 jmmv users 2131 May 11 13:22 core/tests/test_arithmetic_mul.md<br>-rw-r--r-- 1 jmmv users 1466 May 11 13:22 core/tests/test_arithmetic_neg.md<br>-rw-r--r-- 1 jmmv users 2517 May 11 13:22 core/tests/test_arithmetic_pow.md<br>-rw-r--r-- 1 jmmv users 2208 May 11 13:22 core/tests/test_arithmetic_sub.md<br>-rw-r--r-- 1 jmmv users 20388 May 11 13:22 core/tests/test_arrays.md<br>-rw-r--r-- 1 jmmv users 5885 May 11 13:22 core/tests/test_assignments.md<br>endbasic$ █Where each file acts as a container of one or more test cases:<br>endbasic$ grep '^# ' core/tests/test_end.md | head -n 5<br># Test: Call to END and nothing else<br># Test: Exit code is an integer immediate<br>#...