Designing Loops for Production-Grade Work — Blog — Liquid AI

Connect

h2]:clear-both [&>h3]:clear-both">In late 2025, we ran an experiment to answer one question: “Can coding agents autonomously solve a production-grade problem from scratch on their own?”<br>For this, we tasked two agents with the (at that time) best publicly available coding models with a real problem and a real deadline. The result of this experiment is a tokenizer trainer called toktoktok, and is now open source on GitHub.<br>In this article, we share what we learned about designing effective loops that allow agents to autonomously solve production-grade problems: how to specify a goal for multi-domain experts and how to set up the verification infrastructure.<br>Why testing autonomy needs a real target<br>As part of our research on the impact of vocabulary size on edge LLMs, we needed a byte-pair encoding (BPE) tokenizer trainer that could run trillions of tokens on a single machine. However, the tokenizer training landscape is thin:sentencepiece was optimized for non-BPE tokenizers and is slow, Hugging Face tokenizers ran out of memory on our corpora, and tiktoken has no training capability at all.<br>That’s why we needed to build a production-grade BPE tokenizer trainer . From our experience with existing libraries, we also knew memory size was the real bottleneck , and they were missing two features we needed: a warm start from an existing tokenizer (vocabulary extension) and a per-language vocabulary budget. This gave us a concrete task, with a real deadline, and an effective way to answer whether coding agents are reliable enough to autonomously solve a task, because it met the following criteria:<br>Production-grade. How agents are commonly used to autonomously solve a problem can't answer this question. First, they are often used for prototyping and never held to a production bar. Second, they reimplement something already existing in a different language, such as “Port SQLite to Rust” or “write a C compiler in Zig,” which is a translation of something the model has likely seen during pretraining. Unlike either of these, ours had a clear ship-to-production goal, and because BPE tokenizer training is recent enough with few public reference implementations, it made an ideal “test distribution” problem sample.<br>Multi-domain expertise. At Liquid AI, our experts run deep, but each in a single domain. Our ML researchers can tell you from memory why OpenAI’s cl100k reserves ranks for every three-digit number, but they’ve never written a line of Rust. Our Rust engineers write exactly the kind of memory-aware, multi-threaded systems code this problem needs, but they’ve never trained a tokenizer.<br>These are two disjoint sets of people, and neither can solve this problem alone. Both human workarounds are lossy: either one of them learns the other’s half first, or we staff it as a collaboration and pay the coordination overhead instead. This is the gap we pointed the agent at: “Can it cover a span of expertise no single one of our engineers has?"<br>Externally verifiable. The artifact must load in tiktoken and Hugging Face tokenizers. Because of this interoperability with third-party software, the work can be checked by code the agent can’t modify. Success isn’t self-reported but rather whether two third-party libraries either produce the right tokens or not.<br>While the details of what had to be built are interesting on their own, what matters for this article is that a task that is real production-grade, hard for our single-domain experts, and externally verifiable is the only honest way to answer whether an agent can do the job without any human oversight.<br>Setting up the experiment<br>For this experiment, we chose Claude Opus 4.5 and Codex with GPT-5.2, the two strongest publicly available coding models in late 2025, as the coding agents and let both work in their planning modes.  Before either agent wrote a single line of code, we set up two things around it: a goal to aim at, and a way to verify whether it had gotten there.

The goal is described in a specification file. It's a single AGENTS.md / CLAUDE.md document written by the operator, describing the outcome and its constraints, not an implementation: The primary architectural constraint is memory. The design spends its complexity budget on memory, so a corpus far larger than RAM stays fairly represented. Compute and I/O are secondary and get straightforward treatment: a system-level programming language (Rust) and multi-threading should be sufficient.<br>To verify whether the agent has reached the specified goal, we gave it two things it couldn’t influence:<br>Production data : We gave the agents sandboxed access to our production training dataset and a machine capable of handling it, specifically an AMD EPYC 9755 with 128 cores, 256 threads, and 2 TB of memory.<br>External verification harness :  The trained vocabulary had to be loaded by tiktoken and Hugging Face tokenizers, and checked both for encode and decode...