-->

Don't Build an RL Environment Startup

Don't Build an RL Environment Startup

Don't sell blood to vampires.

Posted Sep 7, 2025 by Benjamin Anderson

The first person who sold an RL environment to a frontier AI lab must have felt like they discovered an infinite money glitch. It's no longer a secret that frontier AI labs regularly pay hundreds of thousands, and sometimes millions, for clones of Linear and Salesforce. If you're reading this, you've probably thought about quitting your day job and starting a company that builds these unusually lucrative Next.js apps. In this post, I'll argue that you should hesitate before hopping on the bandwagon.

Background: What's an RL environment?

For those unfamiliar, an RL (reinforcement learning) environment is like a sandbox for AI models like Claude and GPT to learn from. It keeps track of an internal state, prompts the AI to take actions to complete a task, and assigns a score based on the outcome. The most obvious kind is a clone of a popular website or enterprise software tool like Doordash, Linear, or Amazon, which teaches the AI to click around and order pizza. It can also be text-only, like the TextArena project, which teaches AIs to play games like Set and Blackjack.

RL environments are having a moment because of the rise of a new paradigm in LLM post-training that teaches models new skills based on verifiable rewards. OpenAI realized sometime in 2023 that if you ask a model to "think" before solving a math problem, and reinforce the "thoughts" that led to right answers, the model learns to think more useful thoughts, and gets really good at math. Since then, labs have been rushing to generalize this paradigm, teaching AIs to use the computer, conduct research online, and work on long-horizon coding tasks.

As a result, a new slew of startups has cropped up whose sole purpose is to feed the voracious labs with ever more arcane and challenging environments. The theory is that that the machine learning bit is solved now; the only remaining work is to provide models with more realistic "simulations" to learn from, so that they don't exhibit weird behaviors when we drop them into the "real world" and try to make them do economically valuable work.

If you're a solo researcher or you have a job that gives you a lot of free time, there's basically no downside to trying to whip up an RL environment and sell it to OpenAI or Anthropic (or Amazon and Meta, the dogs nipping at their heels). You might just be able to make a million bucks and retire to Tulum at age 35.

On the other hand, if you're trying to build a generational business that stands the test of time, I would not touch RL environments with a 1-million token stick.

The Rise and Fall of Crowdworkers

Before RL became the dominant post-training paradigm, all we had was a thing called supervised fine-tuning. It was much simpler back then—you would teach a model trained to predict the next word how to be a chatbot by showing it examples of chat conversations. Of course, someone had to write all those chat conversations; then once it was easy enough to make models chatty, someone had to rank all the chat transcripts, so labs could use only the finest chats to make the AI have the Correct left-liberal political opinions and not give detailed instructions on to make meth. That work was predominantly done by sub-minimum-wage crowdworkers in the developing world.

At some point in 2023, studies started to come out showing that LLMs were better than crowdworkers at various text-annotation tasks. Anthropic had already famously been using "Constitutional AI" to improve its models, relying on AI to rate the quality of AI outputs against a rubric. Mistral AI, which got started a bit later, bragged publicly that they hadn't used human data to train their initial models. (Rumors later circulated they were using their competitors' outputs instead). Human data was still being consumed and purchased in vast quantities, but the quality floor had begun to rise rapidly, because the easy stuff could just be done by LLMs.

These days, despite all the synthetic data, human-data providers are bigger than ever, but they're not hiring low-wage Kenyan and Filipino crowdworkers anymore—they're hiring software engineers and PhDs for $50-200 an hour. But when those PhD skills are no longer useful for improving AI models, they'll be discarded just like the crowdworkers were.

When it comes down to it, building an RL environment for an AI lab puts you in the same precarious situation as those crowdworkers of 2023: useful for now, disposable later. You are providing a rung on the ladder to Artificial General Intelligence, but once everyone has climbed past that rung, they won't need you anymore. Once AI models have a skill, the value of datasets and environments that teach that skill tend to 0, as models can now provide unlimited, cheap data as a substitute.

Free Money Spigots are Short-Lived

"But Ben," you might say, "I know...