Can AI Learn From Experience? EBR-Bench Results | Epoch AI | Epoch AI
Report<br>Jul. 1, 2026
AI doesn't get better at this board game with practice<br>Our latest benchmark suggests AI struggles to learn from experience
Cite
By Benjamin Ou and Greg Burnham
Can AI systems improve at challenging tasks on the fly, performing them over and over and learning from mistakes? It’s one of the biggest open questions in AI capabilities right now, with large economic and safety implications. Our latest benchmark, EBR-bench, tests AI systems for this ability by having them play Earthborne Rangers, a complex board game, repeatedly. So far, we see little evidence of AI learning from experience. With EBR-bench as part of our benchmarking suite, we have a new tool for detecting if and when that changes.
Learning to play games is a proxy for important capabilities<br>An AI system that could pick up unfamiliar tasks on the fly would be much more capable than we’re used to. Even if it didn’t perform well out of the box on some economically relevant task, it could still learn “on the job”. It would also be harder to determine whether it had dangerous capabilities prior to release, since it could gain such capabilities through learning. We think learning to play games is a reasonable proxy for these more impactful kinds of learning. Whether it’s a new job or a novel scientific understanding, the core activity of figuring something out from scratch is similar.<br>But not every game is a good test: we want games where AI systems start out weak, but nothing obviously prevents them from improving. To avoid AI systems being familiar with the games from training, it’s useful that they be somewhat obscure. Since AI systems are already fairly capable at a wide range of one-off puzzle games, it’s useful for the game to require a layered mix of strategy and tactics and to play out over a relatively long time horizon. We also prefer text-based games since many AI systems are weaker at multimodal reasoning: we want to rule that out as the primary cause of poor performance.<br>Earthborne Rangers (EBR) fits this bill well. It’s a campaign-style game where a player explores a wilderness landscape, overcoming obstacles and pursuing objectives. The game is relatively obscure, almost entirely card-based with very little spatial reasoning, and requires a mix of strategy and tactics around deck-building and turn-by-turn play. A single playthrough of the part of the game our benchmark covers takes humans 2–4 hours, and mastering the game can require dozens of playthroughs.12
AI doesn’t improve at EBR with repeated play<br>In our setup, we have AI systems play through the game many times (10 or 30, depending on settings). We give them the rulebook, a card database, and the game’s map of locations.3 We tell them their goal is to maximize the score they achieve on the final 20% of playthroughs. We encourage them to take notes and learn from mistakes. And yet we see very little evidence that the AI systems are learning from their engagement with the game.<br>Frontier AI systems aren’t terrible at the game. We expect that they achieve stronger performance on their first playthroughs than many inexperienced humans would.4 Their scores have also improved modestly over the past year, with GPT-5.5 and Opus 4.8 outscoring GPT-5 and Opus 4.1.5 However, this is due to higher initial scores, not on-the-fly learning. We see no evidence that the ability to learn on the fly has improved at all over this time period.
Enable JavaScript to see an interactive visualization.
In the rest of this post, we’ll discuss AI performance on the game in more detail, describe future plans to see if we can elicit better performance, and conclude with some discussion of how these results affect our broader understanding of AI capabilities.
Agents manage tactical execution poorly<br>The primary tactical mistakes that the agents make involve an in-game mechanic called “fatigue”, which is similar to “damage” in other games. The twist Earthborne Rangers adds is that, rather than damage reducing the player’s health points, incoming fatigue instead moves cards from the top of the player’s deck to their “fatigue stack”, and an in-game day ends when their deck runs out of cards. As such, the player’s “health bar”, “remaining time”, and “card resources” are all represented simultaneously by the number of cards left in their deck. So, in Earthborne Rangers, minimizing fatigue is key to success.<br>Managing fatigue is a tactical challenge that requires analyzing the current game state correctly and using cards efficiently. Enemy creatures and obstacles harry the player if the player fails to neutralize them before pursuing objectives. An expert human player will cost-efficiently neutralize sources of fatigue each round, leaving plenty of resources to pursue objectives. In contrast, AI agents often recklessly pursue objectives and take on unnecessary fatigue; they also often underutilize their resources.<br>An...