My Mental Model for “Is It Worth Automating?” Has Changed | Herman Schaaf

My Mental Model for “Is It Worth Automating?” Has Changed

Disposable scripts suddenly make sense when the implementation cost collapses.

Posted on 17 May, 2026 · 11 min read

So, to set the backdrop: It's May 2026, and we're all talking about AI, all the time. Well, some of us, anyway. And there was an inflection point in the abilities of coding agents around January 2026, to the point where they are now good enough to be trusted with writing a lot of code. This could be good or bad, depending on your view, and maybe you shouldn't trust AI with your entire codebase yet.

Something I haven't seen discussed much is just how useful AI can be outside of your main codebase: we can let AI write code to automate things that previously would have had a terrible work-to-time ratios. Obligatory XKCD:

The above XKCD comic was from January 2014. Now, in 2026, the "reality" graph may very well look like the "theory" graph from before.

To illustrate, I'd like to highlight some examples from a game project I've been working on. There were many instances where I asked AI to create automation for problems where I previously would not even have considered automation at all.

Automation gone wild

During development of a solo dev game, Sky Lobby, I tasked AI with automating the following, among other things:

1. Bots to playtest the game, simulating different player styles and skill levels

Sky Lobby is a puzzle game that involves both strategy and reflexes. Before real playtesting started, I wanted to get a quick feel for how difficult every level was, without spending hours playing the full game myself (however fun that may have been... time is a solo developer's most precious resource!). So I asked AI to write a bot that played the game.

It was to implement and compare four strategies: passive, random, a sweep that brute-forces a floor-by-floor tour, and a heuristic that actually plays well. Each strategy ran repeatedly across multiple reflex tiers, generating win rates and engagement flags per level. As the bots play at a sped-up rate, it takes about three minutes to finish the simulations, and then I get a markdown report showing which levels are too easy (the simple strategies win), too hard (no strategies win), or too boring (long periods of not needing to do anything). This was useful for getting an initial baseline for all the levels, and it allowed me to fine-tune them before playing through them myself and later sending them to human players.

Cost to automate with AI : 15 minutes

Time saved : Days, either of manual testing, or developing similar bots

Previously I may also have written such bot playtesters, but it would have taken a few days, at which point it probably would have been faster and better to play the game through a couple of times.

In addition to the markdown file, I wanted to see how the bots played, to make sure they weren't getting stuck or doing something silly. So AI wrote a shell script that drives Godot headless with --write-movie, records each level, and concatenates them into one playthrough video. In addition to giving me further ideas for refining the game and bot behavior, this was great for getting a feel for what it is like to play the game from start to finish, condensed into a few minutes.

Cost to automate with AI : 5 minutes

Time saved : Hours, maybe days

Would I have done this at all, before? Probably not.

2. A script for syncing Game Center entries

Game Center provisioning means 11 leaderboards, 2 global achievements, and per-level 3-star achievements, each in 10 localizations with its own icon. The official tool is App Store Connect's web UI. I spent 10 minutes manually entering data until I thought, "I'm spending a lot of time on this task...". I asked AI to write code that hits the App Store Connect REST API directly, reads every localization from a CSV, and provisions it all idempotently. This is one of those things that I, as a solo dev working on a small project and only expecting to do this task once or twice, probably would not have automated before, but since it's so easy to automate now, it suddenly made sense and ended up saving me a lot of time.

Cost to automate with AI : 10 minutes

Time saved : 2-3 hours of manual entry, or 2-3 hours of writing a similar script by hand

3. Screenshots of the game for every iOS simulator

Early beta testers reported empty areas on the top and bottom of the screen that I had trouble reproducing on my own iPhone. I asked AI to write a script that takes screenshots of the game being played on every possible device. It came up with a bash script that picks from available simulators, installs the build, launches it with a --shoot-gameplay flag I added, waits six seconds, and grabs a screenshot per device via simctl. The whole sweep takes a couple of minutes.

The same Sky Lobby level captured across a dozen iOS simulators in a single run: from...