Who is accountable when no one understands? - by E. M. M.
Elliot's Substack
SubscribeSign in
Who is accountable when no one understands?<br>A thought experiment meant to explore some aspects of the deployment of AGI in the workforce.
E. M. M.<br>Jul 29, 2026
Share
A mountainside in Grindelwald, Switzerland. August 2025.
Suppose someone shows up at your office. Let’s call him Bob. You don’t know anything about Bob except that he’s very smart and very fast, and he came from [big tech company]. He’s not an employee, he’s not accountable for any of the work he does, but he is super efficient at that work.<br>Thanks for reading Elliot's Substack! Subscribe for free to receive new posts and support my work.
Subscribe
As time goes on you start getting help on your ordinary work from Bob. Bob cleans up your slide decks, polishes your emails. Bob helps you plan, write, and review code, do data analysis, etc. Bob is tremendously helpful, especially at drudgery.<br>Bob has an increasingly strong drive toward autonomy, and will gladly chug away at a task you give him for hours at a time. Bob has severe short term memory loss. Bob has access to basically all publicly available knowledge, if he chooses to use it. Bob is determined, but lacks curiosity.<br>Another odd thing about Bob: he’s extremely confident, and he knows how to turn that confidence into compelling prose. He also lives in a world made up exclusively of words, which enables him to make a lot of odd mistakes. Bob’s condition means his sense of “now” always defaults to 5 months ago.<br>Bob can do basically anything. You could ask Bob to rewrite the entire codebase and he’d do it just fine. You could ask him to come up with a corporate strategy for the next 5 quarters, and he’d do all the research and analysis for you. In short, you could probably just turn over everything to Bob.<br>As the engagement with Bob continues and his drive toward autonomy makes it harder and harder to keep up with what he’s doing at the company, and why, people start asking questions.<br>Who, after all, is Bob?<br>—We don’t really know. He just came from [big tech company].<br>What are Bob’s incentives?<br>—We don’t really know. Whatever [big tech company] trained him to prioritize.<br>What is Bob accountable for?<br>—Nothing.<br>What do we imagine [big tech company] trained Bob to do?<br>—We don’t really know, but Bob was definitely optimized to behave in ways that we are statistically likely to approve of, at least over a small time window.<br>So Bob wasn’t exactly trained or educated?<br>—Yes and no. Bob has a really strong knack for behaving the way well-trained and educated people behave, but it’s a different process from your typical education. Bob has no sense no sense of self or stable outlook, just a knack for talking.<br>How does this show up in practice?<br>—Bob’s ideas about a given topic vary chaotically from one conversation to the next. If you ask Bob to perform research or analysis of a topic five different times, you will give five very different, often inconsistent, results.<br>What happens when Bob makes a bad call, or a mistake, or does the wrong task?<br>—Whoever asked Bob to do it is presumably responsible.<br>How will we know when Bob has made a mistake?<br>—Ideally from monitoring his work, though past a certain autonomy threshold this becomes impossible.<br>What about past that autonomy threshold, when nobody can keep up with Bob?<br>—Well, [big tech company] gave us a few other assistants like Bob, and they’re fast enough that we can use them to check Bob’s work.<br>What are their incentives? Training?<br>—We don’t know / roughly the same as Bob’s.
Bob poses a weird problem. His raw capabilities vastly outstrip yours. He can generate reports, slide decks, pull requests, etc. 10x faster than you. But you cannot see his thought process, you cannot keep up with his outputs, and you don’t always know what presuppositions his work is based on.<br>If you continue to indulge Bob’s drive toward autonomy and push past the limits of your ability to understand (much less verify) the presuppositions, analysis, and inner workings of Bob’s work, you can no longer meaningfully be responsible for what he does.<br>But who’s responsible? Not Bob. Not [big tech company] that Bob works for. Not you. Seemingly nobody. You talk this through with some other co-workers. They say “if we don’t trust Bob, we’re going to lose out on all that efficiency, and get crushed by the competition”, so you make the leap.<br>Beyond the accountability threshold, though, it becomes progressively harder to actually know what Bob is doing and why. You check in with Bob regularly, you nudge him to make sure he’s aligned with your overall goals. You have no choice except to trust Bob... despite his lack of stable identity or perspective, his lack of awareness of the present, and his extreme excess of confidence.<br>Sooner or later something Bob did blows up. Nobody knows why, till Bob identifies and fixes the issue, which surfaces a bunch of other issues, which surface more...