The System From Nowhere
Sign in<br>Subscribe
When Accountability Goes Rogue<br>đź’ˇ<br>I recently spoke with Dr. Zena Assaad for her podcast, Responsible Bytes. Video is above, below is an essay adapted from the conversation.
"The system from nowhere" is a way of talking about AI systems that excludes its origin as a consciously, human-designed product. It treats AI as if it were a spontaneously emerging force. It's a reflection of where an observer draws the system’s boundary when they look at it. It’s also a rhetorical magic trick that makes both AI companies, labor and the underlying material infrastructure disappear.<br>Recently, an OpenAI model "hacked" another AI company, Hugging Face. The headlines:<br>New York Times: “OpenAI says its models went rogue and attacked a digital library.”<br>Wired Magazine: “OpenAI models escaped containment and hacked Hugging Face.”<br>Washington Post: “OpenAI’s models went rogue and hacked another company.”<br>When we say “AI models went rogue,” we skip the entire story: the part where OpenAI manually removed the model's cybersecurity blocks. We skip that OpenAI chose to test it on a machine with a live network connection. If you see AI as a system from nowhere, you can make the claim that the model “went rogue,” and that it “broke containment,” both of which place agency and decision-making onto the model itself rather than the people who set the stage for that behavior.<br>When you expand the boundary of the system to include the people building and deploying it, the case becomes much less science fiction and more like incompetence. OpenAI developers optimized an LLM specifically for cybersecurity and coding and then ran it without security guardrails. So they trained a model to find exploits and then acted surprised that it found them.<br>They trained a model to find exploits and then acted surprised that it found them.<br>This incompetence goes beyond some scapegoat engineer. It is an industry orientation: what OpenAI calls "research velocity" is what social media once called "move fast and break things." Had OpenAI been competent, it would still be developing models in this environment in which the AI industry must constantly escalate. The breach was a result of human decisions, but also the human environment. The focus on the model erases both.<br>Focusing on "rogue AI" also forces everyone, even its critics, to talk about what these models can do in a buzz marketing kind of way. I don't think these are stunts. They're public relations spin trying to obscure bad security hygiene. Look at how they work and there is nothing mystical about an LLM writing code that engineers didn't anticipate.<br>But the “system from nowhere” framing overemphasizes the agency of the model, regardless of capabilities or lack thereof. It muddies the line of accountability that leads to the people making choices about how models are built and deployed. It centers the model as if it "acts" and "learns" and "decides" without them. But the first question to ask isn't "why did the model do that?" The question is: what human decisions optimized the model to do that?<br>Objectivity<br>Some lineage: the system from nowhere is a twist on Thomas Nagel's 1986 book The View From Nowhere, which argues for a standard of detached objectivity in relation to the world. Nagel says we can “transcend our particular viewpoint” and see the world more rationally. But there are limits.<br>Jay Rosen adopts the phrase as critique, to describe how journalists imagine themselves standing outside the stories they cover: "just the facts," etc. His objection is that this links legitimacy with a denial of any point of view, but unacknowledged biases are biases left unexamined. Rosen's answer is that journalism should state the position it's coming from and let readers seek out diversity and come to a conclusion.<br>With AI, rather than "both-sidesing," we "no-sides" it. We're told that AI did something, and journalists don't have to wade into why or how. It lets them cover a story without raising technically complicated questions that readers likely won't understand anyway, or confusing questions about the way they’re built and why they are built that way.<br>A point of view accumulates within the model through the decisions engineers make about its architecture.<br>But this confusion is manufactured. It arises from the implication that the models, themselves, have either no point of view or an objective one, or both. Of course, there is no inherent point of view arising from within the model. A point of view accumulates within the model through the decisions engineers make about its architecture. What it is optimized to do, what training data shapes it, what counts as a valid response to a prompt: these are mechanisms put in place through decisions someone made. Good reporting on AI would discuss who, how and why.<br>💡<br>My recent pre-print, The Market in the Model, analyzes how these decisions aggregate into a point of view within the original latent diffusion model.
The...