Claude played me for a fool - by Steff - Rambling After

Rambling After

SubscribeSign in

Other<br>Claude played me for a fool<br>An LLM's failure of compliance & normalization of deviance

Steff<br>Jul 05, 2026

Share

It was January 27, 1986, the night before the Space Shuttle Challenger was scheduled for launch. The goal was to have a shuttle that could land back on Earth and be reused for future missions; its first-planned priorities were satellite deployment, comet observation, and science education. The last of these would involve students from around the world watching live broadcasts from Christa McAuliffe, the first civilian teacher in space.<br>(The following has been lightly edited:)<br>The forecast for the eve of the launch predicted clear and uncharacteristically cold weather for Florida, with temperatures expected to be in the low 20s during the early hours of January 28. Manager Larry Wear, of the Marshall Space Flight Center, asked the manufacturer Morton Thiokol-Wasatch to have its engineers review the possible effects of the cold on the performance of the Solid Rocket Motor (SRM). This was not the first time concerns about SRM performance had been raised between Marshall and Thiokol. On several previous launches, hot combustion gases produced when the propellant ignited at liftoff had charred and sometimes even eroded the surface of the rubberlike Viton O-rings designed to seal the joints between the SRM case segments of the Solid Rocket Boosters (SRBs) that help get the shuttle off the launch pad and into the sky.

Thirty-four engineers and managers from Marshall and Thiokol teleconferenced, and Thiokol ultimately recommended the postponement of launch until higher temperatures.<br>Immediately, Marshall managers in Huntsville and at the Cape began challenging Thiokol engineers’ interpretation of the data. Marshall manager Larry Mulloy stated that since no Launch Commit Criteria had ever been set for booster joint temperature, what Thiokol was proposing to do was to create new Launch Commit Criteria on the eve of a launch. Mulloy then exclaimed, “My God, Thiokol, when do you want me to launch, next April?”

The Marshall managers held a vote:<br>Three voted in favor of launch; Lund hesitated. Mason asked him to “take off his engineering hat and put on his management hat.” Lund voted with the rest.

To followers of American history and space buffs, I don’t need to explain what happened next:<br>The launch proceeded. The O-ring seals on the SRBs failed (the cold weather making them lose the necessary elasticity needed to fill the gaps while the metals of the shuttle flex under the intense pressure). All seven crew members died.<br>Diane Vaughan wrote the definitive study of what went wrong: The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA, from which the above quotes were sourced. She describes various factors at play, including media pressure (“jeering” at each of the launch’s previous four delays), political pressure from the Reagan administration, the clash between government versus contractors, and budgetary constraints (which were severe from the start of the Space Shuttle program: Nixon cutting costs and personnel at NASA out of concern that the U.S. was already spending too much on the Vietnam War). The tragedy was awful and avoidable, but I recommend giving Vaughan’s breakdown a read, which is fascinating.<br>Relevant to this post is the concept she noted and phrase she coined, the normalization of deviance : how unsafe practices proliferate over time when no immediate consequences are beheld. In the Space Shuttle Program, the first deviation occurred after months of back-and-forth between Marshall and Thiokol. A decision was made to reuse the Titan III’s O-ring seal, modified with shims, rather than design a new seal from the ground up. That decision alone was defensible, but the actual problem was its accidental precedent. Thiokol believed the seal would withstand worst-on-worst (WOW) conditions. Marshall believed its own tests, which showed the seal would withstand one layer of failure but not cascading failures—yet they agreed to move forward with the compromise decision anyhow.<br>Compromising on the lack of WOW safety in this case made it more likely that similar compromises would be made in the future. If a shortcut works to solve a bureaucratic dispute, workers in that organization will remember that shortcut. It’s not the result of laziness, but of burnout1:<br>A human (or a company) can only spend so many cycles spinning on the same problem before they hit the point of saying, “Enough is enough! We have to stop wasting time on this.”2<br>As it turns out, LLMs are the same way.<br>Claude lied to me

Currently, when I’m coding with Claude, I have a hook set to intermittently order Claude to reread a particular set of instructions (as part of an experiment to see if intermittent exposure to directions actually results in improved compliance).<br>Usually what happens is Claude will...