How Complex Systems Fail
Complex systems are intrinsically hazardous systems.
All of the interesting systems (e.g. transportation, healthcare, power<br>generation) are inherently and unavoidably hazardous by the own<br>nature. The frequency of hazard exposure can sometimes be changed but<br>the processes involved in the system are themselves intrinsically and<br>irreducibly hazardous. It is the presence of these hazards that drives<br>the creation of defenses against hazard that characterize these<br>systems.
Complex systems are heavily and successfully defended against<br>failure
The high consequences of failure lead over time to the construction of<br>multiple layers of defense against failure. These defenses include<br>obvious technical components (e.g. backup systems, ‘safety’ features<br>of equipment) and human components (e.g. training, knowledge) but also<br>a variety of organizational, institutional, and regulatory defenses<br>(e.g. policies and procedures, certification, work rules, team<br>training). The effect of these measures is to provide a series of<br>shields that normally divert operations away from accidents.
Catastrophe requires multiple failures – single point failures are<br>not enough.
The array of defenses works. System operations are generally<br>successful. Overt catastrophic failure occurs when small, apparently<br>innocuous failures join to create opportunity for a systemic accident.<br>Each of these small failures is necessary to cause catastrophe but<br>only the combination is sufficient to permit failure. Put another way,<br>there are many more failure opportunities than overt system accidents.<br>Most initial failure trajectories are blocked by designed system<br>safety components. Trajectories that reach the operational level are<br>mostly blocked, usually by practitioners.
Complex systems contain changing mixtures of failures latent within<br>them.
The complexity of these systems makes it impossible for them to run<br>without multiple flaws being present. Because these are individually<br>insufficient to cause failure they are regarded as minor factors<br>during operations. Eradication of all latent failures is limited<br>primarily by economic cost but also because it is difficult before the<br>fact to see how such failures might contribute to an accident. The<br>failures change constantly because of changing technology, work<br>organization, and efforts to eradicate failures.
Complex systems run in degraded mode.
A corollary to the preceding point is that complex systems run as<br>broken systems. The system continues to function because it contains<br>so many redundancies and because people can make it function, despite<br>the presence of many flaws. After accident reviews nearly always note<br>that the system has a history of prior ‘proto-accidents’ that nearly<br>generated catastrophe. Arguments that these degraded conditions should<br>have been recognized before the overt accident are usually predicated<br>on naïve notions of system performance. System operations are dynamic,<br>with components (organizational, human, technical) failing and being<br>replaced continuously.
Catastrophe is always just around the corner.
Complex systems possess potential for catastrophic failure. Human<br>practitioners are nearly always in close physical and temporal<br>proximity to these potential failures – disaster can occur at any time<br>and in nearly any place. The potential for catastrophic outcome is a<br>hallmark of complex systems. It is impossible to eliminate the<br>potential for such catastrophic failure; the potential for such<br>failure is always present by the system’s own nature.
Post-accident attribution to a ‘root cause’ is fundamentally<br>wrong.
Because overt failure requires multiple faults, there is no isolated<br>‘cause’ of an accident. There are multiple contributors to accidents.<br>Each of these is necessarily insufficient in itself to create an<br>accident. Only jointly are these causes sufficient to create an<br>accident. Indeed, it is the linking of these causes together that<br>creates the circumstances required for the accident. Thus, no<br>isolation of the ‘root cause’ of an accident is possible. The<br>evaluations based on such reasoning as ‘root cause’ do not reflect a<br>technical understanding of the nature of failure but rather the<br>social, cultural need to blame specific, localized forces or events<br>for outcomes. 1
1 Anthropological field research provides<br>the clearest demonstration of the social construction of the notion of<br>‘cause’ (cf. Goldman L (1993), The Culture of Coincidence: accident<br>and absolute liability in Huli, New York: Clarendon Press; and also<br>Tasca L (1990), The Social Construction of Human Error, Unpublished<br>doctoral dissertation, Department of Sociology, State University of<br>New York at Stonybrook)
Hindsight biases post-accident assessments of human performance.
Knowledge of the outcome makes it seem that events leading to the<br>outcome should have appeared more salient to practitioners at the time<br>than was actually the case. This means that<br>ex post facto accident analysis of human...