Wednesday, November 18, 2026

All Complex Systems Tend to Crash, Even With Redundancies and Backups

It might be normal to wonder whether there is something other than random failure going on when widely-disparate systems crash, such as Microsoft productivity apps; banking app functions,  air traffic control communication systems,  national security systems or other software. 


Engineers of complex systems will not be surprised, as any complex system, especially when relying on software, is very hard to keep running “all the time.” Engineers refer to “availability” of components and systems as the percentage of time a component or full system is available for use. 


And the reliability of a system made up of many components can be substantially lower than the reliability of any individual component.


Reliability of each component

Probability entire system works

System availability

99%

Effectively zero

~0%

99.9%

Effectively zero

~0%

99.99%

0.0045%

~0%

99.999%

0.674%

0.67%

99.9999%

90.48%

90.48%

99.99999%

99.005%

99.0%

99.999999%

99.900%

99.90%


The implication: even if every individual component is 99.9999 percent reliable, the complete system is only about 90.5 percent reliable.


Of course, what really matters for whole-system reliability is whether component failures are independent, and whether the system can continue operating when individual components fail.


Assume a system has 100,000 potential points of failure, and that:

  • Each point has an equal probability of failure.

  • Failures are statistically independent.

  • Failure of any one point causes the entire system to fail.

  • All components must function for the system to operate.


For a system with 100,000 independent components and some amount of component dependency, the availability of each component has to be ery high to reach even 90-percent system availability, and super-high to reach system availability at a level when the whole system is unavailable only a few minutes a year.

Desired overall system reliability

Required reliability per component

90%

99.999895%

95%

99.999949%

99%

99.999990%

99.9%

99.999999%

99.99%

99.9999999%


To be sure, most complex systems are designed to avoid situations where there are single points of failure. So we build backup systems. We try to create architectures that protect against single component failures causing cascading outages. 


Consider an example closer to any typical user’s experience of using the internet and apps accessed from the internet. That is a complex system involving your own power supply, your devices, the apps you use and how often you use them. Your experience might also be affected by the way you create backups and redundancies. 


Still, a typical U.S. internet access user can expect the internet to be available and usable well over 99 percent of the time, but the reliability of any particular website, application, transaction, or online activity is lower.


What that means is that a typical user, using a system that is 99-percent available, will still experience more than seven hours of downtime every month. 


Availability

Downtime per year

Downtime per month

99%

3.65 days

7.3 hours

99.5%

1.83 days

3.65 hours

99.9%

8.76 hours

43.8 minutes

99.95%

4.38 hours

21.9 minutes

99.99%

52.6 minutes

4.4 minutes

99.999%

5.26 minutes

26 seconds



Element

Illustrative availability

Downtime/month

Downtime/year

Household electrical power

99.95%

22 min

4.4 hrs

Computer/device hardware

99.9%

44 min

8.8 hrs

Operating system/browser

99.9%

44 min

8.8 hrs

Home router/Wi-Fi

99.5%

3.65 hrs

43.8 hrs

Broadband ISP access

99.9%

44 min

8.8 hrs

ISP backbone/routing

99.95%

22 min

4.4 hrs

DNS and internet interconnection

99.95%

22 min

4.4 hrs

Cloud hosting infrastructure

99.95%

22 min

4.4 hrs

Individual applications

99.9%

44 min

8.8 hrs


The point is that any complex system is going to crash, sometimes, even when redundancies and backups are part of the architecture. And some use cases do not justify engineering much-higher levels of redundancy and backup. 


How many homes have two or more internet access provider accounts, just in case one supplier has an outage? How many routinely keep backup PCs, tablets and smartphones at the ready? How many have power generators? 


And most other parts of the experience are not controllable to any great extent. 


So it might seem as though multiple complex system failures are the result of some deliberate action and actor. Maybe not. We simply live in a world reliant on many complex systems, all of which have some propensity to crash.


No comments:

All Complex Systems Tend to Crash, Even With Redundancies and Backups

It might be normal to wonder whether there is something other than random failure going on when widely-disparate systems crash , such as Mic...