Distributed Systems: Another Great Jeff Dean Quote
During the first year of a typical Google datacenter, there will be five rack-wide outages, three router failures large enough to require diverting processing away from connected machines, and eight network scheduled maintenance windows, half of which cause 30-minute random connectivity losses. At the same time 1 to 5 percent of all disks will die and each machine will crash at least twice (2 to 4 percent failure rate).Dean, J. (2009), Designs, lessons and advice from building large distributed systems.