Hurricane Sandy (2012) proved that disaster recovery is a very real concern which has the potential to exceed even our worst fears. Overnight millions of dollars of hardware was lost at dozens of data centers throughout New York and New Jersey. Over the following few days, millions more were spent keeping those data centers alive, but running on fumes.
Cloud, colocation and private data centers alike were severely impacted. To their credit, the staff of the affected data centers performed near miraculous acts to prevent loss of data and reduce downtime. However, the scarcity, cost and environmental concerns raised by the use of emergency gas generators should give pause to anyone creating a disaster recovery plan.
The manager of one of the data centers affected by Hurricane Sandy, a significant component in the technology infrastructure of a large international company, was interviewed about the his staff’s experiences during Hurricane Sandy.
He described a nightmarish situation,
“The campus was swamped with six to seven feet of water. Both its power substations were under water, as were the diesel fuel pumps. UPS batteries were nearing their end of life. Street power was out, and access to the facility was hindered by partially collapsed road.” (Source: Data Center Knowledge 2013)
Even the best facility designs couldn’t prevent the damage caused to the local roads and numerous days without power. No amount of on-site preparation can assuage the financial pain of running emergency generators for days on end or ordering emergency replacements during a crisis.
Even if you can source replacement gear, paying for it can be challenging. “I spent close to $2 million in four days…That’s tricky when your company card limit is $50,000 and your internal procurement systems are down”. (Source: LifeHacker 2013)
Colocation providers like Internap and Peer1 suffered tremendous financial losses while dealing with the hurricane damage in New York. Internap alone burned almost 20,000 gallons of fuel to keep their generators functioning. Staff at Peer1 recalls the extreme measures they were forced to take,
“They had their brigade of people carrying five-gallon buckets [of gasoline] up 18 flights of stairs or whatever it was to get to their auxiliary generator.” (Source: ArsTechnica 2012)
Data centers managers in New York and New Jersey could have saved millions of dollars if they had moved data and applications to remote disaster recovery sites before the storm. Their data center staff could have focused their energy on protecting their equipment, applications and data; but instead they were forced to go days without food in flooded buildings carrying buckets of gasoline to generators struggling to keep up with the load.
Many survivors of hurricane Sandy publicly hailed cloud computing as the stop gap solution for disaster recovery; but why should you suddenly feel comfortable exposing your data to the public cloud simply because Mother Nature overwhelmed your disaster plan? This strategy may be perfectly adequate for email and other non-essential data; but may not be appropriate for sensitive or mission-critical data. In terms of privacy and security, deploying your own assets to a disaster recovery site in a remote location is preferable to the public cloud when storing any kind of sensitive data.
Emergency measures, as impressive and well-intentioned as they may be, don’t replace the need for a real disaster plan. In a recent interview with The Data Center Journal, Dr. Alex Winokur (CTO and cofounder of Axxana), outlines a thoughtful and sensible approach to disaster recovery planning for data centers,
“The question shouldn’t be, “How do you mitigate disaster risks to your data centers?” but rather, “How do you mitigate disaster risks to your applications and your data?” The only way I know to do that is to have two data centers a long distance apart…If your data center is in a disaster zone, and it’s your only data center, and it has your only copies of the data, then you may never recover.” (Source: The Data Center Journal 2013)
The challenges faced by companies wanting to establish a reliable and cost-effective disaster recovery site was one of the inspirations behind the creation of DartPoints’ network of data centers. DartPoints’ Private Colo solution gives you the same privacy and security of your own corporate data center in the location of your choice, but offers this service at the price point of traditional multi-tenant data centers.
Two data centers really are better than one…
This post was written by Nathan Binford, to reflect the views and opinions of the team at DartPoints. Nathan is the primary blogger for DartPoints and contributes to a variety of other sites and publications as well.