Near our home is a Dam about to break. It was built long ago, and even though millions were taxed and spent for it, little to none of that money ever got to the actual shoring up and maintaining the dam over time (don’t get me started). Anyway, it’s in sad shape, and an entire valley is about to be underwater. It’s very crowded here, and there are hundreds of thousands of homes at risk and many businesses as well. My house isn’t one of those, but I have to drive that way to get to the deathstar.
I was speaking with a gentleman at the gym this morning and he explained some of the expensive, sensitive, one-of-a-kind pieces of equipment in one of those plants, something that just can’t be moved – and they are right in the path of the floodwaters. They aren’t sure what they are going to do.
I’ve seen this in computer data centers before. In one case where I worked as an Oracle DBA on HP-UX (a UNIX variant) the DBA’s and other administrators kept track of the backups, we had off-site tape storage, we practiced our restores, we had a good RPO and RTO defined, everything was ready. Then a pipe leaked in the data center, and we tried to implement the plan. It did not go well.
What we missed was the tape drive. I was young in my career and didn’t know that the tape drive system we were using was unusual. It was also a long time ago and standard tape formatting wasn’t in use – the format was largely a function of the drive. And we couldn’t get another one of those! We finally found one a few days later at a government surplus shop – a very time-consuming, expensive, frustrating experience.
So what am I saying? Put your physical plant into the DR plan. And don’t build something you can’t replicate – if you do, you’ve effectively locked yourself into a location that just might be part of some really poor government maintenance.