Sandy’s Lessons

Softwhite is located on Long Island. As you know by now, Long Island got wupped hard by Sandy.  Many people lost homes and have suffered.  Some are still suffering.  I live on Long Island as do some of Softwhite’s personnel.  I lost power for 8 days at my home but my house was not damaged and no one was hurt.  Nor were any of Softwhite’s staff’s homes or loved ones harmed. Others are much less fortunate.  100 or so people died in this storm and countless are now homeless.

The day after the storm, Long Island and surrounding communities could not physically go to work. In fact no one could work at all. 90% of the power customers on Long Island lost service. Cell phone service was poor to non-existent. Street lights were not working; power lines were down; the stock market did not open for two days. Several client locations were flooded.

This is what happened next and what I learned from this storm.

Day One of Sandy

The Softwhite data center is fully operational with the exception that the main email server is offline. Softwhite’s inbound email is stored in a server in California, yet sending and receiving of email is held up and some of our client web sites are down. Our policy is honesty and transparency and we are going to report what we know as soon as we know it.  We post a bulletin on our web site landing page acknowledging the issue and promising to post updates.

No contact from clients who are dealing with their own efforts to stand things up. We are all dealing with no power at home, no way to make phone calls and an inability to travel too far from our homes.

Day Two of Sandy

Softwhite’s Lead Network Technician (who is also without power at home) and I meet at the data center facility and we go to work. What we find makes us feel foolish. The data center is powered and sustains no damage. The unresponsive email server is cut off from the internet because our T1 links are down. Power is out somewhere along the way between the Telco Central Office and our location. We feel foolish because we have a 6Mbps fiber optic link which is fully operational. We spend the next few hours moving network interfaces and making the necessary DNS changes.  By 10:30 am, all servers and services are functioning. Email begins to flow.

The Lessons

Email is more important in an emergency than phones.  Communicating with employees and loved ones is critical. One day after the storm, Softwhite restored full services to its customers. Files were backed up off location and safe. The mail was cached in a 4 day queue window. Recovery was in process. This is good, just not good enough.

Softwhite’s disaster recovery procedures were tested and we learned about reacting in a natural disaster. Should there be a next time, we will be more prepared. Should there be a next time, we won’t be at the mercy of a mail server.  Should there be a next time, we will swing over to the backup servers immediately and then go to work restoring regular service.  Next time, we will be better.

 

Leave a Reply