Each month, we ask our roster of HDI featured contributors and IT service and support thought leaders a question about their work or the industry. We then share their collective wisdom with you.
This month’s question is:
Aside from the pandemic, think back to a business disaster that you and your team faced. What are some lessons you can share that may apply to others in the field?
Roy Atkinson
Principal Advisor, Clifton Butterfield
At about 5:15 PM on a Friday, we noticed some odd behavior on our network. Two computers were spitting out more traffic than they should have, and we quickly realized they were infected. We cut them off the network as quickly as we could, but we already had about 20 machines (a tiny fraction of the total) exhibiting bad behavior. We captured samples of the executable that appeared to be causing the problem and sent them off to our anti-malware vendor; they had never seen it before.
In other words, we had a zero day, and it was going to be a long weekend.
Part of that weekend—aside from executing on our documented plan for an event—was spent making posters. We taped a poster to every entrance of every building, explaining that we’d be working across the entire campus to ensure that the infection had not and would not spread, and telling people not to start their computers until we had cleared them.
We had to think outside the normal communication methods to let people know what to do. We contained the zero day, worked with our vendor on a solution, and were able to return to normal operations on Monday afternoon.
My takeaways?
- Have a plan and execute it.
- Work with trusted partners to remedy the situation.
- Use every communication method available to you to get your message to those who need to hear it.
Pierre Bernard
Global ITSM Process Manager, IKO Industries
The power outage of 2003 in Ontario and Northeastern USA affected more than 50 million people.
For disasters, the IT industry strongly advocates for strong continuity plans, fail-safes, and data protection measures. The whole exercise is predicated on having employees using the available IT services. But what happens when there are no end users?
In the above situation or in a major storm, for people whose work is not in the emergency services sectors, computer work is not high on their list of priorities. I am not advocating that IT departments stop working on their continuity plans for disasters. Rather, I am suggesting that businesses and IT work together to discuss and plan for such events and determine the best course of action if people decide that typical computer-based work is not important at the moment. The continuity plans that were invoked during the power outage worked well all around, but many companies chose not to invoke their continuity plans at all, as their end users and their customers had no power or ability to use the services in the first place.
In summary work was delayed, many items were late or postponed, or had to be redone from the beginning. Eventually power was restored and life as we knew it resumed.
Patrycja Sobera
Global VP, Digital Workspace, Unisys
Life in the service and support industry, particularly on a 24x7 service desk, is not for the faint of heart. We face business issues, outages, escalations, and system downtime every day! In fact we often hear more about the bad and the broken than the positive feedback on awesome support our team delivers.
My favorite phrase attributed to Winston Churchill is that we should never allow a good crisis to go to waste. These moments provide valuable lessons which have the power to test teamwork and resilience, and to shape the culture and collaboration of any team.
My team and I are tested daily on how we respond to crises, and the biggest learning for us is in the areas of communication, collaboration, accountability, and structure. We never lack passion and volunteers to fix issues, but what we sometimes lack is a coherent plan and structure to tackle a crisis in the most effective way. Too many cooks not only spoil the broth, they also can lead to wasted time and delayed resolution.
Every crisis needs a leader - a single point of accountability pulling all resources together, motivating the troops, providing updates to stakeholders, communicating and over-communicating to all involved, checking progress, and driving closure. Tools and technology are always there to support us, yes, but it’s the power of strong leadership that makes or breaks our ability to deal with a crisis successfully.