IT directors and others responsible for technology aren’t usually given to exaggeration. In many areas of business, the phrase “major incident” is a dramatic way of moaning about whatever’s creating stress this week. But in an IT director’s lexicon, “major incident” means “having one of the worst days of my entire career.”
That’s what one Netfor client experienced on a recent Saturday evening. In addition to its headquarters staff, the company employs several thousand people at nearly a dozen production sites nationwide. At around 8:00 p.m., all the technology at those places suddenly stopped working. The culprit was the Cryptolocker ransomware, which plunged the company into chaos.
Immediate Response and Transition to Full Support
Netfor previously offered top-tier support after hours for the company. The IT leader quickly contacted our Technical Account Manager (TAM) to report the issue. The company’s IT incident management team was scrambling to restore operations. Netfor needed to be ready to handle a flurry of support calls. They also asked Netfor to begin providing full 24/7 support by end of day Monday so their internal team could concentrate on recovery.
Creating a Rapid Transition Plan
The company’s situation was desperate, to say the least, and Netfor was eager to help a valued client get back up to speed. The TAM contacted Netfor’s top management team, who jumped on an early Sunday morning call to craft a plan for transitioning to 24/7 support on a significantly faster-than-normal timeline.
As the plan came together, the TAM collaborated with her client contact to gather specifics about where Netfor should focus its IT incident Management efforts. That was critical because we had been delivering support on several thousand devices. From solving problems with networked large equipment to walking employees through updates to their Outlook accounts.
Managing the Crisis: IT Incident Management Process
Enhancing Support and Flexibility
We spent several hours talking with the company’s leadership team to get a better understanding of the pain they were experiencing. Since we had only been providing after-hours support, we weren’t entirely sure what their daily volume would look like.
We asked some agents to work extra hours. We also rearranged resources to support the company. Our goal is to maintain the service level agreement (SLA) and quality that our other clients depend on. Ultimately, our role became helping the company bring the chaos under control by stepping in and helping their end-users.
Data Recovery and Cleanup Efforts
While the Netfor team wasn’t directly involved in data recovery (aside from helping the company retrieve some files from locked-down servers), we assisted with the cleanup and ongoing support. As a result of the incident, the company stepped up its cybersecurity, and we guided end-users through the installation process. The company’s leaders saw how our step-by-step approach and knowledge base worked with every system.
A Permanent Shift to Enhanced Support
Eradicating the ransomware and getting all the systems and networks back to normal took about two weeks. When the work was complete, the company’s leadership asked Netfor to continue the 24/7 support on a permanent basis. They were impressed with our team’s adaptability and were willing to provide enhanced support when they needed it most. They also recognized the inherent flexibility of our services and their ability to grow as the company’s requirements changed.
Most of all, they saw that when their pain was at its most intense, the Netfor team’s incident response immediately helped them get through the crisis. By making their problem our problem, we helped them get back to business in less time and with less lost revenues. It’s why Netfor clients truly consider us more of a partner than a vendor.
