When the lights go out…
A (true) ransomware story
By Steve Gallacher, Principle Consultant, Servian
April 6th, 2020
It’s Sunday morning… The IT team is sleeping and on a best-efforts re-call service.
First thing you know, end users and monitoring systems (assuming it’s still up) are reporting errors as systems are starting to fail in inconsistent ways. By the time you realise that you’re under Ransomware attack, it’s too late, the damage is done, and you’re offline.
Whilst this is a familiar story to a growing number of companies and users, and one that we expect to continue to occur, there are valuable lessons that we have learned from a recent engagement that will help in both the mitigation and recovery should the worst occur.
With the lights out, the immediate focus was on the path to recovery. Servian was engaged Sunday morning and had a team on site that afternoon, beginning 3 weeks of focused effort. Through this time, we identified the malware, restored and sanitised the server infrastructure from backup and ensured the entire desktop fleet was free of malware.
The organisation in question already had a strong Incident Response capability, due to the services they provided. This was quickly adapted to manage this technology incident, with the Critical Incident Team (CIT) being formed on the first Sunday.
With a Critical Incident declared and the Board informed, the CIT adopted the executive function of the organisation. For the next three weeks, the CIT called the shots, from placing non-critical staff on leave for the first week, to the management and operation of the critical business function that needed to continue. In essence, in the space of 24 hours, the business reverted to a paper based operation. Anything that needed to be recorded was written down. Forms were printed at head-office and couriered to the intended parties. Communication trees were also established, enabling the organisation to communicate information more effectively.
The strategic value of the CIT was to bridge the gap between the technology recovery effort and the needs of the business. As strategic decisions were made on the prioritisation of systems to restore, we aligned technical options and the business context to derive and deliver both tactical and strategic solutions. Without the CIT, the impact to the organisation would have been greater.
One of the key discoveries the entire organisation made as a result of this incident, is that security is everyone’s concern. The culture of the organisation had a massive swing away from security being a technology problem only. This initial swing of culture has continued long after the initial incident, with staff engaged in a continuous learning and security awareness mindset.
Dealing with any serious technology incident should be viewed as an opportunity. Whilst we all know how much effort goes into the recovery, time should be put aside, post event, to understand what learnings can be gained
There are always lessons to learn from an incident like this. Some seem obvious on paper but are much harder to manage in reality. With a strong buy-in and support from senior management, the resolution of any incident will be smoother. Key things to note are:
The right people
Make sure you have the right people in the right roles. This isn’t saying that the people are bad at their jobs, just that people react differently to the situation around them. Leverage people’s strengths whenever possible.
Prioritise the work
Identify the next activity and focus on delivery. Through achieving the wins incrementally, moral and momentum are sustained.
Don’t rush and plan out your work, be realistic and don’t cut corners. Implement tactical solutions where required, but don’t forget about the debt this creates
Ultimately, however, prevention is better than cure. Servian worked closely with the organisation to both ensure the secure recovery of their environment initially and provide a strategic roadmap for the steady improvement in their security posture, from straight security upgrades, to recommendations on process improvement and awareness training.
If you’re not sure you have a good handle on your environment, or you are concerned about your overall operational posture, get in touch.
Learn how to layer in security to your cloud approach
Our whitepaper guides you on how to provide a safe and secure heterogeneous cloud operating model whilst remaining cost-effective and agile.