How to survive and recover after a ransomware attack, by SITC’s Paul Meyers

When a ransomware attack strikes, every second counts. Few know this better than Paul Meyers, Managing Director of Stock In The Channel, a leading B2B ecommerce platform serving IT resellers and MSPs around the world. After a ransomware attack hit the company, Stock In The Channel was back online within just 24 hours—a turnaround that many organisations can only dream of.

In this candid account, Paul shares how the attack unfolded, the critical decisions that enabled such a rapid recovery, and the lessons MSPs and IT leaders can take away to prepare for their own “worst day.”

Catch Paul at MSP GLOBAL this week—scroll down for the when and where.

How it unfolded

August 11^th, in the evening—a date I won’t forget in a hurry. We run a business with customers in Australia and the US, so we’re on 24 hours. Around 10:00pm, we started getting reports from our customers in Australia that stuff wasn’t working or loading properly. That’s when we started investigating.

It got worse. By about midnight, everything went offline. We didn’t know what it was at the time. So we thought there’s nothing else we can do: we just have to get to the data center and see the actual machines and what’s happening. When we got there and we fired up one of the machines, there was a ransom note from Akira Ransomware. My heart sank seeing that.

It was a horrific moment. We’d been expecting it because so many other companies had experienced it. We knew that we were a target, but seeing it is a horrific experience.

Assessing the damage

The way that the ransomware actors work is they cut off access to the system. So we were increasingly blind.

The first thing we did was to look to see if the attack was still ongoing. We realized that the attackers were still active on our network. The first thing we did was shut off all the affected machines because they were working to encrypt all the data to then exfiltrate it. Shutting everything down gave us some breathing space to decide what to do. Then we tentatively started looking at turning on machines and assessing the extent of the damage.

I think that’s when it can be easy to be tempted to rush into things instead of taking a step back. Is everything affected? Is everything encrypted or is it only partial and what can be salvaged and what can’t? We spent a lot of time going through and auditing the extent of the damage before taking any action.

Back-ups and back online in 24 hours

At the end of the day the best way that anybody can respond to a ransomware attack comes down to whether they have copies of the data that they can recover. If not, you’re into having to negotiate with the ransomware actors.

For us, we had off-site backups, so we knew that we had all our critical data sorted and we were able to start getting it back.

We have a great team, and once we realized what had happened, we had several guys who worked for 24 hours.

I was first on the scene, saying, ‘We can handle this. It’s not good. But we’ve got this,’ when inside, I’m thinking, ‘This is terrible!’ I spent all day at the data center just getting coffees for the guys and bringing them sandwiches and just encouraging them, because they were heroes.

They worked and worked until it was back online and didn’t stop—the hackers had completely compromised our system. Well over half our servers had been encrypted, so there was a huge amount of damage, and even the ones that weren’t encrypted had still been damaged and compromised.

We had done our disaster recovery practice, but until you see whether the backup and restore works, there’s that stress.

The importance of incident response plans

We’d thought about our incident response plan, and we’d discussed it internally, and made sure that our structure was good, that we were backed up and that we would be able to recover. But it wasn’t formalized enough. There were bits that were missing, that could have been done quicker, bits that we lost and information that we could have had to hand that we didn’t have. There were certainly learnings from it of how it could be done better.

Formalizing everything is the big thing, but also make sure you have your backup plan sorted. Make sure that your backup is offsite because the hackers will log in and the first thing they’ll do is delete or format your backup drives, which is what they did to us. So format your main backup.

Make sure that you’ve got a variety of your crucial data stored in different places so that it’s not accessible and hackers aren’t going to be able to find it all, even if they get control over your main network.

Cybersecurity: in-house vs. outsourcing

We have our cybersecurity and network people in-house. In the last three or four years, we’ve decided that cybersecurity is so crucial that we need to have it as part of our core competencies. It’s too important.

After the attack, we had an external forensics agency come in and validate that our response and processes had been correct—we have some very big customers and they have their own cybersecurity teams and their own due diligence to do, so we had to do a lot of proving that yes, we were hacked, but it wasn’t through our negligence, and that the way that we responded to the attack was appropriate.

If you’re outsourcing cybersecurity, they must really understand the business. A cybersecurity company will do pen testing and say, “Have you got Cyber Essentials, are you ISO 27001 compliant?” But that won’t save you. They’ve got to be thinking how your business works and not just ‘yes, you’ve got the latest patches on your servers, yes, you’re following all the things’. That’s not enough.

We had all the latest patches. We follow all the best advice for security best practice, but they still got in. There are two sides to it: the cybersecurity, trying to stop people getting in; but then assuming they’ve got in despite your best efforts, and what happens next.

Backing up the build

The hackers had done a very thorough job corrupting everything. So we had to rebuild everything from scratch: reinstall operating systems, recover the data and recommission all the servers, and put everything together.

One of the things we could have done better was that while we backed up all our critical data, we hadn’t backed up a lot of the configuration files. So we spent a lot of time having to recreate from scratch all the configuration that wires everything together. We’ve now made sure that we’ve got extra bits around our core data and that it’s also backed up.

What to say to clients—and when

Once we’d discovered the attack, there was a temptation to give too much information. We kept it generic: there’s a problem. It’s a cyberattack. We’re working on it. We didn’t give timescales. We had some customers asking for a detailed blow-by-blow, but we didn’t give that because it would have been counterproductive. During the first few hours, the situation was fluid and changing: what we said at 10:00am could have been different by 11:00am. Once we had an idea, then we said more.

I think it’s important to admit what’s happened. Being cyber-attacked is not a matter of shame. It’s an attack by sophisticated criminals. We framed it as that. The following day when we were back online, we were much more forthcoming in terms of what had happened, and we shared the information. Our customers appreciated that transparency.

You didn’t fail—it’s inevitable

From the meetings I had with all the big companies and their cybersecurity teams, people were genuinely very sympathetic because they saw that as, “It’s me next, isn’t it?” So I think that there is a general acceptance that this is something that is happening, and it doesn’t reflect badly on companies. There are a lot of extremely sophisticated and highly motivated people who are constantly trying to find vulnerabilities in systems, and you just have to accept that that’s the way things are now.

My advice? Plan and back up

I would say to an MSP: assume it’s going to happen. Plan for the worst, get an incident response plan together, and make sure that you have multiple offsite backups. If you’ve got that, you’ll survive it.

Catch Paul along with a stellar line-up of panelists for Surviving Your Worst Day: Incident Responses That Worked at the Elevator Stage, Thursday October 23^rd, 2:10pm-2:40pm.