Our managed service provider's remote-management vendor appears to have been breached, and our environment was subsequently hit with ransomware. The investigation is still underway, so we are waiting for forensic findings. We have an onsite backup server and an offsite backup, and the provider says they are usable, but we have only performed limited restoration tests rather than a complete recovery exercise.
We are about a year into a broader migration, so we plan to use this incident as a downtime and disaster-recovery exercise. We want to document which systems were affected, identify gaps in our understanding of the environment, and improve how we manage and protect our data. Assuming we can eventually restore operations, what should our recovery and post-incident steps look like?
We also need to assess our relationship with the MSP and its technology vendor. What questions should we ask about the breach, the timeline, data access, credentials, backups, notification process, and their plans to prevent a recurrence?
4 Answers
Treat every credential and access path associated with the MSP or its remote-management platform as potentially compromised until proven otherwise. Rotate shared passwords, service-account credentials, VPN secrets, privileged accounts, API keys, and any credentials stored or accessible through the management tools. Remove persistent remote access, rebuild or disable affected agents, enforce MFA, and closely monitor authentication and administrative activity.
Ask specifically whether the RMM had access to privileged or Tier 0 systems, whether credentials were stored in it, and whether the attacker accessed or exfiltrated data. A system being encrypted is not the same as knowing what information was viewed or stolen. Put all requests to the MSP in writing so there is a clear record.
Contact your cyber-insurance provider immediately, if you have coverage. They can require specific forensic, legal, notification, and restoration procedures, and failing to follow them could complicate a claim. It is also worth consulting appropriate government or law-enforcement resources and retaining counsel familiar with ransomware incidents.
Do not rush into restoration before investigators have collected the evidence they need and determined how the attacker got in. Otherwise, you may restore systems only to have them compromised again. Preserve logs, images, ransom notes, indicators of compromise, and communications, and document every decision.
The waiting period can be extremely frustrating, but forensic collection really does need to happen before recovery. During a previous incident, the evidence-gathering phase was one of the hardest parts, but skipping it could have meant repeating the entire recovery.
For the MSP and its vendor, ask when they first detected the intrusion, when they confirmed the scope, and when they notified you. Request a precise timeline, affected systems and customers, indicators of compromise, evidence of data access or exfiltration, the role of the RMM, and the containment and remediation steps already taken. Also ask whether their own backup systems, credential stores, administrative servers, and internal infrastructure were affected.
Review the contract, incident-notification obligations, security commitments, liability limits, and insurance coverage with your lawyer. Their communication, transparency, and willingness to provide evidence over the next few weeks should help determine whether the relationship can continue. Use the incident to build an accurate asset inventory, map system dependencies, define a minimally functional operating state, and update your disaster-recovery plan.
Verify the backups yourself rather than relying on a vendor’s statement that they are fine. Confirm when the compromise began, which restore points predate it, whether the backup infrastructure and repositories were reachable by the attacker, and whether immutable or offline copies exist. Restore representative servers and applications in an isolated clean environment, test dependencies and data integrity, and keep recovered systems separated until they are cleared.
A backup existing is not the same as having a working business-continuity plan. Afterward, consider independent backup coverage using a 3-2-1 approach, with at least one offline or immutable copy, and perform full recovery tests instead of only checking that backup jobs report success.
Some providers use their own management tooling on highly privileged internal servers. That can create a path from the provider’s platform into the customer environment, so the scope of those administrative connections needs to be established before trusting any restored system.