MSP Incident Response Guide

Business IT/MSP Talk
1

MSP Incident Response Guide

Disclaimer:
The information in this guide is for general educational purposes only and is not legal advice, insurance advice, or a substitute for professional counsel.
Every incident is unique, and applicable laws, regulations, and contractual obligations can vary.
Before taking any action during a security incident, consult with your attorney, cyber insurance provider, and qualified incident response professionals.
And remember no good legal defense starts with “This person on YouTube said…”

First rule: Don’t panic.
A incident is a high-stress event, but panic wastes energy and clouds judgment. Getting through it requires staying calm, rational, and methodical.

:zero: Start Here: Stay Calm & Document Everything

  • Pause before reacting
    • Don’t send client updates or making hasty changes.
    • Step away for a few minutes if needed, planning requires a clear head.
    • Remember: most incidentes take days to resolve, not hours. Pace yourself.
  • Journal every action
    • Start a incident log immediately. (pen and paper are likely a good idea)
    • Include time, date, and description for each action taken, observation made, or decision reached.
    • This log is critical for:
      • Legal defense.
      • Insurance claims.
      • Forensics and post-mortem reviews.
  • Rest & get help
    • Eat, hydrate, and sleep. Decision-making suffers when exhausted.
    • Have a friend or family member not involved in the incident bring food or simply talk to you. The stress relief is valuable just being able to talk through problems

:one: Assess the Situation

  • Try Identify how the incident occurred
    • Is it still ongoing?
    • Has the attacker maintained persistence in your systems?
    • If you don’t have definitive evidence, do not assume. Clearly note any unknowns so you don’t waste time chasing false leads or mistakenly believe the threat actor has been removed when they haven’t.
  • Work from a clean system
    • If your workstation, RMM, or management tools are compromised, do not roll passwords or touch client systems from one using the tools
    • Use an uncompromised clean machine for all remediation work, if every machine you have is using your tool stack that was used in the incident you may have load a clean system
  • Immediate containment steps to be done from a clean system
    • Lock accounts.
    • Roll passwords.
    • Disable compromised integrations, APIs, and service accounts.
    • Isolate compromised systems from the network (pull network cable, disable NIC, VLAN quarantine).
    • DO NOT DELETE OR DESTROY ANY EVIDENCE! You will feel compelled to start fixing things but for legal and other reasons evidence must be preserved.

:two: Get Key People in the Loop

  • Call your attorney and cyber insurance inform them that an incident is occurring. They can:
    • Advise on compliance and legal obligations.
    • Help coordinate Incident Response (IR) without making costly missteps.
  • Use out-of-band communications for coordination
    • Assume internal systems may be compromised.
    • Use secure, external tools (e.g., Signal Messenger) for incident coordination until your environment is verified clean.
    • Avoid discussing sensitive details over email, corporate chat, or systems that might be monitored.
  • Notify internal leadership & decision makers at your MSP
    • Owners, partners, and executives should be briefed so they can approve actions, budgets, and client communications.
  • Engage your Incident Response team or trusted security partner
    • If you already have a pre-arranged IR vendor, get them engaged early so evidence isn’t lost and mistakes aren’t made.
  • Contact compliance officers or regulatory points of contact at your MSP
    • If you handle data under HIPAA, PCI, ITAR, CMMC, etc., ensure the correct compliance liaison is aware they may have specific reporting timelines.
  • Notify key vendors if they may be the vector of the attack or at risk
    • RMM providers, cloud service providers, or critical SaaS platforms may need to be looped in for forensic or containment assistance.
  • :warning: Who Not to Call Yet
    • Clients (until the message is finalized)
      • Avoid ad-hoc or panic updates. Wait until you have a prepared, consistent statement.
    • The media
      • Any public comment can be misquoted or misinterpreted.

:three: Control the Message

  • Have a prepared statement for clients cleared by your legal counsel
    For example: “We are aware of an issue affecting some systems and are actively working to resolve it. We will provide updates as more information becomes available.”

  • Consistent communication:

    • ALL staff interacting with clients must use the same wording.
    • Update your phone system’s greeting/message with this statement if needed.
    • Avoid speculation and stick to facts.
    • Keep written records of what was communicated, when, and to whom.
    • Designate a single point of contact for client-facing updates to reduce the risk of mixed messages.
    • Avoid over-promising timelines be honest about ongoing investigation and uncertainty.

:four: Gather Critical Information for the IR Team

  • Systems Impact List:
    • MSP-owned systems impacted.
    • Client systems impacted.
    • When each system was last known good (helps prioritize recovery).
  • Legal & Compliance Docs for your clients impacted:
    • Master Service Agreement (MSA).
    • Business Associate Agreements (BAA) for HIPAA clients.
    • Any agreements for government, ITAR, CMMC, or other special compliance.
  • Per Client Damage Assessment:
    • Number of servers/desktops encrypted.
    • Number of servers/desktops not encrypted but displaying a threat actor message.
    • Systems showing signs of compromise without encryption (e.g., backdoors, credential theft).
  • Recovery Resources:
    • Are the data backups available?
    • Are re-image files or gold images available?
    • Will rebuilds of systems be from scratch?
    • Availability of offline or immutable backups, and their last verification date.
  • Evidence Preservation
    • Do not take action without permission from your IR team incorrect handling can destroy critical forensic evidence.
    • Save copies of ransom notes, threat actor communications, and any indicators of compromise (IOCs).
  • Network & Credential Data: have this ready for the teams helping with the response
    • Admin account lists (with roles and permissions).
    • Remote access logs (VPN, RDP, RMM).
    • MFA configuration lists.
  • Communication Records:
    • Any client notifications already sent.
    • Internal incident logs or incident journal entries.

:five: Prioritize Clients for Recovery

  • During normal operations, MSPs can service many clients at once but a incident is not normal operations. You will need to work in a strict priority order to make the best use of limited resources.

    Common factors to consider when ranking clients:

    • Type of company – Critical infrastructure, healthcare, financial, or regulated industries may have legal reporting deadlines or public safety concerns.
    • Size of company – Larger organizations may have more complex recovery needs or wider impact.
    • Legal exposure – Likelihood the client will take legal action against you.
    • Business retention risk – Likelihood the client will terminate services.

  • Once ranked, create a clear, documented order of recovery. Recovery can’t start until their a plan.

  • For each client, determine which systems are truly essential for their operations such as domain controllers, line-of-business applications, ERP systems, email servers, or critical file shares.

  • From that, build a per-client restore order so you bring back the most business-critical systems first, rather than spending time on non-essential resources while core operations remain down.

:six: If You Don’t Have Cyber Insurance

You have two potential options: Self Fund or let clients use their insurance

  • Self-fund an IR team

    • Costs can easily reach hundreds of thousands of dollars. Incident response is very specialized work combined with staffing an on demand high skilled labor force so costs will be very high.
    • You’ll need to have funding available immediately to avoid delays that can make the incident worse.

  • If you Inform clients you cannot assist: They must activate their own cyber insurance.

    • This carries a high risk of subrogation which means the insurer may seek reimbursement from your company as the at-fault party.
    • If possible, still provide clients with any documentation and evidence they’ll need to engage their IR team quickly.

  • Additional considerations:

    • Unless you are very specifically trained and experienced at this DO NOT attempt to handle complex forensics or negotiations on your own. Mistakes can and will increase liability. Even if you are trained on this type of work, being that you dealing with the incident in a high stress situation this can lead to it not being done very well.
    • Another reminder to consult with your attorney before sending any formal notification to clients.

:seven: After the incident

  • Review your incident log and compile a complete timeline of events include detection, containment, recovery, and communications.

  • Conduct a full incident post-mortem with your team and IR vendor:

    • Identify the root cause(s) and how the incident was detected.
    • Document what worked, what failed, and where delays occurred.
    • Record lessons learned and assign owners for remediation actions.

  • Implement security improvements:

    • Strengthen controls to prevent similar incidents.
    • Update monitoring, alerting, and incident response playbooks.
    • Verify and test backup and recovery procedures.

  • Prepare for potential litigation:

    • Work with your attorney to preserve all incident-related evidence and communications.
    • Retain copies of client contracts, SLAs, and compliance agreements.
    • Avoid casual discussions about fault assume everything could be discoverable in court.

  • Repair client relationships:

    • Communicate transparently about the resolution and improvements made.
    • Offer post-incident reviews for impacted clients.
    • Where appropriate, provide service credits, security training, or added monitoring as goodwill.
1 Like