Writing Alert Runbooks That Analysts Actually Use During Incidents

Alert runbooks fail when they are written as documentation rather than decision support. The distinction matters because documentation answers "what is this?" while decision support answers "what should I do right now?"

Too much background, not enough decision. A runbook that opens with three paragraphs explaining what PowerShell is, why encoded commands are suspicious, and how the detection was built is not useful to an analyst who needs to determine in the next 10 minutes whether to escalate or close the alert. Put the triage decision tree at the top, not background information.

Evidence sources listed without interpretation. "Check the process tree in the EDR" is not useful. "Check the process tree in the EDR -- if the parent process is Word.exe or Excel.exe, the user opened a malicious document; if the parent is cmd.exe spawned by a scheduled task, look for scheduled task persistence" is useful. Evidence sources without interpretation guidance require the analyst to have the same mental model as the detection engineer.

Escalation criteria that require judgment. "Escalate if suspicious" or "escalate if it looks real" puts the decision-making burden on the analyst without providing the criteria. The analyst who is unsure escalates everything to be safe (alert fatigue for the escalation path) or escalates nothing (missed incidents). Escalation criteria should be explicit: "Escalate if the parent process is a productivity application AND the encoded command contains a download from an external URL."

No false positive disposal procedure. A runbook that tells analysts how to investigate a real alert but does not tell them how to close a false positive creates inconsistency. Different analysts close false positives differently, and without a documented disposal procedure, the SIEM accumulates unclosed alerts.

Every alert runbook should contain seven fields in this order, optimized for reading under time pressure.

1. Alert name and trigger condition. One sentence: what the detection rule fires on and what SIEM/EDR/platform generates it. Analysts who work across multiple tools need to know where to start. Example: "This alert fires in Microsoft Sentinel when Windows Event ID 4104 logs a PowerShell script block with base64-encoded content exceeding 500 characters."

2. What benign looks like. Before anything else, tell the analyst what a false positive looks like for this specific alert. Example: "Common false positives include Microsoft Intune management scripts, ConfigMgr deployment commands, and scheduled tasks from endpoint management tools. Check the parent process first -- if it is MEM.exe or ccm.exe, this is likely a false positive."

3. What malicious looks like. One to three specific indicators that distinguish a real finding. Example: "Malicious indicators include encoded commands downloading from external URLs (confirm by decoding the base64 payload), network connections from PowerShell.exe to external IPs after execution, and child processes (cmd.exe, mshta.exe, wscript.exe) spawned by the PowerShell process."

4. Required evidence to collect. List exactly what to collect and from where, in the order the analyst should collect it. "1. Process tree from EDR for the powershell.exe process. 2. Network connections from the same host in the 10 minutes surrounding the event. 3. Decoded base64 payload (use CyberChef From Base64 recipe). 4. User authentication activity from Entra ID for the logged-in user in the past 24 hours."

5. Triage decision tree. A branching yes/no structure with explicit criteria at each branch point. Written as conditions, not judgments. See the sample runbook section for the format.

6. Escalation criteria and procedure. The specific conditions that require escalation, who to escalate to, and what information to include in the escalation. "Escalate to Tier 2 if: (a) the decoded payload contains an external download URL, OR (b) the PowerShell process spawned network connections within 60 seconds of execution. Include: host name, user name, decoded payload, and process tree screenshot."

7. False positive disposal procedure. Exactly how to close a confirmed false positive. Which fields to fill in the ticket, what comment to add, and whether to add any exception to prevent the same alert from firing again for the same source. Consistent disposal prevents alert backlog and enables accurate FP rate tracking.

This sample demonstrates the seven-field format applied to a common detection.

Alert: PowerShell Encoded Command Execution Platform: Microsoft Sentinel | Rule: PSEncodedCommand-001

What benign looks like:

• Parent process is MEM.exe, ccm.exe, or a known IT management tool
• Endpoint management scripts from Intune (check for OMA-DM or Intune strings in user data)
• Scheduled maintenance tasks (verify in Task Scheduler on the host)

What malicious looks like:

• Decoded payload downloads from an external URL (IEX, Invoke-Expression, DownloadString, WebClient)
• Child processes (cmd.exe, mshta.exe, wscript.exe, csc.exe) spawned after execution
• Network connections from powershell.exe to external IPs within 60 seconds

Required evidence (collect in order):

1. From EDR: Process tree for the powershell.exe PID
2. From EDR: Network connections for the same host, 5 minutes before and after the event
3. Decode the payload: copy the base64 string from the alert, go to CyberChef, use "From Base64" recipe
4. From Entra ID: Sign-in logs for the user name in the alert, past 24 hours

Triage decision tree:

Is the parent process a known IT management tool (MEM.exe, ccm.exe)?

• YES: Check if the decoded payload matches known management scripts. If yes, close as FP. If no, continue.
• NO: Continue.

Does the decoded payload contain IEX, Invoke-Expression, DownloadString, or an external URL?

• YES: ESCALATE immediately. Do not wait for network connection confirmation.
• NO: Continue.

Did powershell.exe spawn any child processes (cmd.exe, mshta.exe, wscript.exe, csc.exe)?

• YES: ESCALATE.
• NO: Continue.

Did powershell.exe make network connections to external IPs?

• YES: ESCALATE.
• NO: MONITOR. Add to watchlist for 24 hours. If no additional activity, close as low confidence.

Escalation procedure: Escalate to Tier 2 on-call via Slack #soc-escalation. Include: host name, user name, decoded payload text, and process tree screenshot. Ticket must include all four items before paging.

False positive disposal: If confirmed FP: Set ticket status to Resolved, Resolution = False Positive, add comment with the specific reason (e.g., "Intune management script -- parent process MEM.exe confirmed"). No exception needed unless this fires more than twice per week for the same host.

The triage decision tree is the most important part of the runbook because it is what the analyst uses under time pressure. The format must be unambiguous -- no "maybe" branches, no judgment calls without explicit criteria.

Use yes/no branches only. Every branch point should be answerable with yes or no based on observable data. "Is the parent process cmd.exe?" is answerable. "Does this look like lateral movement?" is not.

Lead with the most discriminating check. Order branches so the most reliable false positive indicator comes first. This lets analysts quickly dispose of the majority of benign alerts before doing deeper investigation for the minority that look real.

Provide the specific lookup for each branch. Every branch point should tell the analyst exactly where to find the answer: which tool, which field, which time range. "Check the parent process" is not actionable. "In the EDR alert detail, look at the Parent Process Name field" is actionable.

Terminate every branch. Every path through the decision tree must end with a clear action: escalate, monitor, close. Branches that end with "investigate further" or "use your judgment" are incomplete.

Timebox each step. At the top of the decision tree, state the total time budget for triage. "Complete triage in under 15 minutes. If any evidence step takes longer than 5 minutes, escalate and continue investigation." This prevents analysts from spending 45 minutes on a single alert while the queue grows.

The only reliable test for a runbook is giving it to someone who has never seen it before and timing how long it takes them to reach a triage decision.

Cold-read test procedure. Select an analyst who did not write the runbook. Give them the runbook and a sample alert (use a historical alert from your SIEM or create a synthetic one). Measure: (1) How long until they start investigating (time to understand the runbook). (2) How long to reach a triage decision. (3) Whether they reached the correct decision based on the sample alert. (4) How many questions they asked. A runbook that produces correct decisions in under 15 minutes without questions passes the cold-read test.

Friction points reveal runbook failures. When the test analyst asks a question, the answer to that question should become part of the runbook. When they spend extra time looking something up, that lookup process should be documented. Questions and hesitations are debugging data for the runbook, not failures by the analyst.

Test with real false positives, not just true positives. Most alerts are false positives. A runbook that guides analysts to correct escalation decisions on true positives but sends them in circles on false positives is not complete. Test the runbook against confirmed false positive examples and verify that the disposal procedure is reached cleanly.

Measure runbook effectiveness with ongoing metrics. After deployment, track per-alert: mean time to triage (from alert creation to analyst decision), escalation rate (what percentage of alerts are escalated vs. closed), false positive escalation rate (what percentage of escalated alerts turn out to be FP), and analyst disagreement rate (do different analysts reach different conclusions from the same alert). Significant disagreement between analysts on the same alert type indicates the runbook's criteria are ambiguous.

Runbooks are not static documents. They require maintenance as detections change, new false positive patterns emerge, and the environment evolves.

Trigger: false positive rate above 40%. When more than 40% of an alert type is false positive, the runbook's disposal procedure needs to be faster and the detection may need tuning. Review the false positive patterns and update the "What benign looks like" section with the specific new FP patterns. If the same FP pattern is generating alerts repeatedly, add an exclusion to the detection rule rather than requiring analysts to repeatedly execute the same disposal procedure.

Trigger: false negative confirmed. When an incident postmortem reveals that an alert fired but was incorrectly closed as a false positive, the runbook's decision tree failed. Review the triage path the analyst took, identify which branch point led to the wrong conclusion, and add discriminating criteria or additional evidence steps at that branch point.

Trigger: escalation path changes. When the team structure, on-call rotation, or tooling changes, the escalation section becomes stale. Assign one team member ownership of each runbook and require them to verify escalation path accuracy quarterly.

Trigger: detection rule changes. When the detection logic is modified (new threshold, new exclusions, new data sources), the runbook must be updated to reflect the new trigger condition and any changed false positive patterns.

Quarterly audit. Even without a specific trigger, review all runbooks quarterly: confirm that the evidence sources listed still exist and are accessible, verify that the escalation contacts are current, and check whether new false positive patterns should be documented based on recent tickets.