PagerDuty Security Incidents Failing to Trigger: Escalation Policy Gaps and Webhook Auto-Disable
Reach 50,000+ security professionals
Sponsor a newsletter slot and put your brand in front of CISOs, engineers, and IT decision-makers.
PagerDuty's Events API accepts alerts and returns HTTP 202 (Accepted) even when the alert will not create an incident. A 202 response from PagerDuty only confirms that the event was received; it does not confirm that an incident was triggered or that anyone was paged. Integrations that only check for HTTP 202 on delivery have no visibility into what happened to the alert after it was accepted. For detection pipeline health across SIEM and downstream alerting, see this detection posture management guide.
Root Causes: Five Reasons an Alert Does Not Become an Incident
Cause 1: Service is in Suppression mode. PagerDuty services have an alert behavior setting with three states: Create Incidents (normal), Create Alerts Only (alerts without incidents), and Suppress Alerts. A service in Suppress state accepts all events and logs them but creates no incidents and pages no one. This state is often set during maintenance and forgotten. It is the most common cause of genuine security alerts silently disappearing.
Cause 2: Event rules dropped or re-routed the alert. PagerDuty's Global Event Rules (at the account level) and Service-level Event Rules can match patterns in incoming events and apply suppress actions, route-to-different-service actions, or change the severity level. A broad regex that was intended to suppress noise may match the payload of a real security alert and suppress it before it reaches the service.
Cause 3: Deduplication merged the alert into an existing open incident. PagerDuty deduplicates alerts using the dedup_key field. If an alert arrives with the same dedup_key as an existing open incident, it is merged into that incident rather than creating a new one. If the existing incident is acknowledged or resolved at the wrong moment, or if the dedup_key is shared across multiple distinct alert types, genuine new incidents can be silently merged.
Cause 4: Escalation policy has a gap or is unassigned. A service with no escalation policy, or an escalation policy whose on-call schedule has no one scheduled during the alert time, produces an incident with no one to notify. The incident is created, but no page fires. This is distinct from an incident failing to trigger; the incident exists but notification fails.
Cause 5: Webhook auto-disabled at the source. If PagerDuty is receiving alerts via a webhook subscription from a source system (Splunk, Sentinel, PagerDuty extensions), PagerDuty auto-disables webhook subscriptions after repeated delivery failures. The subscription stays in a disabled state until manually re-enabled, and the source system receives no indication that its deliveries are being dropped.
Diagnostics: Finding the Specific Cause
Step 1: Check the service's alert behavior setting.
In PagerDuty, navigate to Services > Service Directory > your service > Settings. Under 'Create Incidents', verify the setting is 'Create Incidents', not 'Create Alerts Only' or 'Suppress Alerts'. A service in Suppress mode has a banner visible in the service settings page.
Step 2: Check the PagerDuty Event Log.
For an event that should have triggered an incident, go to Services > Event Rules and check the 'Event Log' or use the PagerDuty API to retrieve recent events for the service:
curl -H 'Authorization: Token token=<api-key>' \
'https://api.pagerduty.com/services/<service-id>/integrations'
Then check the event log for suppressed events:
curl -H 'Authorization: Token token=<api-key>' \
'https://api.pagerduty.com/incidents?service_ids[]=<service-id>&statuses[]=suppressed'
Step 3: Check Global Event Rules for suppress actions.
In PagerDuty, navigate to Events > Event Rules. Review each rule that could match the failing alert's payload. Look for rules with a 'Suppress' action and test your alert payload against each rule's conditions. PagerDuty's Event Rules page includes a test tool where you can paste an event payload and see which rules match.
Step 4: Check escalation policy on-call coverage.
Navigate to Services > Escalation Policies and review the escalation policy assigned to the service. Check each schedule referenced at each escalation level to verify there is an on-call person scheduled for the time the alert failed to page. Gaps in on-call schedules (overlaps in timezone handoff, holidays, out-of-office without coverage configured) are a common silent failure mode.
Step 5: Check webhook subscription status at the source.
For alerts routed through webhooks, PagerDuty marks subscriptions as disabled after delivery failures. In PagerDuty, navigate to Integrations > Webhooks and check for any subscriptions showing a Disabled or Failed status. For third-party sources (Splunk, Sentinel), check the source system's integration logs for recent delivery failures to PagerDuty's Events API endpoint.
Briefings like this, every morning before 9am.
Threat intel, active CVEs, and campaign alerts, distilled for practitioners. 50,000+ subscribers. No noise.
Fixes for Each Root Cause
Fix for Suppressed service: change alert behavior.
In the service settings page, change 'Create Alerts' behavior back to 'Create Incidents'. Review who changed this setting and when -- PagerDuty's audit log (Admin > Audit Trail) records configuration changes with timestamps and user attribution.
Fix for Event Rules suppressing real alerts: narrow the suppress condition.
Identify the matching rule and tighten its condition to exclude the payload patterns from real security alerts. Add an explicit allowlist condition before the suppress rule: event rules are evaluated in order, so an 'Allow' rule that matches high-severity security alerts and routes them to the right service takes precedence over a downstream suppress rule.
Fix for deduplication merging distinct alerts: use unique dedup_keys.
Review the integration sending the failing alerts. If it uses a static or near-static dedup_key (such as the alert name or rule ID without any time component), multiple distinct incidents are merged into one. Add a time component or a unique alert identifier to the dedup_key to ensure each alert creates or updates its own distinct incident.
Fix for escalation policy gap: audit and fill on-call coverage.
For each schedule in the escalation policy, check the schedule's calendar view for the period when the alert failed to page. Add override schedules for gaps. Configure a final escalation level that always notifies a manager or a team email rather than a schedule that could be empty.
Fix for auto-disabled webhook: re-enable and fix delivery.
In PagerDuty Integrations > Webhooks, re-enable the disabled subscription. Then investigate why delivery was failing: the most common causes are network connectivity from the source to PagerDuty's Events API, authentication token expiry, or payload format errors. Enable webhook delivery logging at the source to capture the specific error returned by PagerDuty on failed deliveries.
Adding Verification So Missed Pages Are Detected
Use PagerDuty's Dead Man's Switch pattern.
Configure a test alert that fires every 15 minutes from your detection platform and creates a PagerDuty incident on a low-urgency service with a very short escalation timeout. The test incident should auto-resolve after 5 minutes. If the test incident stops appearing in PagerDuty, the integration is broken. This pattern detects broken webhooks and stopped services without waiting for a real security incident to reveal the gap.
Set up a CloudWatch or Prometheus metric on PagerDuty Events API responses.
If your SIEM or SOAR sends alerts to PagerDuty via the Events API, instrument the sender to track the response code and log it. HTTP 202 means accepted; anything else means the event was rejected. Alert on non-202 responses from PagerDuty's Events API as a first-class reliability signal.
Audit the suppressed alerts log regularly.
PagerDuty logs suppressed alerts even when no incident is created. Build a weekly audit of the suppressed events log and review whether any suppressed events match the payload patterns of your known-high-severity alert types. A suppressed critical security alert appearing in this log before a genuine incident would have revealed the misconfiguration.
Known Edge Cases
Low urgency alert hitting a high-urgency schedule. PagerDuty's urgency settings determine which notification rules apply. An alert classified as low urgency only triggers low-urgency notification rules (typically no phone call, no SMS). If the on-call engineer's low-urgency notification rules are configured to only notify via mobile push, and they have do-not-disturb enabled, no alert fires. For security incidents, explicitly set the alert severity to critical in the event payload to trigger high-urgency notification rules.
Organization multi-account PagerDuty. Large orgs sometimes run multiple PagerDuty accounts. An alert routed to the wrong account (a staging PagerDuty account rather than production) fails silently from the perspective of the incident response team. Confirm the Events API key in your integration points to the correct account.
Rate limiting. PagerDuty's Events API enforces rate limits. During a mass alert storm, PagerDuty may respond with HTTP 429 (Too Many Requests). Integrations that do not handle 429 responses may drop alerts rather than queuing and retrying them. For security SIEMs, implement exponential backoff on 429 responses from PagerDuty's Events API.
When to Escalate
Contact PagerDuty Support when:
- The service is not in suppression, Event Rules show no matching suppression, the escalation policy has on-call coverage, and incidents are still not being created for events you can confirm were delivered with HTTP 202.
- PagerDuty's status page shows an active incident affecting event processing.
- The PagerDuty Event Log shows the event was received but marks it as 'trigger_unprocessed' or a similar internal error state.
For a support case, provide: the service ID, the event payload that failed to trigger, the timestamp, the HTTP 202 response from the Events API, and the Event Log entry for the same event ID.
The bottom line
PagerDuty incidents that fail to trigger are almost always caused by a configuration state rather than a system failure: a service in suppression mode, an Event Rule with a suppress action that matches real security alerts, dedup_key collision, an escalation policy with schedule gaps, or a webhook that PagerDuty auto-disabled after delivery failures. Each is diagnosable from the PagerDuty UI and API in under 10 minutes. Add a dead-man's-switch test alert that fires every 15 minutes to detect the break before a real incident does. For SIEM detection quality that feeds the PagerDuty integration, see this Microsoft Sentinel vs IBM QRadar SIEM comparison.
Frequently asked questions
Why is PagerDuty not creating an incident from my security alert?
PagerDuty accepts an event (returns HTTP 202) but may not create an incident for several reasons: the service is in Suppress Alerts mode, an Event Rule is matching the alert and applying a suppress action before it reaches the service, the alert is being deduplicated into an existing open incident using a matching dedup_key, or the escalation policy has no one scheduled on-call at the time the alert fires. Each cause produces the same visible outcome (no incident) but requires a different fix, so check the service alert behavior setting first, then Event Rules, then the escalation policy on-call coverage.
What is PagerDuty service suppression and how do I check it?
PagerDuty service suppression is a service-level setting called 'Suppress Alerts' under the alert behavior configuration. When enabled, PagerDuty accepts all events for the service and logs them but does not create incidents or trigger any escalation policy. No one is paged. To check: go to Services > Service Directory, select the service, go to Settings, and look at the Create Incidents section. If it shows 'Suppress Alerts', the service is silently absorbing all alerts. Change it to 'Create Incidents' to restore normal behavior.
What causes PagerDuty webhooks to be automatically disabled?
PagerDuty auto-disables webhook subscriptions after repeated consecutive delivery failures, where failures are defined as the target endpoint returning an error response (non-2xx status code) or timing out. This protects PagerDuty's delivery infrastructure from sending to unreachable endpoints. The subscription stays disabled until manually re-enabled by an account admin. The source system (the SIEM, SOAR, or monitoring tool sending to PagerDuty via webhook) typically does not receive notification that its subscription was disabled -- it just stops getting alerts.
How do I make sure PagerDuty alerts are always critical urgency for security incidents?
Set the severity field in your PagerDuty Events API payload to 'critical' for security incidents that must trigger high-urgency notification rules (phone call and SMS). Payloads without a severity field default to 'error' in PagerDuty, which may map to low urgency depending on your service configuration. In your SIEM or SOAR integration, map your critical security alert severity levels to 'critical' in the PagerDuty event payload. Also verify the service's urgency settings are configured to treat 'critical' severity as high urgency -- this mapping is set per service in the service settings page.
How do I detect a broken PagerDuty integration before a real incident exposes it?
Implement a dead-man's-switch test: configure a scheduled alert from your detection platform that fires every 15 minutes, creates a PagerDuty incident on a dedicated low-urgency test service, and auto-resolves after 5 minutes. If the test incidents stop appearing in PagerDuty, the integration has broken. Set a PagerDuty alert on the test service that pages on-call if no test incident has been created in the last 30 minutes. This pattern catches broken webhooks, stopped services, and auto-disabled subscriptions before a genuine security incident reveals the gap.
What is the difference between a PagerDuty alert and a PagerDuty incident?
In PagerDuty, an alert is an event received from a monitoring tool and stored in the system. An incident is the actionable record that triggers an escalation policy and pages the on-call engineer. A service configured in 'Create Alerts Only' mode generates alerts but not incidents -- the alert appears in the alerts list but no one is paged. A service in 'Create Incidents' mode converts each alert into an incident (unless deduplication merges it into an existing one). Security integrations should almost always point to services configured in 'Create Incidents' mode to ensure every alert results in a page.
Sources & references
Free resources
Critical CVE Reference Card 2025–2026
25 actively exploited vulnerabilities with CVSS scores, exploit status, and patch availability. Print it, pin it, share it with your SOC team.
Ransomware Incident Response Playbook
Step-by-step 24-hour IR checklist covering detection, containment, eradication, and recovery. Built for SOC teams, IR leads, and CISOs.
Daily briefings + Google Top Stories in one step.
Enter your email and we'll subscribe you to the daily briefing and add Decryption Digest to your Google Search feed at the same time.
Founder & Cybersecurity Evangelist, Decryption Digest
Cybersecurity professional with expertise in threat intelligence, vulnerability research, and enterprise security. Covers zero-days, ransomware, and nation-state operations for 50,000+ security professionals every morning.