Recipe: Incident response and on-call paging
1min read
TL;DR: When a critical alert fires in Sentry or Datadog, this playbook opens a PagerDuty incident, creates a Slack war-room channel, posts the alert context and runbook links, and pages the on-call engineer. Trigger is a monitoring webhook; outputs are a PagerDuty incident and a Slack channel.
Who this is for
Engineering and reliability teams that need fast, consistent incident kickoff.
What you'll need
- Connectors: Sentry or Datadog, PagerDuty, Slack.
- Secrets: Monitoring API key, PagerDuty routing key, Slack token.
- Trigger: Webhook from Sentry/Datadog on a critical alert.
How to build it with Ask Ballet
- Open the Console on Home.
- Paste the prompt below; set severity rules and runbook links.
- Review the plan and confirm.
- Connect the monitoring tool, PagerDuty, and Slack, then add the alert webhook.
Build a playbook that triggers on a Datadog alert webhook with severity critical.
Create a PagerDuty incident on the platform on-call service with the alert title and
link. Create a Slack channel named incident-<date>-<short-id>, post the alert summary,
affected service, dashboard link, and our runbook, and invite the on-call engineer and
#eng-leads. Update the PagerDuty incident with the Slack channel link.
What the playbook does
- Webhook trigger — receives the critical alert.
- Code step (parse) — extracts service, severity, and links.
- HTTP step (PagerDuty) — opens the incident and pages on-call.
- HTTP step (Slack) — creates the war-room channel and posts context.
- HTTP step (link back) — adds the Slack channel to the PagerDuty incident.
Variations and tips
- Gate channel creation on severity so only criticals create war-rooms.
- For customer-facing impact, notify support via Escalations and SLA breaches.
- Add a post-incident step that drafts a timeline from the Slack channel.