Writing a Runbook Your On-Call Actually Opens

A runbook is meant to be the thing you open when a system is misbehaving and you need to act, possibly half asleep, possibly not the person who built it. Most fail that test. They are long, general, and stale, so the responder ignores them and pages someone instead.

One page per alert

Structure the runbook around the alerts that actually fire. Each alert links to a page for that specific alert. The responder does not read a manual, they land on “high error rate on the payments service” and see what to do about that. If your alerts do not link to runbook pages, fixing that is the highest-value change you can make.

Each page answers four things

What does this alert mean, in one line. What is the likely impact on users right now. What should I check first, as concrete steps: this dashboard, this log query, this command. And how do I mitigate: restart this, scale that, flip this feature flag, and who to escalate to if none of it works. Keep it to what a responder needs in the first fifteen minutes. Background reading goes elsewhere.

Concrete beats complete

“Check the database” is useless. “Run this query to see current connections; if it is near the limit, the fix is to restart the worker pool with this command” is a runbook. Paste the actual commands and links. The responder should be able to copy and act, not interpret.

Keep it honest

The fastest way to a stale runbook is to make updating it a separate chore. Make it part of closing an incident: whoever resolved it updates the relevant page while the detail is fresh, in the same sitting. If an alert fired that had no page, they write one. Over a few months this fills in the gaps that matter, because the gaps that matter are the ones that caused incidents.

Make it findable

The runbook lives where people already work: linked from the alert, linked from the service’s page, searchable. A brilliant runbook in a wiki nobody opens is not a runbook.

Test it occasionally

Run a game day. Trigger a known failure in a safe environment and have someone who did not write the system follow the runbook to resolve it. Every gap they hit is a fix. It is far cheaper to find those on a Tuesday afternoon than during a real incident.

The point

A runbook earns its keep only if a tired person under pressure can open it and know what to do. Short, specific, per-alert, and kept current by the people who use it.

Common questions

One big runbook or many small ones?

Many small ones, each tied to a specific alert. When an alert fires, the responder should land on the page for that alert, not scroll a document.

Who keeps it current?

The person who resolves an incident updates the runbook as part of closing it out, while the detail is fresh. If it is a separate later task it will not happen.

What if there is no runbook for the alert that fired?

Write one during or right after the incident. The first responder to a new problem is the best-placed person to document it.