case_study:
id: case-04
title: Alerting that respects on-call
scope: observability
engagement: audit + rebuild
status: draft
problem: |
Alerts nobody trusted, a channel nobody read, and an on-call rota
that existed mainly to absorb noise at night.
what_i_did:
- step: 01
title: Signal inventory
description: |
Every alert checked against one question: does this indicate a
real failure? The rest were removed or turned into dashboards.
- step: 02
title: Runbook per alert
description: |
Each surviving alert got a runbook with the first move, so the
response does not depend on who happens to be awake.
result:
summary: |
Fewer alerts, each with an owner and a known first response. On-call
is a rota with a plan, not a lottery.
stack:
signals: [OpenTelemetry, Prometheus, Loki, Tempo]
alerting: [SLO and burn-rate based, routed to owners]
response: [Runbooks, first move, reviewed after incidents]