The 2 AM Post-Mortem: What One Team Learned When They Stopped Rotating On-Call

A reader we'll call Priya wrote to us in late spring with a spreadsheet and a grudge. She runs platform engineering for a 40-person logistics software company, and for eighteen months her team had been carrying their own pager. The numbers told a familiar story: alert volume climbing, MTTR drifting upward, and two senior engineers quietly updating their resumes. What made her email interesting wasn't the burnout — we've all read that story — but the specific accounting she'd done before and after handing the pager to an outside partner. We asked if we could follow the project and write it up. She said yes, on the condition that names and vendors stay vague. Fair enough.

The setup: a stack that had outgrown its owners

The company runs a fairly standard modern stack — Kubernetes on managed nodes, Terraform for everything provisioned, Prometheus and Grafana for observability, PagerDuty for routing. Nothing exotic. The problem was volume. By February, the on-call engineer was averaging 11 pages per week, roughly a third of them firing between 1 AM and 4 AM. Priya's own post-mortem notes flagged something uncomfortable: of the last 60 incidents, only 14 required a code change. The rest were configuration drift, certificate renewals, noisy thresholds, and one recurring disk-pressure alert that everyone had learned to snooze.

That ratio — mostly operational noise, occasionally real fire — is the classic signal that a team needs either a bigger platform group or a different arrangement. Priya had budget for neither a fourth engineer nor a full SRE hire, so she started evaluating embedded partners in March.

The decision point: embedded, not outsourced

She looked at three options. The first was a traditional managed-service contract, which quoted a monthly rate but routed incidents through a ticket queue with a 30-minute first response. The second was a fractional SRE arrangement — one person, part-time, no 24/7 coverage. The third was SteelRats, which pitched something narrower: senior-only platform engineers who join your on-call rotation rather than sitting behind a portal.

The distinction mattered more than Priya expected. A queue-based provider can acknowledge an alert, but it can't safely roll back a bad Terraform apply at 2 AM without understanding the blast radius. Embedded engineers, in theory, can. She signed a three-month pilot in April with two explicit success criteria: cut MTTR by at least half, and reduce overnight pages to fewer than three per week.

The first six weeks: ugly, then better

Week one was mostly archaeology. The incoming engineers spent four days reading runbooks, tracing alert routes, and — this is the part Priya didn't expect — deleting things. Twenty-two alerts were retired outright because nobody could explain what action they were supposed to trigger. Nine more were rewritten with tighter thresholds. One PagerDuty escalation policy had been misconfigured since 2022, silently paging a distribution list that no longer existed.

Week three brought the first real test: a node pool failure in the Frankfurt cluster during a Tuesday night batch job. The on-call engineer — now a SteelRats platform engineer paired with one of Priya's staff — diagnosed a Terraform state lock collision, drained the affected nodes, and restored service in 19 minutes. The previous comparable incident had taken 94 minutes. Priya's log of that night is two sentences long, which she considers the highest praise available.

Weeks four through six were quieter but more consequential. The team rebuilt the alerting hierarchy around symptom-based signals instead of cause-based ones, migrated two legacy cron jobs into Kubernetes CronJobs with proper retry policies, and wrote runbooks for the five incident classes that accounted for 80% of historical pages.

The measured results, twelve weeks in

At the end of the pilot, Priya pulled the numbers:

• MTTR across all incidents: down 71%, from 78 minutes to 22 minutes.
• Overnight pages (1 AM–4 AM): down from roughly 11 per month to 2.
• Alerts retired or rewritten: 31 of 47.
• Engineer-reported on-call stress, on an internal 1–5 scale: from 4.1 to 2.3.

The 71% figure is the one that got circulated internally, but Priya argues the alert count matters more. Fewer pages means fewer interruptions, which means the remaining incidents get better attention. That's the compounding effect that a spreadsheet won't show you until month four.

What we took from it

Three things stood out to us as editors. First, the pilot succeeded because it had numeric exit criteria — vague satisfaction surveys would have let a mediocre engagement limp along for a year. Second, the early work was deletion, not addition; the fastest MTTR improvement came from silencing alerts that should never have existed. Third, embedding changed the economics. A remote queue can triage, but it can't safely touch infrastructure, and infrastructure is where most 2 AM incidents actually live.

Priya's team now runs a hybrid rotation: two internal engineers and one embedded platform engineer per week, with SteelRats handling the deep overnight window and the Terraform and Kubernetes work that the internal team doesn't have bandwidth to own. Her two flight-risk engineers stayed. The spreadsheet, she says, is finally boring — which is exactly the point.

For teams weighing a similar move, the useful exercise isn't comparing vendor feature grids. It's counting your own pages, classifying them by whether they needed a code change, and asking honestly whether your current rotation can absorb the difference. If the answer is no, the arrangement described here is worth a closer look — the embedded on-call model and how the rotation is structured lays out the mechanics without much sales language.