Most were noise rather than serious outages, but at two in the morning, being overly cautious seemed preferable to letting a genuine problem slip through the cracks. We've since reduced both unnecessary overnight interruptions and recurring issues by iterating on triage and following investigations through to fixes.
Specialized AI agents help us investigate incidents, organize follow-up work, and prepare changes, with engineers reviewing and correcting the work along the way. What surprised us was how much more comfortable we became relying on the process.
Engineers could inspect an assessment, challenge its assumptions, and use those corrections to improve how we handled later incidents. As follow-up fixes addressed recurring issues, we felt more confident about when to wake the on-call engineer.
That experience has made us question the shape of the work we give our agents. A capable coding agent could potentially investigate an incident, raise a ticket, and prepare a fix, so why divide those responsibilities?
