Build Sustainable On-Call for Software Teams

As CTO of an outsourcing company, not a product company, I see on-call a little differently. We don't come into every project assuming, "Okay, now this team owns production 24/7." Sometimes that is true. Often it is not. The client may have their own platform team, their own NOC, their own cloud team, or simply their own rules about who is allowed to touch production. So before we talk about rotations, we need to answer a less glamorous question: who is responsible for what?

If Innowise is involved in production support, the scope has to be clearly defined: what counts as an emergency, who is the first responder, when our team should be escalated to, and what can safely wait until business hours. Without that, on-call becomes vague and inefficient for everyone.

When we help clients design these processes, we usually separate real incidents from ordinary engineering signals. A production outage, failed critical business operation, security issue, or serious degradation may require immediate escalation. A temporary spike, warning, or non-critical bug should usually become a ticket (not a night call by any means).

One change I always recommend is to stop treating all alerts as equal. Create tiers. There should be a very small category of alerts that can wake someone up. Then there should be a second category that creates a ticket or a message for the next working window. This sounds simple, almost too simple, but it changes behavior. People stop reacting emotionally to every signal, and they start trusting the escalation process. As simple as that.

The other underrated part is handoff. In outsourcing, context can disappear very quickly. Someone on the client side knows one piece, our engineer knows another piece, DevOps knows a third piece, and suddenly the incident becomes difficult organizationally (not technically). So handoff should be boring and written: what changed, what was deployed, what was observed, what is still risky, what was already tried, and who owns the next move.

For me, the healthiest on-call model is not the one where everyone is always available. That is just a slow way to burn out good engineers. The healthier model is where the truly urgent issues move fast because the process is clear, and the non-urgent ones are allowed to wait. It protects people, but it also protects reliability. Tired engineers make worse decisions.

Sustainable on-call treats it as real, paid work with time to rest after hard shifts. Clear pay rates, weekend differentials, and simple rules per hour or per page avoid stress and doubt. Recovery time after night duty protects sleep, mood, and judgment.

Caps on page volume and a sane rotation keep the load fair across the group. Extra support like food credit or a ride home during late work shows respect for the strain. Publish a simple on-call pay and recovery policy and walk through it with the team.

Service level objectives turn user needs into clear goals for uptime and speed. Error budgets show how much risk is left before users feel real pain. When the budget runs low, slow or pause risky changes and harden weak parts.

Tie rollout rules and fast rollbacks to each service’s budget so action is automatic. Shared dashboards and simple reviews keep everyone aligned on the same signal. Define one SLO for your top user flow and agree on an error budget rule before the next sprint.

Follow-the-sun on-call spreads pages across time zones so alerts reach rested responders. This cuts night work, lowers stress, and improves focus during incidents. Smooth handoffs matter, so each shift should have a short overlap for status updates.

Clear notes and shared dashboards keep context from getting lost between regions. Fair rotations and backup plans protect coverage when someone is out. Map team time zones, set small overlap windows, and pilot a follow-the-sun schedule next quarter.

Automation turns common fixes into quick, safe actions that run at any hour. Strong runbooks define triggers, steps, and rollback so the system can heal itself with guardrails. Start with noisy alerts and add checks to confirm a real issue before each change.

Bots can gather logs, tag owners, and post links to past fixes to help the human on call. Track time saved and re-page rates so weak automations get tuned or removed. Pick your top three alerts and turn them into tested runbooks with auto-fix steps this month.

Regular incident drills build skill and calm before true outages happen. Short, real-looking exercises help responders build muscle memory and faster judgment. Rotate roles so people learn to lead, to triage, and to give clear updates.

Keep the review blameless and use findings to sharpen runbooks and alerts. Let new joiners shadow first, then handle small tasks, and grow into full duty. Schedule a monthly drill and assign a lead for the first one.

• How Do You Balance Innovation With Maintaining Day-to-Day IT Operations?
• Making API Changes Without Breaking Partner and Internal Apps
• Shipping Accessibility in Web and Mobile Product Development