Set Practical Reliability Targets for Cloud Services Without Slowing Delivery

A practical approach to setting reliability targets for cloud services starts with understanding the business impact of downtime rather than chasing unrealistic uptime numbers. According to Google Cloud's Site Reliability Engineering research, exceeding reliability expectations beyond what users actually notice often leads to diminishing returns and significantly higher operational costs. In enterprise learning environments, stability expectations tend to vary based on usage patterns, with mission-critical certification exams and live virtual training sessions requiring stricter service-level objectives than standard content access.

One meaningful change involved shifting from feature-based release deadlines to error-budget-driven deployments. Instead of pushing releases based solely on roadmap timelines, deployments were temporarily paused whenever system reliability metrics exceeded predefined thresholds tied to latency spikes, failed API calls, or platform interruptions. Google's SRE framework has consistently shown that error budgets create a healthier balance between innovation speed and operational stability. After implementing this policy adjustment, incident frequency during peak usage periods dropped significantly while release velocity remained consistent because engineering decisions became tied to measurable service health rather than subjective urgency. The result was a stronger platform trust without creating bottlenecks in delivery cycles.

I set reliability targets by tying them to user-facing outcomes and by making ownership and success criteria explicit so teams can weigh the cost of safeguards against customer impact. We prioritize critical flows for higher targets and accept lower targets for noncritical components, keeping work focused where it matters. In line with my emphasis on clear ownership and expectations, we changed our release policy to require the designated owner to define acceptance criteria and success signals before any deployment. That change concentrated engineering effort on realistic safeguards, improved stability, and avoided adding extra approvals that would slow delivery.

The mistake most cloud services make with reliability targets is treating them as aspirational rather than contractual. Saying you want 99.99 percent uptime sounds great in a planning meeting, but if you have not modeled the engineering cost of that extra nine, you are making a promise your infrastructure cannot keep without grinding feature development to a halt.

For GpuPerHour, we set reliability targets by working backward from customer impact rather than forward from engineering ambition. We asked: what does a minute of downtime actually cost our customers? For teams running ML training jobs that take 8 to 12 hours, a brief API blip that does not interrupt running jobs is tolerable. A full node outage that kills a training run midway costs them potentially hundreds of dollars in wasted compute and days of lost progress. That distinction drove us to set different reliability targets for different system layers. Our job scheduling API targets 99.95 percent availability. Our node-level compute infrastructure targets 99.99 percent because the cost of failure is asymmetric.

The specific change that improved stability without slowing delivery was introducing a deploy freeze window tied to customer workload patterns rather than calendar sprints. We analyzed our usage data and found that 60 percent of long-running training jobs launch between 9 PM and 2 AM UTC. So we moved our deployment window to 10 AM to 2 PM UTC when utilization is lowest. This meant that if a deploy introduced a regression, the blast radius was smaller and we had a full engineering team awake to respond.

Before this change, we had two incidents in a single quarter where a deployment during peak hours disrupted active training jobs. After implementing workload-aware deploy windows, we went seven months without a customer-impacting deployment incident. The delivery velocity stayed the same because we did not reduce the number of deploys. We just moved when they happened.

The best way to manage cloud reliability vs safeguard costs is to set up your SLA to target real user sessions, and not to overprovision the infrastructure to capture bot sessions. The common pattern seen in scaling SaaS platforms is that sudden spikes of data load and site slowness issues come from bad bots, not real growth.

Rather than spend money on excess compute to maintain a rigid uptime metric, instead do aggressive filtration of traffic as a proactive measure. Test site speed and use AI algorithms to detect anomalous engagement spikes, as these often precede bot infection or DDoS attacks, and neutralize the impact before it hits your database resource metrics or your website quality score.

As a policy change, one of the most effective strategies is to move from highly automated rate limiting towards a tiered escalation policy that involves AI and humans.

In one cloud software environment example, safeguards were highly automated and ended up dropping real verified user sessions during spike events, causing a huge impact on customer trust. Instead of changing the delivery pipeline, they modified the reliability targets to add anomalous traffic monitors that flagged spikes of bots, placing them into a queue.

But crucially, this had to be validated by human DevOps that then put the IP range permanently into a kill list. By automatically filtering early bad agents with AI but maintaining human context on the kill switch, average system latency during traffic spike events went from 1.2 seconds down to 0.8 seconds (a 30% reduction in slowness).

And because this was handled by AI in the early detection and quarantine, the core engineering teams were less involved in firefighting infrastructure lockups and could maintain their rapid daily feature release cadence while preserving platform stability.

Reliability goals set at the infrastructure level are not really what matter. The right question is what failure actually costs the user.

A delay of one second when syncing in the background is not as bad as a one-second delay during checkout. We changed our approach from having the reliability standards across the whole platform to having different standards based on how critical each part is to the user. We set standards for the parts of our system that users notice and were more flexible, with the parts they don't notice. This one change gave us a lot resources to work with without affecting how much users trust us. The key takeaway is that all reliability is not the same. You should set goals based on where users feel the impact of failure not where the system detects it.

Reliability targets should be derived, not guessed. I start with three questions. First, how critical is the use case: is this service on the core business path, or peripheral? Second, what does the cost-of-unavailability curve look like against duration? Some workloads degrade linearly, while others (payments, real-time messaging, safety-critical pipelines) climb exponentially once an outage crosses a threshold. Third, where does this service sit in the larger system? A component embedded in a system with a 99.9% target must itself be measurably stricter, because dependent failures compound. The reliability target is then balanced against the marginal cost of each additional "nine", and safeguards are implemented only up to the point where they remain cheaper than the expected cost of the outages they prevent.
A concrete example: I worked with a customer whose workload could not tolerate more than five minutes of downtime. A RabbitMQ broker on Amazon MQ sat on the critical path. To control cost, they had originally deployed a single-instance broker, but the service's weekly maintenance window rebooted the broker and produced roughly 25 minutes of unavailability, well beyond their tolerance.
The fix was an infrastructure adjustment rather than an application rewrite: we migrated to a new Amazon MQ broker deployed as a three-node cluster. Amazon MQ's maintenance process only takes one node offline at a time, so two nodes always remain available, and patching no longer counts as user-visible downtime. Because this was purely a topology change, no application code, client libraries, or release pipelines had to change, so delivery velocity was unaffected. We then sized up the instance class to absorb future growth.
In summary, when a maintenance policy is the primary source of unavailability, changing the infrastructure topology into a more robust one usually yields more availability than engineering software-level workarounds, and such changes rarely slow the team down.

Reliability targets at a cloud company start with uptime. We commit to 99.95% on our standard VPS plans and hold ourselves to that number. But uptime is only one piece of what customers actually experience. Reliability also means the server is reachable the moment they hit deploy, SSH connects in a couple of seconds, the network performs at the speed we sold them, the backup they paid for restores when the day comes and a real person picks up the support ticket fast when something breaks.

Balancing trust against the cost of safeguards is mostly about choosing where to spend. Every protective measure has a real price attached, from redundant power feeds and hot-spare hardware to replicated storage and an in-house support team staffed across multiple time zones instead of outsourced offshore. We invest selectively. The bigger share of the budget goes to safeguards that prevent failures customers do not forgive. A few hours of outage handled with honest communication is something most customers will absorb. Losing data they believed was being protected is a failure they carry with them for years and tell their networks about. That is the category we spend most aggressively to prevent, even when the per-customer math looks heavy.

The release-policy change I would point to: when we rebuilt our Performance VPS tier in 2025, we shipped automatic daily backups as a feature included by default rather than as an optional add-on at checkout. Before that change, most customers skipped the backup option during purchase and discovered why it mattered only after they had broken something months later. We absorbed the support cost trying to help and usually lost the customer anyway. Making backups default-on cost us a small amount of storage per server and did not slow provisioning. The unrecoverable-data ticket category shrank substantially in the months that followed and renewal rates on the affected plans moved up the next quarter. The change to our release process was a single line in the provisioning template. My take from the experience is that the most effective reliability gains in our market come from changing what customers receive by default rather than from launching new optional features.

I usually start reliability targets from the user's point of view, not from the infrastructure dashboard.

A cloud service can have dozens of internal components, but not all failures hurt users equally. Authentication, payments, data writes, API availability, and deployment safety usually deserve stronger targets than internal tools or low-risk background jobs. Once you separate those paths, reliability planning becomes more practical and less expensive.

One release policy change I like is risk-based approvals. Instead of making every change go through the same heavy process, we classify changes by blast radius. A tag update or isolated service change should not be treated the same as a database migration, IAM change, networking change, or production Terraform apply.

That simple split improves stability without slowing the team down. Routine changes keep moving, but risky changes get proper review, rollback notes, and post-release checks.

The goal is not perfect uptime at any cost. That usually turns into waste. The goal is to put the strongest safeguards around the parts of the service where failure would actually break user trust.

Muhammad Hassaan Javed
Founder, InfraForge
https://infraforge.agency

Reliability is a promise, and the price of breaking it is paid by your customers, not by you. We hold a 99.9% uptime commitment at Paperless Pipeline, and the way we got there is boring on purpose.

Some context. We process about 6% of every U.S. home sale. 1,700+ brokerages, 90,000+ users, more than 4.6 million transactions closed through the platform since 2009. When we go down, real estate deals stop. That cost is what sets the target, not an SLA marketing page.

How we set the number. We started by asking what an unreliable hour actually costs a broker. RE/MAX Plus in Rochester runs a top-100 brokerage and saves $2,000 to $2,500 per month by replacing one admin role with Pipeline. An outage during a closing window costs that brokerage more than a month of subscription. So our reliability target had to be high enough that downtime is rare enough to be forgivable, but realistic enough that we are not burning engineering hours chasing nines we cannot deliver. 99.9% sits at that line for us.

The one change that improved stability without slowing delivery: the 6-week release cadence. We have shipped product upgrades every six weeks since 2009. Every year. Without fail. Before that, releases were ad hoc, and the failures clustered around the unplanned big ones. Locking the cadence forced smaller change sets. Smaller change sets caused fewer regressions. Fewer regressions meant faster rollbacks when something did slip.

The second change: every release now has a kill switch on every new feature, defaulting to off for the first 24 hours. We turn it on for a slice of brokerages, watch error rates, then expand. If the slice misbehaves, the rollback is one config flag, not a redeploy.

Admit the limit. We are not Stripe. We do not need four nines. Setting a target you cannot deliver damages trust more than picking the honest one.