Availability is often discussed as if it were a property of infrastructure: add replicas, distribute them across zones, and uptime will follow. Infrastructure matters, but it is only one part of the system. Real availability emerges from product decisions, architecture, rollout controls, monitoring, incident response, and the organisation’s ability to learn.

A highly redundant service can still fail globally if one unsafe configuration reaches every replica at once. A technically recoverable database can remain degraded for hours if its recovery procedure has never been rehearsed. Availability is therefore not something an operations team adds after delivery. It is a capability the whole organisation builds.

Begin with a user-facing promise

A service level objective should describe what users experience, not whether individual servers are healthy. Useful indicators include successful requests, completed transactions, correctly processed messages, and latency within an acceptable threshold.

Google’s SRE guidance on service availability also recommends aggregate unavailability—such as the percentage of failed operations—when a service can remain partially available. This prevents a healthy replica count from hiding failed customer journeys.

The difference between two apparently similar objectives is substantial:

Choosing the extra nine is a product decision. Does the user benefit justify the engineering cost, slower delivery, and operational complexity? If nobody has answered that question, the reliability target is probably an inherited aspiration rather than a deliberate commitment.

Turn reliability into a decision system

An error budget converts the SLO into a shared constraint. Product, engineering, and operations agree how much failure is acceptable during a window and what happens as that allowance is consumed.

Google’s chapter on embracing risk and error budgets gives a precise example: a quarterly SLO of 99.999% permits a failure rate of 0.001%. An incident that fails 0.0002% of expected queries consumes 20% of the quarterly budget.

The policy should be explicit enough to guide a release without convening an emergency meeting:

yaml
service: customer-api
slo:
  indicator: successful_valid_requests / valid_requests
  objective: 99.9
  window: 30d

release_policy:
  budget_remaining_below_25_percent: slow_rollouts
  budget_exhausted: freeze_nonessential_changes
  exceptions:
    - security_fixes
    - reliability_fixes

This is organisational governance expressed in operational terms. When the budget is healthy, teams can experiment. When it is nearly exhausted, rollouts become smaller and verification becomes stricter. When it is gone, reliability work takes priority. The same rule applies regardless of who argues most persuasively in a meeting.

Blast radius is a design choice

The most revealing incidents are rarely caused by one extraordinary mistake. They happen when several ordinary safeguards fail together.

During Cloudflare’s July 2, 2019 outage , a WAF rule containing a badly behaving regular expression was distributed globally. Serving CPU rose to nearly 100%, customers received 502 responses, and Cloudflare reported losing 80% of its traffic. The core service was down for 27 minutes, although the first synthetic alert fired only three minutes after deployment.

The regular expression was the trigger, but the global blast radius came from the surrounding system:

  • The rule could reach the worldwide fleet without a staged rollout.
  • Tests did not detect excessive CPU consumption.
  • The regular-expression engine offered no suitable complexity guarantee.
  • Rollback was slower than the distribution of the faulty change.
  • Some internal response tools depended on the affected edge.

Cloudflare’s corrective actions therefore went beyond changing one pattern. They added performance profiling, moved toward engines with runtime guarantees, introduced staged rollouts, improved emergency access, and automated status updates. The durable lesson is that deployment speed needs an equally fast containment mechanism.

Dependencies turn small faults into long recoveries

GitHub’s October 21, 2018 incident analysis shows a different failure shape. A 43-second loss of connectivity between network locations triggered database topology changes and ultimately 24 hours and 11 minutes of degraded service.

The original interruption was brief. Recovery was long because the two sides had accepted writes that were not mutually replicated. GitHub intentionally paused webhook delivery and GitHub Pages builds, froze deployments, and prioritised data integrity over faster restoration. No user data was lost, but database clusters ranging from hundreds of gigabytes to nearly five terabytes made restoration and reconciliation slow; restoring remote backups took hours.

This is why dependency design must include impaired modes. Ask not only “Is the dependency redundant?” but also:

  1. What happens when it is reachable from only part of the system?
  2. Can the data plane continue serving if the control plane is unavailable?
  3. Which functions can degrade safely?
  4. How is conflicting state reconciled?
  5. Has recovery been timed with production-scale data?

AWS calls one useful approach static stability : provision the capacity and information required for recovery before an impairment occurs. In its three-Availability-Zone example, AWS overprovisions by 50%, operating each zone at 66% of its load-tested capacity so the remaining zones can absorb one zone’s loss without first depending on a scaling control plane.

Rehearse the organisation, not only the technology

A recovery plan that has never been executed is a hypothesis. AWS recommends regular game days that simulate realistic faults and involve the same teams, procedures, monitoring, and decision paths used in production.

A useful exercise tests more than automatic failover:

  • Did monitoring identify user impact?
  • Was incident ownership established quickly?
  • Could responders access tooling during the impairment?
  • Did the runbook match the actual system?
  • Were rollback and degraded-mode decisions understood?
  • Did communications reach customers and stakeholders?

Every exercise should begin with a hypothesis, define safety guardrails, and end with owned improvements. Repeating an unchanged game day is valuable only if the previous weaknesses have been removed.

Convert incidents into organisational memory

Google’s postmortem culture guidance treats postmortems as instruments for improving the system, not assigning blame. A useful review quantifies impact, distinguishes the trigger from contributing conditions, records recovery decisions, and produces measurable actions with owners and priorities.

Blamelessness does not mean avoiding accountability. It means assuming people acted reasonably with the information and tools available, then asking why the system made an unsafe action possible or a correct action difficult.

Availability improves when the learning loop closes:

  1. Define what users need.
  2. Measure it with an SLO.
  3. Govern change through an error budget.
  4. Limit blast radius by design.
  5. Rehearse dependency failures.
  6. Learn from incidents and close the resulting actions.

That loop crosses team boundaries, budgets, roadmaps, and leadership incentives. This is precisely why availability is an organisational capability: infrastructure can survive a fault, but only an organisation can repeatedly decide, practise, measure, and learn its way toward resilience.