When a region goes down: disaster recovery that works

Cloud regions are incredibly reliable. Until they aren't. Large outages at major providers happen every now and then, sometimes taking down a whole region's worth of services for hours. And disasters aren't only about providers: a deleted production database, a ransomware attack, a bad deployment that corrupts data, or an expired payment method that suspends an account.

Disaster recovery (DR) is about answering one question honestly: if the worst happens, how quickly can we be back, and how much data will we lose?

RPO and RTO, simply

Two numbers define your DR goals:

• RPO (Recovery Point Objective): how much data can you afford to lose? If your RPO is one hour, you need backups or replication at least every hour.
• RTO (Recovery Time Objective): how long can you be down? If your RTO is four hours, you must be able to restore service within four hours.

Lower numbers cost more. An RPO of zero and an RTO of minutes means active-active systems in multiple regions, which is expensive and complex. An RPO of 24 hours and an RTO of a day might be fine for an internal tool.

Set them per system, with the business. Not everything needs the same level.

Four strategies

From cheapest to most expensive:

1. Backup and restore. Regular backups stored in another region or account. In a disaster, you rebuild infrastructure (with infrastructure as code) and restore data. Cheap, but recovery takes hours. RPO depends on backup frequency.

2. Pilot light. The core components (like a replicated database) run in the second region at minimal size. Everything else is ready to be created quickly from code. Faster recovery, moderate cost.

3. Warm standby. A scaled-down but fully working copy of the system runs in the second region. In a disaster, you scale it up and switch traffic. Recovery in minutes to an hour.

4. Active-active (multi-site). The system runs fully in multiple regions at once, serving traffic. A region failure just shifts traffic. Near-zero RTO, but the most expensive and complex, especially for data consistency.

Strategy	Typical RTO	Typical RPO	Cost
Backup and restore	Hours to a day	Hours	Low
Pilot light	Tens of minutes to hours	Minutes	Medium
Warm standby	Minutes	Seconds to minutes	Higher
Active-active	Near zero	Near zero	Highest

Disaster recovery is not a document. It's a capability, and capabilities need practice.

The foundations

Whatever strategy you choose, these basics matter:

• Infrastructure as code for everything. Rebuilding infrastructure by hand during a disaster is slow and error-prone.
• Backups in a different region and a different account, ideally with immutability (object lock) so ransomware or a compromised admin can't delete them.
• Automated, monitored backups: alert when a backup fails or is missing.
• DNS and traffic management that can switch regions quickly (low TTLs, health-checked failover records, global load balancers).
• Secrets, certificates and container images available in the recovery region too. A common surprise: the image registry or secret store only existed in the failed region.
• Runbooks that someone other than the original author can follow.

Kubernetes backups with Velero

For Kubernetes, Velero is the standard open-source tool. It backs up Kubernetes resources (deployments, services, config) and persistent volumes (with snapshots or file-level backup) to object storage, and restores them, even into a different cluster.

# Daily backup of the shop namespace, kept for 30 days
velero schedule create shop-daily \
  --schedule="0 2 * * *" \
  --include-namespaces shop \
  --ttl 720h

# Restore into a new cluster
velero restore create --from-backup shop-daily-20261020020000

With GitOps, your Kubernetes resources are already in Git, so the critical part Velero adds is persistent data. For databases, prefer the database's own backup tools (or an operator like CloudNativePG) for consistent, point-in-time recovery, and use Velero as an extra layer.

Testing it for real

This is the part most teams skip, and the part that matters most.

• Restore backups regularly into a test environment, automatically if possible. Check the data is actually usable.
• Run DR drills: once or twice a year, simulate losing a region or a critical system, and actually recover. Time it. Compare with your RTO.
• Start small: restore one database, then one service, then a whole environment.
• Write down every surprise during a drill and fix it. There will be surprises: missing permissions, forgotten dependencies, outdated runbooks, quotas in the recovery region.
• Chaos experiments like failing an availability zone help build confidence too.

So, where do I start?

Agree RPO and RTO for your three most important systems. Check that their backups exist, are in another region and account, and are immutable. Then restore one of them into a fresh environment this month, and time it. That single exercise will tell you more about your real DR readiness than any document.

My take: you don't have disaster recovery until you've recovered from a disaster, even a pretend one.

— Desly