A staged path to multi-region on AWS

“We need multi-region” is almost never a requirement. It is a proposed solution, and the requirement underneath it is usually one of three things: a compliance clause about data residency, a board-level anxiety after someone else’s outage, or a genuine revenue exposure to downtime that somebody has actually quantified.

Those three want different architectures, and only the third one sometimes justifies the fourth stage below. This guide is the sequence we walk through, what each stage removes, and what each stage costs.

First, name the failure

Resilience work is insurance. You cannot price insurance without naming the event. These are the events, roughly in descending order of how often they actually take systems down:

Read that table honestly and the uncomfortable conclusion is that most multi-region programmes are insuring against the sixth row while the first three keep causing the actual incidents. We have walked into more than one engagement where the second region was half-built and the restore procedure had never been run once.

If you have not tested a restore in the last quarter, you do not have a backup. Fix that before you buy a region.

Stage 0 — one region, done properly

Before any geographic distribution, the single-region system needs to survive its own deploys. This is unglamorous and it prevents more downtime than anything else on this page.

• Health checks that assert dependencies, not just process liveness. A /healthz that returns 200 while the database pool is exhausted is worse than no health check.
• Progressive delivery — canary or linear, with automatic rollback on error rate and latency alarms.
• Every outbound call has a timeout, a retry budget with jitter, and a defined behaviour when the retry budget is exhausted.
• Infrastructure fully in Terraform. If your region is not reproducible from code, you cannot build a second one — you can only build something that resembles it.

That last point is the real gate on this whole path. A second region built by hand is not redundancy; it is a second system with its own distinct bugs and no shared verification.

Stage 1 — multi-AZ and a restore you have actually run

Removes: instance loss, AZ loss, operator error. RTO:~60–120 s automatic for AZ loss; 15–60 min for a restore. RPO: ~5 min via point-in-time recovery. Cost: roughly 1.15× single-AZ.

resource "aws_db_instance" "primary" {
  identifier     = "platform-primary"
  engine         = "aurora-postgresql"
  instance_class = "db.r6g.xlarge"

  # Synchronous standby in a second AZ. Failover is automatic and
  # typically completes in 60-120s.
  multi_az = true

  # PITR window. This is your RPO for operator error, which is the
  # failure mode this stage is really about.
  backup_retention_period = 14
  backup_window           = "03:00-04:00"

  # Do not let a "terraform destroy" in the wrong workspace be an incident.
  deletion_protection      = true
  skip_final_snapshot      = false
  final_snapshot_identifier = "platform-primary-final"

  storage_encrypted = true
  kms_key_id        = aws_kms_key.db.arn
}

# The restore path, exercised on a schedule rather than trusted.
# A backup you have never restored is a hypothesis.
resource "aws_scheduler_schedule" "restore_drill" {
  name                = "monthly-restore-drill"
  schedule_expression = "cron(0 4 1 * ? *)"
  flexible_time_window { mode = "OFF" }

  target {
    arn      = aws_lambda_function.restore_drill.arn
    role_arn = aws_iam_role.scheduler.arn
  }
}

The scheduled restore drill is the part teams skip and the part that matters. It restores the latest snapshot into a throwaway instance, runs a row-count and checksum assertion against known tables, records the wall-clock duration, and tears it down. That duration is your restore RTO. Anything else is a guess, and it is always optimistic.

Stage 2 — warm standby in a second region

Removes: regional degradation and regional loss. RTO: 5–15 min, mostly human decision time. RPO: seconds, bounded by replication lag. Cost:roughly 1.4–1.6× stage 1.

This is the stage most teams should reach and stop at. It survives losing a region, requires no change to your data model, and keeps a single writer — so you never have to reason about write conflicts.

The shape: full infrastructure in the second region, running at reduced capacity, with a read-only database replica promoted on failover.

# One Aurora global cluster; the secondary region carries a read-only
# replica that can be promoted. Replication is storage-level and typically
# lags well under a second.
resource "aws_rds_global_cluster" "platform" {
  global_cluster_identifier = "platform-global"
  engine                    = "aurora-postgresql"
  engine_version            = "16.4"
  storage_encrypted         = true
}

resource "aws_rds_cluster" "secondary" {
  provider                  = aws.eu_west_1
  cluster_identifier        = "platform-eu-west-1"
  global_cluster_identifier = aws_rds_global_cluster.platform.id
  engine                    = "aurora-postgresql"

  # Deliberately smaller. Scaled up as part of the failover runbook,
  # which is a tested step, not an assumption.
  db_cluster_instance_class = "db.r6g.large"
}

# Failover is DNS. Health checks decide; humans confirm.
resource "aws_route53_health_check" "primary" {
  fqdn              = "api-us-east-1.platform.internal"
  type              = "HTTPS"
  resource_path     = "/healthz/deep"
  failure_threshold = 3
  request_interval  = 10

  # Evaluate from multiple regions so one observer's network problem
  # cannot trigger a failover on its own.
  regions = ["us-east-1", "eu-west-1", "ap-southeast-1"]
}

resource "aws_route53_record" "api_primary" {
  zone_id = aws_route53_zone.public.zone_id
  name    = "api.platform.example"
  type    = "A"

  failover_routing_policy { type = "PRIMARY" }
  set_identifier  = "us-east-1"
  health_check_id = aws_route53_health_check.primary.id

  alias {
    name                   = aws_lb.us_east_1.dns_name
    zone_id                = aws_lb.us_east_1.zone_id
    evaluate_target_health = true
  }
}

The three things that break at this stage

The infrastructure is the easy part. These are what actually bite, and none of them are visible in a diagram:

1. Promotion is not idempotent and not reversible.Promoting the secondary breaks the global cluster’s replication topology. Failing back means rebuilding the old primary as a replica. Write that down in the runbook before you need it, because you will be reading it at 3 a.m.
2. Everything stateful outside the database. S3 needs Cross-Region Replication with a documented lag. Secrets need replicating. Long-lived sessions in a single-region cache vanish on failover. Queues are the worst case: in-flight messages in the failed region are simply gone unless you designed for it.
3. TTL is your real RTO floor. A 300 s DNS TTL means five minutes of failed requests no matter how fast your automation is, and resolvers that ignore TTLs make it longer. Set it to 60 s well in advance — changing TTL during an incident does not help, because the old value is already cached.

Stage 3 — active-active, and what it really costs

Removes: failover latency and capacity risk. RTO: near zero. RPO: near zero for committed local writes. Cost:roughly 2.2–2.8× stage 1, plus a permanent tax on every future feature.

Active-active is not an infrastructure decision. It is a data-model decision, and it is irreversible in practice. Two regions accepting writes means one of the following is true, and you must choose deliberately:

If your domain includes a ledger, a stock count, or a uniqueness constraint that matters — and most regulated domains do — then last-writer-wins is out, and you are choosing between write forwarding and partitioning. Write forwarding is usually correct, and it is worth noticing that it gives you a single logical writer, which is what stage 2 already had.

Which is the argument for stopping at stage 2 unless you have quantified revenue exposure to the five to fifteen minutes that stage 3 buys back.

A capability you have not tested is a claim

The failure mode we see most often is a second region that exists, is paid for, and has never taken traffic. The runbook has drifted, the secondary is two schema migrations behind, and the one person who understood the promotion step has left.

Minimum viable discipline:

• Quarterly game day. Fail over in production, on purpose, during business hours, with a comms plan. If that sentence is unthinkable, you do not have a failover capability — you have a hope.
• Continuous drift detection. A scheduled job asserting schema version, config hash, and image digest parity across regions. Alert on divergence, not on failover.
• Replication lag as an SLO. Page on sustained lag above your stated RPO. Lag is a leading indicator; discovering it during an incident is discovering it too late.
• Time the drill and publish the number. Measured RTO, on the wiki, dated. It is the only honest answer when someone asks.

How to decide

Four questions, in order. Stop at the first “no”.

1. Have you tested a restore in the last quarter? If not, that is the entire project until it is true.
2. Is your infrastructure fully reproducible from code? If not, a second region will be a second snowflake.
3. Can you state the revenue or regulatory cost of a four-hour regional outage? If not, stage 1 is the correct place to stop, and saying so is more useful than a proposal.
4. Is that cost greater than roughly 0.5× your current infrastructure spend per year? If yes, stage 2. If it is greater than 1.5×, and your data model can take it, discuss stage 3.

We have run this sequence with teams who arrived certain they needed active-active and left with a tested restore, a warm standby, and a materially smaller bill. That is a good outcome, and it is the one this guide is designed to reach.