Your disaster recovery plan has probably never been run
Almost every organisation has a disaster recovery document. Far fewer have a recovery path anyone has executed. The gap between those two things is discovered on precisely the day you cannot afford to discover it.
This is not negligence. Nothing breaks while DR is ignored, so it loses every prioritisation conversation to work with a visible deadline. It usually surfaces as a procurement blocker — an enterprise client’s security review asks for evidence of tested recovery, and there is none.
Recovery targets are a business decision, not a technical one
Most DR documents apply one tier to everything: four-hour RTO across the board, because somebody wrote it in a template. That is simultaneously too expensive for the internal reporting tool and far too slow for the payment path.
The useful exercise is per service, with the people who own the revenue in the room. How long can this be down before it costs us money, a customer, or a regulatory problem? How much data can we afford to lose? Those answers vary enormously across a system, and the variance is where the budget goes.
Expect to find services nobody can justify recovering quickly. That is a real result — it frees spend for the ones that matter.
The dependencies that break a failover
Failover exercises fail in remarkably consistent places, and rarely the database everyone worried about:
- DNS. TTLs long enough that clients keep resolving the dead region for an hour.
- Secrets. The standby environment cannot reach the secret store, or holds credentials that expired months ago because nothing rotated them.
- Certificates. Renewal automation pointed at the primary only.
- Third parties. A payment provider or identity provider with an IP allowlist containing the primary region’s addresses and not the secondary’s.
- Capacity. The standby region has quota for a fraction of production, and the increase requires a support ticket with a lead time.
- Humans. Nobody is sure who is authorised to declare an incident and start a failover, so the first thirty minutes are a conversation.
None of these appear in an architecture diagram. All of them appear in a test.
Backups are not the same as restores
Backup coverage is easy to report and easy to trust. Restores are the thing that matters, and restore paths rot — a schema change, a version upgrade, an encryption key rotation, and the backup that completed successfully every night for a year cannot be restored into the current system.
Test restores on a schedule, into a real environment, and record how long it took. The duration is your actual RPO contribution, not the number in the document.
Producing evidence
If the driver is ISO 27001, SOC 2 or a client questionnaire, the control is only half the deliverable. What the auditor wants is evidence: a dated record of a test, what was tested, what failed, what was fixed, and who signed it off.
Generating that alongside the work costs very little. Reconstructing it afterwards, from memory, during a review, costs a great deal — and tends to produce a document that reads exactly like what it is.
Start with one service
A full multi-region failover exercise is a large piece of work and is often used as a reason to do nothing. Pick the single most business-critical service, define its targets, run one controlled failover, and write down what broke.
The first test always finds something. That is the point of it, and it is a much better outcome than finding it during an actual outage.