The safety net that depends on the same component

There is a second arrangement behind the most important system. A failover, a backup, a second location, a second zone at the supplier. The record says what the arrangement is and where it runs. What it shares with the system whose failure it is meant to absorb, it does not say.

That is what decides whether it works. If both depend on the same component, one event takes them both down. On paper you then have two. In practice you have one.

Four organisations have written down how that worked out for them. I use their own publications, because those are more precise than the reporting about them.

Four times one event took the safety net with it

Toyota, August 2023

During maintenance on a database, accumulated data was deleted and organised. An error arose from insufficient disk space and the system stopped. Production in the Japanese plants came to a standstill. Toyota writes that the servers were running on the same system, that a similar failure therefore occurred in the backup function, and that a switchover could not be made.

UniSuper and Google Cloud, May 2024

A private cloud environment of the Australian pension fund was inadvertently deleted at Google Cloud. In its own analysis Google writes that this concerned one of the customer's private clouds, running across two zones. Operators left a parameter blank. Both zones sat inside what was deleted, so the duplication was deleted with it. What sat outside it survived. According to Google, the backups in Google Cloud Storage in the same region helped the rapid restoration.

Maastricht University, December 2019

After a ransomware attack, Fox-IT established that backup servers were part of the same domain as the encrypted servers, with no segmentation at network level. That allowed the attacker to encrypt several backups. Not all backups were hit. The university writes that a limited number of systems was involved, and it says why its backups were set up that way. Until then it had chosen to use backups mainly to get a working environment back quickly after an outage or a failure.

Atlassian, April 2022

A script meant to remove a single application was given the identifiers of entire customer sites instead of those of the application. In twenty-three minutes, 883 sites belonging to 775 customers disappeared. The same script also deleted those customers' contact details from the production environment. Atlassian writes that it could not reach many customers, because it had lost their contact information along with the sites that were deleted.

What the four have in common

In all four the arrangement existed and was known. Toyota had a backup function. UniSuper had duplication. Maastricht had backup servers. Atlassian had immutable backups and tested them continuously for restoration. In none of the four was the measure missing.

What differed is what the arrangement shared with the system underneath it. At Toyota that was the system both ran on. At UniSuper the environment both zones sat in. At Maastricht the domain and the network. At Atlassian the data the recovery needed.

That last one is the least visible form, and therefore the most instructive. The recovery shared no system with the event. It shared the data that made the affected customers reachable, and that went with them.

Recorded is not proven

An arrangement can be in order and still not be tested against the event that will hit it. Atlassian describes this itself. The script was tested in a staging environment under its own change management. That test could not reveal the fault, because the wrongly supplied identifiers did not exist in that environment.

So there was a test, it passed, and it covered something other than what went wrong. That is not negligence. It is what happens when a test skips the component that carries both sides.

Atlassian also writes what it had prepared for. For infrastructure failures, such as the loss of a database, a service or an availability zone. It could not select a large set of customer sites and restore them to an earlier point in one go. For failure on that scale there were no runbooks ready. The recovery time objective was missed, the recovery point objective was met.

That is the difference between an arrangement that is recorded and an arrangement that is shown to work. The first says something exists. The second says what happens when you call on it, and against which event.

Why the shared component appears nowhere

The shared component belongs to neither side. It is the system underneath, the domain around them, the environment they both sit in, or the data you need to reach those affected. It does not appear in an overview of measures, because it is not a measure. It does not appear in a test report, because that report covers one side.

What results is a picture that is correct line by line and wrong as a whole. Every line is defensible. The sum of them is that nobody can point out which event hits both sides at once.

Toyota writes about the same system, Google about the same environment, Fox-IT about the same domain. Three times a component that belongs in neither column.

What I typically see

the failover runs at the same supplier as the system it replaces, and nobody has ever raised it
the backup sits in the same network and under the same administration as the data it is meant to save
there is an annual test, and it starts at a moment when everything still works
the runbook for an outage sits in the system that is unreachable during an outage
the passwords for the recovery are in a vault that only opens again after the recovery
the supplier provides a second environment, and no contract says on which component it differs from the first
last year's test passed, and nobody can say which event was simulated
the person who has to carry out the recovery is the same person investigating the failure at that moment
the contact details of the people you need during an outage are in the system that fails
the question of what the two sides share reaches the table only after an event, and then under time pressure

Each part is defensible on its own. Together they do not add up to a picture you can steer by.

The questions this raises

Which process must not stop, and for how long exactly must it keep running?
Which arrangement absorbs the failure of that process?
What do the two share: a system, a network, a domain, a supplier, a subscription, an administrator?
Which event hits that shared component, and what is left after it?
Has the arrangement been tested against that event, and not only against a device failing?
What did you need during the recovery, and did it still exist at that moment?
Who establishes that the test passed, and what shows it?

The core

A second arrangement is only a second arrangement when it cannot go down with the first. What decides that is what they share, and that is not in the record.

Recorded means something exists. Proven means you know what happens when you call on it. Until that difference has been made, you do not know whether you have one arrangement or two.

That question is a board question and not a technical one. Which component carries both sides is a fact about your own set-up, and someone can look it up. Whether that happens depends on who asks.