Whoever writes the report decides the outcome

You read in a report that the continuity of order processing is adequately assured. Underneath that sentence lie documents: test reports, a contract with a supplier, an overview of systems. Each document says something about one area. None of them says "adequately assured". That judgement only comes into being when somebody puts the documents side by side and turns them into a single answer.

How that combining is done is rarely stated in the report. And that is exactly where the outcome is decided. I will show it with an example. It is invented, and the figures serve the reasoning rather than a threshold I would recommend.

The example

You want to know whether order processing continues when something fails. The board has set a requirement for that. At any moment of the day, at least 120 orders an hour must be processed.

Orders reach the central order processing system by three routes.

Route A, the web shop. The customer signs in to the web shop and enters the order. The web shop passes it on to order processing. Capacity 100 orders an hour, day and night.
Route B, the customer portal. The customer signs in to the portal and enters the order. The portal passes it on to order processing. Capacity 100 orders an hour, day and night.
Route C, the service desk. The customer calls, and an employee enters the order directly into order processing. One employee handles 10 orders an hour, and during office hours there are always at least five employees. Capacity 50 orders an hour, office hours only.

The central order processing system itself handles at most 200 orders an hour, because of its licence. At the busiest moment of an office day, that is also what has been measured.

Each route has been tested. The web shop reached its 100, the portal its 100, the service desk its 50. Three tests, three passes, at three different moments. There are no other reports. Now three people who draw a conclusion from them, and all three know their trade.

Three ways to read the same documents

The first reader adds up. During the day 100 plus 100 plus 50, capped by the licence at 200. At night 100 plus 100, so 200 as well. The requirement is 120. Every route has been tested and passed, and there are 80 orders an hour between what is possible and what is required. His conclusion is that order processing is assured, with a margin.

The second reader reckons with the loss of a route. The requirement applies at any moment, and which route fails is not known in advance. So he removes one route at a time and looks at what remains. During the day that works out. Without the web shop, the portal and the service desk remain, together 150. At night it does not. The service desk is closed, and without the web shop only the portal remains, 100. The requirement is 120. His conclusion is that order processing is not assured outside office hours, with a shortfall of 20 orders an hour as soon as one of the two routes fails. He advises raising the capacity of the web shop or the portal to 120, or keeping two employees on call at night.

The third reader asks what the routes have in common. The first two readers treat the routes as if they stood apart from each other. Adding up is only right in that case, and removing one route is only right if a failure does indeed hit one route only. Whether that is so appears in none of the test reports, because each test covered one route. This reader asks the administrators.

What then comes to light is this. The web shop and the portal run at the same hosting provider. They use the same service for signing customers in, the same network connection to the data centre, and at the hosting provider the same firewall. And in the data centre there is a firewall that all three routes pass through, the service desk included. Everything is duplicated, so a broken device is covered. But a wrong change in the hosting provider's firewall, an expired certificate or a failure of the sign-in service does not hit one route but the web shop and the portal at once. Then nothing remains at night, and during the day only the service desk with 50 orders an hour. And if it goes wrong in the data centre firewall, all three routes fail, during the day as well. His conclusion is that the documents do not yet carry the requirement. What has been tested is each route on its own. What the requirement rests on is what they share, and that has never been tested.

The same documents, three outcomes

Assured with a margin, not assured outside office hours, not yet demonstrated. Three conclusions from the same three test reports, by three people who know their trade.

The difference is not in the documents, because they were the same for all three. It is not in the requirement, because that was set and on paper. Nor is it in expertise. The difference is in the rule used to combine the documents. Adding up, reckoning with the loss of a route, or first looking for what is shared.

Nobody wrote that rule down. Each reader chose one for himself, and each of the three can defend that choice. But with that choice the outcome was settled. That is what the title says. Whoever writes the report decides the outcome. Not by twisting anything, but by making a choice nobody asked him to make and nobody gets to see.

The right rule is not a matter of opinion

If three expert readings are possible, is it simply a matter of taste which one you choose? No. Which rule is correct follows from how your organisation is put together.

Can a failure hit one route and not the others? That depends on what the routes share, and that is a fact about your connections. What does "duplicated" mean exactly? That the second half takes over when the first breaks, and not that a fault affecting both halves is absorbed. That too is a fact, and it is in the design. How many orders actually come in at night? That is in your own figures, and it determines how much weight the second reader's night-time outcome carries.

So the right rule can be found. But somebody has to look for it, and that only happens when somebody asks how the documents relate to each other rather than what each document says on its own. The first two readers did not ask that question. Not from ignorance, but because the documents gave no reason to. Each test report covered one route and was in order by itself.

The shared component is never in the documents

That is not a coincidence of this example. A test report describes what was tested, and that is one route. The sign-in service, the network connection and the firewall belong to none of the routes and so appear in none of the reports. They are infrastructure, everybody's and therefore nobody's. What they mean for the conclusion exists only in the head of whoever knows all the routes, and it only comes to the table when somebody asks.

Routes that hang on the same component fail together when that component fails. On paper you have three. In reality you then have one, or none, and an overview showing three routes does not reveal that.

When the requirement is missing too

In the example the board had set the requirement at 120 orders an hour, at any moment. If that requirement is not there, the conclusion carries no number but the word "adequate". What adequate means, the writer has then filled in himself. Then both choices sit with the writer of the report, the standard and the rule. How much the organisation must be able to do during a failure, and for how long, belongs among the things the board sets or has set. Often it already follows from a client contract, a legal obligation or an arrangement with the bank.

On the way up, the qualification disappears

Every document says less than the conclusion above it. The test report for the web shop covers one route, on one day, with the other routes out of the picture and with all the shared components working. At every summary somebody drops a qualification, because at that level it looks minor. "Each route tested at its own moment" becomes "tested". "Duplicated" becomes "no risk". "Three passes" becomes "assured, with a margin".

By the end the conclusion reads without qualification, while no document underneath it did. Nothing was invented. Something was left out each time.

Four questions for whoever wrote the report

You do not need to call for a file to do this. Four questions at the table are enough.

Which requirement was this tested against, and who set it?
Which documents carry this judgement, and what exactly was tested in them?
How were the documents combined, and why that way? Added up, reckoned with the loss of a part, or first searched for what is shared?
What do the parts have in common, and was that shared part included in the tests?

The third question is the question of this article. If you get an answer to it, you see the rule and you can argue with it. If you do not, you are reading an outcome without knowing how it came about.

The point

A conclusion about continuity is not the sum of the documents underneath it. It is a judgement about how those documents work together, and that judgement follows a rule.

If the rule is not in the report, it has still been applied. Only, the person who wrote the report chose it, and you did not. As long as that is the case, whoever writes the report decides the outcome.

The requirement belongs to the board, and it should be there before anything is reported. The rule is not something the board has to invent. It follows from how the organisation is put together, and whoever writes the report is the right person to find it and write it down. What the board can demand is that it appears in the report, with the facts it rests on. Then you see not only the outcome but also the road to it, and you can judge the rule instead of only the outcome.