Mike Bursell - Trust in Computer Systems and the Cloud

There is a further complication to be considered, which is how rarely we have control over—and can therefore have significant trust towards—the reporting or monitoring of channels in a relationship to a computer system. Channels are often under the control of the acting party or can be compromised in ways over which I have very little control. Let us look back again at the example of the bank transfer via a web browser. The reporting of the transfer is via the application, which is presumably written by the bank or at least is under the control of the bank: I have no control of this channel. Even if I am using a downloaded application and it is running on my computer or mobile phone—and even if it is open source, 9 meaning I have the chance to check that what it is displaying is “correct”—I cannot be sure that it is “valid” as the information it has comes from the bank, in the final analysis. In terms of compromise, I can at least take some steps to decrease vulnerabilities and, therefore, the chance of compromise, for the pieces over which I have control: I can keep my computer or phone as secure as possible, check that the versions of any downloaded applications—web browser or banking application, for instance—are verified as being from the expected suppliers, and check that the connection they have to the bank is as secure as I can make it.

In the end, however, it is up to my bank to provide valid information and ensure its correctness, though I will be the one who pays for these measures and am likely to bear any cost of invalid data: security economics raises its head again. This discussion about trust chains and monitoring will reappear later in the book as an important issue when designing and managing trust.

One final point to bear in mind about Gambetta's definition of trust is his description of the subjective probability with which an assessment is made about whether actions will be performed. We dropped this phrase in our definition, replacing it with assurance , as subjective probability frankly seems difficult to quantify or apply. The use of the word assurance should not be taken to imply 100% certainty, but words like subjective are problematic. The first reason is that the subject in this case may not be human. The second is because if we are to tie trust relationships back to discussions of risk, then objectivity and external qualifiability are insufficient: we also require external quantifiability.

We have examined, then, a variety of different trust definitions in the human realm, though none of them seems a perfect fit for our needs. Before we throw all of these out with no further consideration, however, there is an interesting question about the overlap between human-to-human relationships and human-to-computer relationships when the computer has a closely coupled relationship with an organisation. This is different to case 3 that we discussed in Chapter 1, when we discussed the relationship between a bank and its systems, and more like case 2, where my trust relationship to the bank, and the bank's relationship to me, are characterised by interactions largely with their computer systems. In this case, punishment or other social impacts (positive or negative) may be more relevant, as we may be able to relate them to people rather than to the computers with which the actual interaction takes place. We will return to this question later, once we have addressed questions around trust to institutions—which is related but distinct—later in this chapter.

Much of the recent discussion around trust in the human-to-human realm has revolved around game theory and cooperation, either separately or in conjunction. Game theory is the study of social interactions from a theoretical point of view, typically looking at strategies for interaction that yield particular outcomes (usually the “best” for an actor or set of actors). Different styles of interaction, competition, and cooperation are studied, and mathematical approaches are applied, usually with the proviso that the actors are “rational” 10 though not necessarily in possession of perfect information about their situation.

The best-known example of game theory is the Prisoner's Dilemma. The classic scenario of the Prisoner's Dilemma is where two members of a criminal gang are arrested and held separately from each other so that they cannot communicate. Each of them is presented with a choice: they may either stay silent or betray the other. Each is told the consequence of whichever action they may decide to take, combined with that of the fellow gang member. There are four outcomes, of which two are the same for both criminals, reducing the total number of distinct outcomes to three:

Both prisoners stay silent, in which case they are both sentenced to one year in prison.

One prisoner stays silent, but their colleague betrays them, in which case the betrayer goes free but the silent prisoner receives a sentence of three years in prison.

Both prisoners betray the other, in which case they both end up in prison for two years.

The rational position for each prisoner to take is to betray the other because betrayal provides a better reward than staying silent. Three interesting facts fall out of this game and the mountains of theoretical and experimental data associated with it:

If the prisoners play repeatedly but know the number of repeated games, then the most rational strategy is to punish the other for bad behaviour and keep betraying.

If they do not know the number of repetitions, then the most rational strategy is to stay silent.

In reality, humans tend to a more cooperative strategy when playing variants of this game, working together rather than betraying each other.

I have attended events where groups of people—unaware of the theory—have participated in multiple rounds of this game or a modified version of the Prisoner's Dilemma (sometimes, for instance, the payoffs are adjusted and counts kept of notional money won and lost). It is fascinating to watch people trying out strategies whilst reacting to past rounds and also being locked into a history of their own behaviour that they cannot change. Much of the foundational modern work around the Prisoner's Dilemma—and broader game theory—was done by Robert Axelrod. 11 He noted the same points and posited that cooperation—in such games or more broadly—is a positive evolutionary trait. It encourages behaviours that are likely to benefit the survival of the species adopting them. He also suggested ways to encourage cooperation, based on computer models contributed by various academic institutions:

Enlarge the shadow of the future (make players more aware of future games and less bound into their—and their fellow player's—histories).

Ensure that the payoffs are immediate, clear, and motivating.

Teach players to care about each other.

Teach reciprocity (rewarding positive actions—typically cooperation—and punishing negative actions—typically betrayal).

Improve recognition abilities (being able to recognise what the other party's strategy is).

Although proposed as a thought experiment, it turns out that the Prisoner's Dilemma can be used to model rather closely a number of different situations. Game theory arguably suffered, in the last years of the twentieth century and the early years of the twenty-first, from being the blockchain of its day: it was seen by some as the future foundation for all rules and types of societal interaction. As Heap and Varoufakis 12 noted, people's motivations are typically more complex than the somewhat simplistic models provided by game theory and are affected by what they called people's social location : the cultures and societies in which they live and their relative positions in terms of wealth, power, etc.