Franco Taroni - Statistics and the Evaluation of Evidence for Forensic Scientists

Judgements are required in all aspect of scientific investigation. The elicitation of probability distributions for uncertain quantities represents a challenging work for scientists and decision‐makers. O'Hagan (2019) recently wrote:

Subjective expert judgments play a part in all areas of scientific activity, and should be made with the care, rigour, and honesty that science demands. (p. 80)

A discussion can be found in Section 1.7.7.

It is not uncommon for subjective (or personal) probabilities to be considered as a synonym for arbitrariness. This is not so; the use of subjectivism does not mean the use of acquired knowledge that is often available for consideration of relative frequencies is neglected. The main source of misunderstanding is concerned with the relationship between frequencies and beliefs . The two terms are, unfortunately, often regarded as equivalent since frequency data can be used to inform probabilities (Lindley 1991) but they are not equivalent. Dawid and Galavotti (2009, p. 100) quoted de Finetti's view:

every probability evaluation essentially depends on two components: (1) the objective component, consisting of the evidence of known data and facts; and (2) the subjective component, consisting of the opinion concerning unknown facts based on known evidence.

As emphasised more recently by D'Agostini (2016)

It is a matter of fact that relative frequency and probability are somehow connected within probability theory, without the need for identifying the two concepts. (p. 13)

It is reasonable to use relative frequencies to inform measures of belief and the relationship takes the form of a mathematical theorem, de Finetti's Representation theorem . According to the theorem, the convergence of one's personal probability towards the value of observed frequencies, as the number of observations increases, is a logical consequence of Bayes' theorem if a condition called exchangeability is satisfied by the degrees of belief prior to the observations (Dawid 2004).

As an illustration of the connection between frequency and probability, consider again an urn containing a certain number of balls, indistinguishable except by their colour, which is either white or black, and the number of balls of each colour being known. The extraction of a ball from this urn defines an experiment having two and only two possible outcomes that are generally denoted as success (say, the withdrawal of a white ball) or failure (say, the withdrawal of a black ball). Let denote the event ‘a white ball is extracted’. Under the circumstances that balls are all indistinguishable from each other except for the colour, the subjective probability to extract a white ball can be assessed as the known proportion of white balls, that is, . Assuming the urn contains a large number of balls, so that the extraction of a few balls does not alter its composition substantially, individual draws (i.e. sampling 9 ) will be considered as with replacement and the probability of extracting a white ball at subsequent withdrawals will still be , independently on previous observations. In this way one realises a series of so‐called Bernoulli trials (Section A.2.1), where the outcome of each trial has a constant probability independent from previous outcomes.

Suppose now the observer does not know the absolute value of balls present, nor the proportion that are of each colour. De Finetti (1931a) showed that every series of experiments having two and only two possible outcomes that can be taken as exchangeable (i.e. the probability assigned to the outcomes of a sequence of trials is invariant to permutation) can be represented as random withdrawals from an urn of unknown composition. If one can assess one's uncertainty in such a way that labelling of the trials is not relevant, then it can be proved that as the number of observations increases the relative frequencies of successes (i.e. the relative frequency of white balls) tend to a limiting value that is the proportion of white balls. A subjective assessment about the outcome of a sequence of Bernoulli trials is equivalent to placing a prior distribution on . According to this, one only needs to model a prior distribution for the possible values that might take: personal beliefs concerning the colour of the next ball extracted can be computed as

(1.2)

The introduction of a prior probability distribution modelling personal belief about may seem, at first sight, in contradiction with statements that probability is a single number. One can have probabilities for events, or probabilities for propositions, but not probabilities of probabilities, otherwise one would have an infinite regression (de Finetti 1976). Confusion may arise from the fact that parameter is generally termed as ‘probability of success’ . However, it is worth noting that, although it is effectively a probability, it represents a chance rather than a belief .

A set of observations is said to be exchangeable – for you, given a knowledge base – if their joint distribution is invariant under permutation. A formal definition is as follows (Bernardo and Smith 2000):

The random quantities , are said to be judged exchangeable under a probability measure if the implied joint degree of belief distribution satisfies for all permutations defined on the set . (p. 169)