Franco Taroni - Statistics and the Evaluation of Evidence for Forensic Scientists

Practically, consider the following hypothetical case example. A laboratory receives a consignment of discrete items whose attributes may be relevant within the context of a criminal investigation. The laboratory is requested to conduct analyses in order to gather information that should allow an inference to be drawn, for example about the proportion of items in the consignment that are of a certain kind (e.g. counterfeit products). The term ‘positive’ is used here to refer to the presence of an item's property that is of interest (e.g. counterfeit); otherwise the result of the analysis is termed ‘negative’. This allows the introduction of a random variable that takes the value 1 (i.e. success) if the analysed unit is positive and 0 (i.e. failure) otherwise. This is a generic type of case that applies well to many situations, such as surveys or, more generally, sampling procedures conducted to infer the proportion of individuals or items in a population who share a given property or possess certain characteristics (e.g. that of being counterfeit). Suppose now that units are analysed, so that there are possible outcomes. The forensic scientist should be able to assign a probability to each of the 1024 possible outcomes. At this point, if it was reasonable to assume that only the observed values matter and not the order in which they appear, the forensic scientist would have a sensibly simplified task. In fact, the total number of probability assignments would reduce from 1024 to 11, since it is assumed that all sequences are assigned the same probability if they have the same number of 1's, (i.e. successes). This is possible if it is thought that all the items are indistinguishable in the sense that it does not matter which particular item produced a success (e.g. a positive response) or a failure (e.g. a negative response). Stated otherwise, this means that one's probability assignment is invariant under changes in the order of successes and failures. If the outcomes were permuted in any way, assigned probabilities would be unchanged. For a coin‐tossing experiment, Lindley (2014) has expressed this as follows:

One way of expressing this is to say that any one toss, with its resulting outcome, may be exchanged for any other with the same outcome, in the sense that the exchange will not alter your belief, expressing the idea that the tosses were done under conditions that you feel were identical. (p. 148)

The role of exchangeability in the reconciliation of subjective probabilities and frequencies in forensic science is developed in Taroni et al. (2018). It is possible to give relative frequency an explicit role in probability assignments but this does not mean that probabilities can only be given when relative frequencies are available.

The existence of relative frequencies is not a necessary condition for the assignment of probabilities. Typically, relative frequencies are not available in the case of single (not replicable) events. Other methods of elicitation, such as scoring rules, can be implemented to deal with such situations. An extended discussion on elicitation is given by O'Hagan et al. (2006).

The use of scores for the assessment of forecasts is described in DeGroot and Fienberg (1983). The association of scores for the assessment of forecasts and the use of scores for the assessment of the performance of methods for evidence evaluation will be made clear later in Section 8.4.3. A score is used to evaluate and compare forecasters who present their predictions of whether or not an event will occur as a subjective probability of the occurrence of that event. A common use for forecasts is that of weather from one day to the next. Let denote a forecaster's prediction of rain on the following day. Let be the forecaster's actual subjective probability of rain for that day. Let an arbitrary function be the forecaster's score if rain occurs and let another arbitrary function be their score if rain does not occur. With an assumption that the forecaster wishes to maximise their score, assume that is an increasing function of and is a decreasing function of . For a prediction of and an actual subjective probability of , the expected score of the forecaster is

(1.3)

A proper scoring rule is one for which ( 1.3) is maximised when . A strictly proper scoring rule is one for which is the only value of that maximises ( 1.3).

One of the earliest scoring rules, proposed for meteorological forecasts, is the quadratic scoring rule (Brier 1950). This score has the property that the forecaster will minimise their subjective expected Brier score on any particular day with a stated prediction of their actual subjective probability of rain on that day. The expected Brier score is then

(1.4)