Daniel J. Denis - Applied Univariate, Bivariate, and Multivariate Statistics

More generally, if T is some statistic, then we can use T as an estimator of a population parameter θ . Whether the estimator T is any gooddepends on several criteria, which we survey now.

On average, in the long run, the statistic T is considered to be an unbiased estimatorof θ if

That is, an estimator is considered unbiased if its expected value is equal to that of the parameter it is seeking to estimate. The biasof an estimator is measured by how much E ( T ) deviates from θ . When an estimator is biased, then E ( T ) ≠ θ , or, we can say E ( T ) − θ ≠ 0. Since the bias will be a positive number, we can express this last statement as E ( T ) − θ > 0.

Good estimators are, in general, unbiased. The most popular example of an unbiased estimator is that of the arithmetic sample mean since it can be shown that:

An example of an estimator that is biased is the uncorrected sample variance, as we will soon discuss, since it can be shown that

However, S 2is not asymptoticallybiased. As sample size increases without bound, E ( S 2) converges to σ 2. Once the sample variance is corrected via the following, it leads to an unbiased estimator, even for smaller samples:

where now,

Consistency 6 of an estimator means that as sample size increases indefinitely, the variance of the estimator approaches zero. That is, as n → ∞. We could also write this using a limit concept:

which reads “the variance of the estimator T as sample size n goes to infinity (grows without bound) is equal to 0.” Fisher called this the criterion of consistency, informally defining it as “when applied to the whole population the derived statistic should be equal to the parameter” (Fisher, 1922a, p. 316). The key to Fisher's definition is whole population, which means, theoretically at least, an infinitely large sample, or analogously, n → ∞. More pragmatically, when we have the entire population.

An estimator is regarded as efficientthe lower is its mean squared error. Estimators with lower variance are more efficient than estimators with higher variance. Fisher called this the criterion of efficiency, writing “when the distributions of the statistics tend to normality, that statistic is to be chosen which has the least probable error” (Fisher, 1922a, p. 316). Efficient estimators are generally preferred over less efficient ones.

An estimator is regarded as sufficientfor a given parameter if the statistic “captures” everything we need to know about the parameter and our knowledge of the parameter could not be improved if we considered additional information (such as a secondary statistic) over and above the sufficient estimator. As Fisher (1922a, p. 316) described it, “the statistic chosen should summarize the whole of the relevant information supplied by the sample.” More specifically, Fisher went on to say:

If θ be the parameter to be estimated, θ 1a statistic which contains the whole of the information as to the value of θ , which the sample supplies, and θ 2any other statistic, then the surface of distribution of pairs of values of θ 1and θ 2, for a given value of θ , is such that for a given value of θ 1, the distribution of θ 2does not involve θ . In other words, when θ 1is known, knowledge of the value of θ 2throws no further light upon the value of θ .

(Fisher, 1922a, pp. 316–317)

Returning to our discussion of moments, the varianceis the second moment of a distribution. For the discrete case, variance is defined as:

while for the continuous case,

Since E ( y i) = μ , it stands that we may also write E ( y i) as μ . We can also express σ 2as since, when we distribute expectations, we obtain:

Recall that the uncorrected and biasedsample variance is given by:

As earlier noted, taking the expectation of S 2, we find that E ( S 2) ≠ σ 2. The actual expectation of S 2is equal to:

which implies the degree to which S 2is biased is equal to:

We have said that S 2is biased, but you may have noticed that as n increases, ( n − 1)/ n approaches 1, and so E ( S 2) will equal σ 2as n increases without bound. This was our basis for earlier writing . That is, we say that the estimator S 2, though biased for small samples, is asymptotically unbiasedbecause its expectation is equal to σ 2as n → ∞.