Daniel J. Denis - Applied Univariate, Bivariate, and Multivariate Statistics

To understand why should be true, consider first how the sample mean is defined:

Incorporating this into the expectation for , we have:

There is a rule of expectations that says that the expectation of the sum of random variables is equal to the sum of individual expectations. This being the case, we can write the expectation of the sample mean as:

Since the expectation of each y 1through y nis E ( y 1) = μ , E ( y 2) = μ , … E ( y n) = μ , we can write

We note that the n values in numerator and denominator cancel, and so we end up with

Using the fact that E ( y i) = μ , we can also say that the expected value of a sampling distribution of the mean is equal to the mean of the population from which we did the theoretical sampling. That is, is true, since given , it stands that if we have say, five sample means , the expectation of each of these means should be equal to μ , from which we can easily deduce . That is, the mean of all the samples we could draw is equal to the population mean.

We now need a measure of the dispersionof a sampling distribution of the mean. At first glance, it may seem reasonable to assume that the variance of the sampling distribution of means should equal the variance of the population from which the sample means were drawn. However, this is not the case. What is true is that the variance of the sampling distribution of means will be equal to only a fractionof the population variance. It will be equal to of it, where n is equal to the size of samples we are collecting for each sample mean. Hence, the variance of means of the sampling distribution is equal to

or simply,

The mathematical proof of this statistical fact is in most mathematical statistics texts. A version of the proof can also be found in Hays (1994). The idea, however, can be easily and perhaps even more intuitively understood by recourse to what happens as n changes. We consider first the most trivial and unrealistic of examples to strongly demonstrate the point. Suppose that we calculate the sample mean from a sample size of n = 1, sampled from a population with μ = 10.0 and σ 2= 2.0. Suppose the sample mean we obtain is equal to 4.0. Therefore, the sampling variance of the corresponding sampling distribution is equal to:

That is, the variance in means that you can expect to see if you sampled an infinite number of means based on samples of size n = 1 repeatedly from this population is equal to 2. Notice that 2 is exactly equal to the original population variance. In this case, the variance in means is based on only a single data point.

Consider now the case where n > 1. Suppose we now sampled a mean from the population based on sample size n = 2, yielding

What has happened? What has happened is that the variance in sample means has decreased by 1/2 of the original population variance (i.e., 1/2 of 2 is 1). Why is this decrease reasonable? It makes sense, because we already know from the law of large numbersthat as the sample size grows larger, one gets closer and closer to the true probability in estimating a parameter. That is, for a consistent estimator, our estimate of the true population mean (i.e., the expectation) should get better and better as sample size increases. This is exactly what happens as we increase n , our precisionof that which is being estimated increases. In other words, the sampling variance of the estimator decreases. It's less variable, it doesn't “bounce around as much” on average from sample to sample.

Analogous to how we defined the standard deviation as the square root of the variance, it is also useful to take the square root of the variance of means:

which we call the standard error of the mean, σ M. The standard error of the mean is the standard deviation of the sampling distribution of the mean. Lastly, it is important to recognize that is not “the” standard error. It is merely the standard error of the mean. Other statistics will have different SEs.

It is not an exaggeration to say that the central limit theorem, in one form or another, is probably the most important and relevant theorem in theoretical statistics, which translates to it being quite relevant to applied statistics as well.