Douglas C. Montgomery - Introduction to Linear Regression Analysis

The least-squares estimators and have several important properties. First, note from Eqs. (2.6)and (2.7)that and are linear combinationsof the observations yi . For example,

where for i = 1, 2, …, n .

The least-squares estimators and are unbiased estimatorsof the model parameters β 0and β 1. To show this for , consider

since E ( εi ) = 0 by assumption. Now we can show directly that and , so

That is, if we assume that the model is correct [ E ( yi ) = β 0+ β 1 xi ], then is an unbiased estimator of β 1. Similarly we may show that of is an unbiased estimator of β 0, or

The variance of is found as

(2.13)

because the observations yi are uncorrelated, and so the variance of the sum is just the sum of the variances. The variance of each term in the sum is , and we have assumed that Var( yi ) = σ 2; consequently,

(2.14)

The variance of is

Now the variance of is just , and the covariance between and can be shown to be zero (see Problem 2.25). Thus,

(2.15)

Another important result concerning the quality of the least-squares estimators and is the Gauss-Markov theorem, which states that for the regression model (2.1)with the assumptions E ( ε ) = 0, Var( ε ) = σ 2, and uncorrelated errors, the least-squares estimators are unbiased and have minimum variance when compared with all other unbiased estimators that are linear combinations of the yi . We often say that the least-squares estimators are best linear unbiased estimators, where “best” implies minimum variance. Appendix C.4proves the Gauss-Markov theorem for the more general multiple linear regression situation, of which simple linear regression is a special case.

There are several other useful properties of the least-squares fit:

1 The sum of the residuals in any regression model that contains an intercept β0 is always zero, that is,This property follows directly from the first normal equation in Eqs. (2.5)and is demonstrated in Table 2.2 for the residuals from Example 2.1. Rounding errors may affect the sum.

2 The sum of the observed values yi equals the sum of the fitted values , orTable 2.2 demonstrates this result for Example 2.1.

3 The least-squares regression line always passes through the centroid [the point ] of the data.

4 The sum of the residuals weighted by the corresponding value of the regressor variable always equals zero, that is,

5 The sum of the residuals weighted by the corresponding fitted value always equals zero, that is,

In addition to estimating β 0and β 1, an estimate of σ 2is required to test hypotheses and construct interval estimates pertinent to the regression model. Ideally we would like this estimate not to depend on the adequacy of the fitted model. This is only possible when there are several observations on y for at least one value of x (see Section 4.5) or when prior information concerning σ 2is available. When this approach cannot be used, the estimate of σ 2is obtained from the residualor error sum of squares,