Yong Chen - Industrial Data Analytics for Diagnosis and Prognosis

Figure 3.1 Two bivariate normal distributions, (a) ρ = 0 (b) ρ = 0.75

Figure 3.2 Contour plots for the distributions in Figure 3.1

We list some of the most useful properties of the multivariate normal distribution. These properties make it convenient to manipulate normal distributions, which is one of the reasons for the popularity of the normal distribution. Suppose the random vector Xfollows a p -dimensional normal distribution Np ( μ,Σ).

Normality of linear combinations of the variables in X. Let c be a vector of constants. From ( 3.3) and ( 3.4), we have E(cT X) = cT μ and var(cT X) (cT Σc. This is true for any random vector X. When X follows a multivariate normal distribution, we have the additional property that cT X also follows a (univariate) normal distribution. That is, if X ∼ Np(μ,Σ, then cT X ∼ N(cT μ, cT Σc). In general, if C is a q × p matrix, CX still follows a multivariate normal distribution. From ( 3.1) and ( 3.2), we have E(CX) = Cμ and cov(CX) = CΣCT. So CX ∼ Nq(Cμ, CΣCT).

Normality of subvectors. Let X1 = (X1, X2,…, Xq) be the subvector of the first q elements of X and X2 = (Xq+1, Xq+2,…, Xp) be the subvector of the remaining p − q elements of X. From ( 3.5) and ( 3.6), μ and Σ can be partitioned as (3.10)where μi and Σii are the mean vector and covariance matrix of Xi, for i = 1, 2. If X follows a multivariate normal distribution, we have the additional property that both X1 and X2 follow a multivariate normal distribution. That is, if X ∼ Np (μ, Σ), then X1 ∼ Nq (μ1, Σ11) and X2 ∼ Np–q (μ2, Σ22). A special case of this property is that each element of X also follows a (univariate) normal distribution. That is, if X ∼ Np (μ, Σ), then Xj ∼ N(μj, σjj), j = 1, 2,…, p. The converse of this result is not true. If each element of a random vector X follows a univariate normal distribution, X may not follow a multivariate normal distribution.

Zero covariance implies independence. If X ∼ Np (μ, Σ) and , the mean vector and covariance matrix of X can be partitioned as in (3.10). The subvectors X1 and X2 are independent if and only if Σ12 = 0. Specifically, for any two elements Xi and Xj of X, Xi and Xj are independent if and only if σij = cov(Xi, Xj) = 0. Note that if Xi and Xj do not follow joint normal distribution, and Xi and Xj are independent, we still have cov(Xi, Xj) = 0. However, the converse is not necessarily true. That is, if cov(Xi, Xj) = 0, Xi and Xj may not be independent.

Conditional distributions are normal. Suppose and the mean vector and covariance matrix of X is given by (3.10). If X1 and X2 are not independent, we have Σ12 ≠ 0 and the conditional distribution of X1 given X2 = x2, is multivariate normal with (3.11) (3.12)Note that the mean vector of the conditional distribution is a linear function of x2. But the covariance matrix of the conditional distribution does not depend on x2. If X1 and X2 are independent, clearly the conditional distribution of X1 given X2 = x2 is simply Nq (μ1, Σ11), the unconditional distribution of X1.

If the population distribution is assumed to be multivariate normal with mean vector μand covariance matrix Σ. The parameters μand Σcan be estimated from a random sample of n observations x 1, x 2,…, x n. A commonly used method for parameter estimation is the maximum likelihood estimation (MLE), and the estimated parameter values are called the maximum likelihood estimates . The idea of the maximum likelihood estimation is to find μand Σthat maximize the joint density of the x’s, which is called the likelihood function . For multivariate normal distribution, the likelihood function is

(3.13)

It is often easier to find the MLE by minimizing the negative log likelihood function , which is given by

(3.14)

Taking the derivative of ( 3.14) with respect to μ, we have

(3.15)

Setting the partial derivative in ( 3.15) to zero, the MLE of μis obtained as

(3.16)

which is the sample mean vector of the data set x 1, x 2,…, x n . The derivation of the MLE of Σis more involved and beyond the scope of this book. The result is given by

(3.17)

where Sis the sample covariance matrix as given in (2.6). Since the MLE uses n instead of n – 1 in the denominator, it is a biased estimator. So the sample covariance matrix Sis more commonly used to estimate Σ, especially when n is small.

One useful property of MLE is the invariance property . In general, let denote the MLE of the parameter vector θ . Then the MLE of a function of θ , denoted by h (θ), is given by h ( ). This result makes it very convenient to find the MLE of any function of a parameter, given the MLE of the parameter. For example, based on ( 3.17), it is easy to see that the MLE of the variance of Xj , the j th element of X, is given by

Then based on the invariance property, the MLE of the standard deviation √σ jjis .

The MLE has some good asymptotic properties and usually performs well for data sets of large sample sizes. For example, under mild regularity conditions, MLE satisfies the property of consistency, which guarantees that the estimator converges to the true value of the parameter as the sample size becomes infinite. In addition, under certain regularity conditions, the MLE is asymptotically normal and efficient. That is, as the sample size becomes infinite, the distribution of MLE will converge to a normal distribution with variance equal to the optimal asymptotic variance. The details of the regularity conditions are beyond the scope of this book. But these conditions are quite general and often satisfied in common circumstances.