Linear Algebra and Its Applications · Problem 3.3.29

Orthogonality — Gilbert Strang

Mathematics · statistics-covariance-matrix

Problem

Second assumption behind least squares: The

m

errors

e_i

are independent with variance

\sigma^2

, so the average of

(b - Ax)(b - Ax)^T

\sigma^2 I

. Multiply on the left by

(A^T A)^{-1} A^T

and on the right by

A(A^T A)^{-1}

to show that the average of

(\hat{x} - x)(\hat{x} - x)^T

\sigma^2 (A^T A)^{-1}

. This is the all-important covariance matrix for the error in

\hat{x}