Linear Algebra and Its Applications · Problem 3.3.4

Orthogonality — Gilbert Strang

Mathematics · normal-equations-calculus

Problem

Write out

E^2 = \|Ax - b\|^2

and set to zero its derivatives with respect to

u

and

v

, if

A = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix}, \quad x = \begin{bmatrix} u \\ v \end{bmatrix}, \quad b = \begin{bmatrix} 1 \\ 3 \\ 4 \end{bmatrix}.

Compare the resulting equations with

A^T A \hat{x} = A^T b

, confirming that calculus as well as geometry gives the normal equations. Find the solution

\hat{x}

and the projection

p = A\hat{x}

. Why is

p = b