Inner Product Spaces and Gram-Schmidt

⚡ Key Takeaways

An inner product space is a vector space equipped with an inner product — a generalization of the dot product that defines length and angle. The Gram–Schmidt process converts any linearly independent set into an orthonormal one spanning the same space. These tools underpin orthogonal projection, least squares, Fourier analysis, and QR factorization. Core to linear algebra since Schmidt (1907) and earlier work by Gram (1883).

💡 The Core Insight

The inner product packs all of Euclidean geometry — length, angle, perpendicularity — into one bilinear (or sesquilinear) operation. Once you have orthonormality, hard problems become easy: the best approximation of a vector by a subspace is just the sum of its projections onto an orthonormal basis. Gram–Schmidt is the constructive engine that manufactures orthonormality from any basis.

DefinitionInner Product Space

Let $V$ be a vector space over the field $\mathbb{F}$ (either $\mathbb{R}$ or $\mathbb{C}$ ). An inner product is a map $\langle\,\cdot\,,\,\cdot\,\rangle : V \times V \to \mathbb{F}$ such that for all $u,v,w \in V$ and $\alpha \in \mathbb{F}$ :

1

\quad \langle u, v\rangle = \overline{\langle v, u\rangle}

Derivation hint: conjugate symmetry; over $\mathbb{R}$ the conjugation is trivial, giving plain symmetry $\langle u,v\rangle = \langle v,u\rangle$ .

2

\quad \langle \alpha u + w, v\rangle = \alpha\langle u, v\rangle + \langle w, v\rangle

Derivation hint: linearity in the first argument; combined with [1] this forces conjugate-linearity in the second argument.

3

\quad \langle v, v\rangle \ge 0, \quad \text{with} \quad \langle v, v\rangle = 0 \iff v = 0

Derivation hint: positive-definiteness; guarantees the norm below is well-defined and nondegenerate. The pair $(V, \langle\cdot,\cdot\rangle)$ is an inner product space. The induced norm is

4

\quad \|v\| = \sqrt{\langle v, v\rangle}

Derivation hint: take the nonnegative square root, valid by [3]; this measures the "length" of $v$ . Here $\overline{\,\cdot\,}$ denotes complex conjugation, $\alpha$ a scalar, and $0$ the zero vector.

DefinitionOrthogonality and Orthonormality

Two vectors $u, v$ are orthogonal, written $u \perp v$ , if $\langle u, v\rangle = 0$ . A set $\{e_1, \dots, e_n\}$ is orthonormal if

5

\quad \langle e_i, e_j\rangle = \delta_{ij} = \begin{cases} 1 & i = j \\ 0 & i \ne j \end{cases}

Derivation hint: each vector is a unit vector ( $\|e_i\|=1$ ) and mutually perpendicular; $\delta_{ij}$ is the Kronecker delta.

TheoremCauchy–Schwarz Inequality

For all $u, v$ in an inner product space,

6

\quad |\langle u, v\rangle| \le \|u\|\,\|v\|

Derivation hint: minimize $\|u - tv\|^2 \ge 0$ over the scalar $t$ ; equality holds iff $u,v$ are linearly dependent. This guarantees that $\cos\theta = \dfrac{\langle u,v\rangle}{\|u\|\|v\|}$ lies in $[-1,1]$ (real case), so the angle between vectors is well defined.

TheoremTriangle Inequality / Norm Axioms

The induced norm satisfies $\|u+v\| \le \|u\| + \|v\|$ .

7

\quad \|u+v\|^2 = \|u\|^2 + 2\,\mathrm{Re}\langle u,v\rangle + \|v\|^2 \le \big(\|u\|+\|v\|\big)^2

Derivation hint: expand $\langle u+v, u+v\rangle$ using [1]–[2], then bound the cross term with Cauchy–Schwarz [6].

TheoremGram–Schmidt Orthogonalization

Let $\{v_1, \dots, v_n\}$ be a linearly independent set in an inner product space $V$ . Define recursively

8

\quad u_k = v_k - \sum_{j=1}^{k-1} \frac{\langle v_k, u_j\rangle}{\langle u_j, u_j\rangle}\, u_j, \qquad e_k = \frac{u_k}{\|u_k\|}

Derivation hint: subtract from $v_k$ its projections onto all previously built orthogonal directions $u_1,\dots,u_{k-1}$ , leaving a remainder orthogonal to them; then normalize. Then $\{e_1, \dots, e_n\}$ is orthonormal and for every $k$ ,

9

\quad \mathrm{span}\{e_1, \dots, e_k\} = \mathrm{span}\{v_1, \dots, v_k\}

Derivation hint: each $u_k$ is a combination of $v_1,\dots,v_k$ with coefficient $1$ on $v_k$ , so the spans match at every stage (induction).

ProofGram–Schmidt produces an orthonormal set

We prove orthogonality of $\{u_1, \dots, u_n\}$ by induction on $k$ ; normalization is then immediate.

Step 1 — Base case. Set $u_1 = v_1$ . Since $\{v_i\}$ is independent, $v_1 \ne 0$ , so $u_1 \ne 0$ and $e_1 = u_1/\|u_1\|$ is a unit vector. A single-element set is trivially orthogonal.

Step 2 — Inductive hypothesis. Suppose $u_1, \dots, u_{k-1}$ are nonzero and mutually orthogonal: $\langle u_i, u_j\rangle = 0$ for $i \ne j < k$ .

Step 3 — Orthogonality of $u_k$ to each $u_m$ , $m < k$ . Take the inner product of [8] with $u_m$ :

10

\quad \langle u_k, u_m\rangle = \langle v_k, u_m\rangle - \sum_{j=1}^{k-1}\frac{\langle v_k, u_j\rangle}{\langle u_j, u_j\rangle}\langle u_j, u_m\rangle

Derivation hint: apply linearity in the first slot [2] term by term. By the inductive hypothesis $\langle u_j, u_m\rangle = 0$ unless $j = m$ , so only the $j=m$ term survives:

11

\quad \langle u_k, u_m\rangle = \langle v_k, u_m\rangle - \frac{\langle v_k, u_m\rangle}{\langle u_m, u_m\rangle}\langle u_m, u_m\rangle = 0

Derivation hint: the single surviving fraction cancels exactly, forcing orthogonality.

Step 4 — $u_k \ne 0$ . If $u_k = 0$ , then [8] expresses $v_k$ as a linear combination of $u_1,\dots,u_{k-1}$ , which (by the inductive span equality) are combinations of $v_1,\dots,v_{k-1}$ . This contradicts linear independence of $\{v_i\}$ . Hence $u_k \ne 0$ and $e_k$ is well defined.

Step 5 — Span preservation. Equation [8] shows $u_k \in \mathrm{span}\{v_1,\dots,v_k\}$ and conversely $v_k \in \mathrm{span}\{u_1,\dots,u_k\}$ . By induction the spans of the first $k$ vectors coincide, proving [9]. $\blacksquare$

∎

ES

Erhard Schmidt

1876–1959

A German mathematician and student of David Hilbert, Schmidt formalized the orthogonalization procedure in his 1907 work on integral equations, where infinite-dimensional inner product spaces (later named Hilbert spaces) demanded systematic orthonormal bases. Though Jørgen Pedersen Gram described the essential idea in 1883 and Laplace used related techniques even earlier, Schmidt's clean recursive formulation became the textbook standard. His broader contributions to functional analysis — including the spectral theory of compact symmetric operators — made orthonormal expansions a cornerstone of modern mathematics and quantum mechanics.

🧠 Intuition

Think of each new vector $v_k$ as having a "shadow" cast onto the subspace already spanned. The projection term $\sum \frac{\langle v_k, u_j\rangle}{\langle u_j, u_j\rangle}u_j$ is that shadow. Subtracting it leaves only the part of $v_k$ that points in a genuinely new direction — perpendicular to everything before it. Geometrically, you are repeatedly "straightening" vectors: take a tilted vector, remove the component lying in the old plane, and what remains sticks straight out. Normalizing just rescales each new direction to unit length. The reason orthonormal bases are so prized: in such a basis, the coordinate of any vector $x$ along $e_i$ is simply $\langle x, e_i\rangle$ — no system of equations to solve, just inner products.

Figure

Gram–Schmidt: subtracting the projection of

v_2

onto

u_1

leaves

u_2 \perp u_1

ExampleGram–Schmidt in

\mathbb{R}^3

with the standard dot product

Given: $v_1 = (1,1,0)$ , $v_2 = (1,0,1)$ , $v_3 = (0,1,1)$ , with $\langle x,y\rangle = x\cdot y$ .

Step 1. $u_1 = v_1 = (1,1,0)$ , $\|u_1\|^2 = 2$ .

Step 2. Compute $\langle v_2, u_1\rangle = 1\cdot1 + 0\cdot1 + 1\cdot0 = 1$ .

u_2 = (1,0,1) - \tfrac{1}{2}(1,1,0) = \left(\tfrac12, -\tfrac12, 1\right), \quad \|u_2\|^2 = \tfrac14+\tfrac14+1 = \tfrac32.

Step 3. $\langle v_3, u_1\rangle = 0\cdot1+1\cdot1+1\cdot0 = 1$ ; $\langle v_3, u_2\rangle = 0\cdot\tfrac12 + 1\cdot(-\tfrac12) + 1\cdot1 = \tfrac12$ .

u_3 = (0,1,1) - \tfrac{1}{2}(1,1,0) - \frac{1/2}{3/2}\left(\tfrac12,-\tfrac12,1\right) = (0,1,1) - (\tfrac12,\tfrac12,0) - \tfrac13(\tfrac12,-\tfrac12,1).

u_3 = \left(-\tfrac12-\tfrac16,\; \tfrac12+\tfrac16,\; 1-\tfrac13\right) = \left(-\tfrac23, \tfrac23, \tfrac23\right).

Normalize:

\boxed{e_1 = \tfrac{1}{\sqrt2}(1,1,0),\quad e_2 = \tfrac{1}{\sqrt6}(1,-1,2),\quad e_3 = \tfrac{1}{\sqrt3}(-1,1,1)}

(Check: each $\|e_i\|=1$ and pairwise dot products are $0$ . Units are dimensionless.)

ExampleLegendre polynomials via Gram–Schmidt on functions

Given: $V =$ polynomials on $[-1,1]$ with inner product $\langle f,g\rangle = \int_{-1}^{1} f(x)g(x)\,dx$ , applied to $v_0=1,\ v_1=x,\ v_2=x^2$ .

Step 1. $u_0 = 1$ , $\langle u_0,u_0\rangle = \int_{-1}^1 1\,dx = 2$ .

Step 2. $\langle x, 1\rangle = \int_{-1}^1 x\,dx = 0$ , so $u_1 = x$ (already orthogonal to constants). $\langle u_1,u_1\rangle = \int_{-1}^1 x^2 dx = \tfrac23$ .

Step 3. $\langle x^2, 1\rangle = \int_{-1}^1 x^2 dx = \tfrac23$ ; $\langle x^2, x\rangle = \int_{-1}^1 x^3 dx = 0$ .

u_2 = x^2 - \frac{2/3}{2}\cdot 1 - 0\cdot x = x^2 - \tfrac13.

These are (up to normalization) the Legendre polynomials $P_0, P_1, P_2$ :

\boxed{u_0 = 1,\quad u_1 = x,\quad u_2 = x^2 - \tfrac13}

The standard scaling $P_2(x) = \tfrac12(3x^2-1)$ is just $\tfrac32 u_2$ . (Quantities are dimensionless functions.)

⚠️ Common mistake

Order matters, and so does normalization timing. Gram–Schmidt depends on the ordering of the input vectors — different orderings yield different orthonormal bases. Also, in formula [8] the projections use the unnormalized $u_j$ with denominator $\langle u_j, u_j\rangle$ . If you normalize first to get $e_j$ , the formula becomes $u_k = v_k - \sum \langle v_k, e_j\rangle e_j$ (denominator $1$ ). Mixing the two — using $e_j$ in the numerator but $u_j$ in the denominator — is a frequent algebra error.

⚠️ Common mistake

A bilinear form is not automatically an inner product. Positive-definiteness [3] is essential. The Minkowski form $\langle x,y\rangle = x_1y_1 + x_2y_2 + x_3y_3 - x_4y_4$ of special relativity is symmetric and bilinear but indefinite — there exist nonzero null vectors with $\langle v,v\rangle = 0$ . Such spaces are pseudo-inner-product spaces, and Gram–Schmidt can break down (a $u_k$ may be nonzero yet have $\langle u_k, u_k\rangle = 0$ , making normalization impossible).

CounterexampleGram–Schmidt fails on linearly dependent input

Take $v_1 = (1,0)$ , $v_2 = (2,0)$ in $\mathbb{R}^2$ . Then $u_1 = (1,0)$ , and

u_2 = (2,0) - \frac{\langle(2,0),(1,0)\rangle}{1}(1,0) = (2,0) - 2(1,0) = (0,0).

The remainder is the zero vector, so $e_2 = u_2/\|u_2\|$ is undefined (division by zero). This is not a flaw: it correctly detects that $v_2$ added no new direction. The theorem's linear-independence hypothesis is exactly what rules this out. In practice, numerical Gram–Schmidt detects near-dependence as catastrophic loss of precision — see Modified Gram–Schmidt below.

connection

Inner products generalize the dot product from Euclidean Space to abstract and infinite-dimensional settings. The norm [4] makes every inner product space a special case of a Normed Vector Space, and when complete it becomes a Hilbert Space. Orthonormal bases produced by Gram–Schmidt are the discrete analogue of Fourier Series expansions, where $\{e^{inx}\}$ is orthonormal under the $L^2$ inner product. The process is computationally equivalent to QR Decomposition: arranging the $v_k$ as columns of $A$ , the $e_k$ form $Q$ and the projection coefficients form the upper-triangular $R$ . Finally, Orthogonal Projection onto a subspace is the conceptual heart of Least Squares Approximation.

🔬 Where this appears

⏳ History

- 1822 — Joseph Fourier publishes Théorie analytique de la chaleur, expanding functions in orthogonal trigonometric series, implicitly using orthogonality of sines and cosines.

- 1828 — Pierre-Simon Laplace uses orthogonalization-like procedures in least-squares and celestial mechanics.

- 1859 — Pafnuty Chebyshev develops orthogonal polynomials for approximation theory.

- 1883 — Jørgen Pedersen Gram, in a paper on least squares, gives the orthogonalization procedure in terms of determinants (the Gram determinant).

- 1907 — Erhard Schmidt presents the clean recursive algorithm while studying integral equations under Hilbert, giving the process its modern form and name.

- 1912 — David Hilbert and Schmidt establish the theory of what becomes Hilbert spaces, where orthonormal expansions are foundational.

- 1927 — John von Neumann axiomatizes Hilbert spaces for quantum mechanics, embedding inner products at the core of physics.

- 1958 — Modern numerical analysis (Householder, Givens) introduces more stable orthogonalization methods, exposing classical Gram–Schmidt's instability.

- 1966 — Å. Björck analyzes the superior numerical stability of Modified Gram–Schmidt, now standard in computation.

open-problem

Numerical instability and infinite dimensions. The classical Gram–Schmidt algorithm [8] is numerically unstable: rounding errors cause the computed $e_k$ to lose orthogonality, especially when input vectors are nearly dependent. The Modified Gram–Schmidt rearrangement — projecting out each $u_j$ from all remaining vectors immediately, rather than accumulating projections — is mathematically identical but far more stable, though even it degrades; Householder reflections are often preferred. In infinite-dimensional Hilbert spaces, Gram–Schmidt extends to countable bases (yielding e.g. orthogonal polynomials), but convergence and completeness of the resulting orthonormal system require careful analysis (Parseval's identity, completeness of $\{e_n\}$ ). Whether a given orthonormal sequence forms a complete basis is a genuinely nontrivial question depending on the space.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also