Vectors and Matrices in Machine Learning

⚡ Key Takeaways

Vectors and matrices are the fundamental data structures of machine learning: a vector is an ordered list of numbers encoding a single example or parameter set, and a matrix is a rectangular array that stores datasets, linear transformations, and learnable weights. Key result in words: nearly every ML model — from linear regression to deep neural networks — is built by repeatedly applying matrix-vector multiplication, transposition, and inner products, making linear algebra the computational backbone of the field. Where it applies: feature representation, regression, classification, dimensionality reduction, neural networks, recommendation systems. Who/when: linear algebra formalized in the 19th–20th centuries (Cayley, Sylvester); adapted to statistical learning from the 1950s onward and to modern deep learning from the 2010s.

💡 The Core Insight

Data, parameters, and transformations all live in the same algebraic language. A learning algorithm reduces to: stack your data into a matrix $X$ , store your model in a parameter vector/matrix $W$ , compute predictions via $XW$ , measure error, and adjust $W$ using gradients that are themselves matrix expressions. Because matrix multiplication composes linear maps, stacking layers is just multiplying matrices, and this single operation — generalized inner product — is the atom from which most of ML is assembled.

DefinitionVector

A vector in $\mathbb{R}^n$ is an ordered tuple

1

\quad \mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}, \qquad x_i \in \mathbb{R}.

Derivation hint: $\mathbb{R}^n$ is the $n$ -fold Cartesian product of $\mathbb{R}$ with itself, equipped with componentwise addition and scalar multiplication. Here $n$ is the dimension (number of features), $x_i$ is the $i$ -th component, and by convention vectors are columns. Each $\mathbf{x}$ represents one example, e.g. a flower described by 4 measurements gives $\mathbf{x}\in\mathbb{R}^4$ .

DefinitionMatrix

A matrix $A \in \mathbb{R}^{m \times n}$ is a rectangular array

2

\quad A = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix},

Derivation hint: a matrix is a function $\{1,\dots,m\}\times\{1,\dots,n\}\to\mathbb{R}$ , $(i,j)\mapsto a_{ij}$ , displayed as a grid. with $m$ rows and $n$ columns; $a_{ij}$ is the entry in row $i$ , column $j$ . In a design matrix $X\in\mathbb{R}^{m\times n}$ , each of the $m$ rows is one example and each of the $n$ columns is one feature.

DefinitionCore operations

The inner (dot) product, matrix-vector product, and matrix-matrix product are

3

\quad \mathbf{a}^\top \mathbf{b} = \sum_{i=1}^{n} a_i b_i, \qquad (A\mathbf{x})_i = \sum_{j=1}^{n} a_{ij} x_j, \qquad (AB)_{ik} = \sum_{j=1}^{n} a_{ij} b_{jk}.

Derivation hint: each output entry is the dot product of a row of the left operand with a column of the right operand; sizes must conform $(m\times n)(n\times p)=(m\times p)$ . The transpose $A^\top$ has entries $(A^\top)_{ij}=a_{ji}$ . The Euclidean norm is $\|\mathbf{x}\|_2=\sqrt{\mathbf{x}^\top\mathbf{x}}$ .

TheoremLinear maps are matrices

Every linear map $T:\mathbb{R}^n\to\mathbb{R}^m$ (satisfying $T(\alpha\mathbf{u}+\beta\mathbf{v})=\alpha T(\mathbf{u})+\beta T(\mathbf{v})$ ) is represented by a unique matrix $A\in\mathbb{R}^{m\times n}$ such that

4

\quad T(\mathbf{x}) = A\mathbf{x}, \qquad A = \big[\,T(\mathbf{e}_1)\;\big|\;T(\mathbf{e}_2)\;\big|\;\cdots\;\big|\;T(\mathbf{e}_n)\,\big].

Derivation hint: write $\mathbf{x}=\sum_j x_j\mathbf{e}_j$ , apply linearity, and read off $T(\mathbf{x})=\sum_j x_j T(\mathbf{e}_j)$ ; the columns of $A$ are the images of the standard basis vectors.

ProofMatrix multiplication = composition of linear maps

We show that multiplying matrices computes the composition of the maps they represent — the fact that makes stacking neural-network layers equivalent to multiplying weight matrices. 1. Let $T_B:\mathbb{R}^p\to\mathbb{R}^n$ have matrix $B\in\mathbb{R}^{n\times p}$ and $T_A:\mathbb{R}^n\to\mathbb{R}^m$ have matrix $A\in\mathbb{R}^{m\times n}$ . The composition $T_A\circ T_B:\mathbb{R}^p\to\mathbb{R}^m$ is linear (composition of linear maps is linear). 2. By [4], $T_A\circ T_B$ has a matrix $C$ whose $k$ -th column is $(T_A\circ T_B)(\mathbf{e}_k)=T_A(B\mathbf{e}_k)=A(B\mathbf{e}_k)$ . 3. Now $B\mathbf{e}_k$ is the $k$ -th column of $B$ , call it $\mathbf{b}_k$ , with entries $b_{jk}$ . Then $A\mathbf{b}_k$ has $i$ -th entry $\sum_{j=1}^n a_{ij} b_{jk}$ . 4. Therefore $C_{ik}=\sum_{j=1}^n a_{ij}b_{jk}$ , which is exactly the definition $(AB)_{ik}$ in [3]. Hence $C=AB$ and $(T_A\circ T_B)(\mathbf{x})=AB\mathbf{x}$ . 5. Consequence: a two-layer linear network $\mathbf{x}\mapsto A(B\mathbf{x})$ collapses to a single matrix $AB$ — which is why nonlinearities between layers are essential. $\blacksquare$

∎

ProofNormal equations for linear regression

We derive the closed-form least-squares solution, the canonical ML use of these objects. 1. Minimize the squared error $L(\mathbf{w})=\|X\mathbf{w}-\mathbf{y}\|_2^2$ where $X\in\mathbb{R}^{m\times n}$ , $\mathbf{y}\in\mathbb{R}^m$ . 2. Expand: $L=(X\mathbf{w}-\mathbf{y})^\top(X\mathbf{w}-\mathbf{y})=\mathbf{w}^\top X^\top X\mathbf{w}-2\mathbf{y}^\top X\mathbf{w}+\mathbf{y}^\top\mathbf{y}$ . 3. Take the gradient with respect to $\mathbf{w}$ : $\nabla_{\mathbf{w}}L=2X^\top X\mathbf{w}-2X^\top\mathbf{y}$ . 4. Set to $\mathbf 0$ : $X^\top X\mathbf{w}=X^\top\mathbf{y}$ (the normal equations). 5. If $X^\top X$ is invertible, $\mathbf{w}^\star=(X^\top X)^{-1}X^\top\mathbf{y}$ . $\blacksquare$

5

\quad \mathbf{w}^\star = (X^\top X)^{-1} X^\top \mathbf{y}.

Derivation hint: differentiate the quadratic form using $\nabla_{\mathbf w}(\mathbf w^\top M\mathbf w)=(M+M^\top)\mathbf w$ and $\nabla_{\mathbf w}(\mathbf c^\top\mathbf w)=\mathbf c$ , with $M=X^\top X$ symmetric.

∎

mathematician · Arthur Cayley | 1821–1895

Cayley introduced matrices as standalone algebraic objects in his 1858 Memoir on the Theory of Matrices, defining matrix multiplication so that it mirrored the composition of linear substitutions — the exact property exploited in deep-network layer stacking. He proved the Cayley–Hamilton theorem (a matrix satisfies its own characteristic equation), establishing matrices as a self-contained algebra rather than mere notation. His collaborator James Joseph Sylvester coined the word "matrix" (Latin for "womb") in 1850. Together they transformed array notation into the algebraic machinery that, a century later, would underpin statistical learning and neural computation.

🧠 Intuition

Think of a vector as an arrow from the origin: its components are how far you walk along each coordinate axis. A matrix is a machine that bends space — it takes every arrow and maps it to a new arrow, possibly rotating, stretching, shearing, or projecting. Crucially, this machine is fully described by what it does to the basis arrows $\mathbf{e}_1,\dots,\mathbf{e}_n$ (the columns of the matrix). In ML, the dot product $\mathbf{w}^\top\mathbf{x}$ measures alignment: it is large and positive when the data point $\mathbf{x}$ points in the same direction as the weight vector $\mathbf{w}$ , which is exactly how a linear classifier decides "yes" vs "no." A whole dataset is a cloud of points (a matrix), and learning is the act of finding directions $\mathbf{w}$ in which the cloud separates, aligns, or varies most.

Figure

A 2×2 matrix transforming the unit square: the columns of A are the images of the basis vectors e₁ and e₂.

ExamplePredicting house price with a dot product

Given: a weight vector $\mathbf{w}=[150,\;-2000,\;30000]^\top$ (price per sq-ft, per mile from city, per bedroom), bias $b=50000$ , and a house $\mathbf{x}=[1200,\;5,\;3]^\top$ (sq-ft, miles, bedrooms).

Steps: 1. Inner product (using [3]): $\mathbf{w}^\top\mathbf{x}=150(1200)+(-2000)(5)+30000(3)$ . 2. $=180000-10000+90000=260000$ . 3. Add bias: $\hat{y}=\mathbf{w}^\top\mathbf{x}+b=260000+50000$ .

Answer:

\boxed{\hat{y} = \$310{,}000}

The single dot product is the entire prediction of a linear regression model.

ExampleBatch predictions and least squares

Given: design matrix and targets

X=\begin{bmatrix}1&1\\1&2\\1&3\end{bmatrix},\quad \mathbf{y}=\begin{bmatrix}2\\2\\4\end{bmatrix}

(first column = bias, second = a feature). Fit $\mathbf{w}=[w_0,w_1]^\top$ via [5].

Steps: 1. $X^\top X=\begin{bmatrix}3&6\\6&14\end{bmatrix}$ , with $\det=3(14)-36=6$ . 2. $(X^\top X)^{-1}=\frac{1}{6}\begin{bmatrix}14&-6\\-6&3\end{bmatrix}$ . 3. $X^\top\mathbf{y}=\begin{bmatrix}1+1+1\\1+2+3\end{bmatrix}\!\cdot\!\mathbf y$ → compute: $\begin{bmatrix}2+2+4\\1(2)+2(2)+3(4)\end{bmatrix}=\begin{bmatrix}8\\18\end{bmatrix}$ . 4. $\mathbf{w}^\star=\frac{1}{6}\begin{bmatrix}14&-6\\-6&3\end{bmatrix}\begin{bmatrix}8\\18\end{bmatrix}=\frac{1}{6}\begin{bmatrix}112-108\\-48+54\end{bmatrix}=\frac{1}{6}\begin{bmatrix}4\\6\end{bmatrix}$ .

Answer:

\boxed{\;w_0=\tfrac{2}{3}\approx 0.667,\quad w_1=1\;}

The fitted line is $\hat y = 0.667 + x$ (dimensionless prediction matching $\mathbf y$ best in squared error).

⚠️ Matrix multiplication is NOT commutative

In general $AB \neq BA$ — and often one product is undefined even when the other exists (shapes must conform). Order encodes which transformation happens first. A frequent ML bug is writing $\mathbf{x}^\top W$ when you mean $W\mathbf{x}$ , flipping the meaning of rows and columns. Also: $(AB)^\top = B^\top A^\top$ (the order reverses), not $A^\top B^\top$ .

CounterexampleStacking linear layers gains nothing without nonlinearity

Consider two "layers" $\mathbf{h}=B\mathbf{x}$ then $\mathbf{z}=A\mathbf{h}$ . One might hope a deeper network is more expressive. But $\mathbf{z}=A(B\mathbf{x})=(AB)\mathbf{x}=C\mathbf{x}$ for a single matrix $C=AB$ . So any number of stacked linear layers is equivalent to one linear layer — it represents exactly the same set of functions. This is precisely why neural networks insert nonlinear activations (ReLU, sigmoid) between matrix multiplies; without them, depth is an illusion.

⚠️ Singular

X^\top X

breaks the normal equations

Equation [5] requires $X^\top X$ to be invertible, which fails when features are collinear or when $n>m$ (more features than examples). In practice one adds regularization: $\mathbf{w}^\star=(X^\top X+\lambda I)^{-1}X^\top\mathbf{y}$ (ridge regression), guaranteeing invertibility for any $\lambda>0$ , or uses the pseudoinverse / gradient descent. Never blindly invert a matrix in code — solve the linear system instead, for numerical stability.

connection

Vectors and matrices are the substrate for nearly every neighbouring topic. Eigenvalues and Eigenvectors reveal the invariant directions of a matrix and power Principal Component Analysis for dimensionality reduction. The Singular Value Decomposition factorizes any matrix and underlies recommendation systems and low-rank approximation. Gradient Descent navigates parameter space using gradient vectors, and Matrix Calculus gives the rules for differentiating the vector/matrix expressions appearing in Backpropagation. The geometry of Inner Product Spaces formalizes the dot products used by Support Vector Machines and kernel methods, while Norms and Distances define the loss functions being minimized.

🔬 Where this appears

⏳ History

- 1843 — Hamilton invents quaternions, an early non-commutative system foreshadowing matrix algebra.

- 1850 — Sylvester coins the term "matrix."

- 1858 — Cayley's Memoir on the Theory of Matrices defines matrix multiplication and the algebra of matrices.

- 1888 — Peano gives the modern axiomatic definition of a vector space.

- 1900s — Hilbert and others develop infinite-dimensional vector (Hilbert) spaces, foundational for functional analysis and later kernel methods.

- 1936 — Eckart and Young prove the low-rank approximation theorem via SVD, basis of modern data compression.

- 1958 — Rosenblatt's perceptron computes

\mathrm{sign}(\mathbf{w}^\top\mathbf{x})

, framing learning as finding a weight vector.

- 1986 — Rumelhart, Hinton & Williams popularize backpropagation, expressing learning as matrix-calculus gradient flow.

- 2012 — AlexNet exploits GPU matrix multiplication to win ImageNet, igniting the deep-learning era; matrix multiply becomes the dominant compute primitive of AI.

open-problem

The relentless growth of model sizes makes the cost of matrix multiplication a central bottleneck and open research frontier. The asymptotic complexity of multiplying two $n\times n$ matrices is still unknown: the naïve algorithm is $O(n^3)$ , Strassen (1969) achieved $O(n^{2.807})$ , and the current record (Williams et al.) is below $O(n^{2.372})$ , yet the true exponent $\omega\ge 2$ remains undetermined. Practically, these fast algorithms are numerically delicate and rarely used in ML, where hardware-optimized $O(n^3)$ kernels dominate. Parallel open questions include: how to compress weight matrices via low-rank or sparse structure without losing accuracy, how to multiply huge matrices in low precision (8-bit, 4-bit) stably, and how to schedule matrix operations across thousands of accelerators. The mathematics of which matrices ML actually needs (highly structured, low effective rank) versus the worst case studied in theory is itself an active area.

Vectors and Matrices in Machine Learning

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also