The Central Limit Theorem in ML

⚡ Key Takeaways

One-line statement: The sum (or average) of many independent random variables, each with finite variance, converges in distribution to a normal (Gaussian) distribution — regardless of the underlying distributions. In words: Averages tend to be bell-shaped, no matter what you average. Where it applies in ML: sampling-based estimators, mini-batch gradients, bootstrap confidence intervals, error analysis, weight initialization, and the justification for Gaussian noise models. Who/when: de Moivre (1733), Laplace (1810), Lyapunov (1901), and Lindeberg–Lévy (1920s).

💡 The Core Insight

Adding up many small independent random contributions averages out their idiosyncrasies. The fluctuations that survive are exactly the ones a Gaussian describes, because the Gaussian is the unique attractor (fixed point) under the operation "add independent copies and rescale." The CLT is the mathematical statement that this attractor is reached, and it explains why estimator noise in ML shrinks like $\sigma/\sqrt{n}$ — the single most important rate in statistical learning.

DefinitionConvergence in Distribution

A sequence of random variables $Z_n$ converges in distribution to a random variable $Z$ , written $Z_n \xrightarrow{d} Z$ , if their cumulative distribution functions satisfy

1

\quad \lim_{n\to\infty} F_{Z_n}(z) = F_Z(z) \quad \text{at every continuity point } z \text{ of } F_Z.

Derivation hint: This is the weakest standard mode of stochastic convergence; it concerns the shape of the distribution, not pointwise values of the variables. The continuity-point restriction handles jumps in $F_Z$ . Here $F_{Z_n}(z) = \Pr(Z_n \le z)$ is the CDF. The CLT is a statement about this mode of convergence.

DefinitionThe Standardized Sample Mean

Let $X_1, \dots, X_n$ be i.i.d. with $\mathbb{E}[X_i] = \mu$ and $\operatorname{Var}(X_i) = \sigma^2 \in (0, \infty)$ . Define the sample mean and its standardized form:

2

\quad \bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i, \qquad Z_n = \frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} = \frac{1}{\sigma\sqrt{n}}\sum_{i=1}^n (X_i - \mu).

Derivation hint: Subtract the mean ( $\mathbb{E}[\bar X_n]=\mu$ ) and divide by the standard deviation of $\bar X_n$ , which is $\sigma/\sqrt n$ since $\operatorname{Var}(\bar X_n)=\sigma^2/n$ for independent terms. The factor $\sqrt{n}$ is the precise scaling that keeps $Z_n$ from collapsing to $0$ (which $\bar X_n - \mu$ does) or exploding (which $\sum(X_i-\mu)$ does).

TheoremLindeberg–Lévy Central Limit Theorem

Let $X_1, X_2, \dots$ be i.i.d. random variables with finite mean $\mu$ and finite, nonzero variance $\sigma^2$ . Then

3

\quad Z_n = \frac{\sqrt{n}\,(\bar{X}_n - \mu)}{\sigma} \xrightarrow{d} \mathcal{N}(0,1) \quad \text{as } n \to \infty,

Derivation hint: Show the characteristic function of $Z_n$ converges to $e^{-t^2/2}$ , the characteristic function of $\mathcal N(0,1)$ , then invoke Lévy's continuity theorem. equivalently $\Pr(Z_n \le z) \to \Phi(z) = \int_{-\infty}^z \frac{1}{\sqrt{2\pi}} e^{-u^2/2}\,du$ for all $z$ . No assumption on the shape of the $X_i$ distribution is needed beyond finite variance.

TheoremLyapunov CLT — dropping identical distribution

Let $X_1, \dots, X_n$ be independent (not necessarily identically distributed) with means $\mu_i$ , variances $\sigma_i^2$ , and set $s_n^2 = \sum_{i=1}^n \sigma_i^2$ . If for some $\delta > 0$ the Lyapunov condition holds,

4

\quad \lim_{n\to\infty}\frac{1}{s_n^{2+\delta}}\sum_{i=1}^n \mathbb{E}\!\left[|X_i - \mu_i|^{2+\delta}\right] = 0,

Derivation hint: This condition forces no single term to dominate the sum; it controls the $(2+\delta)$ -th moments relative to the total variance, which is what makes the Lindeberg argument go through. then $\frac{1}{s_n}\sum_{i=1}^n (X_i - \mu_i) \xrightarrow{d} \mathcal{N}(0,1)$ . This is the version relevant to heterogeneous ML data (e.g., summing differently-scaled features or non-i.i.d. gradient components).

ProofCharacteristic-Function Proof of Lindeberg–Lévy

Assume WLOG $\mu = 0$ (otherwise replace $X_i$ by $X_i - \mu$ ) and $\sigma^2$ finite. Let $Y_i = X_i/\sigma$ , so $\mathbb{E}[Y_i]=0$ , $\operatorname{Var}(Y_i)=1$ . Then $Z_n = \frac{1}{\sqrt n}\sum_{i=1}^n Y_i$ .

Step 1 — Characteristic function of one term. The characteristic function is $\varphi(t) = \mathbb{E}[e^{itY_1}]$ . Since $\mathbb{E}[Y_1^2]<\infty$ , $\varphi$ is twice differentiable at $0$ with a Taylor expansion

5

\quad \varphi(t) = 1 + it\,\mathbb{E}[Y_1] - \tfrac{t^2}{2}\mathbb{E}[Y_1^2] + o(t^2) = 1 - \tfrac{t^2}{2} + o(t^2).

Derivation hint: Differentiate under the expectation: $\varphi'(0)=i\mathbb E[Y_1]=0$ , $\varphi''(0)=-\mathbb E[Y_1^2]=-1$ . The $o(t^2)$ remainder is justified by finite second moment.

Step 2 — Characteristic function of $Z_n$ . By independence the characteristic function of a sum factorizes, and scaling by $1/\sqrt n$ rescales the argument:

6

\quad \varphi_{Z_n}(t) = \mathbb{E}\!\left[\exp\!\left(\tfrac{it}{\sqrt n}\textstyle\sum_i Y_i\right)\right] = \left[\varphi\!\left(\tfrac{t}{\sqrt n}\right)\right]^n.

Derivation hint: $\mathbb E[\prod_i e^{i s Y_i}] = \prod_i \mathbb E[e^{i s Y_i}]$ for independent $Y_i$ , with $s = t/\sqrt n$ identical across terms by i.i.d.

Step 3 — Take the limit. Substitute the expansion at $t/\sqrt n$ :

7

\quad \varphi_{Z_n}(t) = \left[1 - \frac{t^2}{2n} + o\!\left(\tfrac{1}{n}\right)\right]^n \xrightarrow[n\to\infty]{} e^{-t^2/2}.

Derivation hint: Use $\lim_{n\to\infty}(1 + a_n/n)^n = e^{a}$ when $a_n \to a$ ; here $a_n \to -t^2/2$ and the $o(1/n)$ term vanishes after multiplication by $n$ .

Step 4 — Conclude. The limit $e^{-t^2/2}$ is the characteristic function of $\mathcal{N}(0,1)$ . By Lévy's continuity theorem, pointwise convergence of characteristic functions to a function continuous at $0$ implies convergence in distribution. Hence $Z_n \xrightarrow{d} \mathcal{N}(0,1)$ . $\blacksquare$

∎

mathematician · Pierre-Simon Laplace | 1749–1827

Laplace gave the first general (if non-rigorous) statement and proof of the CLT in his 1810 memoir, generalizing de Moivre's 1733 normal approximation to the binomial. He developed the characteristic-function method ("Laplace transform" of distributions) that, made rigorous by Lyapunov and Lévy a century later, became the standard modern proof. Laplace's central interest was error theory: he wanted to know why the average of many astronomical measurements was reliable — exactly the question a machine learning practitioner asks of a validation estimate today.

🧠 Intuition

Think of each $X_i - \mu$ as a tiny random "kick." Summing $n$ kicks, the positive and negative ones partly cancel; what remains is a random walk of typical size $\sqrt n$ (not $n$ , because of cancellation — this is the diffusive scaling). Dividing by $\sqrt n$ in $Z_n$ exactly cancels this growth and exposes a stable shape. Why is that shape Gaussian and nothing else? Because the Gaussian is the only distribution (with finite variance) that reproduces itself when you add independent copies and rescale — it is the fixed point of the "convolve-and-shrink" operation. The CLT says every finite-variance distribution flows toward this fixed point. In ML terms: it does not matter whether your per-example loss is heavy on one side, bimodal, or discrete; once you average enough examples, the average's uncertainty is bell-shaped.

Figure

Convergence of the standardized sample mean toward the standard normal as n grows. Histograms of

Z_n

for a skewed (exponential) source flatten their skew and approach the gold Gaussian curve.

ExampleConfidence Interval on Validation Accuracy

Given: A classifier is evaluated on $n = 2{,}000$ held-out examples drawn i.i.d. from the test distribution. Each example is a Bernoulli trial $X_i$ with $X_i = 1$ if correctly classified. Observed accuracy $\hat p = \bar X_n = 0.86$ .

Goal: A 95% confidence interval for the true accuracy $p$ .

Steps: 1. Each $X_i$ has mean $p$ and variance $p(1-p)$ . By the CLT, $\hat p \approx \mathcal N\!\big(p,\; p(1-p)/n\big)$ for large $n$ . 2. Estimate the standard error with $\hat p$ : $\;\mathrm{SE} = \sqrt{\hat p(1-\hat p)/n} = \sqrt{0.86 \cdot 0.14 / 2000}$ . 3. Compute: $0.86\cdot 0.14 = 0.1204$ ; divide by $2000 = 6.02\times10^{-5}$ ; square root $= 0.00776$ . 4. The 95% interval uses $z_{0.975} = 1.96$ : margin $= 1.96 \times 0.00776 = 0.0152$ . 5. Interval $= 0.86 \pm 0.0152$ .

Answer: $\boxed{p \in [0.845,\ 0.875] \text{ with } 95\% \text{ confidence (dimensionless accuracy)}}$ . The CLT is what licenses the Gaussian $z$ -multiplier; with $n=2000$ the Bernoulli skew is negligible.

ExampleMonte Carlo Estimate of an Expectation

Given: We estimate $\mu = \mathbb{E}[f(W)]$ (e.g., expected reward) by Monte Carlo with $n = 10{,}000$ i.i.d. samples. The sample mean is $\bar f = 4.20$ and the sample standard deviation is $s = 3.0$ (units: reward).

Goal: Report the estimate with a one-standard-error band, and find $n$ needed to halve the error.

Steps: 1. By the CLT, $\bar f \approx \mathcal N(\mu, s^2/n)$ . Standard error $= s/\sqrt n = 3.0/\sqrt{10000} = 3.0/100 = 0.03$ . 2. Report: $\hat\mu = 4.20 \pm 0.03$ (reward). 3. Error scales as $1/\sqrt n$ . To halve $0.03 \to 0.015$ we need $\sqrt{n'} = 2\sqrt n$ , i.e. $n' = 4n$ . 4. Required: $n' = 4 \times 10000 = 40000$ .

Answer: $\boxed{\hat\mu = 4.20 \pm 0.03 \text{ reward};\ \ n' = 40{,}000 \text{ samples to halve the error}}$ . The quadrupling cost to halve error is the practical sting of the CLT's $1/\sqrt n$ rate.

⚠️ What the CLT does NOT say

It does not say the data $X_i$ become normal. The sample mean's distribution becomes normal; the raw data keep whatever shape they had.
It does not require a large individual dataset to be Gaussian — it requires a large number of summands/averages.
" $n \ge 30$ is always enough" is a rule of thumb, not a theorem. Heavily skewed or heavy-tailed (but finite-variance) sources may need thousands of samples for the normal approximation to be accurate in the tails.
The CLT is about convergence in distribution, which is the weakest mode — it says nothing about almost-sure or $L^2$ convergence of $Z_n$ (which in fact does not hold; $Z_n$ keeps fluctuating).

CounterexampleInfinite variance breaks the CLT

Let $X_1, X_2, \dots$ be i.i.d. Cauchy with density $\frac{1}{\pi(1+x^2)}$ . The Cauchy distribution has no finite mean and infinite variance, so the CLT hypotheses fail. Remarkably, the sample mean $\bar X_n$ of $n$ Cauchy variables is itself exactly Cauchy with the same scale — averaging does not concentrate it at all, and $\sqrt n(\bar X_n - \mu)$ never converges to a Gaussian. In ML this matters: gradient noise or importance-sampling weights with heavy tails (Pareto-type, infinite variance) can violate the CLT, so the usual $1/\sqrt n$ error shrinkage and Gaussian confidence intervals are simply wrong. Such cases fall under Generalized Central Limit Theorem with $\alpha$ -stable limits instead of Gaussian.

connection

The CLT sits at the hub of statistical ML reasoning.

It justifies the Law of Large Numbers companion result: while the LLN says $\bar X_n \to \mu$ , the CLT quantifies how fast and with what fluctuations.
It underlies the Bootstrap, which resamples to approximate the sampling distribution that the CLT describes asymptotically.
It motivates Gaussian Distribution noise models and the Maximum Likelihood Estimation asymptotic normality of estimators ( $\hat\theta \approx \mathcal N(\theta, I^{-1}/n)$ ).
In optimization, it frames Stochastic Gradient Descent as a noisy gradient $\nabla = \bar g + \mathcal N(0, \Sigma/B)$ over a mini-batch of size $B$ .
Its failure mode connects to the Generalized Central Limit Theorem and Heavy-Tailed Distributions.

🔬 Where this appears

⏳ History

- 1733 — Abraham de Moivre: approximates the binomial distribution by the normal curve for large

n

— the first special case of the CLT (de Moivre–Laplace theorem).

- 1810 — Pierre-Simon Laplace: states and (heuristically) proves a general CLT using characteristic functions, motivated by the theory of measurement errors.

- 1853 — Augustin-Louis Cauchy & Irénée-Jules Bienaymé: dispute over least squares forces clarification of the conditions and counterexamples (Cauchy distribution).

- 1887 — Pafnuty Chebyshev: develops the method of moments toward a rigorous CLT.

- 1901 — Aleksandr Lyapunov: gives the first fully rigorous proof under his moment ([Lyapunov]) condition, allowing non-identical distributions.

- 1922 — Jarl Waldemar Lindeberg: proves the CLT under the weaker Lindeberg condition, the most general sufficient condition for i.n.i.d. arrays.

- 1925 — Paul Lévy: develops the continuity theorem and the modern characteristic-function machinery; later studies

\alpha

-stable laws (the heavy-tailed generalizations).

- 1935 — William Feller & Lévy: establish that the Lindeberg condition is essentially necessary (Feller–Lévy), closing the classical theory.

- 20th c. onward: extensions to martingales, dependent and mixing sequences, high-dimensional ([Berry–Esseen] rates), and random-matrix CLTs underpinning modern ML theory.

open-problem

Rate of convergence. The Berry–Esseen theorem bounds the error as $\sup_z|F_{Z_n}(z)-\Phi(z)| \le C\,\rho/(\sigma^3\sqrt n)$ where $\rho=\mathbb E|X-\mu|^3$ ; the optimal constant $C$ and tail-uniform rates remain delicate, and the bound degrades for skewed data.
High dimensions. In modern ML, $n$ samples in $d \sim n$ dimensions break classical CLT intuition; multivariate CLTs need conditions on $d$ growing with $n$ , an active area (Chernozhukov–Chetverikov–Kato high-dimensional CLTs).
Dependence. Real data and gradients are correlated (sequential, mixing, or graph-structured). CLTs exist for martingales and weakly dependent sequences, but verifying mixing/ergodicity in deep learning dynamics is largely open.
Heavy tails. When variance is infinite, the limit is $\alpha$ -stable, not Gaussian; characterizing when ML quantities (importance weights, attention scores, gradient noise) are heavy-tailed is an empirical and theoretical frontier.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also