The Law of Large Numbers

⚡ Key Takeaways

One-line statement: As the number of independent trials grows, the sample average converges to the true expected value. Key result in words: Random fluctuations average out: the empirical mean $\bar{X}_n$ of $n$ i.i.d. random variables approaches the population mean $\mu$ as $n \to \infty$ . Where it applies: Probability, statistics, gambling theory, Monte Carlo methods, insurance, statistical mechanics — anywhere repeated random sampling occurs. Who/when: First proved by Jacob Bernoulli (published posthumously, 1713) for the binomial case; generalized through Chebyshev (1867), Khinchin (1929), and Kolmogorov (1930s).

💡 The Core Insight

Individual random outcomes are unpredictable, but their average is not. As you accumulate more independent samples, the contributions of unusual outcomes are diluted by a factor of $1/n$ , while the systematic center $\mu$ stays fixed. The randomness doesn't vanish — it gets divided away. The spread of the sample mean shrinks (its standard deviation falls like $1/\sqrt{n}$ ), forcing the average into an ever-tightening neighborhood of the true mean.

DefinitionSample Mean and Setup

Let $X_1, X_2, \dots, X_n$ be random variables on a common probability space. The sample mean (empirical average) is

1

\quad \bar{X}_n = \frac{1}{n}\sum_{i=1}^{n} X_i.

Derivation hint: This is just an arithmetic average of realized values, promoted to a random variable since each $X_i$ is random. Here:

$X_i$ — the $i$ -th observation, a real-valued random variable;
$n$ — the sample size (number of trials);
$\mu = \mathbb{E}[X_i]$ — the common (population) mean, assumed finite, $|\mu| < \infty$ ;
$\sigma^2 = \operatorname{Var}(X_i)$ — the common variance (assumed finite for the elementary version);
"i.i.d." — independent and identically distributed: the $X_i$ share one distribution and are mutually independent.

TheoremWeak Law of Large Numbers (WLLN)

Let $X_1, X_2, \dots$ be i.i.d. with finite mean $\mu$ . Then $\bar{X}_n$ converges to $\mu$ in probability: for every $\varepsilon > 0$ ,

2

\quad \lim_{n\to\infty} \mathbb{P}\!\left(\,\left|\bar{X}_n - \mu\right| \ge \varepsilon\,\right) = 0.

Derivation hint: Equivalently $\mathbb{P}(|\bar X_n - \mu| < \varepsilon) \to 1$ ; this is the definition of convergence in probability, $\bar X_n \xrightarrow{P} \mu$ . (Khinchin's theorem requires only finite mean; the elementary proof below additionally assumes finite variance.)

TheoremStrong Law of Large Numbers (SLLN)

Let $X_1, X_2, \dots$ be i.i.d. with finite mean $\mu$ (Kolmogorov, 1933). Then $\bar{X}_n$ converges to $\mu$ almost surely:

3

\quad \mathbb{P}\!\left(\lim_{n\to\infty} \bar{X}_n = \mu\right) = 1.

Derivation hint: Almost sure convergence means the set of sample sequences whose averages fail to converge to $\mu$ has probability zero — a strictly stronger conclusion than (2).

⚠️ Weak vs. Strong — They Are Not the Same

The SLLN implies the WLLN, but not conversely. The WLLN says: for each large $n$ , the average is probably close to $\mu$ . The SLLN says: with probability 1, the average settles down and stays close, eventually, forever. There exist sequences satisfying the WLLN but not the SLLN. The distinction is the difference between convergence in probability and almost-sure convergence.

ProofWLLN via Chebyshev's Inequality (finite-variance case)

Step 1 — Mean of the sample mean. By linearity of expectation and the identical distribution,

4

\quad \mathbb{E}[\bar{X}_n] = \frac{1}{n}\sum_{i=1}^{n}\mathbb{E}[X_i] = \frac{1}{n}\cdot n\mu = \mu.

Derivation hint: Expectation is linear regardless of dependence; each $\mathbb{E}[X_i]=\mu$ .

Step 2 — Variance of the sample mean. Using independence (so covariances vanish) and $\operatorname{Var}(cX) = c^2\operatorname{Var}(X)$ ,

5

\quad \operatorname{Var}(\bar{X}_n) = \frac{1}{n^2}\sum_{i=1}^{n}\operatorname{Var}(X_i) = \frac{1}{n^2}\cdot n\sigma^2 = \frac{\sigma^2}{n}.

Derivation hint: $\operatorname{Var}\!\big(\tfrac1n\sum X_i\big)=\tfrac1{n^2}\operatorname{Var}\!\big(\sum X_i\big)$ ; independence makes the variance of the sum the sum of variances.

Step 3 — Apply Chebyshev's inequality. For any random variable $Y$ with mean $\mu_Y$ and variance $\sigma_Y^2$ , and any $\varepsilon>0$ , $\mathbb{P}(|Y-\mu_Y|\ge\varepsilon)\le \sigma_Y^2/\varepsilon^2$ . With $Y = \bar X_n$ :

6

\quad \mathbb{P}\!\left(\left|\bar{X}_n - \mu\right| \ge \varepsilon\right) \;\le\; \frac{\operatorname{Var}(\bar X_n)}{\varepsilon^2} = \frac{\sigma^2}{n\,\varepsilon^2}.

Derivation hint: Substitute (4) and (5) into Chebyshev's bound; the only $n$ -dependence is the $1/n$ factor.

Step 4 — Take the limit. Fix $\varepsilon>0$ . As $n\to\infty$ , the right side $\sigma^2/(n\varepsilon^2)\to 0$ . Since probabilities are nonnegative, the squeeze theorem gives

7

\quad \lim_{n\to\infty}\mathbb{P}\!\left(\left|\bar{X}_n - \mu\right| \ge \varepsilon\right) = 0,

Derivation hint: $0 \le \mathbb{P}(\cdots) \le \sigma^2/(n\varepsilon^2)\to 0$ forces the middle term to 0. which is exactly (2). $\blacksquare$

∎

ProofSketch of the SLLN (Kolmogorov, finite 4th moment shortcut)

The full proof uses Kolmogorov's maximal inequality and the three-series theorem. A digestible route (assuming $\mathbb{E}[X_i^4]<\infty$ , WLOG $\mu=0$ ) bounds the fourth moment of the partial sum $S_n=\sum X_i$ :

8

\quad \mathbb{E}[S_n^4] = n\,\mathbb{E}[X_1^4] + 3n(n-1)\big(\mathbb{E}[X_1^2]\big)^2 \le C\,n^2.

Derivation hint: Expand $(\sum X_i)^4$ ; independence and $\mathbb{E}[X_i]=0$ kill all terms except those with each index appearing an even number of times — giving $n$ fourth-power terms and $3n(n-1)$ paired-square terms. Then by Markov's inequality applied to $\bar X_n^4 = S_n^4/n^4$ ,

9

\quad \sum_{n=1}^{\infty}\mathbb{P}\!\left(|\bar X_n|\ge\varepsilon\right) \le \sum_{n=1}^{\infty}\frac{\mathbb{E}[S_n^4]}{n^4\varepsilon^4} \le \frac{C}{\varepsilon^4}\sum_{n=1}^{\infty}\frac{1}{n^2} < \infty.

Derivation hint: $\mathbb{P}(|\bar X_n|\ge\varepsilon)=\mathbb{P}(\bar X_n^4\ge\varepsilon^4)\le \mathbb{E}[\bar X_n^4]/\varepsilon^4$ , then use (8); the resulting series is a convergent $p$ -series ( $p=2$ ). By the Borel–Cantelli lemma, since the probabilities are summable, $|\bar X_n|\ge\varepsilon$ occurs only finitely often with probability 1. Letting $\varepsilon\to0$ through a sequence yields $\bar X_n\to0$ almost surely. Kolmogorov's achievement was removing the moment assumptions down to just finite mean. $\blacksquare$

∎

mathematician · Jacob Bernoulli | 1655–1705

The Swiss mathematician Jacob (Jakob) Bernoulli proved the first version of the LLN — his "Golden Theorem" — in Ars Conjectandi, published posthumously in 1713 by his nephew Nicolaus. He showed that for repeated trials of a binary event with probability $p$ , the observed frequency converges (in probability) to $p$ , and crucially he derived an explicit bound on how many trials guarantee a given accuracy with given confidence — a remarkably modern, quantitative result. Bernoulli reportedly worked on the theorem for twenty years; he regarded it as so important that he wanted it to anchor probability as a serious mathematical discipline rather than a gambler's toy.

🧠 Why Averages Are Tamer Than Individuals

Imagine each $X_i$ as a noisy vote about the true value $\mu$ . A single vote can be wildly off. But independent errors point in random directions: some too high, some too low. When you average, the errors partially cancel. The key quantitative fact is (5): the standard deviation of the average is $\sigma/\sqrt{n}$ , not $\sigma$ . To halve your uncertainty you need four times the data. This $\sqrt n$ scaling is the price and the promise of the LLN — convergence is guaranteed but slow. A useful mental image: the histogram of a single $X$ has fixed width $\sigma$ . The histogram of $\bar X_n$ is centered at the same place $\mu$ but is narrower by $\sqrt n$ , collapsing toward a spike at $\mu$ .

Figure

Concentration of the sample mean: the distribution of

\bar X_n

narrows around

\mu

as

n

grows

ExampleFair Coin: Frequency of Heads

Given: A fair coin, $\mathbb{P}(\text{heads})=p=0.5$ . Let $X_i=1$ if toss $i$ is heads, else $0$ (a Bernoulli variable). Here $\mu = p = 0.5$ and $\sigma^2 = p(1-p) = 0.25$ . We flip it $n = 10{,}000$ times.

Question: Bound the probability that the observed fraction of heads $\bar X_n$ deviates from $0.5$ by at least $\varepsilon = 0.01$ .

Steps: 1. Variance of the mean (eq. 5): $\operatorname{Var}(\bar X_n) = \sigma^2/n = 0.25/10000 = 2.5\times 10^{-5}$ . 2. Chebyshev bound (eq. 6): $\mathbb{P}(|\bar X_n - 0.5|\ge 0.01) \le \dfrac{0.25}{10000 \times (0.01)^2} = \dfrac{0.25}{1.0} = 0.25.$ 3. Interpretation: at least a 75% chance the head-fraction lands within $[0.49, 0.51]$ . (Chebyshev is loose; the Central Limit Theorem gives the true probability $\approx 0.0455$ , i.e. ~95% inside.)

Answer: $\boxed{\mathbb{P}(|\bar X_n - 0.5|\ge 0.01) \le 0.25}$ (dimensionless probability). The LLN guarantees this bound $\to 0$ as $n\to\infty$ .

ExampleMonte Carlo Estimation of an Integral

Given: Estimate $I = \int_0^1 e^{x^2}\,dx$ by sampling $U_i \sim \text{Uniform}(0,1)$ and setting $X_i = e^{U_i^2}$ . Then $\mu = \mathbb{E}[X_i] = \int_0^1 e^{x^2}dx = I$ exactly, so $\bar X_n \to I$ by the SLLN. Suppose we draw $n = 1{,}000{,}000$ samples and find $\bar X_n = 1.4627$ , with sample standard deviation $s = 0.494$ .

Question: Estimate $I$ and quantify the error.

Steps: 1. Point estimate: $\hat I = \bar X_n = 1.4627$ . 2. Standard error of the mean: $\text{SE} = s/\sqrt n = 0.494/\sqrt{10^6} = 0.494/1000 = 4.94\times 10^{-4}$ . 3. Approximate 95% interval (CLT): $\hat I \pm 1.96\,\text{SE} = 1.4627 \pm 0.00097$ . 4. The true value is $I \approx 1.4627$ , inside the interval.

Answer: $\boxed{I \approx 1.463 \pm 0.001}$ (dimensionless). Reducing the error tenfold requires $100\times$ more samples — the $1/\sqrt n$ tax.

⚠️ The Gambler's Fallacy

The LLN does not say that deviations get corrected. After 10 heads in a row, tails is not "due." The coin has no memory; future tosses are independent. What actually happens is that the absolute number of excess heads can grow without bound — only the fraction tends to $1/2$ , because we divide by an ever-larger $n$ . The imbalance is diluted, not repaired.

⚠️ Finite Mean Is Required

If $\mathbb{E}[|X_i|] = \infty$ , the LLN can fail outright. The sample mean of i.i.d. variables with no finite mean does not converge to anything. Always check the mean exists before invoking the law.

CounterexampleCauchy Distribution — No Convergence

Let $X_1, X_2, \dots$ be i.i.d. standard Cauchy (density $\tfrac{1}{\pi(1+x^2)}$ ). The mean does not exist (the defining integral diverges). A striking fact: the sample mean $\bar X_n$ of $n$ Cauchy variables is itself exactly standard Cauchy, for every $n$ .

10

\quad \bar X_n = \frac{1}{n}\sum_{i=1}^n X_i \;\stackrel{d}{=}\; X_1 \quad \text{for all } n.

Derivation hint: The Cauchy characteristic function is $\varphi(t)=e^{-|t|}$ ; the sum of $n$ of them has c.f. $e^{-n|t|}$ , and dividing by $n$ rescales $t\to t/n$ , returning $e^{-|t|}$ . So averaging more Cauchy samples does not reduce the spread at all — $\bar X_n$ stays as wildly variable as a single observation. The LLN simply does not apply.

connection · Where the LLN Sits in Probability

The LLN is the first of the two great limit theorems. Its partner, the Central Limit Theorem, refines it: while the LLN says $\bar X_n \to \mu$ , the CLT describes the shape of the fluctuation, asserting $\sqrt n(\bar X_n - \mu)/\sigma \to \mathcal N(0,1)$ in distribution. The LLN sets the target; the CLT measures the wobble around it at scale $1/\sqrt n$ . The proof of the WLLN relies on Chebyshev's Inequality and on the Variance of a sum of Independent Random Variables. The SLLN rests on the Borel–Cantelli Lemma. Both laws presuppose a clean notion of Expected Value and the distinction between Convergence in Probability and Almost Sure Convergence.

🔬 Where the Law of Large Numbers Operates

⏳ History

- 1713 — Jacob Bernoulli's Ars Conjectandi (posthumous) proves the "Golden Theorem," the WLLN for binomial trials, with explicit error bounds.

- 1733–1738 — Abraham de Moivre develops the normal approximation to the binomial, foreshadowing both the LLN's refinement and the CLT.

- 1810s — Siméon Denis Poisson coins the phrase "la loi des grands nombres" (the law of large numbers) and extends Bernoulli's result to varying probabilities.

- 1837 — Poisson publishes a generalized version allowing non-identical trial probabilities.

- 1867 — Pafnuty Chebyshev gives a clean, general proof of the WLLN using his inequality, requiring only finite variance and not the binomial structure.

- 1909 — Émile Borel proves a Strong Law for binary digits (normal numbers), the first almost-sure version.

- 1917–1929 — Francesco Cantelli and Aleksandr Khinchin sharpen the strong and weak laws; Khinchin's WLLN needs only a finite mean.

- 1930–1933 — Andrey Kolmogorov proves the definitive SLLN: for i.i.d. variables, finite mean is necessary and sufficient for almost-sure convergence; he also gives his maximal inequality and three-series theorem.

- 1933 — Kolmogorov's Grundbegriffe axiomatizes probability via measure theory, placing the LLN on fully rigorous foundations.

open-problem · Boundaries and Refinements

The classical LLN is settled, but its frontiers remain active. (1) Non-i.i.d. and dependent data: for stationary ergodic sequences, Birkhoff's ergodic theorem plays the LLN's role; for martingales there are martingale strong laws; for mixing or long-range-dependent processes, conditions are delicate and incomplete. (2) Rates of convergence: the LLN gives no rate by itself; the Berry–Esseen theorem, large-deviation principles (Cramér's theorem), and the law of the iterated logarithm (which pins the exact fluctuation size $\sqrt{2\sigma^2 n \log\log n}$ of $S_n$ ) refine it, but optimal constants under minimal moments are still studied. (3) Heavy tails: when variance is infinite but the mean is finite, convergence holds but is slow and CLT-style normalization fails; the right framework is stable laws. (4) High-dimensional and matrix settings: laws of large numbers for random matrices, empirical measures, and U-statistics extend the idea but require new tools and have open quantitative questions.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also