Convergence of Random Variables

⚡ Key Takeaways

Convergence of random variables describes the various precise senses in which a sequence $X_1, X_2, \ldots$ of random variables can "approach" a limiting random variable $X$ . The four principal modes — almost sure, in probability, in $L^p$ (mean), and in distribution — form a strict hierarchy of strength. They underpin the Law of Large Numbers and the Central Limit Theorem, and were forged into rigour by Kolmogorov, Lévy, and others in the early 20th century.

💡 The Core Insight

A random variable is a function on a sample space, so "convergence" inherits the full ambiguity of how a sequence of functions can converge. The question "where does the error live?" — in the sample space (almost sure), in the probability of failure (in probability), in an integral (in mean), or only in the shape of the distribution (in law) — selects the mode. Strength is measured by how much information the mode preserves about the joint behaviour of the $X_n$ and $X$ .

DefinitionAlmost Sure Convergence

$X_n$ converges to $X$ almost surely (written $X_n \xrightarrow{\text{a.s.}} X$ ) if the set of outcomes on which the numerical sequence converges has probability one:

1

\quad \mathbb{P}\left( \left\{ \omega \in \Omega : \lim_{n\to\infty} X_n(\omega) = X(\omega) \right\} \right) = 1.

Derivation hint: This is ordinary pointwise convergence of the functions $\omega \mapsto X_n(\omega)$ , but tolerated to fail on a set of probability zero (a null set). Here $\omega$ is a sample outcome, $X_n(\omega)$ is the realized value, and the limit is the usual limit of real numbers.

DefinitionConvergence in Probability

$X_n$ converges to $X$ in probability (written $X_n \xrightarrow{\mathbb{P}} X$ ) if for every $\varepsilon > 0$ ,

2

\quad \lim_{n\to\infty} \mathbb{P}\left( |X_n - X| > \varepsilon \right) = 0.

Derivation hint: Demand that the probability of an error exceeding any fixed tolerance $\varepsilon$ shrinks to zero as $n$ grows; the failure set may change identity with $n$ . The symbol $\varepsilon > 0$ is an arbitrary error tolerance; the event $\{|X_n - X| > \varepsilon\}$ is the "bad set" at stage $n$ .

DefinitionConvergence in

L^p

(Mean of Order

p

)

For $p \geq 1$ , $X_n$ converges to $X$ in $L^p$ (written $X_n \xrightarrow{L^p} X$ ) if all variables lie in $L^p$ and

3

\quad \lim_{n\to\infty} \mathbb{E}\left[ |X_n - X|^p \right] = 0.

Derivation hint: Measure the error by its $p$ -th moment (an integral against $\mathbb{P}$ ) and require that integrated quantity to vanish; $p=1$ is "in mean", $p=2$ is "in mean square". $\mathbb{E}[\cdot]$ denotes expectation. The case $p=2$ (mean-square convergence) dominates statistics and signal processing.

DefinitionConvergence in Distribution

$X_n$ converges to $X$ in distribution (or in law, or weakly; written $X_n \xrightarrow{d} X$ ) if the cumulative distribution functions $F_{X_n}(x) = \mathbb{P}(X_n \leq x)$ satisfy

4

\quad \lim_{n\to\infty} F_{X_n}(x) = F_X(x) \quad \text{at every } x \text{ where } F_X \text{ is continuous}.

Derivation hint: Compare the distribution functions pointwise, but excuse disagreement at jump points of the limiting law $F_X$ , where a literal demand would be too rigid. Crucially, $X_n$ and $X$ need not share a probability space — only their laws are compared.

TheoremThe Hierarchy of Convergence

The four modes satisfy the following strict implications, none of which reverses in general:

5

\quad \big(X_n \xrightarrow{\text{a.s.}} X\big) \implies \big(X_n \xrightarrow{\mathbb{P}} X\big) \implies \big(X_n \xrightarrow{d} X\big),

6

\quad \big(X_n \xrightarrow{L^p} X\big) \implies \big(X_n \xrightarrow{\mathbb{P}} X\big), \qquad \text{and for } q > p \geq 1,\ \big(X_n \xrightarrow{L^q} X\big) \implies \big(X_n \xrightarrow{L^p} X\big).

Derivation hint: a.s. $\Rightarrow$ in prob. follows from continuity of measure on shrinking tail events; $L^p \Rightarrow$ in prob. is Markov's inequality; in prob. $\Rightarrow$ in dist. follows by squeezing CDFs; the $L^q \Rightarrow L^p$ nesting is Jensen/Lyapunov. There is no general implication between almost sure and $L^p$ convergence: each can hold without the other.

ProofAlmost Sure ⇒ In Probability

Step 1. Fix $\varepsilon > 0$ . Define the tail events

7

\quad A_n = \bigcup_{m \geq n} \{ |X_m - X| > \varepsilon \}.

Derivation hint: $A_n$ is the event that some future term beyond $n$ exceeds the tolerance.

Step 2. The sets $A_n$ are nested decreasing: $A_1 \supseteq A_2 \supseteq \cdots$ . On the event of a.s. convergence, for each such $\omega$ there exists $N(\omega)$ with $|X_m(\omega)-X(\omega)| \le \varepsilon$ for all $m \ge N(\omega)$ , so $\omega \notin A_n$ for $n > N(\omega)$ . Hence

8

\quad \mathbb{P}\left( \bigcap_{n=1}^\infty A_n \right) \leq \mathbb{P}\big(\{X_n \not\to X\}\big) = 0.

Derivation hint: The intersection of all $A_n$ is contained in the divergence set, which has probability zero by hypothesis.

Step 3. By continuity of measure from above (since $\mathbb{P}(A_1) \le 1 < \infty$ ),

9

\quad \lim_{n\to\infty}\mathbb{P}(A_n) = \mathbb{P}\left(\bigcap_n A_n\right) = 0.

Derivation hint: Downward continuity of a finite measure converts the nested intersection into a limit of probabilities.

Step 4. Since $\{|X_n - X| > \varepsilon\} \subseteq A_n$ , monotonicity gives $\mathbb{P}(|X_n - X| > \varepsilon) \le \mathbb{P}(A_n) \to 0$ . As $\varepsilon$ was arbitrary, $X_n \xrightarrow{\mathbb{P}} X$ . $\blacksquare$

∎

ProofIn Probability ⇒ In Distribution

Step 1. Let $x$ be a continuity point of $F_X$ , and fix $\varepsilon > 0$ . Split using the law of total probability on the discrepancy event:

10

\quad F_{X_n}(x) = \mathbb{P}(X_n \le x) \le \mathbb{P}(X \le x+\varepsilon) + \mathbb{P}(|X_n - X| > \varepsilon).

Derivation hint: If $X_n \le x$ but $X > x+\varepsilon$ , then $|X_n - X| > \varepsilon$ ; otherwise $X \le x+\varepsilon$ .

Step 2. Taking $\limsup$ and using convergence in probability,

11

\quad \limsup_{n} F_{X_n}(x) \le F_X(x+\varepsilon).

Derivation hint: The error term vanishes, leaving the shifted CDF.

Step 3. Symmetrically, $\mathbb{P}(X \le x-\varepsilon) \le F_{X_n}(x) + \mathbb{P}(|X_n - X|>\varepsilon)$ , giving $F_X(x-\varepsilon) \le \liminf_n F_{X_n}(x)$ . Combining:

12

\quad F_X(x-\varepsilon) \le \liminf_n F_{X_n}(x) \le \limsup_n F_{X_n}(x) \le F_X(x+\varepsilon).

Derivation hint: A two-sided squeeze brackets the limit between shifted values of $F_X$ .

Step 4. Let $\varepsilon \downarrow 0$ . By continuity of $F_X$ at $x$ , both outer bounds tend to $F_X(x)$ , forcing $F_{X_n}(x) \to F_X(x)$ . $\blacksquare$

∎

Proof

L^p

⇒ In Probability (via Markov)

Step 1. For $\varepsilon > 0$ , apply Markov's inequality to the nonnegative variable $|X_n - X|^p$ :

13

\quad \mathbb{P}(|X_n - X| > \varepsilon) = \mathbb{P}(|X_n-X|^p > \varepsilon^p) \le \frac{\mathbb{E}\big[|X_n - X|^p\big]}{\varepsilon^p}.

Derivation hint: Markov: $\mathbb{P}(Y \ge a) \le \mathbb{E}[Y]/a$ for $Y \ge 0$ , applied with $Y = |X_n-X|^p$ , $a = \varepsilon^p$ .

Step 2. By hypothesis $\mathbb{E}[|X_n - X|^p] \to 0$ , so the right side $\to 0$ for every fixed $\varepsilon$ . Hence $X_n \xrightarrow{\mathbb{P}} X$ . $\blacksquare$

∎

mathematician · Andrey Kolmogorov | 1903–1987

Kolmogorov's 1933 monograph Grundbegriffe der Wahrscheinlichkeitsrechnung placed probability on a measure-theoretic foundation, making " $X_n \to X$ " a precise question about measurable functions and integrals. His Strong Law of Large Numbers is the archetypal almost-sure convergence result, and his Three-Series and zero–one laws are the engines for proving a.s. statements. By identifying random variables with measurable functions, he unified the previously disparate analytic and probabilistic notions of convergence.

🧠 Intuition

Picture each $\omega$ as a fixed "lottery draw" that, once chosen, determines the entire numerical sequence $X_1(\omega), X_2(\omega), \ldots$ .

Almost sure asks: for almost every fixed draw, does the resulting number-sequence settle down? The drama happens inside individual paths.
In probability asks: at each time $n$ , is it unlikely that the current value is far off? It says nothing about whether the same draw keeps misbehaving — the bad set may hop around.
$L^p$ asks: is the average size of the error small? A rare but catastrophic error can spoil $L^p$ while leaving "in probability" intact.
In distribution forgets the joint picture entirely and asks only whether the histograms line up. Two independent fair coins are equal in distribution but never close path-by-path.

The gap between "in probability" and "almost sure" is the difference between the bad set being small at each instant and the bad set eventually vanishing for good.

Figure

The strict hierarchy of convergence modes; arrows point from stronger to weaker. Dashed boxes mark the conditions that reverse an arrow.

ExampleMean-Square Convergence of a Sample Mean

Given: Let $Y_1, Y_2, \ldots$ be i.i.d. with $\mathbb{E}[Y_i] = \mu = 3$ and $\mathrm{Var}(Y_i) = \sigma^2 = 4$ . Define the sample mean $X_n = \frac{1}{n}\sum_{i=1}^n Y_i$ . Does $X_n \xrightarrow{L^2} \mu$ ?

Step 1 — Identify the error. Since $\mathbb{E}[X_n] = \mu$ , the $L^2$ error is the variance:

14

\quad \mathbb{E}\big[(X_n - \mu)^2\big] = \mathrm{Var}(X_n).

Derivation hint: For an unbiased estimator the mean-square error equals the variance.

Step 2 — Compute the variance. By independence,

15

\quad \mathrm{Var}(X_n) = \frac{1}{n^2}\sum_{i=1}^n \mathrm{Var}(Y_i) = \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n} = \frac{4}{n}.

Derivation hint: Variance of a sum of independent terms is the sum of variances; scaling by $1/n$ squares to $1/n^2$ .

Step 3 — Take the limit. $\dfrac{4}{n} \to 0$ as $n \to \infty$ .

\boxed{\ \mathbb{E}\big[(X_n - 3)^2\big] = \dfrac{4}{n} \to 0,\ \text{so } X_n \xrightarrow{L^2} 3.\ }

By the hierarchy [6] this also gives $X_n \xrightarrow{\mathbb{P}} 3$ — the Weak Law of Large Numbers. (Units: if $Y_i$ are in metres, the MSE is in metres².)

ExampleConvergence in Distribution to a Uniform Maximum

Given: Let $U_1, \ldots, U_n$ be i.i.d. Uniform $(0,1)$ and let $M_n = \max_i U_i$ . Define $Z_n = n(1 - M_n)$ . Find the limiting distribution.

Step 1 — CDF of the maximum. For $0 \le m \le 1$ ,

16

\quad \mathbb{P}(M_n \le m) = \prod_{i=1}^n \mathbb{P}(U_i \le m) = m^n.

Derivation hint: The max is $\le m$ iff all $n$ independent variables are $\le m$ .

Step 2 — CDF of $Z_n$ . For $z \ge 0$ , with $m = 1 - z/n$ ,

17

\quad \mathbb{P}(Z_n \le z) = \mathbb{P}\!\left(M_n \ge 1 - \tfrac{z}{n}\right) = 1 - \left(1 - \tfrac{z}{n}\right)^n.

Derivation hint: $Z_n \le z \Leftrightarrow n(1-M_n)\le z \Leftrightarrow M_n \ge 1 - z/n$ ; subtract the lower-tail probability from 1.

Step 3 — Take the limit. Using $\big(1 - z/n\big)^n \to e^{-z}$ :

18

\quad \lim_{n\to\infty} \mathbb{P}(Z_n \le z) = 1 - e^{-z}, \qquad z \ge 0.

Derivation hint: The standard exponential limit $(1 + a/n)^n \to e^a$ with $a = -z$ .

\boxed{\ Z_n = n(1 - M_n) \xrightarrow{d} \mathrm{Exponential}(1).\ }

This is a classic extreme-value limit: the rescaled distance of the maximum from 1 becomes exponential.

⚠️ Common mistake

"Convergence in distribution means the variables get close." False. In-distribution convergence compares laws, not values. If $X \sim N(0,1)$ and $X_n = -X$ , then $X_n \xrightarrow{d} X$ trivially (both are standard normal), yet $|X_n - X| = 2|X|$ never shrinks. Distribution convergence carries no information about the joint behaviour, and is the only mode for which a common probability space is unnecessary.

"Almost sure and $L^p$ are the same." They are logically independent — neither implies the other (see the counterexample). The cause is that $L^p$ penalises rare large errors that a.s. convergence happily tolerates.

Slutsky reminder: $X_n \xrightarrow{d} X$ and $Y_n \xrightarrow{d} Y$ do not imply $X_n + Y_n \xrightarrow{d} X + Y$ unless one limit is a constant or the pair converges jointly.

CounterexampleIn Probability but NOT Almost Surely — the "Typewriter" Sequence

On $\Omega = [0,1]$ with Lebesgue measure, define indicator variables on shrinking, sweeping intervals. For block size $k$ , let $X_{n}$ run through indicators of

19

\quad \left[\tfrac{j}{k}, \tfrac{j+1}{k}\right], \quad j = 0, 1, \ldots, k-1, \quad k = 1, 2, 3, \ldots

Derivation hint: Enumerate intervals of width $1/k$ that sweep across $[0,1]$ , then shrink $k$ ; each $X_n$ is the indicator of one such interval. Then $\mathbb{P}(X_n = 1) = 1/k \to 0$ , so $X_n \xrightarrow{\mathbb{P}} 0$ . But for every fixed $\omega \in [0,1]$ , the sequence $X_n(\omega)$ equals $1$ infinitely often (the sweeping interval revisits $\omega$ in every block) and $0$ infinitely often. Hence $X_n(\omega)$ diverges for every $\omega$ — almost sure convergence fails completely.

CounterexampleAlmost Surely but NOT in

L^1

— Escaping Mass

On $\Omega = [0,1]$ , let $X_n = n \cdot \mathbf{1}_{[0, 1/n]}$ . For any fixed $\omega > 0$ , eventually $1/n < \omega$ , so $X_n(\omega) = 0$ ; thus $X_n \xrightarrow{\text{a.s.}} 0$ . However,

20

\quad \mathbb{E}[X_n] = n \cdot \mathbb{P}\big([0,1/n]\big) = n \cdot \tfrac{1}{n} = 1 \not\to 0.

Derivation hint: A spike of height $n$ on a set of measure $1/n$ carries constant area 1; pointwise it vanishes but its integral does not. So $X_n \xrightarrow{L^1} 0$ fails. The mass "escapes to infinity in height." Adding uniform integrability is exactly the condition that closes this gap, upgrading in-probability convergence to $L^1$ .

connection

Convergence of random variables is the technical backbone of asymptotic probability. The Law of Large Numbers is a statement of in-probability (weak) and almost-sure (strong) convergence of sample means; the <a class="wiki-link" href="/wiki/central-limit-theorem">Central Limit Theorem</a> is a statement of convergence in distribution to a Gaussian. The squeezing argument in the in-probability $\Rightarrow$ in-distribution proof relies on the structure of the Cumulative Distribution Function, while the Markov step rests on Markov's Inequality and its variance-bearing cousin Chebyshev's Inequality. The whole edifice presupposes the framework of Measure Theory and Probability Space, where random variables are measurable functions and expectation is the Lebesgue integral. Convergence in distribution is the probabilistic face of Weak Convergence of Measures, and the bridge back to almost-sure statements is the Skorokhod Representation Theorem.

🔬 Where this appears

⏳ History

- 1713 — Jacob Bernoulli's Ars Conjectandi proves the first Law of Large Numbers: the relative frequency of an event converges (in what we now call probability) to its theoretical chance.

- 1837 — Poisson coins the phrase "law of large numbers" and extends Bernoulli's result.

- 1867 — Chebyshev gives his inequality, yielding a clean, general proof of the Weak Law and the first systematic use of moment bounds for convergence.

- 1900–1901 — Lyapunov proves the Central Limit Theorem under moment conditions, sharpening the meaning of convergence in distribution.

- 1909 — Émile Borel proves the Strong Law for Bernoulli trials, the first genuine almost-sure convergence theorem, introducing measure-zero reasoning.

- 1920s — Paul Lévy develops characteristic functions and the continuity theorem, the master tool for proving convergence in distribution; Khinchin and Kolmogorov refine LLN conditions.

- 1933 — Kolmogorov's Grundbegriffe axiomatises probability via measure theory, making all four convergence modes precise and proving the general Strong Law.

- 1956 — Skorokhod's representation theorem shows convergence in distribution can always be realised as almost-sure convergence on a suitable space, clarifying the relationship between the modes.

- 1968 — Billingsley's Convergence of Probability Measures systematises weak convergence on metric spaces, the modern functional viewpoint.

open-problem

The clean undergraduate hierarchy hides genuine subtleties at the research frontier.

Rates and uniformity. Knowing $X_n \xrightarrow{d} X$ says nothing about how fast. Quantifying the gap (Berry–Esseen bounds, Stein's method, Wasserstein distances) and obtaining sharp, dimension-free rates in high dimensions remains active.
Beyond the real line. For random variables in infinite-dimensional spaces (function-valued, measure-valued), the modes proliferate and tightness conditions become delicate; characterising weak compactness in such spaces is nontrivial.
Mode-mixing under operations. Determining minimal conditions under which a.s. or $L^p$ convergence is preserved through nonlinear functionals, conditioning, or stochastic integration is technically deep (e.g. for solutions of stochastic differential equations).
Almost-sure CLT. There exist refined "almost sure central limit theorems" where time-averages of normalized partial sums converge a.s.; their universality and rate are still being mapped out.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also