Random Variables and Distributions

⚡ Key Takeaways

A random variable is a measurable function assigning numbers to outcomes of a random experiment; its distribution describes how probability is spread over those numbers. Key result: every random variable is fully characterized by its cumulative distribution function (CDF), and integrable functions of it have well-defined expectations. Applies wherever uncertainty is quantified — statistics, physics, finance, machine learning. Foundational since Kolmogorov's 1933 axiomatization.

💡 The Core Insight

A random variable separates the mechanism of randomness (the sample space $\Omega$ and its measure) from the numerical observable we care about. Once we have the distribution of $X$ , the original $\Omega$ can be forgotten entirely. All probabilistic questions about $X$ — its mean, variance, the chance it exceeds a threshold — depend only on the distribution, not on the underlying experiment. This is why a "fair die" and a "spinner landing in six equal sectors" are interchangeable: same distribution, same probabilistic content.

DefinitionProbability Space

A probability space is a triple $(\Omega, \mathcal{F}, P)$ where:

$\Omega$ is the sample space (set of all outcomes),
$\mathcal{F}$ is a $\sigma$ -algebra of subsets of $\Omega$ (the events), closed under complement and countable union,
$P:\mathcal{F}\to[0,1]$ is a probability measure with $P(\Omega)=1$ and countable additivity.

1

\quad P\!\left(\bigcup_{i=1}^{\infty} A_i\right) = \sum_{i=1}^{\infty} P(A_i), \quad \text{for disjoint } A_i \in \mathcal{F}.

Derivation hint: This is the countable additivity axiom itself; disjointness ensures no outcome is double-counted.

DefinitionRandom Variable

A random variable is a function $X:\Omega\to\mathbb{R}$ that is measurable: for every $x\in\mathbb{R}$ ,

2

\quad \{\omega\in\Omega : X(\omega)\le x\} \in \mathcal{F}.

Derivation hint: We require preimages of $(-\infty,x]$ to be events so that $P(X\le x)$ is defined; these sets generate the Borel $\sigma$ -algebra, so all Borel preimages become measurable. Here $X(\omega)$ is the numerical value observed when outcome $\omega$ occurs. The measurability condition guarantees every "interesting" set $\{X \in B\}$ for Borel $B$ is an event.

DefinitionCumulative Distribution Function

The CDF of $X$ is

3

\quad F_X(x) = P(X\le x) = P\big(\{\omega: X(\omega)\le x\}\big).

Derivation hint: Compose the measure $P$ with the preimage from [2]; well-definedness is exactly the measurability requirement. Every CDF satisfies: (i) non-decreasing, (ii) right-continuous, (iii) $\lim_{x\to-\infty}F_X(x)=0$ and $\lim_{x\to+\infty}F_X(x)=1$ .

TheoremCharacterization of Distributions

A function $F:\mathbb{R}\to[0,1]$ is the CDF of some random variable if and only if it is non-decreasing, right-continuous, and has limits $0$ at $-\infty$ and $1$ at $+\infty$ . The distribution (the induced measure $P_X(B)=P(X\in B)$ on Borel sets) is uniquely determined by $F$ .

DefinitionDiscrete and Continuous Types

Discrete: $X$ takes values in a countable set; described by a probability mass function (PMF) $p_X(x)=P(X=x)$ , with $\sum_x p_X(x)=1$ .
Continuous (absolutely continuous): $F_X$ has a probability density function (PDF) $f_X\ge 0$ with

4

\quad F_X(x) = \int_{-\infty}^{x} f_X(t)\,dt, \qquad \int_{-\infty}^{\infty} f_X(t)\,dt = 1.

Derivation hint: $F_X$ absolutely continuous $\Rightarrow$ by Lebesgue's differentiation theorem $f_X = F_X'$ exists a.e. and integrates back to $F_X$ . For continuous $X$ , $P(X=x)=0$ for each point; probability lives in intervals: $P(a<X\le b)=F_X(b)-F_X(a)$ .

DefinitionExpectation and Variance

The expectation (mean) of $X$ , when it exists, is

5

\quad \mathbb{E}[X] = \int_{\Omega} X\,dP = \int_{-\infty}^{\infty} x\,dF_X(x) = \begin{cases}\sum_x x\,p_X(x) & \text{discrete}\\[4pt] \int_{-\infty}^{\infty} x\,f_X(x)\,dx & \text{continuous}\end{cases}

Derivation hint: The middle form is a Lebesgue–Stieltjes integral against $F_X$ ; it specializes to a sum or ordinary integral depending on whether $F_X$ jumps or is smooth. The variance measures spread:

6

\quad \operatorname{Var}(X) = \mathbb{E}\big[(X-\mathbb{E}[X])^2\big] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2.

Derivation hint: Expand $(X-\mu)^2 = X^2 - 2\mu X + \mu^2$ and use linearity with $\mathbb{E}[X]=\mu$ constant.

ProofLaw of the Unconscious Statistician (LOTUS)

Claim: For a measurable $g:\mathbb{R}\to\mathbb{R}$ , $\mathbb{E}[g(X)] = \int_{-\infty}^{\infty} g(x)\,dF_X(x)$ — i.e. we can integrate against the distribution of $X$ without knowing $\Omega$ . 1. By definition $\mathbb{E}[g(X)] = \int_\Omega g(X(\omega))\,dP(\omega)$ , an integral over the abstract space. 2. The random variable $X$ pushes $P$ forward to a measure $P_X$ on $\mathbb{R}$ via $P_X(B)=P(X^{-1}(B))$ . 3. Change of variables for pushforward measures: for any measurable $h\ge 0$ ,

\int_\Omega h(X(\omega))\,dP(\omega) = \int_{\mathbb{R}} h(x)\,dP_X(x).

This holds first for indicators $h=\mathbf{1}_B$ (where it is exactly the definition of $P_X$ ), then for simple functions by linearity, then for general $h\ge 0$ by monotone convergence, then for signed $h$ by splitting into positive and negative parts. 4. Setting $h=g$ and recognizing $P_X$ as the measure with CDF $F_X$ gives

\mathbb{E}[g(X)] = \int_{\mathbb{R}} g(x)\,dP_X(x) = \int_{-\infty}^\infty g(x)\,dF_X(x). \qquad \blacksquare

The phrase "unconscious statistician" reflects that practitioners use this formula instinctively, often unaware that a measure-theoretic theorem justifies it.

∎

AK

Andrey Kolmogorov

1903–1987

Kolmogorov's 1933 monograph Grundbegriffe der Wahrscheinlichkeitsrechnung placed probability on measure-theoretic foundations, defining a random variable precisely as a measurable function on a probability space. Before him, probability lacked a rigorous axiomatic basis and was riddled with paradoxes (Bertrand's paradox, ambiguous "geometric probability"). Kolmogorov's framework unified discrete and continuous probability under one roof, made conditional expectation rigorous, and connected probability to analysis, enabling the modern theories of stochastic processes, ergodic theory, and information.

🧠 Intuition

Think of the sample space $\Omega$ as a bag of all possible "states of the world," each with a probability weight. A random variable is a measuring instrument: you reach into the bag, pull out a state $\omega$ , and read off a number $X(\omega)$ . The distribution is the histogram you'd build by repeating this enormously many times. The CDF $F_X(x)$ answers "what fraction of the weight sits at or below $x$ ?" — it sweeps from 0 (far left) to 1 (far right). A jump in $F_X$ means a lump of probability sits exactly at that point (discrete atom). A smooth rise means probability is smeared out (density). The steepness of the rise is exactly the density $f_X$ : where $F_X$ climbs fast, values are likely. Expectation is the balance point of the histogram — where it would teeter on a knife-edge. Variance is the average squared distance from that balance point: small variance means a tall thin pile near the mean; large variance means a wide flat spread.

Figure

CDF and PDF of a continuous random variable: the density is the slope of the CDF

ExampleDiscrete: Expected value of a biased die

Given: A four-sided die with PMF $p(1)=0.1,\ p(2)=0.2,\ p(3)=0.3,\ p(4)=0.4$ .

Find: $\mathbb{E}[X]$ and $\operatorname{Var}(X)$ .

Steps: 1. Check normalization: $0.1+0.2+0.3+0.4 = 1.0$ ✓ 2. Mean via [5]:

\mathbb{E}[X] = 1(0.1)+2(0.2)+3(0.3)+4(0.4) = 0.1+0.4+0.9+1.6 = 3.0.

3. Second moment:

\mathbb{E}[X^2] = 1(0.1)+4(0.2)+9(0.3)+16(0.4) = 0.1+0.8+2.7+6.4 = 10.0.

4. Variance via [6]: $\operatorname{Var}(X)=10.0 - 3.0^2 = 10.0 - 9.0 = 1.0$ .

Answer: $\boxed{\mathbb{E}[X]=3.0,\quad \operatorname{Var}(X)=1.0,\quad \sigma=1.0}$ (dimensionless pip-count).

ExampleContinuous: Exponential waiting time

Given: $X\sim\text{Exponential}(\lambda)$ with density $f(x)=\lambda e^{-\lambda x}$ for $x\ge 0$ , $\lambda = 2\ \text{hr}^{-1}$ (e.g. customer arrivals).

Find: $F_X$ , $\mathbb{E}[X]$ , and $P(X>1\text{ hr})$ .

Steps: 1. CDF via [4]: $F_X(x)=\int_0^x 2e^{-2t}\,dt = 1 - e^{-2x}$ for $x\ge 0$ . 2. Mean via [5] (integration by parts):

\mathbb{E}[X]=\int_0^\infty x\cdot 2e^{-2x}\,dx = \frac{1}{\lambda} = \frac{1}{2} = 0.5\ \text{hr}.

3. Tail probability: $P(X>1) = 1 - F_X(1) = e^{-2(1)} = e^{-2} \approx 0.1353$ .

Answer: $\boxed{\mathbb{E}[X]=0.5\ \text{hr},\quad P(X>1\ \text{hr}) \approx 0.135}$ .

⚠️ Common mistake

A density is not a probability. For a continuous random variable, $f_X(x)$ can exceed 1 (e.g. a Uniform $(0,0.5)$ density equals 2 everywhere on its support). Only integrals of $f_X$ over intervals are probabilities. Likewise, $P(X=x)=0$ for continuous $X$ does not mean the value $x$ is impossible — it means it has zero measure, a subtle distinction. Impossible events have probability 0, but probability-0 events need not be impossible.

⚠️ Common mistake

Expectation may not exist. Not every random variable has a finite (or even defined) mean. The integral in [5] requires $\mathbb{E}[|X|]<\infty$ . When this fails, statements like "the average of $X$ " are meaningless, and the Law of Large Numbers does not apply in its usual form.

CounterexampleCauchy distribution: no mean

Let $X$ have the standard Cauchy density $f(x)=\dfrac{1}{\pi(1+x^2)}$ . Then

\int_{-\infty}^{\infty} |x|\,\frac{1}{\pi(1+x^2)}\,dx = \frac{2}{\pi}\int_0^\infty \frac{x}{1+x^2}\,dx = \frac{1}{\pi}\big[\ln(1+x^2)\big]_0^\infty = \infty.

So $\mathbb{E}[|X|]=\infty$ and $\mathbb{E}[X]$ is undefined. Sample averages of Cauchy data do not converge to any number — they keep fluctuating with the same Cauchy distribution. This dramatically refutes the intuition that "averaging always stabilizes."

connection

Random variables are the atoms from which the entire probabilistic edifice is built. The CDF connects to Cumulative Distribution Functions and Lebesgue–Stieltjes integration. Sums of independent random variables lead to Convolution of Distributions and the Central Limit Theorem. Conditional structure between several random variables is formalized in Joint Distributions and Independence, while the long-run behavior of averages appears in the Law of Large Numbers. The transform machinery — Moment Generating Functions and characteristic functions — uses LOTUS [3]–[5] to encode distributions analytically.

🔬 Where this appears

⏳ History

- 1654 — Pascal and Fermat exchange letters on the problem of points, computing expected values of gambling outcomes — the implicit birth of expectation.

- 1713 — Jacob Bernoulli's Ars Conjectandi proves the (weak) Law of Large Numbers for binomial trials, the first limit theorem for a random variable's average.

- 1733–1812 — De Moivre and Laplace develop the Normal approximation to the binomial, an early Central Limit Theorem.

- 1809 — Gauss introduces the Normal distribution in the theory of measurement errors.

- 1900s — Borel and Lebesgue develop measure theory and the Lebesgue integral, the analytic tools probability needed.

- 1909 — Borel's strong law for binary digits foreshadows almost-sure convergence.

- 1933 — Kolmogorov's Grundbegriffe axiomatizes probability: random variable = measurable function, distribution = pushforward measure.

- 1937 — Cramér and Lévy systematize characteristic functions and distribution theory.

- 1950s onward — Doob, Itô, and others extend the framework to stochastic processes, turning random variables into the building blocks of modern probability.

open-problem

Several deep subtleties surround even this elementary topic. (1) Non-measurable functions: with the Axiom of Choice one can construct sets (Vitali sets) and functions on $[0,1]$ that are not measurable, hence not random variables — these have no well-defined probability, exposing the role of set-theoretic foundations. (2) Singular continuous distributions: there exist CDFs (e.g. the Cantor function) that are continuous yet have no density and no atoms — they are neither discrete nor absolutely continuous, defying the textbook dichotomy. (3) Existence of moments: characterizing which transforms (moment problem) uniquely determine a distribution from its moments is subtle — the lognormal distribution is famously not determined by its moments. (4) Infinite-dimensional analogues: defining "random functions" or measures on function spaces (Gaussian processes, Brownian motion) requires substantial extra machinery (Kolmogorov extension theorem) and motivates ongoing research in stochastic analysis.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also