Probability Theory

⚡ Key Takeaways

Probability Theory is the mathematical study of randomness and uncertainty. One line: It assigns numbers between 0 and 1 to events to quantify how likely they are, subject to the axioms of Kolmogorov. Key result: A consistent measure-theoretic framework lets us compute expectations, prove limit laws (Law of Large Numbers, Central Limit Theorem), and reason rigorously about chance. Where it applies: statistics, physics, finance, machine learning, information theory, biology, and engineering. Who/when: Foundations laid by Pascal & Fermat (1654); axiomatized by Andrey Kolmogorov (1933).

💡 The Core Insight

Randomness can be tamed by treating probability as a normalized measure on a space of outcomes. Once we accept three simple axioms — non-negativity, normalization, and countable additivity — every intuitive rule of chance (complements, unions, conditioning, independence) follows by deduction, and powerful limit theorems become provable rather than merely observed.

DefinitionProbability Space

A probability space is a triple $(\Omega,\mathcal{F},P)$ where:

$\Omega$ is the sample space, the set of all possible outcomes $\omega$ .
$\mathcal{F}\subseteq 2^\Omega$ is a $\sigma$ -algebra (the events): a collection closed under complement and countable union, containing $\Omega$ .
$P:\mathcal{F}\to[0,1]$ is a probability measure satisfying Kolmogorov's axioms.

The axioms are:

1

\quad P(A)\ge 0 \quad \text{for all } A\in\mathcal{F}

Derivation hint: This is an axiom (non-negativity); probabilities cannot be negative since they measure relative size of event sets.

2

\quad P(\Omega)=1

Derivation hint: Normalization axiom — the certain event has full measure; this fixes the scale.

3

\quad P\!\left(\bigcup_{i=1}^{\infty} A_i\right)=\sum_{i=1}^{\infty} P(A_i)\quad\text{for pairwise disjoint } A_i

Derivation hint: Countable additivity axiom — disjoint events' probabilities add; the countable (not just finite) version is needed to support limit theorems.

DefinitionConditional Probability and Independence

For events $A,B$ with $P(B)>0$ , the conditional probability of $A$ given $B$ is

4

\quad P(A\mid B)=\frac{P(A\cap B)}{P(B)}

Derivation hint: Restrict the sample space to $B$ and renormalize by $P(B)$ so the conditional measure sums to 1. Events $A,B$ are independent iff

5

\quad P(A\cap B)=P(A)\,P(B)

Derivation hint: Equivalent to $P(A\mid B)=P(A)$ when $P(B)>0$ ; knowing $B$ gives no information about $A$ .

DefinitionRandom Variable and Expectation

A random variable $X$ is a measurable function $X:\Omega\to\mathbb{R}$ (so $\{\omega: X(\omega)\le x\}\in\mathcal{F}$ for all $x$ ). Its distribution function is $F_X(x)=P(X\le x)$ . The expectation is

6

\quad \mathbb{E}[X]=\int_\Omega X\,dP=\begin{cases}\sum_x x\,p_X(x) & \text{discrete}\\[4pt]\displaystyle\int_{-\infty}^{\infty} x\,f_X(x)\,dx & \text{continuous}\end{cases}

Derivation hint: The expectation is the Lebesgue integral of $X$ against $P$ ; the two cases are its specializations to probability mass function $p_X$ and density $f_X$ . The variance is $\operatorname{Var}(X)=\mathbb{E}[(X-\mathbb{E}X)^2]=\mathbb{E}[X^2]-(\mathbb{E}X)^2$ .

TheoremBayes' Theorem

For events $A,B$ with $P(A),P(B)>0$ ,

7

\quad P(A\mid B)=\frac{P(B\mid A)\,P(A)}{P(B)},\qquad P(B)=\sum_i P(B\mid A_i)P(A_i)

Derivation hint: Apply [4] twice ( $P(A\cap B)=P(A\mid B)P(B)=P(B\mid A)P(A)$ ) and use the law of total probability for $P(B)$ over a partition $\{A_i\}$ .

TheoremStrong Law of Large Numbers

Let $X_1,X_2,\dots$ be i.i.d. with $\mathbb{E}|X_1|<\infty$ and mean $\mu$ . Then

8

\quad P\!\left(\lim_{n\to\infty}\frac{1}{n}\sum_{i=1}^n X_i=\mu\right)=1

Derivation hint: Sample averages converge almost surely to the mean; proved via truncation and the Borel–Cantelli lemmas (Kolmogorov's proof).

TheoremCentral Limit Theorem

Let $X_1,\dots,X_n$ be i.i.d. with mean $\mu$ and finite variance $\sigma^2>0$ . Then

9

\quad \frac{1}{\sigma\sqrt{n}}\sum_{i=1}^n (X_i-\mu)\ \xrightarrow{\ d\ }\ \mathcal{N}(0,1)

Derivation hint: Show the characteristic function of the standardized sum converges to $e^{-t^2/2}$ via a second-order Taylor expansion of $\log\varphi_X(t/\sqrt{n})$ , then apply Lévy's continuity theorem.

ProofBayes' Theorem from the Axioms

1. By the definition of conditional probability [4], $P(A\cap B)=P(A\mid B)\,P(B)$ . 2. By symmetry of intersection, $A\cap B=B\cap A$ , so also $P(A\cap B)=P(B\mid A)\,P(A)$ . 3. Equate steps 1 and 2: $P(A\mid B)\,P(B)=P(B\mid A)\,P(A)$ . 4. Divide both sides by $P(B)>0$ to obtain $P(A\mid B)=\dfrac{P(B\mid A)P(A)}{P(B)}$ . 5. For the denominator, suppose $\{A_i\}$ partitions $\Omega$ (disjoint, union $=\Omega$ ). Then $B=\bigcup_i(B\cap A_i)$ is a disjoint union, so by additivity [3], $P(B)=\sum_i P(B\cap A_i)=\sum_i P(B\mid A_i)P(A_i)$ . $\blacksquare$

∎

ProofWeak Law of Large Numbers via Chebyshev

1. Assume finite variance $\sigma^2$ for simplicity. Let $\bar X_n=\frac1n\sum_{i=1}^n X_i$ . 2. By linearity of expectation, $\mathbb{E}[\bar X_n]=\mu$ . 3. By independence, variances add: $\operatorname{Var}(\bar X_n)=\frac{1}{n^2}\sum_{i=1}^n\operatorname{Var}(X_i)=\frac{\sigma^2}{n}$ . 4. Chebyshev's inequality states $P(|Y-\mathbb{E}Y|\ge\varepsilon)\le \operatorname{Var}(Y)/\varepsilon^2$ . Apply with $Y=\bar X_n$ :

P(|\bar X_n-\mu|\ge\varepsilon)\le\frac{\sigma^2}{n\varepsilon^2}.

5. As $n\to\infty$ the right side $\to 0$ for every fixed $\varepsilon>0$ , so $\bar X_n\xrightarrow{P}\mu$ (convergence in probability). $\blacksquare$

∎

mathematician · Andrey Kolmogorov | 1903–1987

The Soviet mathematician Andrey Nikolaevich Kolmogorov transformed probability from a collection of intuitive techniques into a rigorous branch of mathematics. In his 1933 monograph Grundbegriffe der Wahrscheinlichkeitsrechnung ("Foundations of the Theory of Probability"), he founded the subject on measure theory, defining probability spaces $(\Omega,\mathcal F,P)$ and the three axioms above. This grounded random variables as measurable functions and expectations as Lebesgue integrals, unifying discrete and continuous cases. Kolmogorov also made deep contributions to stochastic processes, turbulence, algorithmic complexity, and the strong law of large numbers.

🧠 Intuition

Think of a probability measure as distributing one unit of "mass" (or paint) over the sample space. Events are regions; $P(A)$ is the amount of paint that fell on region $A$ . The axioms say: paint amounts are never negative [1], the total is exactly 1 [2], and the paint on disjoint regions adds up [3].

Conditioning is zooming in: when you learn $B$ occurred, you discard all paint outside $B$ and rescale what's left so it again totals 1 — that's why we divide by $P(B)$ .

Independence means knowing one region's mass tells you nothing about another's proportionally — the joint mass factors. The two great limit theorems describe what happens when you average many independent splatters: the average position stabilizes (LLN) and the shape of the fluctuation becomes a bell curve (CLT), no matter what the individual splatter shapes were.

Figure

Standard normal density (limit shape from the CLT) with ±1σ region shaded

ExampleBayes with a Diagnostic Test

Given: A disease affects 1% of a population ( $P(D)=0.01$ ). A test has sensitivity $P(+\mid D)=0.99$ and specificity $P(-\mid \neg D)=0.95$ , so the false-positive rate is $P(+\mid\neg D)=0.05$ . A person tests positive. What is $P(D\mid +)$ ?

Steps: 1. Total probability of a positive test [7]:

P(+)=P(+\mid D)P(D)+P(+\mid\neg D)P(\neg D)=0.99(0.01)+0.05(0.99).

2. Compute: $0.0099+0.0495=0.0594$ . 3. Apply Bayes [7]: $P(D\mid +)=\dfrac{0.99\times 0.01}{0.0594}=\dfrac{0.0099}{0.0594}$ . 4. Evaluate: $\approx 0.1667$ .

Answer: $\boxed{P(D\mid +)\approx 16.7\%}$ — despite the "99% accurate" test, a positive result is more likely false than true because the disease is rare (the base-rate effect).

ExampleExpectation and Variance of a Fair Die

Given: $X$ = outcome of one roll of a fair six-sided die, $p_X(k)=1/6$ for $k=1,\dots,6$ .

Steps: 1. Expectation [6]: $\mathbb{E}[X]=\sum_{k=1}^6 k\cdot\tfrac16=\tfrac{1+2+3+4+5+6}{6}=\tfrac{21}{6}=3.5$ . 2. Second moment: $\mathbb{E}[X^2]=\sum_{k=1}^6 k^2\cdot\tfrac16=\tfrac{1+4+9+16+25+36}{6}=\tfrac{91}{6}\approx 15.1667$ . 3. Variance: $\operatorname{Var}(X)=\mathbb{E}[X^2]-(\mathbb{E}X)^2=\tfrac{91}{6}-\left(\tfrac{7}{2}\right)^2=15.1667-12.25=2.9167$ . 4. Standard deviation: $\sigma=\sqrt{2.9167}\approx 1.708$ .

Answer: $\boxed{\mathbb{E}[X]=3.5,\quad \operatorname{Var}(X)=35/12\approx 2.917,\quad \sigma\approx 1.71}$ (dimensionless pips).

⚠️ Independence ≠ Mutual Exclusivity

These are opposite concepts. Disjoint (mutually exclusive) events with positive probability are never independent: if $A\cap B=\varnothing$ then $P(A\cap B)=0\ne P(A)P(B)>0$ , and learning $B$ occurred makes $A$ impossible. Independence means one event gives no information about the other; mutual exclusivity means one event gives maximal information.

⚠️ The Gambler's Fallacy

The Law of Large Numbers does not say the universe "balances" short-run streaks. After ten heads in a row, a fair coin's next toss is still $P(H)=1/2$ . LLN [8] guarantees the average converges, but it does so by diluting early deviations across an ever-larger $n$ , not by compensating with opposite outcomes.

CounterexamplePairwise Independence Does Not Imply Mutual Independence

Toss two fair coins. Let $A=\{$ first is H $\}$ , $B=\{$ second is H $\}$ , $C=\{$ the two coins differ $\}$ . Each has probability $1/2$ , and every pair is independent: e.g. $P(A\cap C)=P(\text{first H, coins differ})=P(\text{HT})=1/4=P(A)P(C)$ . But the three are not mutually independent: $P(A\cap B\cap C)=P(\text{first H, second H, they differ})=0$ , whereas $P(A)P(B)P(C)=1/8$ . Knowing $A$ and $B$ determines $C$ exactly.

connection

Probability theory sits at a crossroads. It is built on Measure Theory — a probability measure is precisely a measure of total mass 1, and expectation is a Lebesgue integral. It feeds directly into Statistics, where probability models data and we invert the reasoning to estimate parameters. The study of probability over time gives Stochastic Processes (Markov chains, Brownian motion, martingales). Through Information Theory, entropy $H=-\sum p\log p$ quantifies the uncertainty of a distribution. Combinatorics underlies classical equally-likely-outcome probabilities, and the Central Limit Theorem connects probability to the ubiquitous Normal Distribution.

🔬 Where this appears

⏳ History

- 1654 — Blaise Pascal and Pierre de Fermat correspond over gambling problems (the "problem of points"), launching the mathematical theory of chance.

- 1657 — Christiaan Huygens publishes De Ratiociniis in Ludo Aleae, the first printed treatise on probability, introducing expectation.

- 1713 — Jacob Bernoulli's posthumous Ars Conjectandi states the first Law of Large Numbers (the "golden theorem").

- 1733/1738 — Abraham de Moivre derives the normal approximation to the binomial, an early Central Limit Theorem.

- 1763 — Thomas Bayes' essay (published posthumously by Richard Price) introduces inverse probability, later formalized as Bayes' theorem.

- 1812 — Pierre-Simon Laplace's Théorie analytique des probabilités systematizes the classical theory and generating functions.

- 1900s — Émile Borel, Henri Lebesgue, and others develop measure theory, providing the needed analytic foundation.

- 1933 — Andrey Kolmogorov publishes his axiomatic measure-theoretic foundations, the modern basis of the field.

- 1951–1968 — Itô calculus, martingale theory (Doob), and modern stochastic process theory mature, extending probability into continuous time.

open-problem

Foundational tensions remain. The measure-theoretic framework is mathematically settled, but the interpretation of probability is still debated: frequentists read $P(A)$ as a long-run frequency, while Bayesians read it as a degree of belief — these diverge for one-off events ("probability that this hypothesis is true"). On the technical side, non-measurable sets (e.g. Vitali sets, requiring the Axiom of Choice) show that not every subset of $\Omega$ can be assigned a probability, forcing the $\sigma$ -algebra restriction. Defining conditioning on events of probability zero (the Borel–Kolmogorov paradox) shows conditional probability can depend on how a limiting event is approached. And reconciling classical probability with quantum probability — where amplitudes interfere and Kolmogorov's additivity [3] fails for unobserved paths — remains a live conceptual frontier.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also