Probability Foundations

⚡ Key Takeaways

Probability is the mathematical measure of how likely an event is, expressed as a number between 0 (impossible) and 1 (certain). The core result is Kolmogorov's axioms: all of probability theory follows from three simple rules about non-negativity, total certainty, and additivity of mutually exclusive events. It applies everywhere uncertainty appears — gambling, physics, statistics, machine learning, insurance, and medicine. The modern foundations were laid by Andrey Kolmogorov in 1933, building on three centuries of work since Pascal and Fermat (1654).

💡 The Core Insight

Probability is not a property of single events but a consistent system of weights spread across all possible outcomes, summing to 1. Once you fix this distribution of weight over the sample space, the probability of any event is simply the total weight it captures. Everything else — conditional probability, independence, expected value — is bookkeeping built on this single picture of "total certainty = 1, divided up among possibilities."

DefinitionSample Space and Events

A random experiment is any process with an uncertain outcome. The sample space $\Omega$ is the set of all possible outcomes. An event is a subset $A \subseteq \Omega$ . An elementary outcome $\omega \in \Omega$ is a single point.

Tossing a coin: $\Omega = \{H, T\}$ .
Rolling a die: $\Omega = \{1,2,3,4,5,6\}$ ; the event "even" is $A = \{2,4,6\}$ .
$\Omega$ itself is the certain event; $\varnothing$ is the impossible event.

DefinitionProbability Function — Kolmogorov's Axioms

A probability measure $P$ assigns to each event a real number satisfying three axioms:

1

\quad P(A) \ge 0 \quad \text{for every event } A

Derivation hint: This is an axiom (a chosen rule), not a theorem — probabilities cannot be negative because they measure a "size."

2

\quad P(\Omega) = 1

Derivation hint: An axiom asserting that something in the sample space is certain to happen; total weight is normalized to 1.

3

\quad P\!\left(\bigcup_{i=1}^{\infty} A_i\right) = \sum_{i=1}^{\infty} P(A_i) \quad \text{when the } A_i \text{ are pairwise disjoint}

Derivation hint: Axiom of countable additivity — if events cannot overlap, the probability of "any one of them" is the sum of their separate probabilities. Here $A_i$ are events with $A_i \cap A_j = \varnothing$ for $i \ne j$ . Symbols: $P$ = probability function, $\Omega$ = sample space, $A_i$ = events, $\bigcup$ = union ("or"), $\sum$ = sum.

TheoremConsequences of the Axioms

From the three axioms, these follow for all events $A, B$ :

4

\quad P(\varnothing) = 0

Derivation hint: $\Omega = \Omega \cup \varnothing \cup \varnothing \cup \cdots$ with disjoint sets; apply axiom [3] and subtract.

5

\quad P(A^c) = 1 - P(A)

Derivation hint: $A$ and its complement $A^c$ are disjoint and union to $\Omega$ ; apply [2] and [3].

6

\quad P(A \cup B) = P(A) + P(B) - P(A \cap B)

Derivation hint: Split $A \cup B$ into disjoint pieces; the overlap $A \cap B$ gets counted twice and must be subtracted once (inclusion–exclusion).

7

\quad A \subseteq B \implies P(A) \le P(B)

Derivation hint: Write $B = A \cup (B \setminus A)$ as disjoint union; the extra term is $\ge 0$ by [1].

DefinitionClassical (Equally Likely) Probability

When $\Omega$ is finite and every outcome is equally likely:

8

\quad P(A) = \frac{|A|}{|\Omega|} = \frac{\text{number of favorable outcomes}}{\text{total number of outcomes}}

Derivation hint: Each of $|\Omega|$ outcomes carries equal weight $1/|\Omega|$ by symmetry and axiom [2]; summing over $A$ 's outcomes via [3] gives the ratio.

DefinitionConditional Probability and Independence

The probability of $A$ given that $B$ has occurred:

9

\quad P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \quad P(B) > 0

Derivation hint: Restrict the sample space to $B$ (which now plays the role of certainty), then re-normalize the weight of $A \cap B$ by dividing by $P(B)$ . Events $A$ and $B$ are independent if knowing one tells nothing about the other:

10

\quad P(A \cap B) = P(A)\,P(B)

Derivation hint: Substitute [10] into [9]: $P(A\mid B) = P(A)$ , i.e. conditioning on $B$ leaves $A$ 's probability unchanged.

TheoremBayes' Theorem

For events with positive probability:

11

\quad P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}

Derivation hint: Both $P(A\mid B)P(B)$ and $P(B\mid A)P(A)$ equal $P(A\cap B)$ by [9]; equate and divide by $P(B)$ . With the law of total probability for a partition $A_1,\dots,A_n$ of $\Omega$ :

12

\quad P(B) = \sum_{i=1}^{n} P(B \mid A_i)\,P(A_i)

Derivation hint: $B = \bigcup_i (B \cap A_i)$ is a disjoint union; apply [3], then [9] to each term.

ProofDeriving the Complement Rule (Equation 5)

1. By definition, $A$ and its complement $A^c$ satisfy $A \cap A^c = \varnothing$ (they share no outcomes). 2. Their union is everything: $A \cup A^c = \Omega$ . 3. Since they are disjoint, axiom [3] gives $P(A \cup A^c) = P(A) + P(A^c)$ . 4. But $A \cup A^c = \Omega$ , so by axiom [2], $P(A) + P(A^c) = P(\Omega) = 1$ . 5. Rearranging: $P(A^c) = 1 - P(A)$ . $\blacksquare$

∎

ProofDeriving Inclusion–Exclusion (Equation 6)

1. Decompose $A \cup B$ into three disjoint pieces: $(A \setminus B)$ , $(A \cap B)$ , and $(B \setminus A)$ . 2. By axiom [3]: $P(A \cup B) = P(A\setminus B) + P(A\cap B) + P(B\setminus A)$ . 3. Note $P(A) = P(A\setminus B) + P(A\cap B)$ and $P(B) = P(B\setminus A) + P(A\cap B)$ , since each event splits into its overlapping and non-overlapping parts. 4. Substitute $P(A\setminus B) = P(A) - P(A\cap B)$ and $P(B\setminus A) = P(B) - P(A\cap B)$ into step 2. 5. This gives $P(A\cup B) = [P(A) - P(A\cap B)] + P(A\cap B) + [P(B) - P(A\cap B)] = P(A) + P(B) - P(A\cap B)$ . $\blacksquare$

∎

mathematician · Andrey Kolmogorov | 1903–1987

The Russian mathematician Andrey Nikolaevich Kolmogorov gave probability its rigorous modern foundation in his 1933 monograph Grundbegriffe der Wahrscheinlichkeitsrechnung ("Foundations of the Theory of Probability"). His insight was to identify probability with a measure — the same abstract notion of "size" used to define length, area, and volume — defined on a $\sigma$ -algebra of events. This dissolved centuries of paradoxes and made probability a respectable branch of pure mathematics. Kolmogorov also made landmark contributions to turbulence, topology, complexity theory (Kolmogorov complexity), and dynamical systems, making him one of the most versatile mathematicians of the 20th century.

🧠 Intuition

Picture the sample space $\Omega$ as a region of total area exactly $1$ — a unit of "certainty." Each event is a sub-region, and its probability is its area. The axioms become visually obvious:

Areas can't be negative (axiom [1]).
The whole region has area 1 (axiom [2]).
Non-overlapping regions have areas that add (axiom [3]).

Conditional probability $P(A\mid B)$ means: zoom in so that region $B$ now fills your entire view (becomes the new "100%"), and ask what fraction of that view is occupied by $A$ . Independence means $A$ takes up the same fraction of $B$ as it does of the whole space — knowing you're inside $B$ tells you nothing new about $A$ .

Figure

The sample space as area 1, with overlapping events A and B

ExampleTwo Dice — Sum and Conditioning

Given: Roll two fair six-sided dice. The sample space has $|\Omega| = 6 \times 6 = 36$ equally likely outcomes.

Question 1: What is $P(\text{sum} = 7)$ ? Steps: The favorable pairs are $(1,6),(2,5),(3,4),(4,3),(5,2),(6,1)$ — that's $6$ outcomes. By equation [8]:

P(\text{sum}=7) = \frac{6}{36} = \frac{1}{6}.

Question 2: Given that the first die shows a $3$ , what is $P(\text{sum} = 7 \mid \text{first} = 3)$ ? Steps: Use equation [9]. Event $B = \{\text{first}=3\}$ has $6$ outcomes, so $P(B) = 6/36$ . The intersection (first = 3 and sum = 7) is just $(3,4)$ , so $P(A\cap B) = 1/36$ . Then:

P(A\mid B) = \frac{1/36}{6/36} = \frac{1}{6}.

The answers are equal — consistent with independence of the two dice.

Boxed answers:

\boxed{P(\text{sum}=7) = \tfrac{1}{6} \approx 0.167, \qquad P(\text{sum}=7\mid \text{first}=3) = \tfrac{1}{6} \approx 0.167}

ExampleMedical Test — Bayes' Theorem in Action

Given: A disease affects $1\%$ of a population. A test is $99\%$ accurate both ways: $P(\text{positive}\mid \text{disease}) = 0.99$ and $P(\text{negative}\mid \text{no disease}) = 0.99$ (so the false-positive rate is $0.01$ ). A person tests positive. What is the probability they actually have the disease? Steps: Let $D$ = "has disease," $T$ = "tests positive."

Prior: $P(D) = 0.01$ , $P(D^c) = 0.99$ .
$P(T\mid D) = 0.99$ , $P(T\mid D^c) = 0.01$ .

First, total probability of a positive test (equation [12]):

P(T) = (0.99)(0.01) + (0.01)(0.99) = 0.0099 + 0.0099 = 0.0198.

Now apply Bayes' theorem (equation [11]):

P(D\mid T) = \frac{P(T\mid D)\,P(D)}{P(T)} = \frac{(0.99)(0.01)}{0.0198} = \frac{0.0099}{0.0198} = 0.5.

Boxed answer:

\boxed{P(\text{disease} \mid \text{positive test}) = 0.50}

Despite a "99% accurate" test, a positive result means only a 50% chance of disease — because the disease is rare, false positives are as numerous as true positives. This is the famous base-rate fallacy.

⚠️ The Gambler's Fallacy

Independent events have no memory. After five heads in a row with a fair coin, the probability of heads on the next toss is still exactly $\tfrac{1}{2}$ — the coin does not "owe" you a tail. By equation [10], past tosses are independent of future ones, so

P(H_6 \mid H_1\cap\cdots\cap H_5) = P(H_6) = \tfrac{1}{2}.

The belief that "tails is due" confuses independent trials with self-correcting systems.

⚠️ Confusing

P(A\mid B)

with

P(B\mid A)

These are not equal in general — that's exactly why Bayes' theorem (equation [11]) is needed. The medical example shows $P(\text{positive}\mid\text{disease}) = 0.99$ but $P(\text{disease}\mid\text{positive}) = 0.50$ . Reversing the conditioning (the "prosecutor's fallacy") has led to real wrongful convictions.

CounterexamplePairwise Independence ≠ Mutual Independence

Toss two fair coins. Define three events: $A$ = "first is heads," $B$ = "second is heads," $C$ = "the two coins differ." Each pair is independent: $P(A\cap C) = \tfrac14 = P(A)P(C)$ , and similarly for the others. But the three are not mutually independent, because $A \cap B$ forces "same," making $C$ impossible: $P(A\cap B\cap C) = 0 \ne P(A)P(B)P(C) = \tfrac18$ . Pairwise independence does not imply joint independence.

connection

Probability foundations sit at the crossroads of several areas. The classical formula [8] rests on Combinatorics for counting favorable and total outcomes. The axioms are a special case of Measure Theory, where $P$ is a measure with total mass 1. The transition from discrete to continuous sample spaces requires Calculus and the Probability Density Function, replacing sums with integrals. Bayes' theorem [11] is the engine of Bayesian Inference and modern machine learning. The behavior of repeated experiments connects to Law of Large Numbers and the Central Limit Theorem, which explain why averages stabilize and why the Normal Distribution appears so universally.

🔬 Where this appears

⏳ History

- 1654 — Blaise Pascal and Pierre de Fermat exchange letters solving the "problem of points" (how to divide stakes in an interrupted game of chance), founding mathematical probability.

- 1657 — Christiaan Huygens publishes De Ratiociniis in Ludo Aleae, the first printed treatise on probability, introducing expected value.

- 1713 — Jacob Bernoulli's posthumous Ars Conjectandi proves the first Law of Large Numbers.

- 1733–1738 — Abraham de Moivre derives the normal curve as an approximation to the binomial distribution.

- 1763 — Thomas Bayes' theorem on inverse probability is published posthumously by Richard Price.

- 1812 — Pierre-Simon Laplace's Théorie analytique des probabilités systematizes the classical theory and the principle of equally likely outcomes.

- 1900 — Hilbert's sixth problem calls for the axiomatization of physics, explicitly including probability.

- 1933 — Andrey Kolmogorov publishes his measure-theoretic axioms, giving probability rigorous foundations and ending centuries of paradox.

- 1950s–present — Frequentist vs. Bayesian interpretations are debated; computation and machine learning make Bayesian methods practical and dominant.

open-problem

What does probability actually mean? Kolmogorov's axioms tell you how to compute with probabilities, but not what a probability is. Two major interpretations compete:

Frequentist: $P(A)$ is the long-run relative frequency of $A$ in repeated trials. But this struggles with one-time events ("probability it rains tomorrow") and never quite defines the limit rigorously.
Bayesian: $P(A)$ is a degree of rational belief, updated via Bayes' theorem as evidence arrives. But whose belief, and what prior is "correct"?

Deeper subtleties: not every subset of a continuous sample space can be assigned a probability (non-measurable sets, related to the Banach–Tarski paradox), which is why measure theory restricts events to a $\sigma$ -algebra. And the foundations of inference — how much data justifies a conclusion — remain philosophically contested. Probability is mathematically settled but interpretively open.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also