Probability & Statistics

⚡ Key Takeaways

Probability measures how likely an event is on a scale from 0 (impossible) to 1 (certain); statistics is the science of collecting, summarising, and drawing conclusions from data. Key result in words: the long-run frequency of an event converges to its probability, and large random samples reliably reveal the structure of the population they come from. Applies to: gambling, insurance, quality control, medicine, polling, machine learning, and any reasoning under uncertainty. Foundations laid by Pascal & Fermat (1654), formalised by Kolmogorov (1933).

💡 The Core Insight

Randomness in the small produces order in the large. A single coin flip is unpredictable, but the average behaviour of many independent repetitions is governed by precise mathematical laws. Probability builds a deductive model of this randomness from axioms; statistics uses observed averages to work backwards to the model's hidden parameters. The bridge between the two is the idea that relative frequency estimates probability, and that estimate improves predictably as the sample grows.

DefinitionSample Space, Events, and Probability

A random experiment has a sample space $\Omega$ , the set of all possible outcomes. An event $A$ is any subset of $\Omega$ . A probability measure $P$ assigns to each event a number satisfying Kolmogorov's axioms:

1

\quad P(A) \ge 0 \quad\text{for every event } A

Derivation hint: probabilities are non-negative by definition — they count "share of certainty."

2

\quad P(\Omega) = 1

Derivation hint: something in the sample space must occur, so the total probability is 1.

3

\quad P(A \cup B) = P(A) + P(B) \quad\text{if } A \cap B = \varnothing

Derivation hint: for mutually exclusive (disjoint) events, probabilities simply add. Here $\Omega$ is the set of outcomes, $A,B$ are events, $\cup$ is union ("or"), $\cap$ is intersection ("and"), and $\varnothing$ is the empty event (impossible).

DefinitionConditional Probability and Independence

The conditional probability of $A$ given that $B$ has occurred is

4

\quad P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \qquad P(B) > 0

Derivation hint: restrict the sample space to $B$ and renormalise so $P(B\mid B)=1$ . Events $A$ and $B$ are independent when knowing one tells nothing about the other:

5

\quad P(A \cap B) = P(A)\,P(B)

Derivation hint: substitute independence into [4] to get $P(A\mid B)=P(A)$ .

TheoremBayes' Theorem

For events $A$ and $B$ with $P(B)>0$ ,

6

\quad P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}

Derivation hint: write $P(A\cap B)$ two ways using [4] — as $P(A\mid B)P(B)$ and $P(B\mid A)P(A)$ — then solve. If $A_1,\dots,A_n$ partition $\Omega$ , the denominator expands by the law of total probability:

7

\quad P(B) = \sum_{i=1}^{n} P(B \mid A_i)\,P(A_i)

Derivation hint: $B$ is the disjoint union of the $B\cap A_i$ ; apply [3] repeatedly and [4].

DefinitionRandom Variables, Expectation, Variance

A random variable $X$ assigns a number to each outcome. Its expected value (mean) and variance are

8

\quad E[X] = \sum_i x_i\,P(X = x_i) \quad\text{(discrete)}, \qquad E[X] = \int_{-\infty}^{\infty} x\,f(x)\,dx \quad\text{(continuous)}

Derivation hint: a probability-weighted average of all values $x_i$ , with $f(x)$ the probability density.

9

\quad \operatorname{Var}(X) = E\big[(X - \mu)^2\big] = E[X^2] - \mu^2, \qquad \mu = E[X]

Derivation hint: expand $(X-\mu)^2$ and use linearity of $E$ , noting $E[\mu^2]=\mu^2$ . The standard deviation is $\sigma = \sqrt{\operatorname{Var}(X)}$ , in the same units as $X$ .

TheoremKey Distributions

The binomial distribution counts $k$ successes in $n$ independent trials each with success probability $p$ :

10

\quad P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}, \qquad k = 0,1,\dots,n

Derivation hint: each specific sequence has probability $p^k(1-p)^{n-k}$ ; multiply by the $\binom{n}{k}$ orderings. The normal (Gaussian) distribution with mean $\mu$ and standard deviation $\sigma$ has density

11

\quad f(x) = \frac{1}{\sigma\sqrt{2\pi}}\,\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)

Derivation hint: the unique symmetric density whose log is quadratic; the constant $\frac{1}{\sigma\sqrt{2\pi}}$ makes the total area 1.

ProofThe Mean and Variance of a Sample Average

Let $X_1,\dots,X_n$ be independent random variables, each with mean $\mu$ and variance $\sigma^2$ , and let $\bar X = \frac1n\sum_i X_i$ be the sample mean. 1. Mean of the average. By linearity of expectation, $E[\bar X] = \frac1n\sum_i E[X_i] = \frac1n(n\mu) = \mu$ . The sample mean is an unbiased estimator of $\mu$ . 2. Variance of a sum of independent variables. For independent variables, variances add: $\operatorname{Var}\!\left(\sum_i X_i\right) = \sum_i \operatorname{Var}(X_i) = n\sigma^2$ . 3. Scaling. Since $\operatorname{Var}(aY) = a^2\operatorname{Var}(Y)$ with $a = 1/n$ ,

12

\quad \operatorname{Var}(\bar X) = \frac{1}{n^2}\,(n\sigma^2) = \frac{\sigma^2}{n}

Derivation hint: combine additivity of variance for independent terms with the $a^2$ scaling rule. 4. Consequence. The standard deviation of $\bar X$ is $\sigma/\sqrt{n}$ , the standard error. As $n\to\infty$ it shrinks to 0, so $\bar X$ concentrates ever more tightly around $\mu$ — this is the quantitative heart of the Law of Large Numbers. ∎

∎

mathematician · Andrey Kolmogorov | 1903–1987

The Soviet mathematician Andrey Kolmogorov transformed probability from a collection of clever gambling tricks into a rigorous branch of mathematics. In his 1933 monograph Grundbegriffe der Wahrscheinlichkeitsrechnung he proposed that probability is simply a measure — a non-negative function on sets summing to 1 — borrowing the machinery of measure theory developed by Lebesgue. This axiomatic foundation (equations [1]–[3]) settled centuries of philosophical dispute about what probability is by declining to answer it: instead it specified the rules probability must obey, leaving interpretation open. Kolmogorov also made deep contributions to turbulence, dynamical systems, and complexity theory.

🧠 Intuition

Think of probability as shared-out certainty. You have one unit of belief to distribute over all possible outcomes; equation [2] says it all has to go somewhere, and [1] says you can't assign negative belief. Conditional probability [4] is zooming in: once you learn $B$ happened, you throw away the part of the world where it didn't and rescale what remains so it again sums to 1. For statistics, picture a giant opaque urn (the population) and a handful of balls you draw out (the sample). You never see the whole urn, but each handful is a blurry photograph of it. The blur is the standard error $\sigma/\sqrt n$ from [12]: bigger handfuls give sharper photographs, but only as fast as $\sqrt n$ — to halve the blur you must quadruple the sample. This square-root law is why national polls of 1,000 people work surprisingly well, yet doubling accuracy is genuinely expensive.

Figure

The normal distribution: the 68–95–99.7 rule. About 68% of probability lies within one standard deviation of the mean.

ExampleDisease Testing and Bayes' Theorem

Given: A disease affects 1% of a population. A test is 99% sensitive (detects the disease when present) and 95% specific (correctly negative when absent). A randomly chosen person tests positive. What is the probability they actually have the disease?

Steps: Let $D$ = has disease, $T$ = tests positive.

$P(D) = 0.01$ , $P(\neg D) = 0.99$
$P(T\mid D) = 0.99$ , $P(T\mid \neg D) = 1 - 0.95 = 0.05$
Total probability of a positive test [7]:

P(T) = (0.99)(0.01) + (0.05)(0.99) = 0.0099 + 0.0495 = 0.0594

Bayes' theorem [6]:

P(D\mid T) = \frac{(0.99)(0.01)}{0.0594} = \frac{0.0099}{0.0594} \approx 0.167

Answer: $\boxed{P(D \mid T) \approx 0.17 = 17\%}$ — despite the accurate test, most positives are false alarms because the disease is rare. This is the base-rate effect.

ExampleSampling and the Standard Error

Given: Heights of adult women have $\mu = 162$ cm and $\sigma = 7$ cm. A random sample of $n = 49$ women is taken. Within what range does the sample mean $\bar X$ fall about 95% of the time?

Steps:

Standard error from [12]: $\dfrac{\sigma}{\sqrt n} = \dfrac{7}{\sqrt{49}} = \dfrac{7}{7} = 1.0$ cm.
By the Central Limit Theorem, $\bar X$ is approximately normal about $\mu = 162$ cm.
The 95% range is roughly $\mu \pm 2\,(\text{SE}) = 162 \pm 2(1.0)$ .

Answer: $\boxed{160\ \text{cm} \le \bar X \le 164\ \text{cm} \ \text{(about 95\% of samples)}}$ . Note the individual heights vary by $\pm 14$ cm, but the average of 49 varies only by $\pm 2$ cm — averaging suppresses noise.

⚠️ The Gambler's Fallacy

Independent events have no memory. After five heads in a row, a fair coin is still exactly 50% to land heads next — equation [5] guarantees past flips do not change future ones. The belief that tails is "due" confuses the Law of Large Numbers (long-run averaging) with a non-existent self-correcting force. The average drifts toward 0.5 not because future flips compensate for the past, but because past flips become an ever-smaller fraction of the total.

⚠️ Correlation Is Not Causation

Two variables can move together — high correlation — without one causing the other. Ice-cream sales and drowning deaths correlate strongly, but neither causes the other; both rise with summer temperature, a confounding variable. Statistics can establish association; only a controlled experiment or careful causal reasoning can establish cause.

CounterexampleProbability of "At Least One" Is Not a Simple Sum

A common error: with a 1-in-6 chance per roll, "the probability of at least one six in 6 rolls" is not $6\times\frac16 = 1$ . Probabilities of non-exclusive events don't simply add (that violates the disjointness condition in [3]). Compute via the complement instead:

P(\text{at least one six}) = 1 - \left(\tfrac56\right)^6 \approx 1 - 0.335 = 0.665.

The naïve sum even exceeds 1 for 7 rolls, which is impossible by [1]–[2].

connection

Probability is the deductive engine; statistics runs it in reverse for inference. The Central Limit Theorem explains why the Normal Distribution appears everywhere, justifying the confidence intervals and hypothesis tests of Statistical Inference. Bayes' Theorem generalises into Bayesian Inference, an entire framework for updating beliefs with data. The expectation and variance of [8]–[9] connect to Combinatorics (for counting outcomes) and to Calculus (integrals define continuous expectations and densities). Variance and the standard error link to Regression Analysis, where the same square-of-distance ideas underpin least-squares fitting.

🔬 Where this appears

⏳ History

- 1654 — Blaise Pascal and Pierre de Fermat exchange letters solving the "problem of points," founding mathematical probability.

- 1657 — Christiaan Huygens publishes De ratiociniis in ludo aleae, the first printed treatise on probability, introducing expected value.

- 1713 — Jacob Bernoulli's posthumous Ars Conjectandi states and proves the first Law of Large Numbers.

- 1733/1738 — Abraham de Moivre derives the normal curve as an approximation to the binomial, an early Central Limit Theorem.

- 1763 — Thomas Bayes' essay (published posthumously) introduces what becomes Bayes' theorem.

- 1809–1812 — Carl Friedrich Gauss and Pierre-Simon Laplace develop least squares and the general central limit theorem; the Gaussian distribution takes Gauss's name.

- 1900s–1920s — Karl Pearson, Ronald Fisher, and Jerzy Neyman build modern statistics: correlation, the chi-squared test, maximum likelihood, experimental design, and hypothesis testing.

- 1933 — Andrey Kolmogorov axiomatises probability with measure theory, giving the field its rigorous modern foundation.

open-problem

The meaning of probability remains philosophically contested. Frequentists define $P(A)$ as the long-run fraction of times $A$ occurs in repeated trials — but this is awkward for one-off events ("the probability it rains tomorrow"). Bayesians treat probability as a degree of belief, updated by [6] — but then the choice of prior belief is subjective. Both interpretations use the same axioms [1]–[3]; they disagree on what the numbers mean. Deeper still: real data are rarely truly independent or identically distributed, so the clean theorems above are idealisations. Modern statistics wrestles with the replication crisis — many published findings fail to reproduce, partly from misuse of $p$ -values and " $p$ -hacking" — and with making valid inferences from massive, messy, non-random datasets where the classical $\sqrt n$ guarantees may not apply.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also