Foundations of Statistics

⚡ Key Takeaways

Foundations of Statistics concerns the philosophical and mathematical principles that justify how we infer properties of a population from a sample. Key result in words: rival schools (frequentist, Bayesian, likelihoodist) interpret probability and inference differently, but each rests on a coherent axiomatic core — probability theory plus a rule connecting data to belief or decision. Where it applies: every domain that draws conclusions from data — science, medicine, economics, machine learning. Who/when: crystallized 1920s–1950s through R.A. Fisher, Jerzy Neyman, Egon Pearson, Harold Jeffreys, and Abraham Wald.

💡 The Core Insight

All of statistical inference reduces to one question: given data $x$ generated by an unknown distribution $P_\theta$ from a known family $\{P_\theta : \theta \in \Theta\}$ , what can we say about $\theta$ ? The likelihood function $L(\theta) = p(x \mid \theta)$ — the data's probability viewed as a function of the parameter — is the bridge between observation and inference. The deep disagreements in statistics are not about mathematics but about what to do with the likelihood: maximize it (Fisher), embed it in long-run error control (Neyman–Pearson), or combine it with a prior via Bayes' theorem (Bayesian).

DefinitionStatistical Model

A statistical model (or experiment) is a triple $(\mathcal{X}, \mathcal{A}, \{P_\theta : \theta \in \Theta\})$ where $\mathcal{X}$ is the sample space of possible data, $\mathcal{A}$ a $\sigma$ -algebra of events, and $\{P_\theta\}$ a family of probability measures on $(\mathcal{X}, \mathcal{A})$ indexed by a parameter $\theta$ ranging over the parameter space $\Theta$ . The model is parametric if $\Theta \subseteq \mathbb{R}^k$ , nonparametric if $\Theta$ is infinite-dimensional. A statistic is any measurable function $T : \mathcal{X} \to \mathbb{R}^m$ of the data alone.

DefinitionSufficiency

A statistic $T$ is sufficient for $\theta$ if the conditional distribution of the data $X$ given $T = t$ does not depend on $\theta$ . Intuitively, $T$ captures all information in the sample relevant to $\theta$ .

TheoremFisher–Neyman Factorization

A statistic $T$ is sufficient for $\theta$ if and only if the density factorizes as

3

\quad p(x \mid \theta) = g\big(T(x), \theta\big)\, h(x)

where $g$ depends on the data only through $T$ and $h$ does not depend on $\theta$ .

TheoremLikelihood Principle

If two experiments yield data $x$ and $y$ (possibly from different sample spaces) such that the likelihood functions are proportional,

4

\quad L_1(\theta \mid x) = c\, L_2(\theta \mid y) \quad \text{for all } \theta, \ \ c > 0,

then $x$ and $y$ carry identical evidence about $\theta$ , and any inference must be the same. Birnbaum (1962) proved this follows from the Sufficiency Principle plus the Conditionality Principle.

ProofBirnbaum's Theorem — Sufficiency + Conditionality ⟹ Likelihood

1. Setup. Let $E_1, E_2$ be two experiments with the same parameter $\theta$ , yielding outcomes $x$ (from $E_1$ ) and $y$ (from $E_2$ ) with proportional likelihoods, $L_1(\theta\mid x) = c\,L_2(\theta\mid y)$ . 2. Mixed experiment. Define $E^*$ : flip a fair coin; if heads run $E_1$ , if tails run $E_2$ . An outcome of $E^*$ is a pair $(j, z)$ recording which experiment was run and its result. 3. Conditionality Principle (C). Inference from observing $(j, z)$ in $E^*$ must equal inference from running $E_j$ alone and observing $z$ — the unperformed experiment is irrelevant. Thus $\mathrm{Ev}(E^*, (1,x)) = \mathrm{Ev}(E_1, x)$ and $\mathrm{Ev}(E^*, (2,y)) = \mathrm{Ev}(E_2, y)$ . 4. Sufficiency within $E^*$ . Construct a statistic $T$ on $E^*$ that maps both $(1,x)$ and $(2,y)$ to a common value (legitimate because their likelihoods are proportional, so the conditional distribution given $T$ is $\theta$ -free). $T$ is sufficient. 5. Sufficiency Principle (S). Two outcomes with the same value of a sufficient statistic carry the same evidence: $\mathrm{Ev}(E^*, (1,x)) = \mathrm{Ev}(E^*, (2,y))$ . 6. Conclusion. Chaining steps 3 and 5: $\mathrm{Ev}(E_1, x) = \mathrm{Ev}(E_2, y)$ . Proportional likelihoods imply identical evidence. $\blacksquare$

∎

⚠️ Common mistake

Birnbaum's theorem is genuinely unsettling: nearly all statisticians accept Sufficiency and Conditionality as obvious, yet the Likelihood Principle they jointly imply forbids frequentist procedures whose answers depend on the whole sample space (the unobserved data), such as p-values, confidence levels, and stopping rules. The theorem itself has been challenged (Durbin 1970, Evans 2013) on the grounds that the two principles are stated over different domains; the debate is unresolved.

mathematician · Ronald A. Fisher | 1890–1962

The towering figure of 20th-century statistics. Fisher introduced likelihood (distinguishing it sharply from probability), maximum likelihood estimation, sufficiency, information (the Fisher Information), and the design of experiments with randomization and factorial layouts. He fought bitterly with Neyman and Pearson over the meaning of inference, defending fiducial arguments and rejecting their decision-theoretic framing. His 1925 Statistical Methods for Research Workers and 1935 The Design of Experiments shaped how science is done. Brilliant, combative, and also a population geneticist who unified Mendelism with Darwin, Fisher's foundational concepts remain the vocabulary of the field even where his philosophy is rejected.

🧠 Intuition

Think of the parameter $\theta$ as a hidden dial setting on a machine that spits out data. You see the output but not the dial. The likelihood asks, for each candidate dial setting, "how plausibly would this machine have produced exactly what I saw?" A setting under which your data is wildly improbable is a poor explanation; a setting under which it is probable is a good one. The three schools then differ on the final move. The frequentist refuses to assign probability to the dial itself (it has one fixed unknown value) and instead asks: "If I used this procedure repeatedly across many hypothetical datasets, how often would I be right?" — error rates over imagined replications. The Bayesian treats the dial setting as genuinely uncertain, places a prior probability on it, and updates to a posterior using Bayes' rule — probability as degree of belief. The likelihoodist stays minimal: report the likelihood curve itself and let relative likelihoods quantify evidence, refusing both error rates and priors. Same data, same likelihood, three philosophies of what counts as an answer.

Figure

The likelihood function for a binomial sample: 7 successes in 10 trials. The peak is the maximum likelihood estimate.

ExampleMaximum Likelihood for a Binomial Proportion

Given: A coin is flipped $n = 10$ times yielding $k = 7$ heads. Model: $X \sim \text{Binomial}(n, \theta)$ , $\theta$ = probability of heads.

Step 1 — Likelihood.

L(\theta) = \binom{10}{7}\theta^{7}(1-\theta)^{3}.

Step 2 — Log-likelihood (drop constant).

\ell(\theta) = 7\ln\theta + 3\ln(1-\theta) + \text{const}.

Step 3 — Differentiate and set to zero.

\frac{d\ell}{d\theta} = \frac{7}{\theta} - \frac{3}{1-\theta} = 0 \ \Rightarrow\ 7(1-\theta) = 3\theta \ \Rightarrow\ 7 = 10\theta.

Step 4 — Solve. $\hat\theta = 7/10$ . Second derivative is negative, confirming a maximum.

Step 5 — Fisher information & standard error. For one observation $I(\theta)=1/[\theta(1-\theta)]$ ; for $n$ trials the total is $n/[\theta(1-\theta)]$ . At $\hat\theta$ :

\widehat{\mathrm{SE}} = \sqrt{\frac{\hat\theta(1-\hat\theta)}{n}} = \sqrt{\frac{0.7\times0.3}{10}} = \sqrt{0.021} \approx 0.145.

\boxed{\hat\theta = 0.70, \qquad \widehat{\mathrm{SE}} \approx 0.145 \ (\text{dimensionless})}

ExampleBayesian Update with a Beta Prior

Given: Same data ( $k=7$ heads in $n=10$ ). Prior belief: $\theta \sim \text{Beta}(2,2)$ — symmetric, mild preference for fairness.

Step 1 — Prior density. $p(\theta) \propto \theta^{2-1}(1-\theta)^{2-1} = \theta(1-\theta)$ .

Step 2 — Likelihood. $L(\theta) \propto \theta^{7}(1-\theta)^{3}$ .

Step 3 — Posterior (Bayes).

p(\theta \mid x) \propto \theta(1-\theta)\cdot\theta^{7}(1-\theta)^{3} = \theta^{8}(1-\theta)^{4}.

This is $\text{Beta}(9, 5)$ — conjugacy: $\text{Beta}(a,b)$ updates to $\text{Beta}(a+k, b+n-k) = \text{Beta}(2+7, 2+3)$ .

Step 4 — Posterior mean.

\mathbb{E}[\theta\mid x] = \frac{9}{9+5} = \frac{9}{14} \approx 0.643.

Step 5 — 95% credible interval (numerically, Beta(9,5) quantiles): approximately $[0.39,\ 0.86]$ .

\boxed{\text{Posterior} = \text{Beta}(9,5),\quad \mathbb{E}[\theta\mid x]\approx 0.643,\quad 95\%\ \text{CrI} \approx [0.39, 0.86]}

Note the Bayesian estimate (0.643) is pulled below the MLE (0.70) toward the prior mean (0.5).

⚠️ The p-value is not P(H₀ is true)

A p-value is $P(\text{data as or more extreme} \mid H_0)$ — a probability about the data given the hypothesis, computed under the assumption that $H_0$ holds. It is not $P(H_0 \mid \text{data})$ , the probability the hypothesis is true. These can differ enormously: for a borderline result $p = 0.05$ the posterior probability of $H_0$ can exceed 0.30 under reasonable priors (Berger–Sellke 1987). Confusing the two — the "prosecutor's fallacy" — is the single most consequential error in applied statistics.

⚠️ A 95% confidence interval does not contain θ with probability 0.95

Once computed, a confidence interval like $[0.41, 0.99]$ either contains the fixed true $\theta$ or it does not — there is no probability left. The "95%" refers to the procedure: across many hypothetical datasets, 95% of the intervals it generates would cover $\theta$ . The probability lives in the random interval before you see data, not in the realized one. Only a Bayesian credible interval licenses the statement " $\theta$ lies here with probability 0.95."

CounterexampleStopping rules break frequentist inference but not likelihood

Two experimenters both observe 7 heads in 10 flips. Experimenter A fixed $n=10$ in advance (Binomial model). Experimenter B flipped until getting 3 tails (Negative Binomial model). The likelihood functions are proportional — $\propto \theta^7(1-\theta)^3$ — so by the Likelihood Principle their evidence about $\theta$ is identical. Yet the frequentist p-value for $H_0:\theta=0.5$ differs between the two designs ( $0.172$ for Binomial vs $0.0327$ for Negative Binomial), because the "more extreme" sample space differs. The same data yields significance under one stopping rule and not the other — a direct collision between frequentist practice and the Likelihood Principle.

connection

The foundations connect every branch of inference. Maximum Likelihood Estimation operationalizes the likelihood as a point estimator; Fisher Information quantifies the curvature of $\ell(\theta)$ that determines estimator precision via the Cramér–Rao Bound. The Neyman–Pearson Lemma founds optimal hypothesis testing on likelihood ratios. Bayes' Theorem supplies the engine of Bayesian inference, with Decision Theory (Wald) providing a unifying framework where frequentist and Bayesian methods appear as different ways of handling risk. Sufficiency and the Exponential Family structure determine which models admit clean inference, while Asymptotic Theory (consistency, asymptotic normality) explains why all schools tend to agree as data accumulate — a deep reconciliation.

🔬 Where this appears

⏳ History

- 1654 — Pascal and Fermat exchange letters on games of chance, founding probability and implicitly the classical (equiprobable) interpretation.

- 1763 — Thomas Bayes' essay (published posthumously by Richard Price) introduces inverse probability — updating belief about a cause from observed effects.

- 1812 — Laplace's Théorie analytique des probabilités systematizes inverse probability and the principle of insufficient reason; dominates 19th-century inference.

- 1922 — R.A. Fisher's On the Mathematical Foundations of Theoretical Statistics introduces likelihood, sufficiency, efficiency, and consistency, displacing inverse probability.

- 1933 — Neyman and Pearson publish their lemma, founding hypothesis testing on error rates and the long-run frequency interpretation; Kolmogorov axiomatizes probability measure-theoretically.

- 1937 — Neyman introduces confidence intervals as a frequentist alternative to fiducial/Bayesian intervals.

- 1939 — Harold Jeffreys' Theory of Probability revives and rigorizes Bayesian inference with objective priors, against the prevailing frequentist tide.

- 1950 — Abraham Wald's Statistical Decision Functions unifies estimation and testing under decision theory, proving Bayes rules are essentially complete (admissibility).

- 1954 — L.J. Savage's Foundations of Statistics derives subjective probability and expected utility from axioms of rational preference, grounding the modern Bayesian program.

- 1962 — Allan Birnbaum proves that Sufficiency + Conditionality imply the Likelihood Principle, sharpening the conflict between schools.

open-problem

Several foundational questions remain genuinely open or contested. (1) The status of Birnbaum's theorem: its premises are stated over different experimental domains, and critics (Evans, Mayo) argue the proof equivocates — so whether Sufficiency and Conditionality really force the Likelihood Principle is unsettled. (2) The problem of priors: Bayesian inference requires a prior, but "objective" or "non-informative" priors (Jeffreys, reference priors) are not invariant under reparametrization in general and can yield improper or paradoxical posteriors (marginalization paradoxes). (3) Severity and error-statistics: Deborah Mayo's program argues that good inference requires that a claim has survived a severe test, a property invisible to the Likelihood Principle — challenging whether evidence can be likelihood-only. (4) Calibration of subjective belief: how, if at all, personal probabilities connect to objective frequencies (de Finetti's exchangeability gives a partial bridge) remains philosophically live. (5) The replication crisis has reopened all of this: whether the foundations themselves, or merely their misuse, are to blame for irreproducible science is actively debated.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also