Maximum Likelihood Estimation

⚡ Key Takeaways

Maximum Likelihood Estimation (MLE) is a method of estimating the parameters of a statistical model by choosing the values that make the observed data most probable. Key result: the estimate is the argument that maximizes the likelihood function, $\hat\theta = \arg\max_\theta L(\theta\mid x)$ . Applies to: parametric inference across statistics, econometrics, machine learning, physics, and genetics. Who/when: systematized by Ronald A. Fisher in 1912–1922, with roots in Gauss and Laplace.

💡 The Core Insight

Probability and likelihood use the same formula but read it in opposite directions. Probability fixes the parameter and asks how likely various data are; likelihood fixes the observed data and asks how plausible various parameter values are. MLE says: believe the parameter value under which what you actually saw was least surprising. Because products of probabilities are awkward, we almost always maximize the logarithm of the likelihood — turning products into sums without moving the location of the maximum.

DefinitionLikelihood and Log-Likelihood

Let $X_1,\dots,X_n$ be data with joint probability mass or density function $f(x_1,\dots,x_n\mid\theta)$ depending on a parameter $\theta\in\Theta$ (possibly a vector). The likelihood function is the joint density read as a function of $\theta$ :

1

\quad L(\theta\mid x) = f(x_1,\dots,x_n\mid\theta).

Derivation hint: simply relabel the joint density, holding the observed $x$ fixed and letting $\theta$ vary over the parameter space $\Theta$ . If the observations are independent and identically distributed (i.i.d.) with common density $f(x\mid\theta)$ , the likelihood factorizes:

2

\quad L(\theta\mid x) = \prod_{i=1}^{n} f(x_i\mid\theta).

Derivation hint: independence makes the joint density the product of marginals; "identically distributed" makes each factor use the same $f$ . The log-likelihood is

3

\quad \ell(\theta) = \log L(\theta\mid x) = \sum_{i=1}^{n}\log f(x_i\mid\theta).

Derivation hint: take the natural logarithm of [2]; the logarithm of a product is the sum of logarithms.

Symbols: $\theta$ = unknown parameter(s); $\Theta$ = parameter space; $x_i$ = the $i$ -th observation; $n$ = sample size; $f$ = the model's density/mass function; $L$ = likelihood; $\ell$ = log-likelihood.

DefinitionMaximum Likelihood Estimator

The maximum likelihood estimator $\hat\theta$ is any parameter value maximizing the likelihood (equivalently the log-likelihood):

4

\quad \hat\theta = \arg\max_{\theta\in\Theta} L(\theta\mid x) = \arg\max_{\theta\in\Theta} \ell(\theta).

Derivation hint: $\log$ is strictly increasing, so it preserves the location of the maximizer — maximizing $L$ and maximizing $\ell$ give the same $\hat\theta$ . When $\ell$ is differentiable and the maximum is interior to $\Theta$ , $\hat\theta$ satisfies the likelihood (score) equation:

5

\quad \left.\frac{\partial \ell(\theta)}{\partial \theta}\right|_{\theta=\hat\theta} = 0, \qquad \text{with}\quad \left.\frac{\partial^2 \ell}{\partial\theta^2}\right|_{\hat\theta} < 0.

Derivation hint: a smooth interior maximum requires the first derivative to vanish (first-order condition) and the second derivative to be negative (second-order condition for a maximum, not a minimum or inflection).

TheoremAsymptotic Properties of the MLE

Under regularity conditions (the true $\theta_0$ lies in the interior of $\Theta$ , the model is identifiable, the support of $f$ does not depend on $\theta$ , and $\ell$ is sufficiently smooth), the MLE $\hat\theta_n$ from $n$ i.i.d. observations satisfies: 1. Consistency: $\hat\theta_n \xrightarrow{p} \theta_0$ as $n\to\infty$ . 2. Asymptotic normality:

6

\quad \sqrt{n}\,(\hat\theta_n - \theta_0) \xrightarrow{d} \mathcal{N}\!\big(0,\; I_1(\theta_0)^{-1}\big),

Derivation hint: Taylor-expand the score $\ell'(\hat\theta)=0$ about $\theta_0$ , apply the Central Limit Theorem to the score and the Law of Large Numbers to the (negative) curvature, then divide. where $I_1(\theta_0)$ is the Fisher information per observation,

7

\quad I_1(\theta) = \mathbb{E}\!\left[\left(\frac{\partial}{\partial\theta}\log f(X\mid\theta)\right)^{2}\right] = -\,\mathbb{E}\!\left[\frac{\partial^2}{\partial\theta^2}\log f(X\mid\theta)\right].

Derivation hint: differentiate the identity $\int f\,dx = 1$ twice under the integral sign; the score has mean zero, and its variance equals minus the expected curvature. 3. Asymptotic efficiency: $\hat\theta_n$ attains the Cramér–Rao lower bound asymptotically — no consistent, asymptotically unbiased estimator has smaller asymptotic variance.

ProofSketch of Consistency and Asymptotic Normality

Step 1 — Why the maximizer lands on the truth. Define the normalized log-likelihood $\frac{1}{n}\ell(\theta) = \frac1n\sum_i \log f(X_i\mid\theta)$ . By the Law of Large Numbers this converges to $M(\theta) = \mathbb{E}_{\theta_0}[\log f(X\mid\theta)]$ .

Step 2 — The truth maximizes the limit. Consider $M(\theta_0) - M(\theta) = \mathbb{E}_{\theta_0}\!\left[\log\frac{f(X\mid\theta_0)}{f(X\mid\theta)}\right]$ , the Kullback–Leibler divergence, which is $\ge 0$ and equals $0$ only when $f(\cdot\mid\theta)=f(\cdot\mid\theta_0)$ . Hence $\theta_0$ uniquely maximizes $M$ .

8

\quad \mathrm{KL}(\theta_0\,\|\,\theta) = \mathbb{E}_{\theta_0}\!\left[\log\frac{f(X\mid\theta_0)}{f(X\mid\theta)}\right] \ge 0.

Derivation hint: apply Jensen's inequality to the concave $\log$ , using $\mathbb{E}_{\theta_0}[f(X\mid\theta)/f(X\mid\theta_0)] = 1$ . Since the empirical objective converges to one maximized at $\theta_0$ , its maximizer $\hat\theta_n$ converges to $\theta_0$ .

Step 3 — Expand the score equation. Let $s(\theta)=\ell'(\theta)$ be the score. By definition $s(\hat\theta_n)=0$ . Taylor-expand about $\theta_0$ :

9

\quad 0 = s(\hat\theta_n) = s(\theta_0) + s'(\theta_0)\,(\hat\theta_n-\theta_0) + \text{remainder}.

Derivation hint: first-order Taylor expansion of the score function around the true parameter; the remainder vanishes faster under regularity.

Step 4 — Rearrange and rescale. Solving and multiplying by $\sqrt n$ :

10

\quad \sqrt{n}\,(\hat\theta_n-\theta_0) = \frac{\tfrac{1}{\sqrt n}\,s(\theta_0)}{-\tfrac{1}{n}\,s'(\theta_0)}.

Derivation hint: divide [9] by $s'(\theta_0)$ , isolate $(\hat\theta_n-\theta_0)$ , and split the $n$ factors so numerator and denominator each have a standard limit.

Step 5 — Apply limit theorems. The numerator $\frac{1}{\sqrt n}s(\theta_0)$ is a sum of i.i.d. mean-zero score terms, so by the CLT it converges to $\mathcal N(0, I_1(\theta_0))$ . The denominator $-\frac1n s'(\theta_0)$ converges by the LLN to $I_1(\theta_0)$ . By Slutsky's theorem the ratio converges to $\mathcal N\!\big(0, I_1(\theta_0)/I_1(\theta_0)^2\big)=\mathcal N\!\big(0, I_1(\theta_0)^{-1}\big)$ , establishing [6]. $\blacksquare$

∎

mathematician · Ronald Aylmer Fisher | 1890–1962

The British statistician and geneticist Ronald Fisher introduced the term "likelihood" as distinct from probability and developed maximum likelihood into a coherent theory between 1912 and 1922. His 1922 paper "On the Mathematical Foundations of Theoretical Statistics" defined consistency, efficiency, and sufficiency, and introduced what we now call Fisher information. Fisher argued that likelihood, not posterior probability, was the natural vehicle for inference when no prior is available — a stance that anchored the frequentist tradition. He also founded modern experimental design and population genetics, making him arguably the single most influential figure in 20th-century statistics.

🧠 Intuition

Imagine you find a coin on the ground and flip it 10 times, getting 7 heads. Ask: "For what bias $p$ would this exact outcome be least surprising?" If $p=0.1$ , seven heads is wildly unlikely; if $p=0.9$ , it's also somewhat unlikely; somewhere in between the probability of "7 of 10 heads" peaks. MLE finds that peak — here at $p=0.7$ , exactly the observed fraction. The likelihood is not the probability that $p=0.7$ (that question requires a prior); it is a ranking of parameter values by how well each explains the data you hold in your hand. The log turns the unwieldy product of ten coin-flip probabilities into a clean sum, and setting its derivative to zero locates the peak.

Figure

Likelihood of a coin's bias given 7 heads in 10 flips, peaking at p = 0.7

ExampleBernoulli / Binomial Proportion

Given: $n=10$ independent coin flips yield $k=7$ heads. Model each flip as $\text{Bernoulli}(p)$ .

Step 1 — Likelihood. $L(p) = \binom{10}{7}p^{7}(1-p)^{3}$ .

Step 2 — Log-likelihood (drop the constant binomial coefficient):

11

\quad \ell(p) = 7\log p + 3\log(1-p) + \text{const}.

Derivation hint: take $\log$ of the Bernoulli likelihood $p^k(1-p)^{n-k}$ with $k=7$ , $n-k=3$ ; the binomial coefficient is constant in $p$ .

Step 3 — Differentiate and set to zero:

12

\quad \frac{d\ell}{dp} = \frac{7}{p} - \frac{3}{1-p} = 0 \;\Longrightarrow\; 7(1-p) = 3p \;\Longrightarrow\; p = \frac{7}{10}.

Derivation hint: differentiate [11], clear denominators by multiplying through by $p(1-p)$ , and solve the linear equation.

Step 4 — Confirm a maximum: $\ell''(p) = -7/p^2 - 3/(1-p)^2 < 0$ everywhere, so $p=0.7$ is the maximizer.

\boxed{\hat p = \frac{k}{n} = 0.70 \ \ (\text{dimensionless proportion})}

ExampleNormal Distribution with Unknown Mean and Variance

Given: Five measurements (in cm) of a rod: $x = \{10.1, 9.8, 10.0, 10.3, 9.8\}$ , modeled as i.i.d. $\mathcal N(\mu, \sigma^2)$ .

Step 1 — Log-likelihood:

13

\quad \ell(\mu,\sigma^2) = -\frac{n}{2}\log(2\pi) - \frac{n}{2}\log\sigma^2 - \frac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i-\mu)^2.

Derivation hint: take $\log$ of the product of $n$ Gaussian densities $\frac{1}{\sqrt{2\pi\sigma^2}}e^{-(x_i-\mu)^2/2\sigma^2}$ and collect terms.

Step 2 — Solve for $\mu$ : $\partial\ell/\partial\mu = \frac{1}{\sigma^2}\sum(x_i-\mu)=0$ gives

14

\quad \hat\mu = \bar x = \frac{1}{n}\sum_{i=1}^{n} x_i.

Derivation hint: the only $\mu$ -dependent term is the sum of squares; its derivative vanishes when $\mu$ equals the sample mean. Numerically $\hat\mu = (10.1+9.8+10.0+10.3+9.8)/5 = 50.0/5 = 10.00$ cm.

Step 3 — Solve for $\sigma^2$ : $\partial\ell/\partial\sigma^2 = -\frac{n}{2\sigma^2} + \frac{1}{2\sigma^4}\sum(x_i-\hat\mu)^2 = 0$ gives

15

\quad \hat\sigma^2 = \frac{1}{n}\sum_{i=1}^{n}(x_i-\hat\mu)^2.

Derivation hint: treat $\sigma^2$ as a single variable, set the partial derivative to zero, and solve; note the divisor is $n$ , not $n-1$ . Numerically the squared deviations from 10.00 are $0.01, 0.04, 0.00, 0.09, 0.04$ , summing to $0.18$ , so $\hat\sigma^2 = 0.18/5 = 0.036\ \text{cm}^2$ .

\boxed{\hat\mu = 10.00\ \text{cm}, \qquad \hat\sigma^2 = 0.036\ \text{cm}^2 \ \ (\hat\sigma \approx 0.190\ \text{cm})}

⚠️ Common mistake

Likelihood is not probability of the parameter. $L(\theta\mid x)$ is not the probability that $\theta$ takes a given value. Within the frequentist framework $\theta$ is fixed (not random), so it has no probability distribution. The likelihood is the data's density viewed as a function of $\theta$ ; it need not integrate to 1 over $\theta$ , and a "likelihood of 0.3" carries no probabilistic meaning on its own — only ratios of likelihoods are interpretable.

⚠️ Common mistake

The MLE can be biased. Consistency does not imply unbiasedness. The MLE of normal variance, $\hat\sigma^2 = \frac1n\sum(x_i-\bar x)^2$ , has expectation $\frac{n-1}{n}\sigma^2$ — it systematically underestimates. The familiar unbiased estimator uses $n-1$ in the denominator, but that is not the MLE. MLE optimizes likelihood, not unbiasedness, though the bias vanishes as $n\to\infty$ .

CounterexampleWhen the MLE Behaves Badly

Consider $X_1,\dots,X_n$ i.i.d. from the uniform $U(0,\theta)$ . The density is $1/\theta$ for $0\le x\le\theta$ and $0$ otherwise, so the support depends on $\theta$ — violating a key regularity condition. The likelihood $L(\theta)=\theta^{-n}\,\mathbf 1\{\theta\ge \max_i x_i\}$ is not differentiable at its maximizer; setting a derivative to zero fails entirely. Here the MLE is $\hat\theta = \max_i x_i$ , found by direct reasoning (smaller $\theta$ gives zero likelihood; among feasible $\theta$ , $\theta^{-n}$ is largest at the smallest feasible value). This estimator is biased and not asymptotically normal — it converges at rate $n$ , not $\sqrt n$ , and its limiting law is exponential, not Gaussian. The lesson: the smooth asymptotic theory of Theorem [6] assumes a fixed support.

connection

MLE sits at the center of a web of statistical ideas. Under a flat prior, the MLE coincides with the Maximum A Posteriori Estimation mode, linking it to Bayesian Inference. Minimizing the Kullback-Leibler Divergence between the empirical distribution and the model is equivalent to maximizing likelihood, tying MLE to Information Theory. For the Gaussian linear model, MLE reduces exactly to Ordinary Least Squares. The optimality bound it attains is the Cramér-Rao Lower Bound, expressed through Fisher Information. Likelihood ratios drive the Neyman-Pearson Lemma and the Likelihood Ratio Test, and the broader generalization to non-likelihood objectives is M-Estimation.

🔬 Where this appears

⏳ History

- 1755–1760 — Thomas Bayes & Roger Cotes: early reasoning about inverting probability to infer causes from effects foreshadows likelihood thinking.

- 1809 — Carl Friedrich Gauss: in Theoria Motus, derives the method of least squares by assuming the arithmetic mean is the most probable value, implicitly using a likelihood argument for the normal error law.

- 1812 — Pierre-Simon Laplace: develops large-sample (asymptotic) approximations and the central role of the normal distribution in estimation.

- 1821 — Gauss: the Gauss–Markov framework refines least-squares estimation, later seen as Gaussian MLE.

- 1912 — Ronald Fisher: as an undergraduate, proposes maximizing the "absolute probability" of the data — his first formulation of the method.

- 1922 — Fisher: "On the Mathematical Foundations of Theoretical Statistics" coins likelihood, consistency, efficiency, sufficiency, and information; establishes the modern theory.

- 1925 — Fisher: Statistical Methods for Research Workers popularizes the method among practicing scientists.

- 1940s — Harald Cramér & C. R. Rao: prove the Cramér–Rao bound, rigorously establishing the asymptotic efficiency Fisher had asserted.

- 1977 — Dempster, Laird & Rubin: formalize the Expectation–Maximization (EM) algorithm, making MLE tractable for incomplete-data and latent-variable models.

open-problem

Several deep issues qualify the clean asymptotic theory. Non-existence and non-uniqueness: likelihoods can be unbounded (e.g., a Gaussian mixture where one component's variance shrinks to zero, sending the likelihood to infinity), so a global maximizer may not exist; one must seek a sensible local maximum instead. High-dimensional regimes: when the number of parameters grows with $n$ (the Neyman–Scott problem), the MLE can be inconsistent. Boundary and singular models: when $\theta_0$ lies on the boundary of $\Theta$ or the Fisher information is singular (common in mixtures and neural networks), the standard $\sqrt n$ -normal limit fails and is replaced by more delicate theory (singular learning theory, Watanabe's work). Computation: finding the global maximum of a non-concave likelihood is generally intractable, so practitioners rely on EM, gradient ascent, or Newton methods that find only local optima — quantifying the gap between local and global solutions in models like deep networks remains an active research frontier.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also