Comparing Distributions

⚡ Key Takeaways

Comparing distributions means quantifying how similar or different two probability distributions are, using graphical tools (Q–Q plots, ECDF overlays), summary-statistic comparisons (mean, variance, quantiles), formal hypothesis tests (two-sample t, Kolmogorov–Smirnov, Mann–Whitney), and distance/divergence measures (total variation, KL divergence, Wasserstein). Key result: different tools answer different questions — "are these the same distribution?" vs. "are their means equal?" vs. "how far apart are they?" — and choosing the wrong one is the central practical error. Applies across statistics, machine learning, econometrics, and the experimental sciences. Foundational figures: Kolmogorov, Smirnov, Mann, Whitney, and Wasserstein (1969 metric).

💡 The Core Insight

There is no single "distance" between distributions — there is a toolbox, and each tool privileges a different feature. Mean comparisons ask about the first moment; variance comparisons about the second; the Kolmogorov–Smirnov statistic about the largest vertical gap between cumulative distribution functions (CDFs); KL divergence about information loss; Wasserstein distance about the physical cost of moving probability mass. The art of comparing distributions is matching the question ("equal centers?" "same shape?" "indistinguishable laws?") to the right instrument, while respecting whether your data are paired or independent, continuous or discrete, and how heavy the tails are.

DefinitionEquality of Distributions

Two distributions are equal, written $F = G$ , if their CDFs agree everywhere:

1

\quad F(t) = G(t) \quad \text{for all } t \in \mathbb{R}.

Derivation hint: the CDF uniquely determines a probability law on $\mathbb{R}$ , so equality of CDFs is equivalent to equality of distributions; this is the strongest sense in which two distributions can "be the same."

DefinitionEmpirical Cumulative Distribution Function (ECDF)

The ECDF of a sample estimates the true CDF by the fraction of observations at or below each point:

2

\quad \hat{F}_n(t) = \frac{1}{n}\sum_{i=1}^{n} \mathbf{1}\{x_i \le t\},

where $\mathbf{1}\{\cdot\}$ is the indicator function (1 if true, 0 if false). Derivation hint: $\hat F_n(t)$ is an average of i.i.d. Bernoulli $(F(t))$ indicators, so by the law of large numbers $\hat F_n(t) \to F(t)$ ; uniform convergence is the Glivenko–Cantelli theorem.

DefinitionKolmogorov–Smirnov Statistic

The two-sample KS statistic is the supremum (largest) vertical distance between the two ECDFs:

3

\quad D_{n,m} = \sup_{t \in \mathbb{R}} \left| \hat{F}_n(t) - \hat{G}_m(t) \right|.

Derivation hint: because both ECDFs are step functions that jump only at observed data points, the supremum is attained at one of the $n+m$ sample values, making $D_{n,m}$ computable by scanning the pooled sorted data.

DefinitionTotal Variation Distance

For distributions with densities $f$ and $g$ (or PMFs), the total variation distance is

4

\quad \mathrm{TV}(F,G) = \sup_{A} \left| P_F(A) - P_G(A) \right| = \tfrac{1}{2}\int |f(x) - g(x)|\,dx,

where the supremum runs over all measurable events $A$ . Derivation hint: the optimal event is $A^* = \{x : f(x) > g(x)\}$ ; substituting and using $\int(f-g)=0$ over the whole space yields the $\tfrac12\int|f-g|$ form.

DefinitionKullback–Leibler Divergence

The KL divergence from $G$ to $F$ measures expected information lost when $G$ is used to approximate $F$ :

5

\quad D_{\mathrm{KL}}(F \,\|\, G) = \int f(x)\,\log\frac{f(x)}{g(x)}\,dx \;\ge\; 0.

Derivation hint: nonnegativity follows from Jensen's inequality applied to the convex function $-\log$ , since $\mathbb{E}_F[-\log(g/f)] \ge -\log \mathbb{E}_F[g/f] = -\log 1 = 0$ .

DefinitionWasserstein (Earth Mover's) Distance

The $p$ -Wasserstein distance is the minimal cost of transporting mass from $F$ to $G$ :

6

\quad W_p(F,G) = \left( \inf_{\gamma \in \Gamma(F,G)} \int |x-y|^p \, d\gamma(x,y) \right)^{1/p},

where $\Gamma(F,G)$ is the set of joint distributions (couplings) with marginals $F$ and $G$ . Derivation hint: for $p=1$ on $\mathbb{R}$ , the optimal coupling matches quantiles, giving the closed form $W_1 = \int_0^1 |F^{-1}(u) - G^{-1}(u)|\,du = \int_{-\infty}^{\infty}|F(t)-G(t)|\,dt$ .

TheoremGlivenko–Cantelli

The ECDF converges uniformly to the true CDF almost surely:

7

\quad \sup_{t}\left|\hat{F}_n(t) - F(t)\right| \xrightarrow{\text{a.s.}} 0 \quad \text{as } n \to \infty.

Derivation hint: pointwise a.s. convergence from the SLLN plus monotonicity of CDFs lets a finite grid control the whole real line; refining the grid drives the uniform error to zero.

TheoremKolmogorov Limiting Distribution

Under $F = G$ (continuous), the scaled KS statistic has a distribution-free limit:

8

\quad \sqrt{\tfrac{nm}{n+m}}\; D_{n,m} \xrightarrow{d} K, \qquad P(K \le t) = 1 - 2\sum_{k=1}^{\infty}(-1)^{k-1}e^{-2k^2 t^2}.

Derivation hint: the empirical process $\sqrt{n}(\hat F_n - F)$ converges to a Brownian bridge whose supremum has the Kolmogorov distribution; the result is distribution-free because the time-change $t \mapsto F(t)$ removes dependence on $F$ .

ProofThe KS Test Is Distribution-Free Under the Null

Claim: Under $H_0: F = G$ with $F$ continuous, the null distribution of $D_{n,m}$ does not depend on the underlying $F$ . 1. Probability integral transform. If $X \sim F$ with $F$ continuous, then $U = F(X)$ is Uniform $(0,1)$ . Apply $F$ to every observation in both samples. 2. ECDFs transform consistently. Since $F$ is monotone increasing, $x_i \le t \iff F(x_i) \le F(t)$ . Writing $u_i = F(x_i)$ and $s = F(t)$ , the indicator $\mathbf{1}\{x_i \le t\}$ equals $\mathbf{1}\{u_i \le s\}$ . 3. The supremum is invariant. As $t$ ranges over $\mathbb{R}$ , $s=F(t)$ ranges over $(0,1)$ , so

D_{n,m} = \sup_t |\hat F_n(t) - \hat G_m(t)| = \sup_{s\in(0,1)} |\hat F^U_n(s) - \hat G^U_m(s)|,

where the right side uses ECDFs of the transformed uniform samples. 4. Conclusion. Under $H_0$ both transformed samples are i.i.d. Uniform $(0,1)$ , irrespective of the original $F$ . Hence the distribution of $D_{n,m}$ depends only on $n$ and $m$ — not on $F$ . $\blacksquare$

∎

ProofKL Divergence Is Nonnegative (Gibbs' Inequality)

1. Write $D_{\mathrm{KL}}(F\|G) = \mathbb{E}_F\!\left[-\log\frac{g(X)}{f(X)}\right]$ where $X \sim f$ . 2. The function $\phi(u) = -\log u$ is convex on $(0,\infty)$ . 3. By Jensen's inequality, $\mathbb{E}_F[\phi(g/f)] \ge \phi(\mathbb{E}_F[g/f])$ . 4. Compute the inner expectation: $\mathbb{E}_F[g(X)/f(X)] = \int f \cdot (g/f)\,dx = \int g\,dx = 1$ . 5. Therefore $D_{\mathrm{KL}} \ge \phi(1) = -\log 1 = 0$ , with equality iff $g/f = 1$ a.e., i.e. $F = G$ . $\blacksquare$

∎

Figure

Two ECDFs and the Kolmogorov–Smirnov gap

mathematician · Andrey Kolmogorov | 1903–1987

Kolmogorov axiomatized probability theory in 1933 and, the same year, introduced the goodness-of-fit statistic that compares an ECDF to a hypothesized CDF via their maximal gap. His insight that this gap's limiting distribution is governed by the Brownian bridge — and is free of the underlying distribution — made distribution comparison rigorous and universal. Nikolai Smirnov extended the idea to the two-sample case in 1939, giving the tool used to this day for asking "do these two samples come from the same law?"

🧠 Intuition

Picture each distribution as a pile of sand on a number line. Comparing means asks only where the balance point of each pile sits — you could shovel sand around wildly without moving it. Comparing variances asks how spread out each pile is. The KS statistic stacks the sand into a running cumulative wall and asks: what is the biggest height difference between the two walls anywhere? Wasserstein distance asks the most physical question of all: if I literally had to shovel one pile into the shape of the other, what is the minimum total "dirt × distance" work? Notice these can disagree dramatically: two piles with the same balance point can have a huge KS gap (different shapes), and two piles a KS test calls "barely different" can be far apart in Wasserstein if the difference lives in the tails. The choice of tool is a choice of what "different" means.

ExampleTwo-Sample t-Test for Equal Means

Given: A new fertilizer (group A) and control (group B). Yields (kg/plot): A: $n_A = 5$ , mean $\bar x_A = 52$ , sd $s_A = 4$ . B: $n_B = 5$ , mean $\bar x_B = 47$ , sd $s_B = 3$ .

Goal: Test $H_0:\mu_A = \mu_B$ at $\alpha = 0.05$ (two-sided), assuming equal variances.

Step 1 — Pooled variance.

s_p^2 = \frac{(n_A-1)s_A^2 + (n_B-1)s_B^2}{n_A+n_B-2} = \frac{4(16)+4(9)}{8} = \frac{100}{8} = 12.5.

Step 2 — Standard error.

\mathrm{SE} = s_p\sqrt{\tfrac{1}{n_A}+\tfrac{1}{n_B}} = \sqrt{12.5}\,\sqrt{0.4} = 3.536 \times 0.632 = 2.236.

Step 3 — t-statistic.

t = \frac{\bar x_A - \bar x_B}{\mathrm{SE}} = \frac{52-47}{2.236} = 2.236.

Step 4 — Critical value. With $df = 8$ , $t_{0.025,8} = 2.306$ .

Step 5 — Decision. Since $|t| = 2.236 < 2.306$ , we fail to reject $H_0$ .

\boxed{t = 2.24,\quad df = 8,\quad p \approx 0.056 > 0.05 \;\Rightarrow\; \text{no significant difference in mean yield (kg/plot)}}

ExampleTwo-Sample KS Test for Whole-Distribution Difference

Given: Two small samples (sorted). A: $\{1, 2, 4, 7\}$ ( $n=4$ ). B: $\{3, 5, 6, 8\}$ ( $m=4$ ).

Goal: Test $H_0: F_A = F_B$ using $D_{n,m}$ .

Step 1 — Build ECDFs over the pooled sorted points $\{1,2,3,4,5,6,7,8\}$ . Each step is $1/4 = 0.25$ .

$t$	$\hat F_A$	$\hat F_B$	$\|\hat F_A-\hat F_B\|$
1	0.25	0.00	0.25
2	0.50	0.00	0.50
3	0.50	0.25	0.25
4	0.75	0.25	0.50
5	0.75	0.50	0.25
6	0.75	0.75	0.00
7	1.00	0.75	0.25
8	1.00	1.00	0.00

Step 2 — Maximum gap. $D_{4,4} = 0.50$ (attained at $t=2$ and $t=4$ ).

Step 3 — Critical value. For $n=m=4$ at $\alpha=0.05$ , the exact critical value is $D_{\mathrm{crit}} = 3/4 = 0.75$ .

Step 4 — Decision. Since $0.50 < 0.75$ , we fail to reject $H_0$ .

\boxed{D_{4,4} = 0.50 < 0.75 \;\Rightarrow\; \text{samples consistent with the same distribution (small } n \text{ has weak power)}}

⚠️ Common mistake

Equal means do NOT imply equal distributions. A two-sample t-test that "finds no difference" tests only the first moment. Two groups can have identical means and still differ wildly in variance, skew, or modality. Always ask which feature your test is sensitive to before declaring distributions "the same."

⚠️ Common mistake

A non-significant test is not evidence of equality. Failing to reject $H_0:F=G$ means you lacked power to detect a difference, not that the distributions match. With small samples (as in the KS example above), the test can miss even large real differences. "Absence of evidence is not evidence of absence."

⚠️ Common mistake

KL divergence is not a distance. It is asymmetric, $D_{\mathrm{KL}}(F\|G) \ne D_{\mathrm{KL}}(G\|F)$ , fails the triangle inequality, and blows up to $+\infty$ whenever $G$ assigns zero probability where $F$ does not. Do not use it as if it were a metric — use TV or Wasserstein for that.

Counterexample

Identical mean and variance, totally different shape. Let $F = \mathcal{N}(0,1)$ and let $G$ be the two-point mixture placing mass $\tfrac12$ at $-1$ and $\tfrac12$ at $+1$ . Both have mean $0$ and variance $1$ . A t-test and an F-test (variance test) both report "no difference." Yet $G$ is discrete and bimodal while $F$ is unimodal continuous — a KS or Wasserstein comparison detects a large gap. Moment-matching can perfectly disguise a structural difference.

connection

Comparing distributions sits at the crossroads of several areas. The KS test rests on the Empirical Distribution Function and its convergence (Glivenko–Cantelli), while its limiting law is governed by the Brownian Bridge from stochastic processes. Mean comparisons descend from the Central Limit Theorem and Hypothesis Testing. Divergence measures like KL connect to Information Theory (entropy, cross-entropy) and to Maximum Likelihood Estimation, where minimizing KL to the empirical distribution is maximizing likelihood. Wasserstein distance links to Optimal Transport and underlies modern generative modeling. Rank-based comparisons (Mann–Whitney) belong to Nonparametric Statistics.

🔬 Where this appears

⏳ History

- 1908 — William Gosset ("Student") publishes the t-distribution, enabling small-sample mean comparison.

- 1933 — Andrey Kolmogorov introduces the one-sample goodness-of-fit statistic based on the maximal ECDF–CDF gap.

- 1933 — Glivenko and Cantelli prove uniform convergence of the ECDF to the true CDF.

- 1939 — Nikolai Smirnov extends Kolmogorov's idea to the two-sample setting, creating the KS two-sample test.

- 1947 — Henry Mann and Donald Whitney publish the rank-based U test for stochastic ordering of two distributions.

- 1948 — Claude Shannon's information theory provides the conceptual home for cross-entropy and (soon after) KL divergence.

- 1951 — Solomon Kullback and Richard Leibler formalize the directed divergence between distributions.

- 1969 — Leonid Vaserstein (Wasserstein) introduces the metric, rooted in Monge–Kantorovich optimal transport, that now bears his name.

- 2017 — Arjovsky, Chintala, and Bottou introduce Wasserstein GANs, propelling optimal-transport distribution comparison into deep learning.

open-problem

Several deep issues remain practically and theoretically active. (1) High-dimensional comparison: the KS test has no canonical multivariate generalization, and estimating Wasserstein or KL distances in high dimensions suffers a curse of dimensionality (sample requirements grow exponentially). Entropic regularization (Sinkhorn distances) and kernel methods (Maximum Mean Discrepancy) are partial answers, but optimal sample complexity is still studied. (2) Calibrated equivalence testing: standard tests are built to detect differences, not to certify similarity; rigorous "two one-sided test" (TOST) equivalence frameworks exist but choosing the equivalence margin remains a modeling judgment. (3) Distribution-free guarantees under dependence: most theory assumes i.i.d. data, yet real time-series and networked data violate this, and the limiting distributions of comparison statistics under general dependence are incompletely characterized.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also