Sampling Distributions

⚡ Key Takeaways

A sampling distribution is the probability distribution of a statistic (such as the sample mean) computed over all possible samples of a fixed size drawn from a population. Key result: the sample mean of $n$ independent observations has mean $\mu$ and standard deviation $\sigma/\sqrt{n}$ , and for large $n$ its distribution is approximately normal regardless of the population shape. Applied throughout inferential statistics — confidence intervals, hypothesis testing, quality control. Formalized in the early 20th century by Gosset ("Student"), Fisher, and others.

💡 The Core Insight

A statistic is itself a random variable, because it is a function of random data. Repeating the sampling experiment produces a spread of statistic values; that spread — its center, dispersion, and shape — is the sampling distribution. Inference works by treating the one observed statistic as a single draw from this distribution and reasoning backward to the parameter.

DefinitionPopulation, Sample, Statistic, Sampling Distribution

Let a population be modeled by a random variable $X$ with cumulative distribution $F$ , mean $\mu = \mathbb{E}[X]$ , and variance $\sigma^2 = \operatorname{Var}(X)$ . A random sample of size $n$ is a sequence $X_1, X_2, \dots, X_n$ of independent and identically distributed (i.i.d.) copies of $X$ . A statistic is any measurable function $T = g(X_1,\dots,X_n)$ that does not depend on unknown parameters. The sampling distribution of $T$ is the probability distribution of the random variable $T$ . The most important statistic is the sample mean:

1

\quad \bar{X}_n = \frac{1}{n}\sum_{i=1}^{n} X_i

Derivation hint: This is a definition; note it is a linear combination of the $X_i$ , which makes its mean and variance easy to compute. Its first two moments are:

2

\quad \mathbb{E}[\bar{X}_n] = \mu, \qquad \operatorname{Var}(\bar{X}_n) = \frac{\sigma^2}{n}

Derivation hint: Apply linearity of expectation to the sum; for the variance, use independence so cross-covariance terms vanish, giving $\frac{1}{n^2}\sum \operatorname{Var}(X_i) = \frac{n\sigma^2}{n^2}$ . The quantity $\sigma_{\bar X} = \sigma/\sqrt{n}$ is called the standard error of the mean. Here $\mu$ is the population mean, $\sigma^2$ the population variance, $n$ the sample size, and $X_i$ the $i$ -th observation.

TheoremCentral Limit Theorem (CLT)

Let $X_1, X_2, \dots$ be i.i.d. with finite mean $\mu$ and finite variance $\sigma^2 > 0$ . Then the standardized sample mean converges in distribution to a standard normal:

3

\quad Z_n = \frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} \mathcal{N}(0,1) \quad \text{as } n \to \infty

Derivation hint: Show the characteristic function (or moment generating function) of $Z_n$ converges to $e^{-t^2/2}$ via a Taylor expansion to second order; invoke Lévy's continuity theorem. Equivalently, for large $n$ , $\bar{X}_n \approx \mathcal{N}(\mu, \sigma^2/n)$ .

TheoremSampling Distributions Under Normality

If $X_1,\dots,X_n$ are i.i.d. $\mathcal{N}(\mu,\sigma^2)$ , then exactly (not just asymptotically):

4

\quad \bar{X}_n \sim \mathcal{N}\!\left(\mu, \frac{\sigma^2}{n}\right), \qquad \frac{(n-1)S^2}{\sigma^2} \sim \chi^2_{\,n-1}

Derivation hint: $\bar X$ is a linear combination of normals, hence normal. For $S^2$ , use the orthogonal (Helmert) transformation showing $\sum(X_i-\bar X)^2/\sigma^2$ is a sum of $n-1$ squared independent standard normals. where $S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2$ is the sample variance. Moreover $\bar X$ and $S^2$ are independent, and the studentized mean follows a $t$ -distribution:

5

\quad T = \frac{\bar{X}_n - \mu}{S/\sqrt{n}} \sim t_{\,n-1}

Derivation hint: Write $T = Z / \sqrt{V/(n-1)}$ where $Z = (\bar X-\mu)/(\sigma/\sqrt n)\sim\mathcal N(0,1)$ and $V=(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$ are independent; this ratio is the definition of a $t$ variable.

ProofSketch of the CLT via characteristic functions

1. Center and scale: let $Y_i = (X_i - \mu)/\sigma$ , so $\mathbb{E}[Y_i]=0$ , $\operatorname{Var}(Y_i)=1$ . Then $Z_n = \frac{1}{\sqrt n}\sum_{i=1}^n Y_i$ . 2. The characteristic function of $Z_n$ factors by independence:

8

\quad \varphi_{Z_n}(t) = \left[\varphi_Y\!\left(\frac{t}{\sqrt n}\right)\right]^n.

Derivation hint: $\varphi_{\sum}(t)=\prod\varphi(t)$ for independent sums, and $\varphi_{cY}(t)=\varphi_Y(ct)$ . 3. Taylor-expand $\varphi_Y$ near 0 using $\varphi_Y(0)=1$ , $\varphi_Y'(0)=i\,\mathbb E[Y]=0$ , $\varphi_Y''(0)=-\mathbb E[Y^2]=-1$ :

9

\quad \varphi_Y\!\left(\frac{t}{\sqrt n}\right) = 1 - \frac{t^2}{2n} + o\!\left(\frac{1}{n}\right).

Derivation hint: Finite variance guarantees the second-order Taylor term exists with the stated coefficient. 4. Therefore

10

\quad \varphi_{Z_n}(t) = \left(1 - \frac{t^2}{2n} + o(1/n)\right)^n \longrightarrow e^{-t^2/2}.

Derivation hint: Use $\lim_{n\to\infty}(1+a_n/n)^n = e^{a}$ when $a_n\to a$ . 5. The limit $e^{-t^2/2}$ is exactly the characteristic function of $\mathcal N(0,1)$ . By Lévy's continuity theorem, pointwise convergence of characteristic functions implies convergence in distribution. $\blacksquare$

∎

WS

William Sealy Gosset ("Student")

1876–1937

Gosset was a chemist and statistician employed by the Guinness brewery in Dublin, where small-sample experiments on barley and yeast were routine. Forbidden by company policy from publishing under his own name, he wrote as "Student." In his 1908 paper The Probable Error of a Mean he derived the sampling distribution of $T=(\bar X-\mu)/(S/\sqrt n)$ — the Student's $t$ -distribution — recognizing that for small $n$ the substitution of $S$ for the unknown $\sigma$ inflates the tails beyond the normal curve. R. A. Fisher later supplied the rigorous geometric derivation and the degrees-of-freedom concept. Gosset's insight launched the modern theory of small-sample inference.

🧠 Intuition

Imagine a giant urn (the population) with a known average. You scoop out $n$ balls, average their numbers, write the result on a card, and return the balls. Repeat millions of times. The histogram of all those cards is the sampling distribution of the mean — it is not about any single scoop but about the variability of the scooping procedure. Three things happen as $n$ grows: (1) the histogram centers ever more tightly on the true population mean (it stays centered at $\mu$ but narrows); (2) its width shrinks like $1/\sqrt n$ ; and (3) regardless of how lumpy or skewed the urn's contents are, the histogram of averages smooths into a bell. The averaging process "blends" the irregularities of individual draws, and the more you blend, the more normal the result. This is why a single sample, properly analyzed, lets you quantify uncertainty about the whole population.

Figure

Sampling distribution of the mean narrows and normalizes as

n

increases

ExampleStandard error and a normal approximation

Given: A population of light-bulb lifetimes has mean $\mu = 1200$ hours and standard deviation $\sigma = 300$ hours. A sample of $n = 36$ bulbs is taken. Find the mean and standard error of $\bar X$ , and approximate $P(\bar X > 1250)$ .

Steps: 1. Mean of the sampling distribution: $\mathbb E[\bar X] = \mu = 1200$ hours. 2. Standard error: $\sigma_{\bar X} = \sigma/\sqrt n = 300/\sqrt{36} = 300/6 = 50$ hours. 3. By the CLT, $\bar X \approx \mathcal N(1200, 50^2)$ . Standardize:

11

\quad z = \frac{1250 - 1200}{50} = 1.0

Derivation hint: subtract the mean and divide by the standard error to convert to a standard-normal $z$ -score. 4. From normal tables, $P(Z > 1.0) = 1 - 0.8413 = 0.1587$ .

Answer:

\boxed{\;\mathbb E[\bar X] = 1200\text{ h}, \quad \sigma_{\bar X} = 50\text{ h}, \quad P(\bar X > 1250) \approx 0.159\;}

ExampleSmall-sample inference with the

t

-distribution

Given: From a normal population, $n = 10$ measurements give sample mean $\bar x = 52.0$ and sample standard deviation $s = 4.0$ . The population standard deviation $\sigma$ is unknown. Construct a 95% confidence interval for $\mu$ .

Steps: 1. Because $\sigma$ is unknown and $n$ is small, the studentized mean follows $t_{n-1} = t_9$ . 2. Standard error estimate: $s/\sqrt n = 4.0/\sqrt{10} = 1.265$ . 3. Critical value: for 95% confidence with 9 degrees of freedom, $t_{0.025,\,9} = 2.262$ . 4. Margin of error: $E = t \cdot s/\sqrt n = 2.262 \times 1.265 = 2.861$ . 5. Interval:

12

\quad \bar x \pm t_{0.025,9}\,\frac{s}{\sqrt n} = 52.0 \pm 2.861

Derivation hint: invert the pivot $T=(\bar X-\mu)/(S/\sqrt n)\sim t_{n-1}$ to bracket $\mu$ .

Answer:

\boxed{\;(49.14,\; 54.86)\text{ — a 95\% confidence interval for }\mu\;}

⚠️ Common mistake

The sampling distribution is not the population distribution, nor the data distribution. Three distinct objects are routinely confused:

The population distribution describes individual values of $X$ (mean $\mu$ , sd $\sigma$ ).
The data (empirical) distribution is the histogram of your one observed sample.
The sampling distribution describes a statistic across hypothetical repeated samples (e.g., $\bar X$ has sd $\sigma/\sqrt n$ ).

Increasing $n$ makes the sampling distribution narrower and more normal — but it does not change the population's shape, and it does not make your single data histogram normal.

CounterexampleThe CLT does not save a heavy-tailed population

Consider a Cauchy population, whose density is $f(x)=\frac{1}{\pi(1+x^2)}$ . It has no finite mean or variance. The sample mean $\bar X_n$ of $n$ i.i.d. Cauchy variables is itself Cauchy with the same scale — it does not become normal and does not concentrate as $n\to\infty$ :

13

\quad \bar X_n \sim \text{Cauchy}(0,1) \quad\text{for every } n.

Derivation hint: the Cauchy characteristic function is $e^{-|t|}$ ; averaging gives $\left(e^{-|t|/n}\right)^n = e^{-|t|}$ , unchanged. The CLT's finite-variance hypothesis is essential — it is not a mere technicality.

connection

The sampling distribution sits at the hub of inferential statistics. Its existence and shape are dictated by the Central Limit Theorem and the Law of Large Numbers (which guarantees the center converges to $\mu$ ). The exact distributions under normality — the Chi-Squared Distribution for $S^2$ , Student's t-Distribution for the studentized mean, and the F-Distribution for ratios of variances — are all sampling distributions of specific statistics. Every Confidence Interval is built by inverting a pivot whose sampling distribution is known, and every Hypothesis Testing $p$ -value is a tail probability under the sampling distribution assumed by the null. The Standard Error is simply the standard deviation of a sampling distribution, and the Bootstrap Method estimates a sampling distribution empirically by resampling when analytic forms are unavailable.

🔬 Where this appears

⏳ History

- 1733 — Abraham de Moivre approximates the binomial by the normal curve for large

n

, the first special case of the CLT (for sample proportions).

- 1810 — Pierre-Simon Laplace generalizes de Moivre's result toward a broad central limit theorem in his Théorie analytique des probabilités.

- 1900 — Karl Pearson introduces the chi-squared distribution and goodness-of-fit testing, an early use of a statistic's sampling distribution.

- 1908 — William Gosset ("Student") publishes the

t

-distribution for the small-sample mean with unknown variance.

- 1915–1925 — Ronald A. Fisher gives rigorous geometric derivations of the

t

,

\chi^2

, and

F

sampling distributions, formalizes degrees of freedom, and develops likelihood and sufficiency.

- 1922 — Fisher clarifies the conceptual distinction between population parameters and sampling distributions of estimators.

- 1920–1937 — Lindeberg and Lévy establish modern rigorous versions of the CLT, with Lindeberg's condition giving the precise requirements for asymptotic normality of sums.

- 1933 — Neyman & Egon Pearson build hypothesis-testing theory squarely on sampling distributions under null and alternative hypotheses.

- 1979 — Bradley Efron introduces the bootstrap, allowing sampling distributions to be approximated computationally without analytic formulas.

open-problem

Several practical and theoretical issues complicate the clean asymptotic picture. How large is "large $n$ "? The CLT is asymptotic; for highly skewed populations (e.g., income, insurance claims) the sample mean may remain noticeably non-normal even at $n=100$ , and there is no universal cutoff — the required $n$ depends on the population's skewness and kurtosis (quantified via the Berry–Esseen bound, which gives an $O(n^{-1/2})$ convergence rate). Dependence and non-identical distributions: real data are often correlated (time series, clustered surveys) or non-i.i.d., requiring modified CLTs and effective-sample-size corrections. Heavy tails: when variance is infinite or undefined (as for Cauchy or some power-law distributions), the normal CLT fails entirely and stable-law limits with non- $\sqrt n$ scaling take over. Sampling distributions of complex statistics (medians, quantiles, ratios, machine-learning model parameters) often have no closed form, motivating resampling and asymptotic-expansion methods — an active area where exact finite-sample behavior remains incompletely understood.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also