Sampling Methods

⚡ Key Takeaways

Sampling methods are systematic procedures for selecting a subset (sample) from a larger population to estimate population characteristics. The key result: properly randomized sampling allows valid statistical inference with quantifiable error, regardless of population size. Used wherever exhaustive measurement is impossible — surveys, quality control, ecology, polling, and machine learning. Foundations laid by Bowley, Neyman, and Hansen in the early-to-mid 20th century.

💡 The Core Insight

The power of sampling comes not from the sample being "representative" by appearance, but from the randomization of selection. Because each unit's inclusion probability is known by design, we can mathematically reweight the observed data to reconstruct the population total or mean without bias, and we can quantify how much our estimate would vary across hypothetical repetitions of the sampling process. Randomness in selection, not in nature, is what licenses the inference.

DefinitionPopulation, Sample, and Inclusion Probability

Let the population be a finite set of $N$ labeled units $\mathcal{U} = \{1, 2, \dots, N\}$ , each carrying a value $y_i$ of a study variable. A sample $S \subseteq \mathcal{U}$ is a subset, selected by a random mechanism described by a sampling design $p(\cdot)$ , where $p(S)$ is the probability of drawing the particular subset $S$ . The first-order inclusion probability of unit $i$ is

1

\quad \pi_i = \Pr(i \in S) = \sum_{S \ni i} p(S),

and the second-order inclusion probability of the pair $(i,j)$ is

2

\quad \pi_{ij} = \Pr(i \in S \text{ and } j \in S) = \sum_{S \supseteq \{i,j\}} p(S).

Derivation hint: each $\pi_i$ aggregates the design probabilities over all samples containing unit $i$ ; $\pi_{ii} = \pi_i$ by convention.

DefinitionPopulation Parameters and the Sample Mean

The principal targets of estimation are the population total $T$ and population mean $\bar{Y}$ , with population variance $S^2$ :

3

\quad T = \sum_{i=1}^{N} y_i, \qquad \bar{Y} = \frac{1}{N}\sum_{i=1}^{N} y_i, \qquad S^2 = \frac{1}{N-1}\sum_{i=1}^{N}\left(y_i - \bar{Y}\right)^2.

Derivation hint: $S^2$ uses divisor $N-1$ so that the expected sample variance equals it under simple random sampling.

TheoremHorvitz–Thompson Estimator

For any probability sampling design with $\pi_i > 0$ for all $i$ , the Horvitz–Thompson (HT) estimator of the population total,

4

\quad \hat{T}_{HT} = \sum_{i \in S} \frac{y_i}{\pi_i},

is unbiased: $\mathbb{E}[\hat{T}_{HT}] = T$ . Its variance is

5

\quad \mathrm{Var}(\hat{T}_{HT}) = \sum_{i=1}^{N}\sum_{j=1}^{N} \left(\pi_{ij} - \pi_i \pi_j\right)\frac{y_i}{\pi_i}\frac{y_j}{\pi_j}.

Derivation hint: write $\hat{T}_{HT} = \sum_i (y_i/\pi_i)\,\mathbb{1}[i\in S]$ and use $\mathbb{E}[\mathbb{1}[i\in S]] = \pi_i$ and $\mathbb{E}[\mathbb{1}[i\in S]\mathbb{1}[j\in S]] = \pi_{ij}$ .

TheoremSimple Random Sampling Without Replacement: Mean and Variance

Under simple random sampling without replacement (SRSWOR) of size $n$ from $N$ , every unit has $\pi_i = n/N$ , and the sample mean $\bar{y} = \frac{1}{n}\sum_{i\in S} y_i$ is unbiased for $\bar{Y}$ with variance

6

\quad \mathrm{Var}(\bar{y}) = \frac{S^2}{n}\left(1 - \frac{n}{N}\right).

The factor $f = n/N$ is the sampling fraction and $(1 - n/N)$ is the finite population correction (FPC). Derivation hint: expand $\mathrm{Var}(\bar y) = \frac{1}{n^2}\big[\sum_i \mathrm{Var}(\mathbb 1_i)y_i^2 + \sum_{i\neq j}\mathrm{Cov}(\mathbb 1_i,\mathbb 1_j)y_iy_j\big]$ using $\pi_i = n/N$ , $\pi_{ij} = \frac{n(n-1)}{N(N-1)}$ .

ProofUnbiasedness and Variance of the Sample Mean under SRSWOR

Step 1 — Indicator representation. Define the inclusion indicator $a_i = \mathbb{1}[i \in S]$ . Then $\bar{y} = \frac{1}{n}\sum_{i=1}^{N} a_i y_i$ , where the sum now runs over the whole population because $a_i$ is zero outside $S$ .

Step 2 — Inclusion probabilities. By symmetry of SRSWOR, each unit is equally likely to be among the $n$ chosen, so $\mathbb{E}[a_i] = \pi_i = n/N$ . For a pair, the number of size- $n$ samples containing both is $\binom{N-2}{n-2}$ , giving $\pi_{ij} = \binom{N-2}{n-2}/\binom{N}{n} = \frac{n(n-1)}{N(N-1)}$ .

Step 3 — Unbiasedness.

\mathbb{E}[\bar{y}] = \frac{1}{n}\sum_{i=1}^{N} \mathbb{E}[a_i]\,y_i = \frac{1}{n}\cdot\frac{n}{N}\sum_{i=1}^{N} y_i = \bar{Y}.

Step 4 — Variances and covariances of indicators. Since $a_i$ is Bernoulli, $\mathrm{Var}(a_i) = \pi_i(1-\pi_i) = \frac{n}{N}\left(1-\frac{n}{N}\right)$ . The covariance is

\mathrm{Cov}(a_i,a_j) = \pi_{ij} - \pi_i\pi_j = \frac{n(n-1)}{N(N-1)} - \frac{n^2}{N^2} = -\frac{n}{N}\cdot\frac{N-n}{N^2(N-1)}\cdot N = -\frac{1}{N-1}\,\frac{n}{N}\Big(1-\frac{n}{N}\Big).

Step 5 — Assemble.

\mathrm{Var}(\bar{y}) = \frac{1}{n^2}\left[\sum_i y_i^2 \mathrm{Var}(a_i) + \sum_{i\neq j} y_i y_j \mathrm{Cov}(a_i,a_j)\right].

Substituting and using $\sum_{i\neq j} y_i y_j = T^2 - \sum_i y_i^2$ together with $\sum_i (y_i - \bar Y)^2 = (N-1)S^2$ , the algebra collapses to

\mathrm{Var}(\bar{y}) = \frac{S^2}{n}\left(1 - \frac{n}{N}\right). \qquad \blacksquare

∎

mathematician · Jerzy Neyman | 1894–1981

A Polish-American mathematician and statistician, Neyman transformed survey sampling from a debated art into a rigorous theory. His landmark 1934 paper On the Two Different Aspects of the Representative Method established the superiority of randomized (probability) sampling over purposive selection and introduced confidence intervals and optimal allocation in stratified sampling. Working at University College London and later Berkeley, he founded one of the most influential statistics departments in the world. His "representative method" framework underlies essentially all modern survey practice.

🧠 Intuition

Think of estimating the average height of trees in a vast forest. You cannot measure every tree, so you wander in and measure some. If you measure only the tall, easy-to-reach ones near the road, your estimate is biased upward — no amount of arithmetic fixes a selection problem. But if you place a grid over a map and visit randomly chosen cells, then even though any single sample might happen to catch unusually tall or short trees, the procedure has no systematic lean. The FPC term $(1 - n/N)$ encodes a comforting fact: as your sample approaches the whole forest ( $n \to N$ ), uncertainty vanishes to zero, because you are converging on a census. The Horvitz–Thompson trick — dividing each observed value by its inclusion probability $\pi_i$ — is just bookkeeping: a tree sampled with probability $1/100$ "stands in" for 100 trees, so we weight it by 100 to rebuild the total.

Figure

Sampling design hierarchy: how units flow from population to estimate

DefinitionMajor Probability Sampling Designs

Simple Random Sampling (SRS): every subset of size $n$ equally likely; baseline design. With or without replacement.
Systematic Sampling: choose a random start $r \in \{1,\dots,k\}$ where $k = N/n$ , then take every $k$ -th unit. Inclusion probabilities $\pi_i = 1/k = n/N$ .
Stratified Sampling: partition $\mathcal{U}$ into $H$ homogeneous strata of sizes $N_h$ ; sample $n_h$ within each independently. The stratified mean estimator is

7

\quad \bar{y}_{st} = \sum_{h=1}^{H} W_h\,\bar{y}_h, \qquad W_h = \frac{N_h}{N},

with variance $\mathrm{Var}(\bar{y}_{st}) = \sum_h W_h^2 \frac{S_h^2}{n_h}\left(1 - \frac{n_h}{N_h}\right)$ .

Cluster Sampling: sample whole groups (clusters) and measure all units within them; cheaper but usually less precise per unit.
Multistage Sampling: combine the above hierarchically (e.g., sample districts, then households, then persons).
Probability Proportional to Size (PPS): set $\pi_i \propto x_i$ for an auxiliary size measure $x_i$ , improving efficiency when $y_i \approx \beta x_i$ .

Derivation hint: equation [7] is a weighted average; its variance follows from independence across strata, so cross-stratum covariances vanish.

TheoremNeyman Optimal Allocation

For stratified sampling with fixed total sample size $n = \sum_h n_h$ , the allocation minimizing $\mathrm{Var}(\bar{y}_{st})$ is

8

\quad n_h = n \cdot \frac{N_h S_h}{\sum_{k=1}^{H} N_k S_k}.

Strata that are larger or more internally variable receive proportionally more sample. Derivation hint: minimize $\sum_h W_h^2 S_h^2/n_h$ subject to $\sum_h n_h = n$ using a Lagrange multiplier; the optimum satisfies $n_h \propto W_h S_h \propto N_h S_h$ .

ExampleSRSWOR Mean and Its Standard Error

Given: A town has $N = 4{,}000$ households. An SRSWOR of $n = 100$ households records monthly water use; the sample mean is $\bar{y} = 18.0\text{ m}^3$ and sample standard deviation $s = 6.0\text{ m}^3$ .

Step 1 — Point estimate of mean. $\hat{\bar{Y}} = \bar{y} = 18.0\text{ m}^3$ .

Step 2 — Estimate the variance of $\bar y$ . Using $s^2 = 36$ as an estimate of $S^2$ :

\widehat{\mathrm{Var}}(\bar{y}) = \frac{s^2}{n}\left(1 - \frac{n}{N}\right) = \frac{36}{100}\left(1 - \frac{100}{4000}\right) = 0.36 \times 0.975 = 0.351.

Step 3 — Standard error. $\mathrm{SE}(\bar y) = \sqrt{0.351} = 0.592\text{ m}^3$ .

Step 4 — Estimate population total and its SE. $\hat{T} = N\bar y = 4000 \times 18.0 = 72{,}000\text{ m}^3$ ; $\mathrm{SE}(\hat T) = N \times \mathrm{SE}(\bar y) = 4000 \times 0.592 = 2{,}369\text{ m}^3$ .

Step 5 — 95% confidence interval for the mean (large-sample, $z = 1.96$ ):

18.0 \pm 1.96 \times 0.592 = 18.0 \pm 1.16.

\boxed{\bar{Y} \in (16.84,\ 19.16)\ \text{m}^3, \quad \hat T = 72{,}000 \pm 4{,}644\ \text{m}^3}

ExampleNeyman Allocation Across Two Strata

Given: A workforce of $N = 1{,}000$ splits into stratum A (managers, $N_A = 200$ , $S_A = 12$ ) and stratum B (line workers, $N_B = 800$ , $S_B = 4$ ), measuring annual training hours. Total budget allows $n = 100$ .

Step 1 — Compute the allocation weights.

N_A S_A = 200 \times 12 = 2400, \qquad N_B S_B = 800 \times 4 = 3200, \qquad \text{sum} = 5600.

Step 2 — Apply equation [8].

n_A = 100 \times \frac{2400}{5600} = 42.9 \approx 43, \qquad n_B = 100 \times \frac{3200}{5600} = 57.1 \approx 57.

Step 3 — Compare with proportional allocation ( $n_h = n W_h$ ): proportional would give $n_A = 20$ , $n_B = 80$ . Neyman shifts effort toward the small but highly variable managers.

Step 4 — Resulting stratified variance (ignoring FPC for brevity), $W_A = 0.2$ , $W_B = 0.8$ :

\mathrm{Var}(\bar y_{st}) = (0.2)^2\frac{144}{43} + (0.8)^2\frac{16}{57} = 0.04\times3.35 + 0.64\times0.281 = 0.134 + 0.180 = 0.314.

\boxed{n_A = 43,\quad n_B = 57,\quad \mathrm{Var}(\bar y_{st}) \approx 0.314}

⚠️ A large sample does not cure selection bias

The single most dangerous error is believing that more data fixes bad sampling. If the selection mechanism is correlated with $y$ (e.g., only volunteers respond), bias persists no matter how large $n$ grows. The 1936 Literary Digest poll surveyed 2.4 million people yet predicted Landon would defeat Roosevelt — Roosevelt won in a landslide — because the sample was drawn from car and telephone owners, who skewed wealthy during the Depression. A randomized sample of a few thousand would have been far more accurate. Precision shrinks with $n$ ; bias does not.

CounterexampleSystematic sampling can fail catastrophically with periodicity

Systematic sampling (every $k$ -th unit) usually behaves like SRS, but if the population list has a hidden period equal to $k$ , the design can be disastrously biased. Suppose we audit daily sales by sampling every 7th day starting on a Monday: with $k = 7$ we will only ever sample Mondays. If Monday sales systematically differ from the week's average, $\bar{y}$ is biased and its variance estimate (built assuming SRS-like behavior) is misleadingly small. The list order interacting with the step size — not randomness — produces the failure.

⚠️ The sample variance

s^2/n

formula assumes randomization, not "typical-looking" data

Confidence intervals are valid because the selection was randomized, which makes the estimator's sampling distribution computable. They are not justified by the sample "looking representative." Applying the formula $s^2/n$ to a convenience sample (e.g., the first 100 people who walked by) produces a number with the right units but no inferential meaning — the implied population is undefined and the coverage probability is unknown.

connection

Sampling theory sits at the crossroads of probability and inference. The finite-population framework here contrasts with the infinite-population (model-based) view in Statistical Inference, where data are treated as i.i.d. draws from an assumed distribution rather than units of a fixed list. The unbiasedness and variance results rest on Expectation and Variance of indicator random variables. Large-sample confidence intervals invoke the Central Limit Theorem, adapted to without-replacement designs. Stratification and ratio estimation connect to Analysis of Variance (partitioning variability into within- and between-group components) and to Linear Regression through auxiliary-variable methods. The Monte Carlo "sampling" of distributions in computational statistics — Monte Carlo Methods and Markov Chain Monte Carlo — is a conceptual cousin: there one samples from a known distribution to estimate integrals, whereas survey sampling reconstructs an unknown finite population.

🔬 Where this appears

⏳ History

- 1895 — Anders Kiaer (Norway) proposes the "representative method," advocating partial enumeration over full census, sparking international debate.

- 1906 — Arthur Bowley gives the first rigorous treatment of random sampling and standard errors for social surveys in Britain.

- 1925 — The International Statistical Institute formally endorses the representative method after years of controversy.

- 1934 — Jerzy Neyman's foundational paper distinguishes random from purposive sampling, introduces stratified sampling theory, optimal allocation, and confidence intervals.

- 1936 — The Literary Digest fiasco vs. George Gallup's small scientific sample dramatizes the superiority of probability sampling to mass convenience samples.

- 1943 — Mahalanobis develops large-scale survey methods (and pilot/cost studies) at the Indian Statistical Institute.

- 1949 — Hansen and Hurwitz formalize unequal-probability (PPS) sampling and the theory of multistage designs at the U.S. Census Bureau.

- 1952 — Horvitz and Thompson publish their general unbiased estimator for arbitrary inclusion probabilities, unifying the field.

- 1977 — Leslie Kish's Survey Sampling and Cochran's Sampling Techniques (3rd ed.) become the canonical undergraduate/graduate references.

open-problem · Nonresponse, frame errors, and the limits of design-based inference

Classical sampling theory assumes a perfect sampling frame (an accurate list of all units) and full response. In practice, frames are incomplete (undercoverage), units overlap or duplicate, and nonresponse is rarely random — those who decline often differ systematically from those who answer. The HT estimator's unbiasedness collapses when $\pi_i$ is not truly known because some units are unreachable. Adjustments — weighting classes, calibration, raking, and model-assisted estimators — partly repair this but reintroduce modeling assumptions, blurring the clean design-based / model-based distinction. With the decline of telephone landlines and the rise of nonprobability online panels, a live methodological frontier is whether well-calibrated nonprobability samples, paired with auxiliary data and statistical models, can match probability samples — a question without consensus. Estimating second-order inclusion probabilities $\pi_{ij}$ for complex designs, needed for valid variance estimation, also remains computationally and theoretically delicate, motivating resampling methods (jackknife, bootstrap for survey data).

Formal Statement

Why It Holds / Derivation

Building Intuition

Catalogue of Methods

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also