Sampling Bias

⚡ Key Takeaways

Sampling bias occurs when the way a sample is selected from a population systematically over- or under-represents certain groups, so the sample is not representative of the population. Key result: a biased sample produces estimates that are systematically wrong — increasing sample size does NOT fix it. Applies wherever data is collected to make inferences: polls, surveys, medicine, science, machine learning. First rigorously diagnosed in the 1936 Literary Digest poll and 1948 Dewey/Truman failures.

💡 The Core Insight

Statistical inference rests on one bargain: if the sample is representative, the sample statistic estimates the population parameter. Sampling bias voids the bargain. The error introduced is a bias — a fixed offset between expected estimate and true value — which is fundamentally different from random sampling error, which shrinks as nn grows. No amount of data cures a biased process; only a better process does.

Formal Statement

DefinitionPopulation, Sample, Parameter, Statistic
  • Population: the complete set of NN units we want to know about.
  • Sample: the subset of nn units we actually observe.
  • Parameter (θ\theta): a fixed numerical fact about the population (e.g. true mean μ\mu or proportion pp).
  • Statistic (θ^\hat\theta): a number computed from the sample to estimate θ\theta (e.g. sample mean xˉ\bar x or proportion p^\hat p).
  • Sampling bias: a feature of the selection mechanism causing E[θ^]≠θE[\hat\theta] \neq \theta for reasons of who got selected.

The bias of an estimator is its expected deviation from truth:

1
Bias(θ^)=E[θ^]−θ\quad \mathrm{Bias}(\hat\theta) = E[\hat\theta] - \theta

Derivation hint: take the expectation of the estimator over all possible samples the mechanism can produce, then subtract the true parameter. The total error of an estimator decomposes into bias and variance via the mean squared error:

2
MSE(θ^)=(Bias(θ^))2⏟systematic+Var(θ^)⏟random\quad \mathrm{MSE}(\hat\theta) = \underbrace{\big(\mathrm{Bias}(\hat\theta)\big)^2}_{\text{systematic}} + \underbrace{\mathrm{Var}(\hat\theta)}_{\text{random}}

Derivation hint: expand E[(θ^−θ)2]E[(\hat\theta-\theta)^2], add and subtract E[θ^]E[\hat\theta] inside the square, and use that the cross term vanishes because E[θ^−E[θ^]]=0E[\hat\theta-E[\hat\theta]]=0. For a simple random sample of proportion, the random part shrinks with nn:

3
Var(p^)=p(1−p)n\quad \mathrm{Var}(\hat p) = \frac{p(1-p)}{n}

Derivation hint: p^\hat p is the mean of nn independent Bernoulli(pp) indicators; divide the single-trial variance p(1−p)p(1-p) by nn. But sampling bias gives a constant offset that does not depend on nn. If a fraction of the population is selectively included, the limit of the estimate is the parameter of the reachable sub-population, not the whole:

4
lim⁡n→∞p^=preachable≠ppopulation\quad \lim_{n\to\infty}\hat p = p_{\text{reachable}} \neq p_{\text{population}}

Derivation hint: under biased selection the law of large numbers still applies — but it converges to the mean of the distribution you are actually sampling from, which is the biased sub-population, not the target.

TheoremBias Dominates at Large n

If a sampling mechanism has a fixed bias b=E[θ^]−θ≠0b = E[\hat\theta]-\theta \neq 0, then as n→∞n\to\infty the MSE approaches b2>0b^2 > 0. Therefore the estimator is inconsistent: it does not converge to θ\theta no matter how much data is collected. Random error can be beaten by size; bias cannot.

Proof

Step 1. From [2], MSE(θ^)=b2+Var(θ^)\mathrm{MSE}(\hat\theta) = b^2 + \mathrm{Var}(\hat\theta).

Step 2. For most sensible estimators, Var(θ^)→0\mathrm{Var}(\hat\theta)\to 0 as n→∞n\to\infty (e.g. [3] shows variance ∝1/n\propto 1/n).

Step 3. Taking the limit: lim⁡n→∞MSE(θ^)=b2+0=b2\displaystyle\lim_{n\to\infty}\mathrm{MSE}(\hat\theta) = b^2 + 0 = b^2.

Step 4. Since b≠0b\neq 0, we have b2>0b^2>0, so the MSE never reaches zero. The estimator converges to θ+b≠θ\theta + b \neq \theta. ■\blacksquare

∎
GG
George Gallup
1901–1984

Though a market researcher rather than a physicist, Gallup is the pivotal figure. In 1936 he publicly predicted that the Literary Digest's 2.4-million-response poll would wrongly call the U.S. election — and that he could beat it with a far smaller, deliberately representative sample. He was right twice over: he forecast both the actual result AND the Digest's error. Gallup demonstrated empirically what equation [4] states formally: representativeness, not size, determines accuracy.

Building Intuition

🧠 Intuition

Imagine you want the average height of everyone in a city, but you only measure people exiting a basketball arena. Measure 50 people or 50,000 — your estimate stays tall, because the door you chose filters for tall people. The filter is the bias. Random sampling error is like the slight wobble in your tape measure: it averages out with more measurements. Sampling bias is like measuring at the wrong door: more measurements just make you more confident in the wrong number. The cure is never "measure more people at the arena" — it is "choose a door that lets everyone through equally."

Figure
Random error shrinks with n; bias does not — the dartboard view
Error vs. Sample Size sample size n → error bias floor b total error irreducible bias remains small n large n

Worked Examples

ExampleThe Basketball Door

Given: A city's true mean adult height is μ=170\mu = 170 cm. You sample people leaving a basketball arena, whose mean height is 185185 cm. You measure n=10,000n = 10{,}000 people; the sampling SD of xˉ\bar x is 7/10000=0.077/\sqrt{10000}=0.07 cm.

Find: The bias and whether more data helps.

Steps: 1. Expected estimate E[xˉ]=185E[\bar x] = 185 cm (the reachable sub-population mean). 2. Bias =E[xˉ]−μ=185−170=15= E[\bar x] - \mu = 185 - 170 = 15 cm (from [1]). 3. Random error ≈0.07\approx 0.07 cm — negligible. 4. From [2]: MSE=(15)2+(0.07)2=225+0.0049≈225\mathrm{MSE} = (15)^2 + (0.07)^2 = 225 + 0.0049 \approx 225 cm². 5. Doubling nn changes only the 0.00490.0049, never the 225225.

Answer: Bias=+15 cm; MSE≈225 cm2, dominated entirely by bias.\boxed{\text{Bias} = +15\text{ cm; MSE}\approx 225\text{ cm}^2\text{, dominated entirely by bias.}} More data does not help.

ExampleNonresponse in a Survey

Given: A town has true support for a policy p=0.50p = 0.50. Supporters answer phone surveys with probability 0.80.8; opponents with probability 0.40.4. The pollster reaches n=4000n = 4000 people.

Find: The expected sample proportion and the bias.

Steps: 1. Of supporters (50% of town), fraction responding ∝0.5×0.8=0.40\propto 0.5\times 0.8 = 0.40. 2. Of opponents (50%), fraction responding ∝0.5×0.4=0.20\propto 0.5\times 0.4 = 0.20. 3. Among responders, supporter share =0.400.40+0.20=0.400.60=0.667= \dfrac{0.40}{0.40+0.20} = \dfrac{0.40}{0.60} = 0.667. 4. So E[p^]=0.667E[\hat p] = 0.667, while p=0.50p = 0.50. 5. Bias =0.667−0.500=+0.167= 0.667 - 0.500 = +0.167 (from [1]). The 95% interval is p^±1.960.667(0.333)/4000≈p^±0.0146\hat p \pm 1.96\sqrt{0.667(0.333)/4000} \approx \hat p \pm 0.0146 — tight, and centered on the wrong value.

Answer: E[p^]≈0.667, bias≈+16.7 percentage points.\boxed{E[\hat p]\approx 0.667,\ \text{bias}\approx +16.7\text{ percentage points.}} The narrow margin of error gives false confidence.

Common Misconceptions

⚠️ Common mistake

"A bigger sample is always better." False when the process is biased. A larger biased sample has the same bias but smaller variance, which produces a tighter confidence interval around the wrong answer — making the result more misleading, not less. The reported margin of error only accounts for random sampling error (equation [3]); it is silent about bias entirely.

⚠️ Common mistake

"Random sampling error and sampling bias are the same thing." No. Random error is the unavoidable luck-of-the-draw scatter that averages to zero over many samples and shrinks with nn. Sampling bias is a directional error baked into the selection method; its average is nonzero and constant in nn.

CounterexampleThe 1936 Literary Digest Poll

The Literary Digest mailed surveys to 10 million people and received 2.4 million replies — an enormous sample — predicting Alf Landon would beat Franklin Roosevelt 57% to 43%. Roosevelt won 61% to 37%, one of the largest landslides in U.S. history. The cause was sampling bias: the mailing list came from telephone directories, automobile registrations, and magazine subscribers — disproportionately wealthy during the Great Depression, and disproportionately Republican. Non-response bias compounded it (Landon supporters were more motivated to reply). Two and a half million responses could not save a biased frame. This single case is the textbook proof that size ≠ accuracy.

Connections

connection

Sampling bias is one face of a broader family. It contrasts with Random Sampling Error, which is the unbiased scatter that statistical theory handles. It is a precondition violation for the Central Limit Theorem and Confidence Interval machinery, both of which assume representative draws. It overlaps with Selection Bias (a superset including how subjects enter studies) and is distinct from Measurement Bias (errors in recording values rather than selecting units). In machine learning the same phenomenon appears as Dataset Shift and training/deployment mismatch. The corrective methods connect to Stratified Sampling and Weighting and Post-Stratification.

Real-World Applications

🔬 Where this appears

Historical Development

⏳ History
- 1662 — John Graunt analyzes London's Bills of Mortality; early awareness that recorded data may not reflect the true population.
- 1895 — Anders Kiaer (Norway) proposes the "representative method," arguing samples should mirror the population — an early articulation against biased selection.
- 1934 — Jerzy Neyman formalizes the theory of representative sampling and stratification, giving probability sampling a rigorous foundation.
- 1936 — The Literary Digest poll fails spectacularly; George Gallup's smaller quota sample succeeds, dramatizing bias-over-size.
- 1946 — Joseph Berkson identifies the hospital-admission selection bias now bearing his name.
- 1948 — Gallup, Roper, and Crossley all wrongly predict Dewey over Truman; quota sampling's hidden biases force the field toward probability sampling.
- 1949 — Hansen and Hurwitz develop survey-sampling theory with explicit nonresponse handling at the U.S. Census Bureau.
- 1976 — Abraham Wald's WWII work on aircraft armor (survivorship bias) is popularized; planes returning with bullet holes mis-told where to add armor.
- 2016–2020 — Polling misses in U.S./UK elections renew focus on nonresponse and coverage bias in the era of cellphones and declining response rates.

Subtleties & Limits

open-problem

The deepest practical difficulty is that bias is invisible from inside the data. Variance can be estimated from the sample itself; bias generally cannot, because the units you never observed leave no trace. Diagnosing sampling bias requires external information — a known census total, a benchmark, or a model of the selection mechanism — and such information is often unavailable or itself biased. Modern survey research wrestles with falling response rates (sometimes under 5%), where the assumption that nonresponse is "ignorable" becomes increasingly untenable. Methods like multilevel regression and post-stratification (MRP) and inverse-probability weighting can correct known biases, but no method corrects an unknown selection mechanism. The open frontier — sharpened by big "found" data and web scraping — is whether massive non-probability samples can ever be reliably de-biased, or whether, as the Meng (2018) "data quality" identity suggests, even tiny correlations between selection and outcome can swamp the benefit of enormous nn.

See Also

Random Sampling Error · Selection Bias · Survivorship Bias · Nonresponse Bias · Stratified Sampling · Central Limit Theorem · Confidence Interval · Measurement Bias · Dataset Shift · Berkson's Paradox