Study Design

⚡ Key Takeaways

Study design is the systematic blueprint for collecting and analyzing data so that a research question can be answered with minimal bias and maximal inferential power. The core result: only randomized, controlled designs license causal conclusions, while observational designs support association under explicit assumptions. It applies wherever data is gathered to test hypotheses — clinical trials, agriculture, psychology, economics, A/B testing. Foundational work by R. A. Fisher (1920s–30s), Jerzy Neyman, and Austin Bradford Hill (mid-20th century).

💡 The Core Insight

The validity of a conclusion is bounded above by the quality of the design. Randomization breaks the dependence between treatment assignment and potential outcomes, so that the difference in observed means becomes an unbiased estimate of the average causal effect. Control isolates the variable of interest; replication quantifies and shrinks random error; blocking removes known nuisance variation. Every named design is a particular combination of these four principles applied under practical and ethical constraints.

DefinitionPotential Outcomes and Treatment Assignment

For each unit $i \in \{1,\dots,n\}$ define a binary treatment indicator $T_i \in \{0,1\}$ . Each unit has two potential outcomes: $Y_i(1)$ under treatment and $Y_i(0)$ under control. The individual treatment effect is $\tau_i = Y_i(1) - Y_i(0)$ . Only the observed outcome

1

\quad Y_i = T_i\, Y_i(1) + (1 - T_i)\, Y_i(0)

is realized; the other is the counterfactual.

$Y_i(1), Y_i(0)$ — potential outcomes (real numbers)
$T_i$ — assignment indicator (0 or 1)
$\tau_i$ — unit-level causal effect
$Y_i$ — observed/measured outcome

TheoremUnbiasedness Under Randomization

If treatment is assigned independently of potential outcomes, i.e.

3

\quad \big(Y_i(1), Y_i(0)\big) \perp\!\!\!\perp T_i,

then the naive difference-in-means estimator is unbiased for the ATE:

4

\quad \hat\tau = \frac{1}{n_1}\sum_{i:T_i=1} Y_i - \frac{1}{n_0}\sum_{i:T_i=0} Y_i, \qquad \mathbb{E}[\hat\tau] = \mathrm{ATE}.

$n_1, n_0$ — number of treated and control units
$\perp\!\!\!\perp$ — statistical independence

ProofRandomization yields an unbiased ATE estimate

Step 1. By the consistency identity [1], for any treated unit the observed outcome equals its treated potential outcome: $\mathbb{E}[Y_i \mid T_i=1] = \mathbb{E}[Y_i(1) \mid T_i=1].$

Step 2. Randomization [3] makes $T_i$ independent of $(Y_i(1),Y_i(0))$ , so conditioning on $T_i=1$ does not change the distribution of $Y_i(1)$ :

\mathbb{E}[Y_i(1)\mid T_i=1] = \mathbb{E}[Y_i(1)].

Step 3. Symmetrically, $\mathbb{E}[Y_i \mid T_i=0] = \mathbb{E}[Y_i(0)\mid T_i=0] = \mathbb{E}[Y_i(0)].$

Step 4. The sample averages in [4] are unbiased estimators of these conditional expectations: $\mathbb{E}\big[\tfrac{1}{n_1}\sum_{T_i=1}Y_i\big] = \mathbb{E}[Y_i(1)]$ and likewise for control.

Step 5. Subtracting and applying [2]:

\mathbb{E}[\hat\tau] = \mathbb{E}[Y_i(1)] - \mathbb{E}[Y_i(0)] = \mathrm{ATE}. \qquad \blacksquare

The crucial line is Step 2: without randomization, $\mathbb{E}[Y_i(1)\mid T_i=1]\neq \mathbb{E}[Y_i(1)]$ in general, because the units who got treatment may systematically differ — this is confounding.

∎

RA

Ronald Aylmer Fisher

1890–1962

Working at Rothamsted Experimental Station among fields of barley and manure, Fisher invented modern experimental design. His 1925 Statistical Methods for Research Workers and 1935 The Design of Experiments introduced randomization, replication, blocking, factorial designs, and the analysis of variance (ANOVA). His famous "lady tasting tea" thought experiment grounded hypothesis testing in the physical act of randomized assignment. Fisher argued that randomization is not a statistical luxury but the very thing that makes a significance test valid, since it generates the reference distribution under the null.

🧠 Intuition

Imagine you want to know whether a fertilizer raises crop yield. The danger is that the best soil always gets the fertilizer (because the farmer naturally treats his good fields well). Then high yields could come from the soil, not the fertilizer — the two causes are confounded and inseparable. Randomization is like shuffling a deck before dealing: by flipping a coin for each field, fertile and barren plots scatter evenly between treatment and control, so on average the only systematic difference between the two groups is the fertilizer itself. You sacrifice a little efficiency (chance imbalance in any single trial) in exchange for unbiasedness and a known probability model. Replication then lets the law of large numbers shrink the residual chance imbalance, and blocking — grouping similar fields before randomizing within each group — removes nuisance variation you can identify in advance.

Figure

The hierarchy of study designs, ordered by strength of causal evidence

ExampleDifference in means in a randomized trial

Given. A drug trial randomizes 200 patients: $n_1=100$ to treatment, $n_0=100$ to placebo. Outcome = reduction in blood pressure (mmHg). Sample means: $\bar Y_1 = 12.4$ , $\bar Y_0 = 8.1$ . Sample variances: $s_1^2 = 25$ , $s_0^2 = 16$ .

Step 1 — Point estimate of ATE.

\hat\tau = \bar Y_1 - \bar Y_0 = 12.4 - 8.1 = 4.3\ \text{mmHg}.

Step 2 — Standard error of the difference.

\mathrm{SE} = \sqrt{\frac{s_1^2}{n_1} + \frac{s_0^2}{n_0}} = \sqrt{\frac{25}{100} + \frac{16}{100}} = \sqrt{0.41} = 0.640\ \text{mmHg}.

Step 3 — Test statistic (large-sample $z$ ).

z = \frac{\hat\tau}{\mathrm{SE}} = \frac{4.3}{0.640} = 6.72.

Step 4 — Conclusion. $|z| = 6.72 \gg 1.96$ , so we reject $H_0:\mathrm{ATE}=0$ at the 5% level.

\boxed{\hat\tau = 4.3 \pm 1.25\ \text{mmHg (95\% CI: }3.05\text{ to }5.55),\ z=6.72,\ p<0.001}

(95% CI half-width $= 1.96 \times 0.640 = 1.25$ mmHg.)

ExampleSample size for a desired power

Given. Plan a two-arm RCT to detect a difference $\delta = 5$ mmHg, with common SD $\sigma = 15$ , significance $\alpha = 0.05$ (two-sided, $z_{\alpha/2}=1.96$ ) and power $1-\beta = 0.80$ ( $z_\beta = 0.84$ ).

Step 1 — Per-group sample size formula.

5

\quad n_{\text{per arm}} = \frac{2\,(z_{\alpha/2} + z_\beta)^2\,\sigma^2}{\delta^2}.

Step 2 — Substitute numbers.

n = \frac{2\,(1.96+0.84)^2\,(15)^2}{5^2} = \frac{2\,(2.80)^2\,(225)}{25} = \frac{2\,(7.84)(225)}{25} = \frac{3528}{25}=141.1.

Step 3 — Round up. Always round up to guarantee power.

\boxed{n = 142\ \text{patients per arm},\ \text{total } 284}

⚠️ Common mistake

"A large sample fixes bias." It does not. Increasing $n$ shrinks variance (random error) but leaves bias (systematic error from a flawed design) untouched. A biased estimator with $n=10^6$ is just a precisely wrong answer. Equation [4] is unbiased only under the independence condition [3]; no amount of data repairs broken randomization.

⚠️ Common mistake

"Statistical significance proves a large or important effect." The $p$ -value confounds effect size with sample size. With huge $n$ , a clinically meaningless 0.2 mmHg difference can yield $p<0.001$ . Always report effect size and confidence intervals, not just $p$ .

CounterexampleConfounding defeats observational comparison

Suppose smokers ( $T=1$ ) have higher lung-cancer rates than non-smokers ( $T=0$ ), but suppose (hypothetically) that a genetic variant $G$ both increases the desire to smoke and independently causes cancer. Then

\mathbb{E}[Y\mid T=1] - \mathbb{E}[Y\mid T=0] = \underbrace{\mathrm{ATE}}_{\text{causal}} + \underbrace{\big(\mathbb{E}[Y(0)\mid T=1]-\mathbb{E}[Y(0)\mid T=0]\big)}_{\text{selection bias} \neq 0}.

The second term is nonzero precisely because [3] fails: smokers and non-smokers differ in $Y(0)$ even before smoking. The observed association overstates the causal effect. This is why observational data alone cannot settle causation — only randomization (or strong identifying assumptions) zeroes the bias term.

connection

Study design sits at the crossroads of several fields. The inference engine is Hypothesis Testing, which converts the design's reference distribution into $p$ -values; the precision of estimates is governed by Confidence Intervals and the Central Limit Theorem. Sample-size planning [5] draws on Statistical Power. The causal machinery — equations [1]–[4] — is the Potential Outcomes Framework, complemented by Causal Inference tools like Directed Acyclic Graphs for reasoning about confounders. The analysis of multifactor designs uses Analysis of Variance, and the random-assignment mechanism rests on Probability Theory.

🔬 Where this appears

⏳ History

- 1747 — James Lind conducts an early controlled trial on scurvy aboard HMS Salisbury, comparing six dietary treatments among 12 sailors.

- 1843– — Rothamsted Experimental Station founded; long-term agricultural field experiments begin.

- 1923 — Neyman introduces potential outcomes in the context of agricultural experiments (in Polish).

- 1925 — Fisher publishes Statistical Methods for Research Workers, formalizing significance testing.

- 1935 — Fisher's The Design of Experiments establishes randomization, replication, blocking, and factorial design; the "lady tasting tea" example appears.

- 1948 — The British Medical Research Council streptomycin trial for tuberculosis is the first modern randomized clinical trial (designed with Austin Bradford Hill).

- 1950–1954 — Doll & Hill case-control and cohort studies establish the smoking–lung cancer link; Hill later (1965) proposes the "Bradford Hill criteria" for causation from observational data.

- 1974 — Donald Rubin formalizes the Rubin Causal Model, unifying potential outcomes for experiments and observational studies.

- 1995–2009 — Judea Pearl develops the structural causal model and do-calculus, integrating DAGs with design-based inference.

open-problem

External vs. internal validity. Randomization secures internal validity (unbiased causal estimates within the study sample) but says nothing about external validity — whether the result generalizes to a different population, setting, or time. The SUTVA assumptions (no interference between units, no hidden treatment versions) underlying [1] often fail in networked settings (vaccines confer herd immunity; A/B tests leak between users), an active research frontier in interference and spillover effects. Furthermore, observational causal inference under unmeasured confounding remains fundamentally unidentified without untestable assumptions (ignorability, instrumental-variable validity); sensitivity analysis quantifies, but cannot eliminate, this uncertainty. Adaptive and sequential designs that update assignment probabilities mid-trial complicate the clean probability model of [3]–[4] and require careful inference to avoid bias.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also