Hypothesis Testing: Power and Multiple Testing

⚡ Key Takeaways

Power is the probability that a hypothesis test correctly rejects a false null hypothesis: $\text{Power} = 1 - \beta = P(\text{reject } H_0 \mid H_1 \text{ true})$ . Multiple testing addresses the inflation of error rates that occurs when many hypotheses are tested simultaneously — testing 20 true nulls at $\alpha = 0.05$ yields a 64% chance of at least one false positive. Used everywhere from clinical trials to genomics; formalized chiefly by Neyman, Pearson, Bonferroni, and later Benjamini–Hochberg (1995).

💡 The Core Insight

A hypothesis test trades off two errors. Lowering $\alpha$ (false positives) raises $\beta$ (false negatives) unless you compensate with sample size or effect size. Power quantifies the false-negative side. When many tests are run, the per-comparison error $\alpha$ is no longer the right thing to control; you must control a family-wide quantity — either the chance of any error (FWER) or the expected proportion of errors among rejections (FDR). The deep tension: aggressive multiplicity correction protects against false discoveries but destroys power. The entire field is a balancing act between these two failure modes.

DefinitionThe Two Errors and Power

For a test of $H_0$ against $H_1$ with rejection region $R$ :

Type I error (false positive): rejecting $H_0$ when it is true. Probability $\alpha = P(\text{reject } H_0 \mid H_0)$ .
Type II error (false negative): failing to reject $H_0$ when it is false. Probability $\beta = P(\text{fail to reject } H_0 \mid H_1)$ .
Power: the probability of correctly rejecting a false $H_0$ .

1

\quad \text{Power}(\theta) = 1 - \beta(\theta) = P_\theta(\mathbf{X} \in R), \qquad \theta \in \Theta_1

Derivation hint: Power is just the probability of landing in the rejection region, computed under the true (alternative) parameter $\theta$ , not under $H_0$ . Here $\mathbf{X}$ is the data, $R$ the rejection region, $\Theta_1$ the alternative parameter set, and $\theta$ the true parameter value.

DefinitionPower of the One-Sample z-Test

For testing $H_0: \mu = \mu_0$ vs $H_1: \mu = \mu_1 > \mu_0$ with known variance $\sigma^2$ and sample size $n$ , at level $\alpha$ , the power is:

2

\quad \text{Power} = \Phi\!\left( \frac{\mu_1 - \mu_0}{\sigma/\sqrt{n}} - z_{1-\alpha} \right)

Derivation hint: Reject when $\bar X > \mu_0 + z_{1-\alpha}\,\sigma/\sqrt n$ ; standardize $\bar X$ under $\mu = \mu_1$ so $(\bar X - \mu_1)/(\sigma/\sqrt n) \sim N(0,1)$ , then read off the tail probability. $\Phi$ is the standard normal CDF, $z_{1-\alpha}$ the $(1-\alpha)$ quantile, and the term $\delta = (\mu_1-\mu_0)\sqrt n/\sigma$ is the noncentrality parameter — the standardized effect size scaled by $\sqrt n$ .

DefinitionMultiple Testing Error Rates

Suppose $m$ hypotheses are tested, of which $m_0$ nulls are true. Let $V$ = number of true nulls rejected (false discoveries), $S$ = true alternatives rejected, $R = V + S$ = total rejections.

Family-Wise Error Rate: $\text{FWER} = P(V \geq 1)$ — probability of any false positive.
False Discovery Rate: $\text{FDR} = E\!\left[\dfrac{V}{R}\right]$ (with $V/R \equiv 0$ when $R=0$ ) — expected proportion of false positives among rejections.

3

\quad \text{FWER}_{\text{indep}} = 1 - (1-\alpha)^m \approx m\alpha \;\; (\alpha \text{ small})

Derivation hint: Under $m$ independent true nulls each tested at $\alpha$ , the probability that none is rejected is $(1-\alpha)^m$ ; complement gives the chance of at least one false rejection. First-order Taylor expansion gives $m\alpha$ .

TheoremNeyman–Pearson Lemma

Among all tests of a simple null $H_0: \theta = \theta_0$ versus a simple alternative $H_1: \theta = \theta_1$ at level $\alpha$ , the likelihood ratio test that rejects when

4

\quad \Lambda(\mathbf{x}) = \frac{L(\theta_1 \mid \mathbf{x})}{L(\theta_0 \mid \mathbf{x})} > k

Derivation hint: Choose threshold $k$ so the size equals $\alpha$ ; any other level- $\alpha$ test can be shown to have rejection region overlapping less with the high-likelihood-ratio region, hence less power. is most powerful — no other level- $\alpha$ test has greater power. Here $L(\theta\mid\mathbf x)$ is the likelihood and $k$ is set so $P_{\theta_0}(\Lambda > k) = \alpha$ .

TheoremBonferroni Inequality

If each of $m$ tests is conducted at level $\alpha/m$ , then the family-wise error rate is controlled:

5

\quad \text{FWER} = P\!\left(\bigcup_{i=1}^{m} A_i\right) \leq \sum_{i=1}^{m} P(A_i) = m \cdot \frac{\alpha}{m} = \alpha

Derivation hint: $A_i$ is the event "test $i$ falsely rejects." Boole's inequality (union bound) bounds the probability of a union by the sum of probabilities — no independence required.

ProofPower of the z-Test (Equation 2)

Step 1. The level- $\alpha$ rejection rule for $H_1: \mu > \mu_0$ rejects when

\bar X > \mu_0 + z_{1-\alpha}\,\frac{\sigma}{\sqrt n}.

Step 2. Power is the probability of this event computed under the true mean $\mu_1$ :

\text{Power} = P_{\mu_1}\!\left(\bar X > \mu_0 + z_{1-\alpha}\tfrac{\sigma}{\sqrt n}\right).

Step 3. Under $\mu=\mu_1$ , standardize: $Z = (\bar X - \mu_1)/(\sigma/\sqrt n) \sim N(0,1)$ . Subtract $\mu_1$ from both sides inside the probability:

\text{Power} = P\!\left(Z > \frac{\mu_0 - \mu_1}{\sigma/\sqrt n} + z_{1-\alpha}\right).

Step 4. Flip the inequality using symmetry $P(Z > c) = \Phi(-c)$ :

\text{Power} = \Phi\!\left(\frac{\mu_1 - \mu_0}{\sigma/\sqrt n} - z_{1-\alpha}\right). \qquad \blacksquare

This shows power increases with effect size $|\mu_1-\mu_0|$ , sample size $n$ , and decreasing $\sigma$ .

∎

ProofWhy FWER Inflates (Equation 3)

Step 1. Let $m$ true nulls be tested independently, each rejected with probability $\alpha$ under $H_0$ .

Step 2. The event "no false rejection" requires every test to fail to reject:

P(V = 0) = \prod_{i=1}^m (1-\alpha) = (1-\alpha)^m.

Step 3. FWER is the complement:

\text{FWER} = 1 - (1-\alpha)^m.

Step 4. For $m=20,\ \alpha=0.05$ : $\text{FWER} = 1 - 0.95^{20} = 1 - 0.358 = 0.642$ . A 64% chance of at least one false alarm despite each test being "valid." $\qquad \blacksquare$

∎

mathematician · Jerzy Neyman & Egon Pearson | 1894–1981 / 1895–1980

Neyman (Polish-American) and Pearson (British, son of Karl Pearson) collaborated from 1928 to develop the decision-theoretic foundation of hypothesis testing. Their 1933 paper introduced the explicit distinction between Type I and Type II errors, the concept of power, and the Neyman–Pearson Lemma identifying the most powerful test. This reframed Fisher's significance testing into a formal accept/reject framework with controllable error rates — the dominant paradigm of 20th-century applied statistics. Neyman also founded confidence-interval theory; Pearson edited Biometrika for decades.

🧠 Intuition

Picture two overlapping bell curves: one centered at $\mu_0$ (the null) and one at $\mu_1$ (the truth). You draw a vertical line — the critical value — and reject whenever data falls right of it.

The area of the null curve to the right of the line is $\alpha$ (false positives).
The area of the alternative curve to the right of the line is power.
The area of the alternative curve to the left is $\beta$ (misses).

Four levers move power: (1) slide the curves apart (bigger effect), (2) shrink them (more data / less noise), (3) move the line left (raise $\alpha$ — but you trade off more false positives), (4) use a one-sided test if direction is known. For multiple testing, imagine playing the lottery once vs. buying 1000 tickets. Each ticket (test) rarely wins (false positive), but buy enough and a "win" becomes near-certain. Correction shrinks each ticket's win-probability so the whole batch stays honest.

Figure

Power, Type I, and Type II errors as overlapping distributions

```svg

```

ExampleSample Size for Target Power

Given: A drug is expected to lower blood pressure by $\mu_1 - \mu_0 = 5$ mmHg. Population SD $\sigma = 12$ mmHg. We want $\alpha = 0.05$ (one-sided) and power $= 0.80$ . Find required $n$ .

Step 1. Required quantiles: $z_{1-\alpha} = z_{0.95} = 1.645$ ; for power $0.80$ , $z_{\text{power}} = z_{0.80} = 0.842$ .

Step 2. The sample-size formula (rearranging Equation 2 so power $= \Phi(z_{\text{power}})$ ) is:

n = \frac{(z_{1-\alpha} + z_{\text{power}})^2\, \sigma^2}{(\mu_1-\mu_0)^2}.

Step 3. Plug in:

n = \frac{(1.645 + 0.842)^2 \times 12^2}{5^2} = \frac{(2.487)^2 \times 144}{25} = \frac{6.185 \times 144}{25} = \frac{890.6}{25} = 35.6.

Step 4. Round up. > Answer: $\boxed{n = 36 \text{ patients}}$ to achieve 80% power.

ExampleBonferroni vs. Benjamini–Hochberg

Given: $m = 5$ genes tested for differential expression with p-values $0.001,\ 0.008,\ 0.02,\ 0.04,\ 0.20$ . Desired error control $\alpha = 0.05$ .

Step 1 — Bonferroni (FWER). Threshold each p against $\alpha/m = 0.05/5 = 0.01$ .

$0.001 < 0.01$ ✓ reject
$0.008 < 0.01$ ✓ reject
$0.02, 0.04, 0.20 >$ 0.01 ✗

> Bonferroni rejects 2 hypotheses.

Step 2 — Benjamini–Hochberg (FDR). Sort p-values (already sorted), rank $i=1\dots5$ . Compare $p_{(i)}$ to $\frac{i}{m}\alpha$ :

$i=1: 0.001 \le \frac{1}{5}(0.05)=0.010$ ✓
$i=2: 0.008 \le \frac{2}{5}(0.05)=0.020$ ✓
$i=3: 0.02 \le \frac{3}{5}(0.05)=0.030$ ✓
$i=4: 0.04 \le \frac{4}{5}(0.05)=0.040$ ✓
$i=5: 0.20 \le \frac{5}{5}(0.05)=0.050$ ✗

Step 3. BH rejects all up to the largest $i$ satisfying the condition, which is $i=4$ . > Answer: Bonferroni rejects $\boxed{2}$ genes; BH rejects $\boxed{4}$ genes. BH is more powerful but controls only the expected proportion of false discoveries, not the chance of any.

⚠️ Common mistake

"A non-significant result means the null is true." No — failure to reject can simply mean low power. With $n$ too small, even large true effects produce $p > 0.05$ . Always report power or confidence intervals: a wide CI around zero signals an inconclusive (underpowered) study, not evidence of "no effect."

" $\alpha$ is the probability the null is true given a significant result." False. $\alpha = P(\text{reject} \mid H_0)$ , not $P(H_0 \mid \text{reject})$ . Confusing these is the prosecutor's fallacy.

"Power is a single number for a test." Power is a function of the true parameter $\theta$ . A test has high power for large effects and low power for tiny ones. Always specify the effect size at which power is computed.

CounterexampleThe Look-Elsewhere / Garden of Forking Paths

A researcher tests 20 candidate dietary supplements for an effect on weight loss, each at $\alpha = 0.05$ , and finds that "green tea extract" is significant ( $p = 0.03$ ). They publish. But under the global null (no supplement works), the probability of at least one false positive is $1 - 0.95^{20} = 0.64$ . Finding a significant result was the expected outcome of pure chance. Without correction, this single "discovery" is essentially noise — a textbook false positive. Reporting only the one winning test (and hiding the 19 failures) makes the result irreproducible, the central engine of the replication crisis.

connection

Power and multiplicity sit at the heart of inferential statistics. The Neyman–Pearson framework that defines power extends Hypothesis Testing: Fundamentals by adding the alternative-hypothesis perspective. Power calculations rely on the Central Limit Theorem to justify normal approximations for test statistics, and on Effect Size to quantify the signal being detected. The noncentrality parameter links directly to the Noncentral Distributions (noncentral $t$ , $\chi^2$ , $F$ ). Multiple-testing corrections connect to Confidence Intervals via simultaneous coverage (e.g., Bonferroni-adjusted intervals) and to Bayesian Inference, which sidesteps multiplicity by modeling all effects jointly with shrinkage priors. The FDR framework is closely tied to Empirical Bayes Methods.

🔬 Where this appears

⏳ History

- 1900 — Karl Pearson introduces the

\chi^2

goodness-of-fit test, an early formal significance procedure.

- 1925 — R.A. Fisher's Statistical Methods for Research Workers popularizes the p-value and the

\alpha = 0.05

convention, but treats significance as evidence, not decision.

- 1928–1933 — Neyman & Pearson formalize Type I/II errors, power, and the most-powerful-test lemma, founding the accept/reject framework.

- 1935–1936 — Carlo Emilio Bonferroni publishes the inequalities (Boole's union bound applied to simultaneous events) underlying the eponymous correction.

- 1953 — John Tukey develops the Honestly Significant Difference test for multiple pairwise comparisons in ANOVA.

- 1979 — Sture Holm introduces the step-down Holm–Bonferroni method, uniformly more powerful than plain Bonferroni while still controlling FWER.

- 1988 — Yosef Hochberg publishes a step-up FWER procedure exploiting Simes' inequality.

- 1995 — Yoav Benjamini & Yosef Hochberg introduce the False Discovery Rate, revolutionizing high-dimensional testing in genomics and beyond.

- 2001 — Benjamini & Yekutieli extend FDR control to arbitrary dependence; Storey introduces the q-value and positive-FDR, refining the framework for the genomics era.

open-problem

Dependence structure. Most clean results (independent FWER, basic BH) assume independence or specific positive-dependence (PRDS) conditions. Real data — correlated genes, spatially smooth neuroimaging voxels — violate these. The Benjamini–Yekutieli correction guarantees FDR control under arbitrary dependence but at a cost of a factor $\sum_{i=1}^m 1/i \approx \ln m$ , sharply reducing power. Finding tight, dependence-aware procedures remains active research.

Choosing the right error rate. FWER vs. FDR is not a settled question — it depends on the cost of a false discovery relative to a missed one. Hybrid notions (per-family error rate, k-FWER, FDP confidence bounds) proliferate without consensus.

Post-selection inference. When the same data is used to select hypotheses and then test them (data dredging, adaptive analysis), classical guarantees collapse. Selective inference and conditional inference (Lee, Taylor, et al.) are an emerging but technically demanding remedy.

Optimal power under multiplicity. There is no fully general theory of "most powerful" multiple-testing procedure; optimality depends on the unknown configuration of true effects, leaving practical procedure choice partly heuristic.

Hypothesis Testing: Power and Multiple Testing

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also