Chi-Square Inference

⚡ Key Takeaways

One-line statement: Chi-square inference uses the chi-square ( $\chi^2$ ) distribution to test hypotheses about categorical data and variances by comparing observed counts to expected counts. Key result in words: When data are grouped into categories, the sum of standardized squared deviations between observed and expected frequencies follows (approximately) a chi-square distribution, enabling goodness-of-fit, independence, and homogeneity tests. Where it applies: Categorical/count data, contingency tables, variance estimation for normal populations. Who/when: Karl Pearson introduced the chi-square goodness-of-fit test in 1900; R.A. Fisher corrected the degrees-of-freedom theory in the 1920s.

💡 The Core Insight

A single standardized normal deviation squared, $Z^2$ , has a chi-square distribution with 1 degree of freedom. If we standardize each category's deviation $(O - E)/\sqrt{E}$ and square-and-sum across categories, the result behaves—approximately—like a sum of such squared normals. Large discrepancies between what we observe and what we expect produce large $\chi^2$ values; rare large values are evidence against the null hypothesis. The whole apparatus reduces complicated categorical comparisons to one number and one reference curve.

DefinitionThe Chi-Square Distribution

A random variable $X$ has a chi-square distribution with $k$ degrees of freedom, written $X \sim \chi^2_k$ , if it is the sum of squares of $k$ independent standard normal random variables:

1

\quad X = \sum_{i=1}^{k} Z_i^2, \qquad Z_i \stackrel{\text{iid}}{\sim} N(0,1)

Derivation hint: This is a definition, but $Z_i^2$ being the building block follows from the change-of-variables formula applied to a single $N(0,1)$ density. Its probability density function on $x > 0$ is

2

\quad f_k(x) = \frac{1}{2^{k/2}\,\Gamma(k/2)}\, x^{k/2 - 1} e^{-x/2}

Derivation hint: This is the Gamma $(k/2, 2)$ density; obtain it by computing the moment generating function $(1-2t)^{-k/2}$ and recognizing it. Here $\Gamma(\cdot)$ is the gamma function, $k$ is a positive integer (the degrees of freedom), $x$ is the variable, and $e$ is Euler's number. The mean is $k$ and the variance is $2k$ .

DefinitionPearson Chi-Square Statistic

For data sorted into $m$ mutually exclusive categories with observed counts $O_i$ and expected counts $E_i$ (under the null hypothesis), the Pearson statistic is

3

\quad \chi^2 = \sum_{i=1}^{m} \frac{(O_i - E_i)^2}{E_i}

Derivation hint: Each term standardizes the deviation $O_i - E_i$ by its approximate variance $E_i$ (Poisson-like); summing approximately reproduces a sum of squared normals. where $O_i$ is the observed frequency in category $i$ , $E_i = n p_i$ is the expected frequency, $n$ is the total sample size, and $p_i$ is the hypothesized probability of category $i$ .

TheoremPearson's Limit Theorem

Under the null hypothesis $H_0: (p_1, \dots, p_m)$ fixed and known, as the sample size $n \to \infty$ , the Pearson statistic in [3] converges in distribution to a chi-square random variable:

4

\quad \chi^2 \xrightarrow{\;d\;} \chi^2_{m-1}

Derivation hint: The standardized count vector tends to a multivariate normal with one linear constraint ( $\sum O_i = n$ ); the quadratic form of this constrained normal is $\chi^2$ with $m-1$ degrees of freedom. If the null model contains $r$ parameters estimated from the data, the degrees of freedom drop to $m - 1 - r$ .

ProofSketch of Pearson's Theorem

Step 1 — Multinomial setup. The vector of counts $(O_1, \dots, O_m)$ follows a multinomial distribution with parameters $n$ and $(p_1, \dots, p_m)$ , where $\sum p_i = 1$ .

Step 2 — Multivariate CLT. Define the standardized deviations

5

\quad U_i = \frac{O_i - n p_i}{\sqrt{n p_i}}.

Derivation hint: This is $(O_i - E_i)/\sqrt{E_i}$ ; by the multivariate central limit theorem the vector $(U_1,\dots,U_m)$ converges to a multivariate normal.

Step 3 — Covariance structure. The limiting normal has mean $\mathbf{0}$ and covariance matrix $\Sigma = I - \sqrt{p}\,\sqrt{p}^{\,T}$ , where $\sqrt{p}$ is the vector of $\sqrt{p_i}$ . This matrix is a projection of rank $m-1$ .

Step 4 — Quadratic form. The statistic is $\chi^2 = \sum U_i^2 = \mathbf{U}^T \mathbf{U}$ . A standard normal vector projected onto an $(m-1)$ -dimensional subspace yields a quadratic form distributed as $\chi^2_{m-1}$ :

6

\quad \mathbf{U}^T \mathbf{U} \xrightarrow{d} \chi^2_{\,\text{rank}(\Sigma)} = \chi^2_{m-1}.

Derivation hint: The eigenvalues of the projection $\Sigma$ are $m-1$ ones and one zero; a sum of squared normals with these weights is exactly $\chi^2_{m-1}$ .

Step 5 — Estimated parameters. Each independently estimated parameter imposes one further linear constraint, lowering the rank—hence the subtraction of $r$ from the degrees of freedom. $\blacksquare$

∎

Figure

Chi-square densities for several degrees of freedom; the right tail beyond the critical value is the rejection region

KP

Karl Pearson

1857–1936

Karl Pearson, founder of the world's first university statistics department at University College London, introduced the chi-square goodness-of-fit test in his 1900 paper "On the criterion that a given system of deviations...". A polymath trained in mathematics and law, Pearson sought objective criteria for judging how well theoretical curves fit real frequency data—motivated by biological and evolutionary problems from his collaboration with Francis Galton and Walter Weldon. His original derivation, however, used $m-1$ degrees of freedom even when parameters were estimated; this error was decisively corrected by Ronald Fisher.

🧠 Intuition

Imagine rolling a die 600 times. Under fairness you expect 100 of each face. If you get $(105, 98, 110, 92, 95, 100)$ , the deviations are small relative to $\sqrt{100}=10$ , so each standardized term is around 1 or less, and their sum is modest. But $(150, 60, 130, 70, 90, 100)$ produces deviations of 50, 40, etc., whose squares dwarf $E_i$ , inflating $\chi^2$ . The statistic essentially asks: "In units of natural sampling noise, how far off are we, totalled across all categories?" The chi-square distribution tells us how large that total tends to get by chance alone, so an observed value in the far right tail signals a genuine departure from the model.

ExampleGoodness-of-Fit: Testing a Fair Die

Given: A die is rolled $n = 120$ times. Observed counts: $O = (28, 14, 22, 18, 20, 18)$ . Test $H_0$ : the die is fair ( $p_i = 1/6$ ) at significance $\alpha = 0.05$ .

Step 1 — Expected counts. $E_i = n p_i = 120 \times \tfrac{1}{6} = 20$ for each face.

Step 2 — Compute terms.

\frac{(28-20)^2}{20} + \frac{(14-20)^2}{20} + \frac{(22-20)^2}{20} + \frac{(18-20)^2}{20} + \frac{(20-20)^2}{20} + \frac{(18-20)^2}{20}

= \frac{64 + 36 + 4 + 4 + 0 + 4}{20} = \frac{112}{20} = 5.6.

Step 3 — Degrees of freedom. $df = m - 1 = 6 - 1 = 5$ . No parameters estimated.

Step 4 — Critical value. $\chi^2_{0.05, 5} = 11.07$ .

Step 5 — Decision. Since $5.6 < 11.07$ , we fail to reject $H_0$ .

\boxed{\chi^2 = 5.6,\quad df = 5,\quad \text{fail to reject; data consistent with a fair die}}

ExampleTest of Independence: Treatment vs. Outcome

Given: A $2\times 2$ contingency table of patients:

	Recovered	Not recovered	Row total
Drug	50	30	80
Placebo	35	45	80
Col total	85	75	160

Test independence of treatment and outcome at $\alpha = 0.05$ .

Step 1 — Expected counts $E_{ij} = \dfrac{(\text{row}_i)(\text{col}_j)}{n}$ : $E_{11} = \tfrac{80 \times 85}{160} = 42.5$ , $E_{12} = \tfrac{80\times 75}{160}=37.5$ , $E_{21}=42.5$ , $E_{22}=37.5$ .

Step 2 — Statistic.

\chi^2 = \frac{(50-42.5)^2}{42.5}+\frac{(30-37.5)^2}{37.5}+\frac{(35-42.5)^2}{42.5}+\frac{(45-37.5)^2}{37.5}

= \frac{56.25}{42.5}+\frac{56.25}{37.5}+\frac{56.25}{42.5}+\frac{56.25}{37.5} = 1.324+1.5+1.324+1.5 = 5.648.

Step 3 — Degrees of freedom. $df = (r-1)(c-1) = (2-1)(2-1) = 1$ .

Step 4 — Critical value. $\chi^2_{0.05,1} = 3.841$ .

Step 5 — Decision. $5.648 > 3.841$ , so reject $H_0$ .

\boxed{\chi^2 = 5.65,\quad df = 1,\quad \text{reject independence: treatment associated with outcome}}

⚠️ Common mistake

The chi-square test requires sufficiently large expected counts. A common rule of thumb is that every $E_i$ should be at least 5 (some allow a few as low as 1 if most exceed 5). When expected counts are too small, the continuous chi-square approximation to the discrete multinomial is poor and p-values are unreliable. Remedies include combining categories or using Fisher's exact test. Also note: the statistic uses counts, never proportions or percentages—plugging in percentages is a frequent and fatal error.

Counterexample

The chi-square test is not a test of the strength of association. Consider two $2\times2$ tables with identical proportions but different sample sizes. With $n=40$ a moderate association may give $\chi^2 = 1.5$ (not significant), while the same proportions with $n=4000$ give $\chi^2 = 150$ (wildly significant). The statistic scales with $n$ , so a "significant" result with huge samples can reflect a trivially small effect. Significance $\ne$ importance—report an effect-size measure such as Cramér's $V$ alongside the test.

connection

Chi-square inference sits in a web of related ideas. It is a special case of the Likelihood Ratio Test, whose statistic $G^2 = 2\sum O_i \ln(O_i/E_i)$ is asymptotically equivalent to Pearson's $\chi^2$ . The distribution itself is the Gamma Distribution with shape $k/2$ and scale 2, and squaring a single Standard Normal Distribution variable gives $\chi^2_1$ . The ratio of two independent chi-squares (scaled) defines the F-Distribution used in Analysis of Variance, while a normal divided by a scaled chi-square root gives the Student t-Distribution. Inference about a normal variance uses $(n-1)s^2/\sigma^2 \sim \chi^2_{n-1}$ , directly linking it to Confidence Intervals.

🔬 Where this appears

⏳ History

- 1875: Friedrich Helmert derives the distribution of the sample variance for normal data—an early implicit chi-square result.

- 1900: Karl Pearson publishes the goodness-of-fit chi-square test, naming the statistic and tabulating its distribution.

- 1904: Pearson extends the method to contingency tables and association.

- 1908–1915: Refinements by Pearson and others on applying the test to estimated parameters (with the degrees-of-freedom error persisting).

- 1922–1924: R.A. Fisher corrects the degrees of freedom, showing that estimating

r

parameters reduces

df

by

r

, sparking a public dispute with Pearson.

- 1924: Fisher publishes tables and clarifies the relationship among

\chi^2

,

t

, and

F

distributions.

- 1934: Fisher introduces the exact test for small

2\times 2

tables, addressing the small-count limitation.

- 1954: William Cochran articulates practical rules for minimum expected frequencies and validity conditions.

open-problem

The chi-square approximation degrades for sparse tables, and there is no universally agreed threshold for "large enough" expected counts—Cochran's rules are heuristics, not theorems. For multi-way contingency tables, log-linear models and the question of which interaction terms to include push beyond classical chi-square testing into model-selection territory where the asymptotic degrees of freedom may not be integers in practice. Continuity corrections (Yates's correction for $2\times2$ tables) remain debated: they can over-correct and reduce power. Modern alternatives—exact permutation tests, Monte Carlo p-values, and Bayesian model comparison—address these limits but raise their own questions about computational feasibility and prior sensitivity in high dimensions.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also