Regression Inference

⚡ Key Takeaways

Regression inference is the body of statistical methods used to draw conclusions about population regression parameters (slopes, intercepts, and predicted means) from a sample of data. Its key result: under the standard linear model assumptions, the least-squares estimators are normally distributed (or approximately so), enabling confidence intervals, hypothesis tests, and prediction intervals for the unknown true relationship. It applies wherever one models how a response variable depends on one or more predictors. Developed by Galton, Pearson, Fisher, Gauss, and others from the 1800s through the 20th century.

💡 The Core Insight

The estimated coefficients are statistics computed from random data, so they have sampling distributions. Once we know the mean and variance (and shape) of those distributions, every inferential question—Is the slope nonzero? What range plausibly contains it? What range will a future observation fall in?—becomes an exercise in standardizing to a $t$ or $z$ variable. Inference is the bridge from one observed line to a statement about the whole family of lines that could have produced the data.

DefinitionThe Simple Linear Regression Model

We assume the data $(x_i, y_i)$ , $i = 1,\dots,n$ , are generated by

1

\quad Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \qquad \varepsilon_i \overset{\text{iid}}{\sim} \mathcal{N}(0,\sigma^2).

Derivation hint: this is an assumption (the model), not a derivation; the $x_i$ are treated as fixed/known constants and all randomness enters through $\varepsilon_i$ . Symbols:

$Y_i$ — the random response for observation $i$ .
$x_i$ — the (fixed) predictor value.
$\beta_0$ — true intercept (population mean of $Y$ when $x=0$ ).
$\beta_1$ — true slope (change in mean $Y$ per unit $x$ ).
$\varepsilon_i$ — random error; assumed independent, mean zero, constant variance $\sigma^2$ , normally distributed.
$\sigma^2$ — common error variance (usually unknown, estimated).

The assumptions (mnemonic LINE): Linearity of the mean, Independence of errors, Normality of errors, Equal variance (homoscedasticity).

DefinitionLeast-Squares Estimators

The estimators minimizing $\sum_i (y_i - b_0 - b_1 x_i)^2$ are

2

\quad \hat\beta_1 = \frac{\sum_{i}(x_i - \bar x)(y_i - \bar y)}{\sum_i (x_i - \bar x)^2} = \frac{S_{xy}}{S_{xx}}, \qquad \hat\beta_0 = \bar y - \hat\beta_1 \bar x.

Derivation hint: set the two partial derivatives of the sum of squared residuals with respect to $b_0$ and $b_1$ to zero (the normal equations) and solve the resulting $2\times 2$ linear system. Here $S_{xy} = \sum(x_i-\bar x)(y_i-\bar y)$ and $S_{xx}=\sum(x_i-\bar x)^2$ .

TheoremSampling Distribution of the Slope

Under model [1], the slope estimator satisfies

3

\quad \hat\beta_1 \sim \mathcal{N}\!\left(\beta_1,\ \frac{\sigma^2}{S_{xx}}\right).

Derivation hint: write $\hat\beta_1 = \sum_i c_i Y_i$ with weights $c_i = (x_i-\bar x)/S_{xx}$ ; then $\hat\beta_1$ is a linear combination of independent normals, so it is normal with mean $\sum c_i(\beta_0+\beta_1 x_i)=\beta_1$ and variance $\sigma^2\sum c_i^2 = \sigma^2/S_{xx}$ . Because $\sigma^2$ is unknown, we estimate it by the mean squared error

4

\quad s^2 = \frac{\sum_i (y_i - \hat y_i)^2}{n-2} = \frac{\text{SSE}}{n-2},

Derivation hint: dividing by $n-2$ (degrees of freedom = sample size minus the two estimated parameters) makes $s^2$ unbiased for $\sigma^2$ ; the loss of two df comes from estimating $\beta_0$ and $\beta_1$ . and the resulting standardized slope follows a $t$ distribution:

5

\quad T = \frac{\hat\beta_1 - \beta_1}{s/\sqrt{S_{xx}}} = \frac{\hat\beta_1 - \beta_1}{\operatorname{SE}(\hat\beta_1)} \sim t_{n-2}.

Derivation hint: a standard normal (from [3]) divided by $\sqrt{(\text{independent }\chi^2_{n-2})/(n-2)}$ yields a $t$ ; here $\text{SSE}/\sigma^2 \sim \chi^2_{n-2}$ and is independent of $\hat\beta_1$ (Cochran's theorem).

TheoremGauss–Markov

Among all linear unbiased estimators of $\beta_0$ and $\beta_1$ , the least-squares estimators have the smallest variance. They are the Best Linear Unbiased Estimators (BLUE). This requires only mean-zero, uncorrelated, equal-variance errors—normality is not needed for Gauss–Markov, only for the exact $t$ distribution.

ProofConfidence interval for the slope

Step 1. From [3], $\dfrac{\hat\beta_1 - \beta_1}{\sigma/\sqrt{S_{xx}}}\sim\mathcal N(0,1)$ — exact normality of the standardized slope.

Step 2. From [4], $\dfrac{(n-2)s^2}{\sigma^2}\sim\chi^2_{n-2}$ , independent of $\hat\beta_1$ .

Step 3. The ratio of Step 1's normal to $\sqrt{\text{Step 2}/(n-2)}$ cancels the unknown $\sigma$ and gives the pivot [5]: $T\sim t_{n-2}$ .

Step 4. A pivot has a distribution free of unknown parameters, so for $t^* = t_{n-2,\,1-\alpha/2}$ ,

P\!\left(-t^* \le \frac{\hat\beta_1-\beta_1}{\operatorname{SE}(\hat\beta_1)} \le t^*\right) = 1-\alpha.

Step 5. Solve the inequality for $\beta_1$ to obtain the interval

6

\quad \hat\beta_1 \pm t_{n-2,\,1-\alpha/2}\cdot \frac{s}{\sqrt{S_{xx}}}.

Derivation hint: algebraically isolate $\beta_1$ inside the probability statement; the random endpoints define a $(1-\alpha)$ confidence interval.

Step 6. To test $H_0:\beta_1 = 0$ (no linear relationship), set $\beta_1=0$ in [5] and compute the test statistic

7

\quad t = \frac{\hat\beta_1}{\operatorname{SE}(\hat\beta_1)},

Derivation hint: under $H_0$ the numerator of the pivot becomes $\hat\beta_1 - 0$ ; compare $t$ to the $t_{n-2}$ distribution to obtain a p-value. rejecting $H_0$ when $|t|$ exceeds the critical value. $\qquad\blacksquare$

∎

DefinitionInference for the mean response and prediction

For a target predictor value $x_0$ , the fitted value is $\hat y_0 = \hat\beta_0 + \hat\beta_1 x_0$ , with

8

\quad \operatorname{Var}(\hat y_0) = \sigma^2\!\left[\frac{1}{n} + \frac{(x_0-\bar x)^2}{S_{xx}}\right].

Derivation hint: write $\hat y_0$ as a linear combination of the $Y_i$ , or propagate variances of $\hat\beta_0$ and $\hat\beta_1$ accounting for their covariance; the term grows quadratically as $x_0$ departs from $\bar x$ . A confidence interval for the mean response $E[Y\mid x_0]$ uses [8]. A prediction interval for a new individual observation adds the irreducible error $\sigma^2$ :

9

\quad \operatorname{Var}(\text{pred error}) = \sigma^2\!\left[1 + \frac{1}{n} + \frac{(x_0-\bar x)^2}{S_{xx}}\right].

Derivation hint: the new observation $Y_0 = \beta_0+\beta_1 x_0 + \varepsilon_0$ contributes its own independent error $\varepsilon_0$ with variance $\sigma^2$ , added to the estimation variance from [8]. Replacing $\sigma$ by $s$ and using $t_{n-2}$ gives the intervals

10

\quad \hat y_0 \pm t_{n-2,\,1-\alpha/2}\, s\sqrt{\tfrac{1}{n} + \tfrac{(x_0-\bar x)^2}{S_{xx}}}\quad(\text{mean}),\qquad \hat y_0 \pm t_{n-2,\,1-\alpha/2}\, s\sqrt{1+\tfrac{1}{n} + \tfrac{(x_0-\bar x)^2}{S_{xx}}}\ (\text{prediction}).

Derivation hint: substitute $s$ for $\sigma$ in [8] and [9], take square roots for standard errors, and form the usual $t$ -based interval.

mathematician · Ronald A. Fisher | 1890–1962

Fisher gave regression inference its modern rigor. He clarified degrees of freedom, developed the analysis of variance that underlies the regression $F$ -test, derived exact small-sample distributions building on Gosset's $t$ , and championed maximum likelihood and significance testing. The decomposition of total variation into explained and residual sums of squares—and its interpretation through $F$ and $t$ statistics—is largely his legacy, refined at Rothamsted Experimental Station in the 1920s.

🧠 Intuition

Picture drawing your sample over and over. Each sample yields a slightly different best-fit line, pivoting around the center of mass $(\bar x, \bar y)$ . The spread of those slopes is the standard error. Equation [3] says this spread shrinks when (a) the noise $\sigma$ is small, (b) the sample is large, or (c) the $x$ -values are widely spread out (large $S_{xx}$ )—a long lever arm makes the slope easy to pin down. The confidence and prediction bands flare outward at the extremes of $x$ (the $(x_0-\bar x)^2$ term in [8]–[10]) because a small wobble in the slope swings the line most where it is farthest from the pivot. Finally, the prediction band is always wider than the confidence band because predicting one noisy individual is harder than estimating an average.

Figure

Regression line with confidence (inner) and prediction (outer) bands. Bands are narrowest at the mean of x and flare outward at the extremes.

ExampleSlope test and confidence interval

Given: $n=12$ observations. Computed quantities: $S_{xx}=200$ , $S_{xy}=160$ , $\bar x = 5$ , $\bar y = 14$ , and residual sum of squares $\text{SSE}=72$ .

Step 1 — Slope: $\hat\beta_1 = S_{xy}/S_{xx} = 160/200 = 0.80$ .

Step 2 — Intercept: $\hat\beta_0 = \bar y - \hat\beta_1\bar x = 14 - 0.80(5) = 10.0$ .

Step 3 — Error variance: $s^2 = \text{SSE}/(n-2) = 72/10 = 7.2$ , so $s = 2.683$ .

Step 4 — Standard error of slope: $\operatorname{SE}(\hat\beta_1)=s/\sqrt{S_{xx}} = 2.683/\sqrt{200} = 2.683/14.14 = 0.1897$ .

Step 5 — Test $H_0:\beta_1=0$ : $t = 0.80/0.1897 = 4.22$ on $n-2=10$ df. Since $t_{10,\,0.975}=2.228$ , we reject $H_0$ (p $\approx 0.0018$ ).

Step 6 — 95% CI for slope: $0.80 \pm 2.228(0.1897) = 0.80 \pm 0.423$ .

\boxed{\hat\beta_1 = 0.80,\quad t=4.22,\quad \text{95\% CI for }\beta_1 = (0.377,\ 1.223)\ \text{[units: }\Delta y/\Delta x\text{]}}

ExampleConfidence vs. prediction interval at a point

Given: Same fit as above ( $\hat\beta_0=10$ , $\hat\beta_1=0.80$ , $s=2.683$ , $n=12$ , $\bar x=5$ , $S_{xx}=200$ ). Predict at $x_0 = 8$ .

Step 1 — Point estimate: $\hat y_0 = 10 + 0.80(8) = 16.4$ .

Step 2 — Leverage term: $\dfrac{1}{n}+\dfrac{(x_0-\bar x)^2}{S_{xx}} = \dfrac{1}{12}+\dfrac{(8-5)^2}{200} = 0.0833 + 0.045 = 0.1283$ .

Step 3 — Mean-response SE: $s\sqrt{0.1283} = 2.683(0.3582)=0.961$ . With $t_{10,0.975}=2.228$ : margin $=2.141$ .

Step 4 — 95% CI for mean response: $16.4 \pm 2.141 = (14.26,\ 18.54)$ .

Step 5 — Prediction SE: $s\sqrt{1+0.1283}=2.683(1.0623)=2.850$ ; margin $=2.228(2.850)=6.351$ .

Step 6 — 95% prediction interval: $16.4 \pm 6.351 = (10.05,\ 22.75)$ .

\boxed{\text{Mean }E[Y\mid 8]\in(14.26,\,18.54);\quad \text{New }Y\in(10.05,\,22.75)}

The prediction interval is far wider, reflecting the extra $\sigma^2$ of an individual outcome.

⚠️ Common mistake

A confidence interval is not a prediction interval. The CI in [10] (left) covers the average $Y$ at $x_0$ ; the PI (right) covers a single future $Y$ . Reporting the narrow CI when you mean to predict an individual outcome dramatically overstates precision.

A high $R^2$ does not validate inference. $R^2$ measures variance explained, not whether the LINE assumptions hold or whether $\beta_1$ is significant. You can have a tiny $R^2$ with a highly significant slope (large $n$ ) or a large $R^2$ with badly nonlinear residuals.

"Fail to reject" $\ne$ "the slope is zero." A non-significant $t$ means insufficient evidence, not proof of no effect—especially with small $S_{xx}$ or small $n$ , where the SE is large.

Extrapolation invalidates the intervals. Equations [8]–[10] are valid only over the observed range of $x$ ; outside it, the linearity assumption is untested and the flaring bands become meaninglessly wide and unreliable.

CounterexampleSignificance does not imply causation or correct model

Suppose $y$ truly depends on $x$ quadratically, $y = x^2 + \varepsilon$ , but we fit a straight line over $x\in[1,10]$ . The fitted slope is highly significant ( $p<0.001$ ) and the $t$ -test "confirms a relationship." Yet the inference about $\beta_1$ is about a misspecified model: the residuals show a clear U-shape, violating linearity and independence-of-trend. The valid-looking p-value is computed under assumptions that are false, so the standard errors—and hence the entire inference—are not trustworthy. Always inspect residual plots before believing a regression p-value.

connection

Regression inference sits at the meeting point of several pillars. The $t$ pivot in [5] is the same machinery as Student's t-Distribution inference for a mean; in fact testing a single slope is a special case of the general Hypothesis Testing framework. The variance decomposition behind the overall fit ( $\text{SST}=\text{SSR}+\text{SSE}$ ) connects directly to Analysis of Variance and the $F$ -test, which for one predictor satisfies $F=t^2$ . Multiple predictors generalize everything through Ordinary Least Squares in matrix form, $\hat{\boldsymbol\beta}=(X^TX)^{-1}X^TY$ , with covariance $\sigma^2(X^TX)^{-1}$ . The estimator-optimality result is the Gauss-Markov Theorem, while the route to exact distributions when errors are normal is Maximum Likelihood Estimation (least squares = MLE under normal errors). The correlation coefficient and $R^2$ link to Pearson Correlation.

🔬 Where this appears

⏳ History

- 1805–1809 — A.-M. Legendre (1805) publishes the method of least squares; C. F. Gauss (1809) claims prior use and links it to the normal error law.

- 1823 — Gauss proves the Gauss–Markov theorem (optimality of least squares among linear unbiased estimators) in his Theoria Combinationis.

- 1885–1889 — Francis Galton studies heredity, coins "regression toward mediocrity" (the regression effect) while comparing parents' and children's heights.

- 1896 — Karl Pearson formalizes the correlation coefficient and the mathematics of regression, founding biometry.

- 1908 — W. S. Gosset ("Student") derives the $t$ -distribution, enabling exact small-sample inference for means and, later, slopes.

- 1918–1925 — R. A. Fisher introduces analysis of variance, degrees of freedom, and the

F

-distribution; Statistical Methods for Research Workers (1925) systematizes regression inference.

- 1933–1938 — Neyman and Pearson develop the hypothesis-testing framework (Type I/II errors, power); Neyman formalizes confidence intervals (1937).

- Mid-20th century — Matrix formulation of multiple regression and the general linear model become standard; computing makes routine inference feasible.

- 1960s–1980s — Diagnostics (residual analysis, leverage, Cook's distance), robust and heteroscedasticity-consistent standard errors extend inference beyond the classical LINE assumptions.

open-problem

Classical regression inference rests on assumptions that real data routinely break, and handling these robustly remains an active concern:

Heteroscedasticity and autocorrelation. When $\operatorname{Var}(\varepsilon_i)$ varies or errors are correlated (time series), the SEs in [3]–[10] are wrong. White's heteroscedasticity-consistent ("sandwich") estimators and Newey–West corrections patch this, but choosing among them and ensuring valid coverage in finite samples is non-trivial.
Errors-in-variables. If $x$ is itself measured with noise, $\hat\beta_1$ is biased toward zero (attenuation); the standard inference is invalid and identification requires instrumental variables or structural assumptions.
Selective inference / data snooping. Building a model from the data and then testing on the same data invalidates p-values. Post-selection inference (e.g., the work of Berk, Lockhart, Taylor, Tibshirani on selective inference and the LASSO) seeks valid intervals after model selection—an area still developing.
Non-normal, heavy-tailed errors. The $t$ pivot is exact only under normality; the central limit theorem rescues large-sample inference, but small samples with outliers can mislead, motivating robust/quantile regression.
Causal interpretation. A precise CI for $\beta_1$ says nothing about causation absent randomization or a credible identification strategy—confounding remains the deepest practical limit.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also