Regression

⚡ Key Takeaways

Regression finds the line (or curve) that best fits a set of data points, letting you predict one variable from another. The key result: the least-squares line $\hat{y} = a + bx$ minimizes the total squared vertical distance from points to the line. It applies anywhere you have paired numerical data — economics, biology, physics labs, machine learning. First developed by Francis Galton (1880s) and formalized using least squares by Gauss and Legendre (~1805).

💡 The Core Insight

Among all possible straight lines you could draw through a cloud of points, exactly one minimizes the sum of squared vertical errors. That line is unique, computable from simple sums of the data, and always passes through the point $(\bar{x}, \bar{y})$ — the center of mass of the data. Squaring the errors (rather than taking absolute values) makes the problem solvable with calculus and uniquely determined.

DefinitionSimple Linear Regression Model

We assume each observed $y_i$ relates to $x_i$ by a true line plus random error:

1

\quad y_i = \alpha + \beta x_i + \varepsilon_i

Derivation hint: this is the assumed model, not derived — $\varepsilon_i$ captures noise with mean zero. Here $\alpha$ is the true intercept, $\beta$ the true slope, and $\varepsilon_i$ the random error for point $i$ . We estimate $\alpha, \beta$ with $a, b$ , giving the fitted line:

2

\quad \hat{y}_i = a + b x_i

Derivation hint: $\hat{y}_i$ is the predicted value; the hat denotes an estimate. The residual for point $i$ is the gap between observed and predicted:

3

\quad e_i = y_i - \hat{y}_i = y_i - (a + b x_i)

Derivation hint: vertical distance from data point to the fitted line.

DefinitionThe Least-Squares Criterion

We choose $a$ and $b$ to minimize the sum of squared residuals $S$ :

4

\quad S(a,b) = \sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n}\bigl(y_i - a - b x_i\bigr)^2

Derivation hint: total squared vertical error as a function of the two unknowns.

TheoremLeast-Squares Solution

The values of $a$ and $b$ that minimize $S$ are:

5

\quad b = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n}(x_i - \bar{x})^2}

Derivation hint: covariance of $x,y$ divided by variance of $x$ ; comes from setting derivatives to zero.

6

\quad a = \bar{y} - b\,\bar{x}

Derivation hint: forces the line through $(\bar{x},\bar{y})$ . where $\bar{x} = \frac{1}{n}\sum x_i$ and $\bar{y} = \frac{1}{n}\sum y_i$ are the means.

DefinitionCorrelation Coefficient

The strength and direction of the linear relationship are measured by Pearson's $r$ :

7

\quad r = \frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum (x_i-\bar{x})^2}\,\sqrt{\sum (y_i-\bar{y})^2}}

Derivation hint: standardized covariance; always lies in $[-1, 1]$ . The coefficient of determination $r^2$ gives the fraction of variance in $y$ explained by $x$ :

8

\quad r^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}

Derivation hint: 1 minus (unexplained variation / total variation).

ProofDeriving the Least-Squares Formulas

We minimize $S(a,b) = \sum (y_i - a - b x_i)^2$ over $a$ and $b$ .

Step 1. Take the partial derivative with respect to $a$ and set it to zero:

9

\quad \frac{\partial S}{\partial a} = -2\sum (y_i - a - b x_i) = 0

Derivation hint: chain rule, derivative of inner $(\cdot)$ w.r.t. $a$ is $-1$ .

Step 2. Dividing by $-2$ and rearranging gives the first normal equation:

10

\quad \sum y_i = n a + b \sum x_i

Derivation hint: sum the constant $a$ across $n$ points. Dividing through by $n$ yields $\bar{y} = a + b\bar{x}$ , which rearranges to equation [6].

Step 3. Take the partial derivative with respect to $b$ and set it to zero:

11

\quad \frac{\partial S}{\partial b} = -2\sum x_i (y_i - a - b x_i) = 0

Derivation hint: derivative of inner $(\cdot)$ w.r.t. $b$ is $-x_i$ .

Step 4. This gives the second normal equation:

12

\quad \sum x_i y_i = a\sum x_i + b\sum x_i^2

Derivation hint: distribute $x_i$ inside the sum.

Step 5. Substitute $a = \bar{y} - b\bar{x}$ from Step 2 into [12], and after algebra (collecting $b$ terms) you obtain equation [5]. Because $S$ is a sum of squares — a convex paraboloid in $(a,b)$ — this critical point is the unique global minimum. $\blacksquare$

∎

mathematician · Carl Friedrich Gauss | 1777–1855

Gauss claimed to have used the method of least squares as early as 1795 to predict the orbit of the dwarf planet Ceres, famously relocating it after it disappeared behind the Sun. Adrien-Marie Legendre published the method first in 1805, sparking a priority dispute. Gauss later connected least squares to the normal (Gaussian) distribution of errors, showing the method is optimal when errors are normally distributed — a result underpinning the Gauss–Markov theorem.

🧠 Intuition

Picture a scatter of points and imagine threading a rigid rod through them. Connect each point to the rod with a vertical spring. Each spring stores energy proportional to the square of its stretch. The least-squares line is the resting position where total spring energy is minimal — the rod settles into the configuration where it pulls equally hard from above and below. Why square the errors rather than just add absolute distances? Squaring (1) makes large errors count disproportionately, so the line "cares" about outliers; (2) gives a smooth function we can differentiate; and (3) produces a unique, closed-form answer. The slope $b$ is essentially "how much $y$ tends to move per unit move in $x$ ," computed as the covariance of $x$ and $y$ normalized by the spread of $x$ .

Figure

Least-squares line minimizing squared vertical residuals

```svg

```

ExamplePredicting Exam Score from Study Hours

Given 5 students with hours studied $x$ and score $y$ :

$x$	1	2	3	4	5
$y$	52	58	67	71	82

Step 1 — Means: $\bar{x} = 3$ , $\bar{y} = (52+58+67+71+82)/5 = 330/5 = 66$ .

Step 2 — Deviations and products:

$x_i-\bar x$	$y_i-\bar y$	product	$(x_i-\bar x)^2$
$-2$	$-14$	$28$	$4$
$-1$	$-8$	$8$	$1$
$0$	$1$	$0$	$0$
$1$	$5$	$5$	$1$
$2$	$16$	$32$	$4$

Sums: $\sum(x-\bar x)(y-\bar y) = 73$ , $\sum(x-\bar x)^2 = 10$ .

Step 3 — Slope (eq [5]): $b = 73/10 = 7.3$ .

Step 4 — Intercept (eq [6]): $a = 66 - 7.3\times 3 = 66 - 21.9 = 44.1$ .

Fitted line:

\boxed{\hat{y} = 44.1 + 7.3x \ \text{(points)}}

Prediction for 6 hours: $\hat{y} = 44.1 + 7.3(6) = 87.9$ points.

ExampleComputing the Correlation Coefficient

Given the same data. We need $\sum(y_i-\bar y)^2 = (-14)^2+(-8)^2+1^2+5^2+16^2 = 196+64+1+25+256 = 542$ .

Step 1 — Apply eq [7]:

r = \frac{73}{\sqrt{10}\cdot\sqrt{542}} = \frac{73}{\sqrt{5420}} = \frac{73}{73.62} = 0.9916.

Step 2 — Coefficient of determination:

\boxed{r^2 = 0.9916^2 \approx 0.983}

Interpretation: about 98.3% of the variation in exam scores is explained by hours studied — a very strong positive linear relationship.

⚠️ Correlation Is Not Causation

A high $r$ means $x$ and $y$ move together — it does not prove that $x$ causes $y$ . Ice-cream sales correlate with drowning deaths, but both are driven by a hidden third variable (hot weather). Regression describes association, not mechanism. Establishing causation requires controlled experiments or careful causal reasoning, not just a good fit.

⚠️ Do Not Extrapolate Beyond the Data

The fitted line is only trustworthy within the range of observed $x$ values. Predicting an exam score for 20 hours of study with our line gives $\hat{y} = 44.1 + 7.3(20) = 190$ — impossible, since scores cap at 100. Extrapolation assumes the linear trend continues forever, which it almost never does.

CounterexampleAnscombe's Quartet

In 1973 statistician Frank Anscombe constructed four datasets with identical means, identical slope ( $b = 0.5$ ), identical intercept, and identical $r = 0.816$ — yet they look completely different. One is a clean line, one is a perfect parabola, one is a line plus a single wild outlier, and one is a vertical cluster dragged by one extreme point. Lesson: never trust regression numbers without plotting the data first.

CounterexampleRegression of

y

on

x

≠ Regression of

x

on

y

The line minimizing vertical errors (predicting $y$ from $x$ ) is not the same as the line minimizing horizontal errors (predicting $x$ from $y$ ). They have different slopes and only coincide when $r = \pm 1$ . Choosing which variable is the response matters.

connection

Regression sits at a crossroads of several ideas. The slope formula in [5] is the covariance of $x$ and $y$ over the variance of $x$ , linking directly to Variance and Standard Deviation and Covariance. The derivation uses Partial Derivatives to minimize a function of two variables — an application of Optimization. The correlation coefficient $r$ relates to the Pearson Correlation and, when squared, to the analysis of variance decomposition of total variation. Extending [1] to several predictors gives multiple regression, expressible compactly with Matrix Algebra and the normal equations $X^\top X \mathbf{b} = X^\top \mathbf{y}$ . Regression is also the simplest member of Supervised Learning in machine learning.

🔬 Where this appears

⏳ History

- 1805 — Adrien-Marie Legendre publishes the method of least squares in an appendix on comet orbits.

- 1809 — Carl Friedrich Gauss publishes least squares, claims prior use (1795) for Ceres, links it to the normal distribution.

- 1821–1823 — Gauss proves the Gauss–Markov theorem: least squares is the best linear unbiased estimator.

- 1877–1886 — Francis Galton studies heredity, coins "regression toward mediocrity" while studying parent–child heights.

- 1896 — Karl Pearson formalizes the correlation coefficient

r

and the modern theory of regression.

- 1900s — Pearson and G. Udny Yule extend regression to multiple variables.

- 1922 — Ronald Fisher develops the statistical framework (variance, significance testing) underpinning regression inference.

- 1973 — Frank Anscombe publishes Anscombe's Quartet, dramatizing the need to visualize data.

- 1970s–present — regression generalizes into ridge/lasso, logistic, and the regression layers inside modern neural networks.

open-problem

Ordinary least squares assumes errors are independent, have constant variance (homoscedasticity), and that $x$ is measured without error. Real data routinely violate all three: heteroscedasticity (spreading variance), autocorrelation (in time series), and measurement error in predictors each bias or destabilize estimates. Outliers can dominate the fit because errors are squared, motivating robust regression methods. With many correlated predictors, multicollinearity makes coefficients unstable — an active area mitigated by regularization (ridge, lasso). And linear regression can only capture linear structure; choosing the right transformation or model form for nonlinear data remains as much art as science.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also