Joint Distributions, Covariance, and Correlation

⚡ Key Takeaways

Joint distributions describe how two or more random variables behave together; covariance $\operatorname{Cov}(X,Y)$ measures the linear co-movement of $X$ and $Y$ ; correlation $\rho_{XY}$ is its dimensionless, scale-free normalization confined to $[-1,1]$ . Used everywhere: statistics, finance, signal processing, machine learning. Concepts crystallized by Galton and Pearson (1880s–1890s).

💡 The Core Insight

The full joint distribution contains everything about how variables relate, but it is a complicated object (a surface or a table). Covariance and correlation compress that interdependence into a single number capturing only the linear trend. This is both their power (one interpretable scalar) and their danger (they are blind to nonlinear structure). Correlation answers exactly one question: "As $X$ rises above its mean, does $Y$ tend to rise or fall above its mean, and how strongly — measured in standard-deviation units?"

DefinitionJoint Distribution

For two random variables $X,Y$ :

Discrete case — the joint PMF is

1

\quad p_{XY}(x,y) = P(X=x,\ Y=y), \qquad \sum_x\sum_y p_{XY}(x,y)=1.

Derivation hint: probabilities of a complete, mutually exclusive set of outcomes sum to 1.

Continuous case — the joint PDF $f_{XY}(x,y)\ge 0$ satisfies

2

\quad P\big((X,Y)\in A\big)=\iint_A f_{XY}(x,y)\,dx\,dy,\qquad \int_{-\infty}^\infty\!\!\int_{-\infty}^\infty f_{XY}=1.

Derivation hint: probability is the volume under the joint density surface over region $A$ . The marginal distributions recover each variable alone by summing/integrating out the other:

3

\quad f_X(x)=\int_{-\infty}^\infty f_{XY}(x,y)\,dy,\qquad p_X(x)=\sum_y p_{XY}(x,y).

Derivation hint: collapse the joint over all values of the variable being eliminated (law of total probability). The conditional density is

4

\quad f_{Y\mid X}(y\mid x)=\frac{f_{XY}(x,y)}{f_X(x)},\qquad f_X(x)>0.

Derivation hint: restrict to the slice $X=x$ and renormalize so the slice integrates to 1. Here $x$ are realized values, $y$ are realized values, $A$ a region in the plane.

DefinitionIndependence

$X$ and $Y$ are independent iff the joint factors into marginals for all $x,y$ :

5

\quad f_{XY}(x,y)=f_X(x)\,f_Y(y)\quad\Longleftrightarrow\quad f_{Y\mid X}(y\mid x)=f_Y(y).

Derivation hint: substitute the product form into the conditional definition [4]; the marginal cancels.

DefinitionCovariance

The covariance of $X$ and $Y$ with means $\mu_X=E[X],\ \mu_Y=E[Y]$ is

6

\quad \operatorname{Cov}(X,Y)=E\big[(X-\mu_X)(Y-\mu_Y)\big].

Derivation hint: average the product of deviations from the mean — positive when they deviate together, negative when oppositely. An equivalent, computation-friendly form:

7

\quad \operatorname{Cov}(X,Y)=E[XY]-E[X]\,E[Y].

Derivation hint: expand the product in [6], use linearity of expectation, and cancel $-\mu_X\mu_Y$ terms. The expectation of a product for the continuous case is

8

\quad E[XY]=\int_{-\infty}^\infty\!\!\int_{-\infty}^\infty xy\,f_{XY}(x,y)\,dx\,dy.

Derivation hint: law of the unconscious statistician applied to the function $g(x,y)=xy$ .

DefinitionCorrelation Coefficient

The (Pearson) correlation coefficient is the normalized covariance:

9

\quad \rho_{XY}=\frac{\operatorname{Cov}(X,Y)}{\sigma_X\,\sigma_Y},\qquad \sigma_X=\sqrt{\operatorname{Var}(X)},\ \sigma_Y=\sqrt{\operatorname{Var}(Y)}.

Derivation hint: divide covariance by the product of standard deviations to cancel all units and bound the result. Symbols: $\sigma_X,\sigma_Y$ are standard deviations, $\operatorname{Var}(X)=\operatorname{Cov}(X,X)=E[(X-\mu_X)^2]$ .

TheoremProperties of Covariance and Correlation

For random variables and constants $a,b,c,d$ :

10

\quad \operatorname{Cov}(aX+b,\;cY+d)=ac\,\operatorname{Cov}(X,Y).

Derivation hint: additive shifts $b,d$ leave deviations unchanged; scale factors pull out of expectation.

11

\quad \operatorname{Var}(X+Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y).

Derivation hint: expand $E[((X+Y)-(\mu_X+\mu_Y))^2]$ and collect the cross term.

12

\quad -1\le \rho_{XY}\le 1.

Derivation hint: apply Cauchy–Schwarz to the centered variables (see proof below).

13

\quad |\rho_{XY}|=1\iff Y=aX+b\ \text{almost surely, with }a\ne0.

Derivation hint: equality in Cauchy–Schwarz holds iff the centered variables are linearly dependent.

ProofCorrelation is bounded:

|\rho_{XY}|\le 1

We prove [12] via the Cauchy–Schwarz inequality on centered variables.

Step 1. Define the centered variables $U=X-\mu_X$ , $V=Y-\mu_Y$ , so $E[U]=E[V]=0$ , $\operatorname{Cov}(X,Y)=E[UV]$ , $\operatorname{Var}(X)=E[U^2]$ , $\operatorname{Var}(Y)=E[V^2]$ .

Step 2. For any real $t$ , the quantity $E[(U-tV)^2]\ge0$ since it is the expectation of a square. Expand:

14

\quad E[(U-tV)^2]=E[U^2]-2t\,E[UV]+t^2E[V^2]\ge 0.

Derivation hint: square the binomial inside, apply linearity of expectation.

Step 3. This is a quadratic in $t$ that is always $\ge0$ , so its discriminant must be $\le 0$ :

15

\quad \big(2E[UV]\big)^2-4\,E[V^2]\,E[U^2]\le 0 \;\Longrightarrow\; \big(E[UV]\big)^2\le E[U^2]\,E[V^2].

Derivation hint: a non-negative parabola opening upward has at most one real root; $b^2-4ac\le0$ .

Step 4. Substitute back:

16

\quad \big(\operatorname{Cov}(X,Y)\big)^2\le \operatorname{Var}(X)\operatorname{Var}(Y)\;\Longrightarrow\; \rho_{XY}^2\le 1.

Derivation hint: divide [15] by $\operatorname{Var}(X)\operatorname{Var}(Y)$ and take square roots.

Step 5 (equality case). Equality holds iff the discriminant is zero, i.e. $E[(U-t^*V)^2]=0$ for some $t^*$ , which forces $U=t^*V$ almost surely — a perfect linear relationship $X-\mu_X=t^*(Y-\mu_Y)$ , proving [13]. $\blacksquare$

∎

ProofIndependence ⟹ Zero Covariance

Step 1. If $X,Y$ independent, $f_{XY}=f_Xf_Y$ by [5].

Step 2. Then

17

\quad E[XY]=\iint xy\,f_X(x)f_Y(y)\,dx\,dy=\Big(\int x f_X\,dx\Big)\Big(\int y f_Y\,dy\Big)=E[X]E[Y].

Derivation hint: the double integral separates into a product of single integrals because the integrand factors.

Step 3. By [7], $\operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y]=0$ , hence $\rho_{XY}=0$ . $\blacksquare$

Caution: the converse is false — see the counterexample below.

∎

mathematician · Karl Pearson | 1857–1936

Pearson, building directly on Francis Galton's work on heredity, gave the correlation coefficient its modern mathematical definition (the "product-moment correlation") in the 1890s. He founded the world's first university statistics department at University College London (1911) and the journal Biometrika. Pearson formalized many tools — the chi-squared test, the method of moments, the correlation coefficient $r$ named for him — though his reputation is shadowed by his promotion of eugenics. His insistence that statistical relationships could be quantified without invoking causation set the philosophical tone for modern applied statistics.

🧠 Intuition

Picture a cloud of data points on the $X$ – $Y$ plane. Draw the two mean lines $x=\mu_X$ and $y=\mu_Y$ , splitting the plane into four quadrants. Covariance is essentially the average signed area of the rectangles formed by each point's deviations $(X-\mu_X)(Y-\mu_Y)$ :

A point in the upper-right (both above mean) or lower-left (both below) contributes a positive product.
A point in the upper-left or lower-right contributes a negative product.

If the cloud tilts up-right, positives dominate → positive covariance. If it tilts down-right, negatives dominate → negative. If it's a symmetric blob, contributions cancel → zero. Correlation then asks: "After stretching/squashing each axis to standard-deviation units, how tight is the tilt?" $\rho=\pm1$ means the points fall on a perfect line; $\rho=0$ means no linear tilt at all (but possibly a curved relationship you can't see in $\rho$ ).

Figure

Sign of covariance by quadrant: products of mean-deviations

ExampleDiscrete joint PMF — full computation

Given. A joint PMF table over $X\in\{0,1\}$ , $Y\in\{0,1\}$ :

	$Y=0$	$Y=1$
$X=0$	$0.1$	$0.2$
$X=1$	$0.3$	$0.4$

Step 1 — marginals. $P(X=1)=0.3+0.4=0.7$ , $P(X=0)=0.3$ . $P(Y=1)=0.2+0.4=0.6$ , $P(Y=0)=0.4$ .

Step 2 — means. $E[X]=1\cdot0.7=0.7$ , $E[Y]=1\cdot0.6=0.6$ .

Step 3 — $E[XY]$ . Only the $(1,1)$ cell gives $xy\ne0$ : $E[XY]=1\cdot1\cdot0.4=0.4$ .

Step 4 — covariance via [7]: $\operatorname{Cov}=0.4-0.7\cdot0.6=0.4-0.42=-0.02$ .

Step 5 — variances. $\operatorname{Var}(X)=E[X^2]-E[X]^2=0.7-0.49=0.21$ (since $X^2=X$ for 0/1). Likewise $\operatorname{Var}(Y)=0.6-0.36=0.24$ .

Step 6 — correlation via [9]: $\rho=\dfrac{-0.02}{\sqrt{0.21}\sqrt{0.24}}=\dfrac{-0.02}{\sqrt{0.0504}}=\dfrac{-0.02}{0.2245}$ .

\boxed{\operatorname{Cov}(X,Y)=-0.02,\qquad \rho_{XY}\approx -0.089\ \text{(dimensionless)}}

A weak negative linear relationship.

ExampleContinuous joint density — covariance via integration

Given. $f_{XY}(x,y)=x+y$ on the unit square $0\le x\le1,\ 0\le y\le1$ (zero elsewhere). Verify it's valid, then find $\rho$ .

Step 1 — validity. $\int_0^1\!\int_0^1 (x+y)\,dx\,dy=\int_0^1(\tfrac12+y)\,dy=\tfrac12+\tfrac12=1$ . ✓

Step 2 — marginal of $X$ . $f_X(x)=\int_0^1(x+y)\,dy=x+\tfrac12$ .

Step 3 — $E[X]$ . $\int_0^1 x(x+\tfrac12)\,dx=\tfrac13+\tfrac14=\tfrac{7}{12}$ . By symmetry $E[Y]=\tfrac{7}{12}$ .

Step 4 — $E[X^2]$ . $\int_0^1 x^2(x+\tfrac12)\,dx=\tfrac14+\tfrac16=\tfrac{5}{12}$ . So $\operatorname{Var}(X)=\tfrac{5}{12}-\left(\tfrac{7}{12}\right)^2=\tfrac{60}{144}-\tfrac{49}{144}=\tfrac{11}{144}$ . Same for $Y$ .

Step 5 — $E[XY]$ via [8]: $\int_0^1\!\int_0^1 xy(x+y)\,dx\,dy=\int_0^1\!\int_0^1(x^2y+xy^2)\,dx\,dy$ . Inner over $x$ : $\tfrac13 y+\tfrac12 y^2$ . Outer over $y$ : $\tfrac13\cdot\tfrac12+\tfrac12\cdot\tfrac13=\tfrac16+\tfrac16=\tfrac13$ .

Step 6 — covariance [7]: $\operatorname{Cov}=\tfrac13-\left(\tfrac{7}{12}\right)^2=\tfrac{48}{144}-\tfrac{49}{144}=-\tfrac{1}{144}$ .

Step 7 — correlation [9]: $\rho=\dfrac{-1/144}{11/144}=-\dfrac{1}{11}$ .

\boxed{\operatorname{Cov}(X,Y)=-\tfrac{1}{144}\approx-0.00694,\qquad \rho_{XY}=-\tfrac{1}{11}\approx -0.0909}

⚠️ Common mistake

Correlation ≠ causation. $\rho\ne0$ tells you two variables move together; it says nothing about why. A lurking confounder, reverse causation, or pure coincidence (spurious correlation) can all produce strong $\rho$ .

Zero correlation ≠ independence. Correlation captures only the linear component of dependence. A perfectly predictable nonlinear relationship can have $\rho=0$ .

Independence ⟹ zero covariance, but NOT the reverse. The implication is one-directional.

Correlation is not robust. A single outlier can drag Pearson's $\rho$ dramatically; it also assumes the relationship is linear and the second moments exist (correlation is undefined for, e.g., a Cauchy distribution with infinite variance).

CounterexampleUncorrelated but dependent

Let $X$ be uniform on $\{-1,0,1\}$ (each probability $\tfrac13$ ) and define $Y=X^2$ . Then $Y$ is completely determined by $X$ — maximal dependence. Compute: $E[X]=0$ . $E[XY]=E[X^3]=\tfrac13(-1)+\tfrac13(0)+\tfrac13(1)=0$ . Therefore $\operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y]=0-0=0$ , so $\rho_{XY}=0$ .

Yet $X$ and $Y$ are not independent: e.g. $P(Y=0\mid X=0)=1\ne P(Y=0)=\tfrac13$ . The covariance is blind to the parabolic relationship because the symmetric positive and negative deviations cancel.

connection

Joint distributions, covariance, and correlation sit at the hub of probability and statistics:

They generalize Variance and Standard Deviation — variance is $\operatorname{Cov}(X,X)$ .
For many variables, covariances assemble into the covariance matrix, central to Principal Component Analysis and the Multivariate Normal Distribution.
They underpin Linear Regression, where the slope equals $\operatorname{Cov}(X,Y)/\operatorname{Var}(X)$ and $\rho^2$ is the coefficient of determination.
The conditional density [4] connects to Conditional Expectation and Bayes' Theorem.
Independence (eq. [5]) relates to Independence of Random Variables and the Law of Large Numbers.

🔬 Where this appears

⏳ History

- 1846 — Auguste Bravais studies the bivariate normal and "correlated errors," anticipating the correlation formula in an astronomical context.

- 1877 — Francis Galton introduces "reversion" (later "regression") while studying sweet-pea seed sizes.

- 1888 — Galton publishes Co-relations and their Measurement, defining the correlation index

r

geometrically from scatter diagrams.

- 1892 — Edgeworth develops the mathematics of the multivariate normal correlation.

- 1896 — Karl Pearson gives the rigorous "product-moment" definition of

r

(eq. [9]), establishing the modern formula.

- 1904 — Charles Spearman introduces rank correlation, a nonparametric alternative.

- 1920 — Pearson's "Notes on the History of Correlation" surveys the field's origins.

- 1936 — Hotelling generalizes to canonical correlation between two sets of variables.

- 1950s — Markowitz embeds the covariance matrix at the core of modern portfolio theory, winning a 1990 Nobel Prize.

open-problem

Beyond linear correlation. Pearson's $\rho$ is fundamentally a linear and second-moment measure. Capturing general dependence rigorously remains active:

Existence. Correlation requires finite variances; heavy-tailed distributions (Cauchy, some financial returns) break it entirely.
Nonlinear dependence measures. Distance correlation (Székely, 2007) equals zero iff variables are independent — a genuine improvement over Pearson's. Mutual information and maximal information coefficient (MIC) attempt to capture arbitrary relationships, but each has estimation and interpretability trade-offs.
Copulas. Sklar's theorem separates marginals from the dependence structure; the "right" scalar summary of a copula is non-unique (Kendall's $\tau$ , Spearman's $\rho$ , tail-dependence coefficients all coexist).
High dimensions. Estimating large covariance matrices from limited samples is ill-conditioned; shrinkage and sparse estimation (graphical lasso) are ongoing research areas.
Causal inference. Recovering causal structure from observational correlations (the do-calculus of Pearl) is a deep, still-evolving program.

Joint Distributions, Covariance, and Correlation

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also