Expectation, Variance, and Moment Generating Functions

⚡ Key Takeaways

Expectation, Variance, and Moment Generating Functions are the three foundational tools for summarizing the location, spread, and full distributional shape of a random variable. The expectation $E[X]$ is the probability-weighted average (the center of mass), the variance $\operatorname{Var}(X)$ measures squared dispersion about that center, and the moment generating function $M_X(t)=E[e^{tX}]$ encodes ALL moments at once and uniquely determines the distribution when it exists. They apply across all of probability, statistics, physics, finance, and engineering. Core ideas trace to Huygens (1657), Chebyshev, and later formalization by Kolmogorov (1933).

💡 The Core Insight

A distribution can be probed at increasing depth: the expectation pins its center, the variance pins its scale, higher moments pin skew and tail weight, and the moment generating function assembles them into a master object. Operations that are hard on distributions — adding independent variables, scaling, shifting — become trivial on these summaries. This is the same philosophy as a Fourier transform: replace a complicated function (the density) by a transform (the MGF) where the hard operation (convolution) becomes a simple one (multiplication).

DefinitionExpectation

For a discrete random variable $X$ with probability mass function $p(x)=P(X=x)$ over support $\mathcal{S}$ , the expectation is

1

\quad E[X]=\sum_{x\in\mathcal{S}} x\,p(x),

Derivation hint: weight each attainable value by its probability and sum; this is the definition of a probability-weighted average (center of mass). For a continuous random variable with probability density function $f(x)$ ,

2

\quad E[X]=\int_{-\infty}^{\infty} x\,f(x)\,dx,

Derivation hint: replace the sum over discrete masses by an integral over the density; $f(x)\,dx$ is the infinitesimal probability of the interval $[x,x+dx]$ . Here $x$ ranges over outcomes, $p(x)$ and $f(x)$ are nonnegative and integrate/sum to 1. The expectation exists when $\sum |x|p(x)<\infty$ or $\int|x|f(x)\,dx<\infty$ (absolute convergence).

TheoremLaw of the Unconscious Statistician (LOTUS)

For any (measurable) function $g$ ,

3

\quad E[g(X)]=\sum_{x} g(x)\,p(x)\quad\text{or}\quad E[g(X)]=\int_{-\infty}^{\infty} g(x)\,f(x)\,dx.

Derivation hint: one need NOT find the distribution of $Y=g(X)$ first; weight $g(x)$ by the probability of $x$ directly, justified by pushing forward the measure of $X$ .

DefinitionVariance and Standard Deviation

The variance is the expected squared deviation from the mean $\mu=E[X]$ :

4

\quad \operatorname{Var}(X)=E\big[(X-\mu)^2\big]=E[X^2]-\big(E[X]\big)^2.

Derivation hint: expand $(X-\mu)^2 = X^2 - 2\mu X + \mu^2$ , take expectation termwise using linearity, and substitute $E[X]=\mu$ . The standard deviation $\sigma=\sqrt{\operatorname{Var}(X)}$ restores the original units. Variance has units of $X$ squared; standard deviation has the units of $X$ .

DefinitionMoments and Central Moments

The $n$ -th raw moment is $\mu_n'=E[X^n]$ ; the $n$ -th central moment is $\mu_n=E[(X-\mu)^n]$ . Thus $\mu_1'=E[X]$ and $\mu_2=\operatorname{Var}(X)$ . The skewness and kurtosis are

5

\quad \gamma_1=\frac{\mu_3}{\sigma^3},\qquad \gamma_2=\frac{\mu_4}{\sigma^4}-3.

Derivation hint: normalize the third and fourth central moments by powers of $\sigma$ to obtain scale-free shape descriptors; subtracting 3 makes the normal distribution's excess kurtosis zero.

DefinitionMoment Generating Function

The moment generating function of $X$ is

6

\quad M_X(t)=E\!\left[e^{tX}\right]=\begin{cases}\sum_x e^{tx}p(x)&\text{(discrete)}\\[4pt]\displaystyle\int_{-\infty}^{\infty} e^{tx}f(x)\,dx&\text{(continuous)}\end{cases}

Derivation hint: apply LOTUS with $g(x)=e^{tx}$ ; the MGF exists only for those $t$ in an open interval about 0 where the sum/integral converges. When $M_X$ exists in a neighborhood of $t=0$ , moments are recovered by differentiation:

7

\quad E[X^n]=M_X^{(n)}(0)=\left.\frac{d^n}{dt^n}M_X(t)\right|_{t=0}.

Derivation hint: expand $e^{tX}=\sum_{n\ge0} \tfrac{t^n}{n!}X^n$ , take expectation termwise (justified by absolute convergence near 0), so $M_X(t)=\sum_n \tfrac{t^n}{n!}E[X^n]$ ; the coefficient of $t^n/n!$ is the $n$ -th moment.

TheoremKey Properties

For constants $a,b$ and independent variables $X,Y$ :

8

\quad E[aX+b]=aE[X]+b,\qquad \operatorname{Var}(aX+b)=a^2\operatorname{Var}(X),

Derivation hint: linearity of expectation gives the first; for variance, the additive constant $b$ shifts the mean but not the spread, and the factor $a$ pulls out as $a^2$ because variance is quadratic.

9

\quad M_{aX+b}(t)=e^{bt}M_X(at),\qquad M_{X+Y}(t)=M_X(t)\,M_Y(t)\ \text{ if } X\perp Y.

Derivation hint: $E[e^{t(aX+b)}]=e^{bt}E[e^{(at)X}]$ ; for the sum, independence factorizes $E[e^{tX}e^{tY}]=E[e^{tX}]E[e^{tY}]$ .

10

\quad \operatorname{Var}(X+Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y),

Derivation hint: expand $E[((X+Y)-\mu_X-\mu_Y)^2]$ and collect the cross term, which defines covariance; if independent, the covariance vanishes.

ProofVariance Shortcut Formula

We prove $\operatorname{Var}(X)=E[X^2]-(E[X])^2$ from equation [4]. 1. Start from the definition $\operatorname{Var}(X)=E[(X-\mu)^2]$ with $\mu=E[X]$ . 2. Expand the square: $(X-\mu)^2 = X^2 - 2\mu X + \mu^2$ . 3. Apply linearity of expectation: $E[(X-\mu)^2]=E[X^2]-2\mu\,E[X]+\mu^2$ . 4. Substitute $E[X]=\mu$ : the middle term becomes $-2\mu\cdot\mu=-2\mu^2$ . 5. Combine: $E[X^2]-2\mu^2+\mu^2 = E[X^2]-\mu^2 = E[X^2]-(E[X])^2$ . $\blacksquare$

∎

ProofMGF Generates Moments

We justify equation [7]. 1. Write the power series $e^{tX}=\sum_{n=0}^{\infty}\frac{t^n X^n}{n!}$ , valid for all real arguments. 2. Provided $M_X(t)$ converges in an interval $(-h,h)$ , we may interchange expectation and summation (dominated convergence), giving $M_X(t)=\sum_{n=0}^\infty \frac{t^n}{n!}E[X^n]$ . 3. This is a Taylor series in $t$ ; by uniqueness of power-series coefficients, $E[X^n]$ is the coefficient of $t^n/n!$ . 4. Differentiating $n$ times and setting $t=0$ kills all lower terms and the factorials of higher terms vanish at $t=0$ , isolating $E[X^n]=M_X^{(n)}(0)$ . $\blacksquare$

∎

Figure

Mean (center of mass),

\pm\sigma

spread, and the role of the MGF as a transform

mathematician · Pafnuty Chebyshev | 1821–1894

The Russian mathematician Pafnuty Chebyshev placed expectation and variance at the heart of rigorous probability. His Chebyshev inequality, $P(|X-\mu|\ge k\sigma)\le 1/k^2$ , derived purely from the existence of a finite variance, gave the first distribution-free tail bound and provided the engine for his proof of the law of large numbers. He trained a school (Markov, Lyapunov) that turned moments into the principal tool of limit theory, paving the road to the central limit theorem via the method of moments.

🧠 Intuition

Think of a probability distribution as a thin metal rod whose mass is spread along the number line according to $f(x)$ . The expectation is the balance point — the center of mass where the rod would balance on a knife edge. The variance is the rod's moment of inertia about that balance point: it measures how the mass is distributed away from the center, with mass far out penalized quadratically. Standard deviation rescales this back to ordinary distance units, answering "how far, typically, are we from the center?" The MGF is best understood as a generating gadget. Imagine you want to know all the moments at once. Rather than computing infinitely many integrals, you compute one function $M_X(t)$ , and then each moment falls out by differentiation, like extracting coefficients from a polynomial. The exponential $e^{tX}$ acts as a "moment vacuum cleaner": its Taylor expansion $1+tX+\tfrac{t^2}{2}X^2+\cdots$ sucks in every power of $X$ simultaneously, and the parameter $t$ tags each moment with its own power so they can be separated later.

SM

Statistical Mechanics Viewpoint

A physicist immediately recognizes $M_X(t)=E[e^{tX}]$ as a partition-function-like object. In statistical mechanics, $Z(\beta)=\sum_i e^{-\beta E_i}$ generates the energy moments: $\langle E\rangle = -\partial_\beta \ln Z$ and $\operatorname{Var}(E)=\partial_\beta^2 \ln Z$ . The same algebra appears here through the cumulant generating function $K_X(t)=\ln M_X(t)$ , whose derivatives at zero yield cumulants: $K'(0)=\mu$ and $K''(0)=\sigma^2$ . That energy fluctuations relate to heat capacity is precisely the statement that the second cumulant is the variance — thermodynamics and probability share one generating-function skeleton.

ExampleExponential Distribution: Mean, Variance, MGF

Given: $X\sim\text{Exponential}(\lambda)$ with density $f(x)=\lambda e^{-\lambda x}$ for $x\ge0$ , $\lambda=2\ \text{s}^{-1}$ (e.g., inter-arrival time of customers).

Step 1 — MGF. For $t<\lambda$ ,

M_X(t)=\int_0^\infty e^{tx}\lambda e^{-\lambda x}\,dx=\lambda\int_0^\infty e^{-(\lambda-t)x}\,dx=\frac{\lambda}{\lambda-t}.

Step 2 — Mean via $M'(0)$ . $M'(t)=\dfrac{\lambda}{(\lambda-t)^2}$ , so $E[X]=M'(0)=\dfrac{1}{\lambda}$ .

Step 3 — Second moment via $M''(0)$ . $M''(t)=\dfrac{2\lambda}{(\lambda-t)^3}$ , so $E[X^2]=M''(0)=\dfrac{2}{\lambda^2}$ .

Step 4 — Variance. $\operatorname{Var}(X)=\dfrac{2}{\lambda^2}-\dfrac{1}{\lambda^2}=\dfrac{1}{\lambda^2}$ .

Numbers ( $\lambda=2$ ): $E[X]=0.5\ \text{s}$ , $\operatorname{Var}(X)=0.25\ \text{s}^2$ , $\sigma=0.5\ \text{s}$ .

\boxed{E[X]=\tfrac{1}{\lambda}=0.5\ \text{s},\quad \operatorname{Var}(X)=\tfrac{1}{\lambda^2}=0.25\ \text{s}^2}

ExampleSum of Two Independent Dice: Linearity and Variance Addition

Given: Two fair six-sided dice $X_1,X_2$ , independent. For one die: $E[X_i]=\tfrac{1+2+\cdots+6}{6}=3.5$ .

Step 1 — Single-die variance. $E[X_i^2]=\tfrac{1+4+9+16+25+36}{6}=\tfrac{91}{6}\approx15.167$ . Then $\operatorname{Var}(X_i)=15.167-3.5^2=15.167-12.25=2.9167$ .

Step 2 — Sum's mean (linearity, no independence needed). $E[X_1+X_2]=3.5+3.5=7$ .

Step 3 — Sum's variance (independence ⇒ covariance 0). $\operatorname{Var}(X_1+X_2)=2.9167+2.9167=5.8333$ .

Step 4 — Standard deviation. $\sigma=\sqrt{5.8333}\approx 2.415$ .

\boxed{E[X_1+X_2]=7,\quad \operatorname{Var}(X_1+X_2)=\tfrac{35}{6}\approx5.833,\quad \sigma\approx2.42}

Note that variances added but standard deviations did not ( $2.415\ne 1.708+1.708$ ).

⚠️ Common mistake

Variance does NOT add for the standard deviation, and only adds for variance when variables are uncorrelated. A frequent error is writing $\sigma_{X+Y}=\sigma_X+\sigma_Y$ . Standard deviations add in quadrature: $\sigma_{X+Y}=\sqrt{\sigma_X^2+\sigma_Y^2}$ only when independent. A second trap: $E[g(X)]\ne g(E[X])$ in general — for nonlinear $g$ , Jensen's inequality gives $E[g(X)]\ge g(E[X])$ when $g$ is convex (e.g., $E[X^2]\ge (E[X])^2$ , which is exactly why variance is nonnegative). Third: the MGF need not exist; if the defining integral diverges for all $t\ne0$ , no moments can be extracted this way.

CounterexampleA Distribution with No MGF and Even No Mean

The Cauchy distribution with density $f(x)=\dfrac{1}{\pi(1+x^2)}$ has $\int_{-\infty}^\infty |x|f(x)\,dx=\infty$ , so $E[X]$ does not exist, the variance is undefined, and $M_X(t)=E[e^{tX}]$ diverges for every $t\ne0$ . Yet the Cauchy distribution is perfectly well-defined, symmetric, and bell-shaped — a vivid reminder that "looks like a bump" does not guarantee finite moments. Its characteristic function $\varphi_X(t)=E[e^{itX}]=e^{-|t|}$ exists, which is why characteristic functions (always defined) supersede MGFs in rigorous theory.

connection

Expectation and variance are the first two members of an infinite hierarchy unified by the MGF, which is the real analogue of the Characteristic Function (use $it$ instead of $t$ ). The logarithm of the MGF gives the Cumulant Generating Function, whose derivatives produce cumulants that add under independence. Variance addition under independence is the seed of the Law of Large Numbers and the Central Limit Theorem, both of which Chebyshev's variance bound and the method of moments help prove. Covariance generalizes variance to the joint setting and underlies Linear Regression and the Multivariate Normal Distribution. The MGF's multiplicative property mirrors the Convolution Theorem — sums of independent variables convolve densities and multiply transforms.

🔬 Where this appears

⏳ History

- 1654 — Pascal and Fermat correspond on the "problem of points," implicitly computing expected values of gambling outcomes.

- 1657 — Christiaan Huygens publishes De Ratiociniis in Ludo Aleae, the first printed treatise giving expectation ("expectatio") a formal definition.

- 1713 — Jacob Bernoulli's Ars Conjectandi proves the (weak) law of large numbers, tying long-run averages to expectation.

- 1733–1812 — De Moivre and Laplace develop the normal approximation, implicitly using mean and variance of binomial sums; Laplace introduces generating functions.

- 1867 — Chebyshev publishes his inequality, giving the first variance-based tail bound and a clean proof of the law of large numbers.

- 1900s — Markov and Lyapunov refine the method of moments, using moment convergence to prove central limit theorems.

- 1920s — R. A. Fisher systematizes moments, cumulants, and the term "variance" (coined 1918) in mathematical statistics.

- 1933 — Kolmogorov's Grundbegriffe axiomatizes probability via measure theory, defining expectation as a Lebesgue integral and securing all earlier results.

- 1937 — Cramér and others advance large-deviation theory using MGFs and Chernoff-type exponential bounds.

open-problem

The MGF is powerful but fragile: it may fail to exist (heavy-tailed laws like Cauchy, log-normal, and Student- $t$ ), so moment-based identification breaks down. Even when all moments are finite, they need not determine the distribution — the moment problem (Stieltjes, Hamburger) shows the log-normal distribution shares its entire moment sequence with infinitely many distinct distributions, so equal moments do not imply equal laws. The characteristic function repairs existence (it is always defined and uniquely determines the distribution), but inverting it constructively, and characterizing precisely which moment sequences correspond to a unique distribution (the determinacy conditions of Carleman and Krein), remain technically delicate. Generalizing moments and concentration bounds to high-dimensional and dependent data is an active research frontier.

Expectation, Variance, and Moment Generating Functions

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also