Characteristic Functions and Moment Generating Functions

⚡ Key Takeaways

Characteristic functions and moment generating functions are integral transforms of a probability distribution that encode all its information in a single function. The characteristic function $\varphi_X(t) = E[e^{itX}]$ always exists; the moment generating function $M_X(t) = E[e^{tX}]$ may not. They linearize convolution (sums of independent variables become products), generate moments by differentiation, and uniquely determine the distribution. Used everywhere in probability theory, statistics, and the proof of the Central Limit Theorem; developed by Lyapunov, Lévy, and others in the late 19th–early 20th century.

💡 The Core Insight

A probability distribution is fully captured not only by its density or CDF but by any one of several transforms. By replacing the variable $X$ with $e^{itX}$ or $e^{tX}$ and taking expectations, we convert convolution into multiplication and moments into derivatives. This is exactly why these transforms are the natural tool for studying sums of independent random variables — the engine behind limit theorems.

DefinitionMoment Generating Function

For a random variable $X$ , the moment generating function is

1

\quad M_X(t) = E\!\left[e^{tX}\right] = \begin{cases} \displaystyle\sum_x e^{tx}\,p_X(x) & \text{(discrete)} \\[2mm] \displaystyle\int_{-\infty}^{\infty} e^{tx}\,f_X(x)\,dx & \text{(continuous)} \end{cases}

Derivation hint: Apply the Law of the Unconscious Statistician (LOTUS) to the function $g(x)=e^{tx}$ , summing or integrating against the distribution. Here $t \in \mathbb{R}$ , $p_X$ is the probability mass function, and $f_X$ is the probability density function. The MGF is defined only for those $t$ where the sum/integral converges; we require convergence on an open interval containing $0$ .

DefinitionCharacteristic Function

For any random variable $X$ , the characteristic function is

2

\quad \varphi_X(t) = E\!\left[e^{itX}\right] = E[\cos tX] + i\,E[\sin tX], \qquad t \in \mathbb{R}.

Derivation hint: Use Euler's formula $e^{i\theta}=\cos\theta+i\sin\theta$ and split the expectation into real and imaginary parts. Here $i^2 = -1$ . Because $|e^{itX}| = 1$ , the integrand is bounded, so $\varphi_X(t)$ exists and is finite for every $t$ and every distribution.

TheoremMoments from Derivatives

If $M_X$ exists in a neighborhood of $0$ , then $X$ has finite moments of all orders and

3

\quad E[X^n] = M_X^{(n)}(0) = \left.\frac{d^n}{dt^n}M_X(t)\right|_{t=0}.

Derivation hint: Differentiate $e^{tX}$ under the expectation $n$ times to bring down a factor $X^n$ , then set $t=0$ so $e^{tX}\to 1$ . Equivalently, the Taylor series $M_X(t)=\sum_{n=0}^\infty \frac{E[X^n]}{n!}t^n$ holds. For the characteristic function, when $E[|X|^n]<\infty$ ,

4

\quad E[X^n] = \frac{1}{i^n}\,\varphi_X^{(n)}(0) = (-i)^n\,\varphi_X^{(n)}(0).

Derivation hint: Differentiate $e^{itX}$ to pull down $iX$ each time; after $n$ derivatives the prefactor is $i^n$ , so divide by it.

TheoremUniqueness and Convolution

Uniqueness. The characteristic function determines the distribution uniquely: if $\varphi_X = \varphi_Y$ then $X$ and $Y$ have the same distribution. The same holds for the MGF when it exists on an open interval around $0$ .

Convolution / Independence. If $X$ and $Y$ are independent, then for the sum $S=X+Y$ ,

5

\quad \varphi_{X+Y}(t) = \varphi_X(t)\,\varphi_Y(t), \qquad M_{X+Y}(t) = M_X(t)\,M_Y(t).

Derivation hint: $E[e^{it(X+Y)}]=E[e^{itX}e^{itY}]$ ; independence factors the expectation of a product into a product of expectations.

Linear transformation. For constants $a,b$ ,

6

\quad \varphi_{aX+b}(t) = e^{itb}\,\varphi_X(at), \qquad M_{aX+b}(t) = e^{tb}\,M_X(at).

Derivation hint: $E[e^{it(aX+b)}]=e^{itb}E[e^{i(at)X}]$ ; pull the constant exponent out and rescale the argument.

ProofMoments by Differentiation

We justify equation [3], assuming $M_X(t)$ converges in $(-\delta,\delta)$ for some $\delta>0$ .

Step 1. Write the definition $M_X(t)=E[e^{tX}]$ .

Step 2. Expand the exponential in its power series: $e^{tX}=\sum_{n=0}^\infty \frac{(tX)^n}{n!}$ .

Step 3. Convergence of $M_X$ on an open interval permits interchanging expectation and summation (dominated convergence / Fubini):

M_X(t)=\sum_{n=0}^\infty \frac{E[X^n]}{n!}\,t^n.

Step 4. This is a power series in $t$ . A power series equals the Taylor series of the function it represents, so the coefficient of $t^n$ is $M_X^{(n)}(0)/n!$ .

Step 5. Matching coefficients: $\dfrac{E[X^n]}{n!}=\dfrac{M_X^{(n)}(0)}{n!}$ , hence $E[X^n]=M_X^{(n)}(0)$ . $\blacksquare$

∎

ProofConvolution Theorem for Sums

We justify equation [5] for independent $X,Y$ .

Step 1. By definition, $\varphi_{X+Y}(t)=E\!\left[e^{it(X+Y)}\right]$ .

Step 2. Algebra of exponents: $e^{it(X+Y)}=e^{itX}\cdot e^{itY}$ .

Step 3. Since $X$ and $Y$ are independent, the functions $e^{itX}$ and $e^{itY}$ are independent, so the expectation of the product factors:

E\!\left[e^{itX}e^{itY}\right]=E\!\left[e^{itX}\right]\,E\!\left[e^{itY}\right].

Step 4. Recognize each factor: $=\varphi_X(t)\,\varphi_Y(t)$ . The identical argument with $t$ in place of $it$ proves the MGF version. $\blacksquare$

∎

mathematician · Paul Lévy | 1886–1971

Paul Lévy was the central architect of the modern theory of characteristic functions. Building on Lyapunov's use of them in proving the Central Limit Theorem, Lévy established the continuity theorem (pointwise convergence of CFs corresponds to convergence in distribution), developed the inversion formula in full generality, and classified the infinitely divisible and stable distributions through their characteristic functions. His 1925 Calcul des probabilités made the characteristic function the standard tool of rigorous probability. Lévy also pioneered stochastic processes (Lévy processes, Brownian motion), and his name is attached to a vast range of probabilistic objects.

🧠 Intuition

Think of a probability distribution as a "signal" — a shape spread over the real line. The characteristic function is its Fourier transform: it re-expresses the same information in terms of frequencies $t$ rather than positions $x$ . Two facts then become intuitive. First, why sums become products. Adding independent random variables means convolving their densities (smearing one shape over the other). The Fourier transform turns convolution into ordinary multiplication — so in the transform domain, addition of variables is just multiplication of functions. This is the same principle that makes Fourier analysis so useful in signal processing. Second, why derivatives give moments. Near $t=0$ , $e^{itX}\approx 1 + itX - \tfrac{1}{2}t^2X^2+\cdots$ . Taking expectations, the coefficients are precisely $1, iE[X], -\tfrac12 E[X^2],\dots$ The local behavior of the transform at the origin encodes the moments — small $t$ probes large-scale features (mean, variance) of the distribution. The MGF does the same thing but along the real axis, which is why it can diverge: $e^{tX}$ grows without bound, so heavy-tailed distributions overwhelm it. The CF's $e^{itX}$ merely rotates on the unit circle, never growing, so it is always tame.

Figure

The two transforms: MGF integrates against a growing exponential (may diverge), CF against a bounded oscillation (always exists)

```svg

```

ExampleExponential Distribution: MGF and Moments

Given: $X\sim\text{Exp}(\lambda)$ with density $f_X(x)=\lambda e^{-\lambda x}$ for $x\ge 0$ , $\lambda=2$ .

Step 1 — Set up the MGF.

M_X(t)=\int_0^\infty e^{tx}\,\lambda e^{-\lambda x}\,dx=\lambda\int_0^\infty e^{-(\lambda-t)x}\,dx.

Step 2 — Evaluate (converges only for $t<\lambda$ ).

M_X(t)=\lambda\cdot\frac{1}{\lambda-t}=\frac{\lambda}{\lambda-t},\qquad t<\lambda.

Step 3 — First derivative for the mean.

M_X'(t)=\frac{\lambda}{(\lambda-t)^2},\qquad E[X]=M_X'(0)=\frac{\lambda}{\lambda^2}=\frac{1}{\lambda}.

Step 4 — Second derivative for $E[X^2]$ .

M_X''(t)=\frac{2\lambda}{(\lambda-t)^3},\qquad E[X^2]=M_X''(0)=\frac{2\lambda}{\lambda^3}=\frac{2}{\lambda^2}.

Step 5 — Variance and numbers with $\lambda=2$ :

E[X]=\tfrac12=0.5,\quad E[X^2]=\tfrac{2}{4}=0.5,\quad \mathrm{Var}(X)=0.5-0.25=0.25.

\boxed{M_X(t)=\frac{2}{2-t}\ (t<2),\quad E[X]=0.5,\quad \mathrm{Var}(X)=0.25}

ExampleStandard Normal: Characteristic Function and Sum of Normals

Given: $Z\sim N(0,1)$ with density $f(z)=\frac{1}{\sqrt{2\pi}}e^{-z^2/2}$ .

Step 1 — Set up the CF.

\varphi_Z(t)=\int_{-\infty}^{\infty}e^{itz}\,\frac{1}{\sqrt{2\pi}}e^{-z^2/2}\,dz.

Step 2 — Complete the square in the exponent.

itz-\tfrac{z^2}{2}=-\tfrac12\big(z^2-2itz\big)=-\tfrac12\big(z-it\big)^2-\tfrac{t^2}{2}.

Step 3 — Integrate (the shifted Gaussian integrates to $\sqrt{2\pi}$ ):

\varphi_Z(t)=e^{-t^2/2}\cdot\frac{1}{\sqrt{2\pi}}\int_{-\infty}^\infty e^{-(z-it)^2/2}\,dz=e^{-t^2/2}.

Step 4 — Apply to a sum. Let $X_1,\dots,X_n$ be i.i.d. $N(0,1)$ and $S=\sum X_k$ . By the convolution property [5],

\varphi_S(t)=\prod_{k=1}^n e^{-t^2/2}=e^{-nt^2/2}.

Step 5 — Recognize the result. $e^{-nt^2/2}$ is the CF of $N(0,n)$ (a normal with variance $n$ ). For $n=4$ : variance $=4$ , standard deviation $=2$ .

\boxed{\varphi_Z(t)=e^{-t^2/2},\qquad S=\sum_{k=1}^4 Z_k\sim N(0,4),\ \sigma_S=2}

⚠️ Common mistake

The MGF need not exist. Many students assume every distribution has an MGF. It does not. The MGF requires $E[e^{tX}]<\infty$ on an open interval around $0$ , which fails for heavy-tailed distributions. The Cauchy distribution, the log-normal, and any distribution with infinitely many infinite-or-undefined moments have no MGF (or only the degenerate value at $t=0$ ). The characteristic function, by contrast, always exists — this is its primary advantage and the reason rigorous probability theory prefers it. A second pitfall: equation [3] gives moments only if the MGF is finite in a neighborhood of $0$ . Finiteness at a single point $t=0$ (where $M_X(0)=1$ trivially) tells you nothing.

CounterexampleDistributions with Equal Moments but Different Laws

A common belief is that "knowing all the moments determines the distribution." This is false without an analyticity condition. The classic counterexample uses the log-normal distribution. Let $X$ be log-normal. There exist densities of the form

f_a(x)=f_X(x)\big(1+a\sin(2\pi\ln x)\big),\qquad |a|\le 1,

which are genuine probability densities (different for different $a$ ) yet share exactly the same moments of every order as the log-normal. Since they have identical moment sequences but different distributions, the moment sequence does not determine the law. The reason: the log-normal's MGF does not exist (its tails are too heavy), so its moments grow too fast for the Carleman condition $\sum_n (E[X^{2n}])^{-1/2n}=\infty$ to hold. The characteristic functions of these distributions do differ — and the CF, unlike the moment sequence, always determines the law uniquely.

connection

The characteristic function is, up to sign conventions, the Fourier transform of the density: $\varphi_X(t)=\hat f_X(-t)$ . This ties probability directly to harmonic analysis — the inversion formula, Parseval/Plancherel identities, and convolution theorems all carry over. See Fourier Transform. The MGF is the two-sided Laplace transform of the density evaluated along the real axis, linking it to Laplace Transform methods used in differential equations and queueing theory. The cumulant generating function $K_X(t)=\log M_X(t)$ generates cumulants and is additive for independent sums, connecting to Cumulants. The whole apparatus is the proof engine behind the Central Limit Theorem: standardizing a sum and showing its CF converges to $e^{-t^2/2}$ , then invoking Lévy's continuity theorem and Convergence in Distribution. The closely related probability generating function $G_X(z)=E[z^X]$ serves integer-valued variables; see Probability Generating Function.

🔬 Where this appears

⏳ History

- 1810s — Laplace. Pierre-Simon Laplace uses what are effectively generating-function and transform techniques in his probability work, including early versions of the central limit phenomenon.

- 1853 — Cauchy. Augustin-Louis Cauchy studies functions of the form

E[e^{itX}]

and the distribution now bearing his name, an early example of a law with no mean and no MGF.

- 1900–1901 — Lyapunov. Aleksandr Lyapunov gives the first rigorous proof of the Central Limit Theorem under his condition, using characteristic functions as the key tool.

- 1920s — Lévy. Paul Lévy develops the systematic theory: the inversion formula, the continuity theorem, and the role of CFs in classifying stable and infinitely divisible laws.

- 1922 — Pólya. George Pólya proves criteria (Pólya's theorem) giving sufficient conditions for a function to be a characteristic function.

- 1932–1934 — Cramér & Bochner. Harald Cramér advances analytic methods; Bochner's theorem characterizes exactly which functions are characteristic functions (continuous, positive-definite, value 1 at 0).

- 1935 — Lévy–Cramér theorem. If

X+Y

is normal with

X,Y

independent, then both

X

and

Y

are normal — a striking decomposition result proved via CFs.

- 1937 — Khinchin. Aleksandr Khinchin systematizes infinitely divisible distributions, completing the Lévy–Khinchin representation of their characteristic functions.

- Mid–20th century onward. Characteristic-function and transform methods become standard in textbooks (Feller, Loève, Billingsley) and the default rigorous tool in modern probability.

open-problem

The moment problem. Given a sequence $\{m_n\}$ , when is it the moment sequence of some distribution, and is that distribution unique? The Hamburger, Stieltjes, and Hausdorff moment problems address existence on $\mathbb{R}$ , $[0,\infty)$ , and $[0,1]$ respectively. Uniqueness (determinacy) is delicate: the Carleman condition $\sum_n m_{2n}^{-1/2n}=\infty$ is sufficient but not necessary, and there is no simple necessary-and-sufficient closed criterion. Constructing explicit non-uniqueness families (as for the log-normal) remains an active area.

Reconstructing the distribution from a truncated transform. In practice one knows the CF only on a bounded range or with noise. Stable numerical inversion (regularized Fourier inversion, the Gil-Pelaez formula in practice) is genuinely hard for heavy-tailed or oscillatory CFs, and quantifying optimal recovery rates is an ongoing statistical question.

Characterizing characteristic functions. Bochner's theorem describes CFs abstractly (positive-definite functions), but deciding whether a given explicit function is a CF — and reconstructing the law efficiently — can be subtle, with Pólya-type sufficient conditions filling part of the gap.

Characteristic Functions and Moment Generating Functions

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also