Discrete Distributions

⚡ Key Takeaways

A discrete distribution assigns probabilities to a countable set of outcomes via a probability mass function (PMF). Key result: the probabilities are nonnegative and sum to 1, and all moments are computed by summation rather than integration. Where it applies: counting problems, success/failure trials, queueing, genetics, reliability, and any phenomenon with separate, enumerable outcomes. Who/when: developed by Bernoulli, Poisson, Pascal, and Fermat (17th–19th centuries), formalized by Kolmogorov (1933).

💡 The Core Insight

A discrete distribution is completely specified by a list of (value, probability) pairs that are nonnegative and total 1. Every probabilistic question reduces to summing the right subset of these probabilities. Continuous distributions integrate; discrete distributions sum — but the conceptual structure (sample space, measure, expectation) is identical.

DefinitionDiscrete Random Variable and Probability Mass Function

Let $(\Omega, \mathcal{F}, P)$ be a probability space. A random variable $X: \Omega \to \mathbb{R}$ is discrete if its image $S = \{x_1, x_2, \dots\}$ (the support) is a countable set. Its probability mass function (PMF) is

1

\quad p_X(x) = P(X = x) = P(\{\omega \in \Omega : X(\omega) = x\}).

Derivation hint: the PMF is the pushforward measure $P_X = P \circ X^{-1}$ evaluated on the singleton $\{x\}$ . The PMF satisfies two axioms:

2

\quad p_X(x) \ge 0 \quad \text{for all } x, \qquad \sum_{x \in S} p_X(x) = 1.

Derivation hint: nonnegativity follows from $P \ge 0$ ; the sum equals $P(\Omega) = 1$ because the events $\{X = x_i\}$ are disjoint and exhaust $\Omega$ up to a null set, then apply countable additivity. The cumulative distribution function (CDF) is a right-continuous step function:

3

\quad F_X(x) = P(X \le x) = \sum_{x_i \le x} p_X(x_i).

Derivation hint: write $\{X \le x\} = \bigcup_{x_i \le x} \{X = x_i\}$ as a disjoint union and apply countable additivity.

DefinitionExpectation, Variance, and Moments

For a discrete random variable $X$ and a function $g$ , the expected value is

4

\quad \mathbb{E}[g(X)] = \sum_{x \in S} g(x)\, p_X(x),

provided $\sum_{x} |g(x)|\, p_X(x) < \infty$ (absolute convergence). In particular the mean is $\mu = \mathbb{E}[X]$ and the variance is

5

\quad \operatorname{Var}(X) = \mathbb{E}[(X-\mu)^2] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2.

Derivation hint: expand $(X-\mu)^2 = X^2 - 2\mu X + \mu^2$ , take expectation termwise using linearity, and use $\mathbb{E}[X]=\mu$ . The probability generating function (PGF), defined for integer-valued $X \ge 0$ , encodes the whole distribution:

6

\quad G_X(s) = \mathbb{E}[s^X] = \sum_{k=0}^{\infty} p_X(k)\, s^k, \qquad |s| \le 1.

Derivation hint: this is the definition of expectation [4] applied to $g(X)=s^X$ ; $G_X(1)=1$ by normalization [2].

TheoremMoments from the PGF

If $X$ is a nonnegative integer-valued random variable with PGF $G_X(s)$ , then

7

\quad \mathbb{E}[X] = G_X'(1), \qquad \mathbb{E}[X(X-1)] = G_X''(1),

and consequently $\operatorname{Var}(X) = G_X''(1) + G_X'(1) - [G_X'(1)]^2$ .

ProofMoments from the PGF

Step 1. Differentiate [6] term by term (valid inside the radius of convergence, which is $\ge 1$ ):

G_X'(s) = \sum_{k=1}^{\infty} k\, p_X(k)\, s^{k-1}.

Step 2. Set $s = 1$ :

G_X'(1) = \sum_{k=1}^{\infty} k\, p_X(k) = \mathbb{E}[X].

Step 3. Differentiate again:

G_X''(s) = \sum_{k=2}^{\infty} k(k-1)\, p_X(k)\, s^{k-2}, \qquad G_X''(1) = \mathbb{E}[X(X-1)].

Step 4. Combine with the variance identity [5]:

\operatorname{Var}(X) = \mathbb{E}[X^2] - \mu^2 = \big(\mathbb{E}[X(X-1)] + \mathbb{E}[X]\big) - \mu^2 = G_X''(1) + G_X'(1) - [G_X'(1)]^2. \qquad \blacksquare

∎

SD

Siméon Denis Poisson

1781–1840

Poisson introduced the distribution bearing his name in his 1837 treatise Recherches sur la probabilité des jugements en matière criminelle et en matière civile as a limiting case of the binomial when the number of trials grows large and the success probability shrinks. He was studying the rate of wrongful convictions in French courts. Ironically, the distribution gained fame much later through Ladislaus Bortkiewicz's 1898 analysis of Prussian soldiers killed by horse kicks — a textbook example of rare-event counting that vindicated Poisson's mathematics decades after his death.

🧠 Intuition

Picture a histogram made of vertical spikes at the support points, each spike's height equal to its probability. The constraint that "all spike heights sum to 1" is just conservation of probability: the random outcome must land somewhere. The mean is the balance point (center of mass) of these spikes treated as point masses; the variance measures how far they spread from that balance point. The key mental shift from continuous distributions: there is no probability density and no smooth curve. $P(X = x)$ is a genuine, often nonzero, probability — unlike the continuous case where $P(X = x) = 0$ always. This is why discrete distributions have CDFs that jump by exactly $p_X(x)$ at each support point, like climbing a staircase. A useful slogan: the binomial is "n fixed trials," the geometric is "trials until first success," and the Poisson is "events in continuous time." All three are discrete because you are counting integer-valued things.

Figure

PMF spikes and the corresponding step CDF for a Poisson(λ=2) distribution

ExampleBinomial — Defective Components

Given: A factory produces components, each independently defective with probability $p = 0.05$ . A box contains $n = 20$ components. Let $X$ = number of defectives. $X \sim \text{Binomial}(20, 0.05)$ with PMF

p_X(k) = \binom{20}{k}(0.05)^k(0.95)^{20-k}.

Find: (a) $P(X = 0)$ , (b) $P(X \ge 2)$ , (c) the mean and standard deviation.

Step 1 — No defectives:

P(X=0) = \binom{20}{0}(0.05)^0(0.95)^{20} = (0.95)^{20} = 0.3585.

Step 2 — Exactly one defective:

P(X=1) = \binom{20}{1}(0.05)(0.95)^{19} = 20 \cdot 0.05 \cdot 0.3774 = 0.3774.

Step 3 — At least two (use complement):

P(X \ge 2) = 1 - P(X=0) - P(X=1) = 1 - 0.3585 - 0.3774 = 0.2641.

Step 4 — Moments: for Binomial, $\mu = np = 20(0.05) = 1$ defective; $\operatorname{Var}(X) = np(1-p) = 1 \cdot 0.95 = 0.95$ , so $\sigma = \sqrt{0.95} = 0.975$ .

\boxed{P(X=0)=0.3585,\quad P(X\ge 2)=0.2641,\quad \mu = 1.0 \text{ defect},\ \sigma = 0.975 \text{ defects}}

ExamplePoisson — Call Center Arrivals

Given: A call center receives calls at an average rate of $\lambda = 3$ calls per minute, modeled as $X \sim \text{Poisson}(3)$ with PMF

p_X(k) = \frac{e^{-\lambda}\lambda^k}{k!} = \frac{e^{-3}\,3^k}{k!}.

Find: (a) $P(\text{exactly 2 calls})$ , (b) $P(\text{at least 1 call})$ , (c) verify mean and variance.

Step 1 — Exactly two calls:

P(X=2) = \frac{e^{-3}\,3^2}{2!} = \frac{9 e^{-3}}{2} = \frac{9 (0.049787)}{2} = 0.2240.

Step 2 — At least one (complement):

P(X \ge 1) = 1 - P(X=0) = 1 - e^{-3} = 1 - 0.049787 = 0.9502.

Step 3 — Mean and variance. For Poisson, the PGF is $G_X(s) = e^{\lambda(s-1)}$ . Then $G_X'(s) = \lambda e^{\lambda(s-1)}$ , so $\mathbb{E}[X] = G_X'(1) = \lambda = 3$ . Also $G_X''(1) = \lambda^2$ , giving $\operatorname{Var}(X) = \lambda^2 + \lambda - \lambda^2 = \lambda = 3$ . (Mean = variance is the Poisson signature.)

\boxed{P(X=2)=0.2240,\quad P(X\ge 1)=0.9502,\quad \mu = \sigma^2 = 3 \text{ calls/min}}

⚠️ PMF values can exceed... no — but densities can

A common error is confusing the PMF with a probability density. For a discrete distribution, $p_X(x)$ is itself a probability, so $0 \le p_X(x) \le 1$ always. This differs from continuous densities $f(x)$ , which can be larger than 1. Conversely, never write $P(X = x) = 0$ for a discrete variable at a support point — that is a continuous-distribution fact and is false here.

⚠️ Convergence of the mean

The expectation $\mathbb{E}[X] = \sum x\, p_X(x)$ requires absolute convergence. If the series only converges conditionally or diverges, the mean is undefined or infinite. Plugging values into a formula without checking can give nonsense. Always verify $\sum |x|\, p_X(x) < \infty$ for infinite supports.

CounterexampleA discrete distribution with infinite mean — St. Petersburg

Define $X$ on the support $\{2, 4, 8, 16, \dots\} = \{2^k : k \ge 1\}$ with $p_X(2^k) = 2^{-k}$ . This is a valid PMF: $\sum_{k=1}^\infty 2^{-k} = 1$ . But the mean is

\mathbb{E}[X] = \sum_{k=1}^\infty 2^k \cdot 2^{-k} = \sum_{k=1}^\infty 1 = \infty.

Every outcome is finite, the probabilities are perfectly well-defined, yet the expected value diverges. This refutes the naïve belief that "discrete = tame" and underlies the famous St. Petersburg paradox where a "fair" game has infinite expected payoff.

connection

Discrete distributions are the countable-support special case of the general theory in Probability Theory, built on Measure Theory via pushforward measures. They contrast with Continuous Distributions, where integration over densities replaces summation over PMFs — and the two unify under Lebesgue–Stieltjes Integration applied to the CDF. The Central Limit Theorem connects them dynamically: a sum of many independent discrete variables (e.g., binomial) converges to the continuous Normal Distribution. The Poisson Distribution arises both as the rare-event limit of the Binomial Distribution (law of small numbers) and from the Poisson Process in continuous time. Generating-function techniques tie this topic to Combinatorics and the algebra of formal power series.

🔬 Where this appears

⏳ History

- 1654 — Blaise Pascal and Pierre de Fermat exchange letters on the "problem of points," founding the systematic calculation of discrete probabilities for games of chance.

- 1657 — Christiaan Huygens publishes De ratiociniis in ludo aleae, the first printed probability text, introducing expectation.

- 1713 — Jacob Bernoulli's posthumous Ars Conjectandi presents the binomial distribution and proves the (weak) law of large numbers.

- 1718 — Abraham de Moivre's The Doctrine of Chances develops the normal approximation to the binomial — an early central limit result.

- 1837 — Siméon Denis Poisson derives the Poisson distribution as a binomial limit in his study of judicial decisions.

- 1898 — Ladislaus Bortkiewicz publishes Das Gesetz der kleinen Zahlen ("Law of Small Numbers"), popularizing the Poisson distribution with the Prussian horse-kick data.

- 1909 — Agner Krarup Erlang applies Poisson and related distributions to telephone-traffic engineering, founding queueing theory.

- 1920 — George Udny Yule and others develop the negative binomial and overdispersion modeling in biology.

- 1933 — Andrey Kolmogorov's Grundbegriffe der Wahrscheinlichkeitsrechnung axiomatizes probability measure-theoretically, placing discrete distributions on rigorous countable-additivity foundations.

open-problem

Several deep questions surround discrete distributions at and beyond the undergraduate boundary. Mixed and singular distributions: a random variable can be partly discrete and partly continuous (e.g., insurance payouts with a point mass at zero plus a continuous tail), requiring the general Lebesgue decomposition rather than a pure PMF. Heavy tails: distributions like the zeta (discrete power-law) distribution can have infinite variance or even infinite mean, complicating estimation and breaking the standard CLT; characterizing convergence for such cases (to stable laws) is subtle. Maximum-entropy characterization: which discrete distribution maximizes entropy under given moment constraints is well understood (geometric under fixed mean, etc.), but the interplay with model selection in machine learning remains an active applied area. Infinite divisibility: classifying which discrete distributions can arise as the distribution of a sum of $n$ i.i.d. parts for every $n$ (the Poisson and negative binomial qualify; the binomial does not) connects to Lévy processes and is a rich research thread.

Formal Statement

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also