Probability Distributions

⚡ Key Takeaways

A probability distribution describes how the total probability (always summing to 1) is spread across all possible outcomes of a random variable. Key result in words: every random quantity has a rule — a table, a function, or a curve — that assigns probabilities to its values, and from that rule we compute means, spreads, and the likelihood of any event. Where it applies: statistics, physics, finance, biology, quality control, machine learning. Who/when: developed from Pascal & Fermat (1654) through Gauss, Poisson, and formalized by Kolmogorov (1933).

💡 The Core Insight

A probability distribution is one whole unit of certainty (probability = 1) divided up among possibilities. For discrete outcomes you split it into chunks; for continuous outcomes you smear it as a density. Knowing how it is split lets you answer every probability question about that variable — and two simple summary numbers, the mean (center) and variance (spread), capture its essential character.

DefinitionRandom Variable and Distribution

A random variable $X$ is a function assigning a real number to each outcome of a random experiment. A probability distribution specifies $P(X = x)$ (discrete) or a density $f(x)$ (continuous) for all values $x$ .

Discrete case — the probability mass function (PMF) $p(x) = P(X = x)$ satisfies:

1

\quad p(x) \ge 0 \quad\text{for all } x, \qquad \sum_{x} p(x) = 1

Derivation hint: probabilities cannot be negative, and exactly one of the mutually exclusive outcomes must occur, so they sum to certainty.

Continuous case — the probability density function (PDF) $f(x)$ satisfies:

2

\quad f(x) \ge 0, \qquad \int_{-\infty}^{\infty} f(x)\,dx = 1, \qquad P(a \le X \le b) = \int_a^b f(x)\,dx

Derivation hint: total area under the density equals total probability 1; probability of an interval is the area above it. Here $x$ ranges over possible values, $p(x)$ is a probability (dimensionless), and $f(x)$ has units of probability per unit of x.

DefinitionCumulative Distribution Function

The cumulative distribution function (CDF) gives the probability of being at or below a value:

3

\quad F(x) = P(X \le x) = \begin{cases} \displaystyle\sum_{t \le x} p(t) & \text{(discrete)} \\[2ex] \displaystyle\int_{-\infty}^{x} f(t)\,dt & \text{(continuous)} \end{cases}

Derivation hint: accumulate all probability from the smallest values up to $x$ ; for the continuous case differentiating gives back $F'(x)=f(x)$ . $F$ is non-decreasing, with $F(-\infty)=0$ and $F(+\infty)=1$ .

DefinitionMean and Variance

The expected value (mean, $\mu$ ) is the probability-weighted average:

4

\quad \mu = E[X] = \sum_{x} x\,p(x) \quad\text{or}\quad \int_{-\infty}^{\infty} x\,f(x)\,dx

Derivation hint: average each value by how likely it is — the "balance point" of the distribution. The variance ( $\sigma^2$ ) measures spread about the mean:

5

\quad \sigma^2 = E[(X-\mu)^2] = \sum_x (x-\mu)^2 p(x) = E[X^2] - \mu^2

Derivation hint: average squared distance from the mean; expand $(X-\mu)^2$ and use linearity of expectation to get the shortcut $E[X^2]-\mu^2$ . The standard deviation is $\sigma = \sqrt{\sigma^2}$ , in the same units as $X$ .

TheoremThe Normal Distribution

The most important continuous distribution is the normal (Gaussian) distribution with mean $\mu$ and standard deviation $\sigma$ :

6

\quad f(x) = \frac{1}{\sigma\sqrt{2\pi}}\,\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)

Derivation hint: the $1/(\sigma\sqrt{2\pi})$ factor is the normalizing constant making the area equal 1, via the Gaussian integral $\int e^{-u^2}du = \sqrt{\pi}$ . About 68% of probability lies within $\mu \pm \sigma$ , 95% within $\mu \pm 2\sigma$ , and 99.7% within $\mu \pm 3\sigma$ (the empirical rule).

mathematician · Carl Friedrich Gauss | 1777–1855

Gauss derived the normal distribution while studying observational errors in astronomy. Seeking a curve for which the arithmetic mean is the most probable estimate of measurements, he arrived at equation [6] in 1809. The "bell curve" became so tied to his name it is called the Gaussian — though Abraham de Moivre had found it earlier (1733) as an approximation to coin-flip counts. Gauss's broader contributions to least-squares estimation made the normal distribution the cornerstone of statistics and physics.

🧠 Intuition

Think of probability as one liter of paint you must spread along the number line. A discrete distribution drops the paint in discrete blobs (a heavy blob over "3" means a 3 is likely). A continuous distribution smears it into a thin layer whose thickness is the density — thick where outcomes cluster, thin in the tails. The total paint is always exactly one liter. The mean is where the painted strip would balance on a knife edge; the variance tells you how widely the paint is smeared. For continuous variables, asking "what is the probability $X = 3.000\ldots$ ?" is like asking the area of a line with zero width — it's zero. You can only ask about intervals, which have area.

Figure

A normal probability density showing the mean and the 68–95–99.7 empirical rule

ExampleA Fair Six-Sided Die (Discrete)

Given: $X$ = result of rolling a fair die, $p(x) = \tfrac{1}{6}$ for $x = 1,\dots,6$ .

Find the mean. Using equation [4]:

\mu = \sum_{x=1}^{6} x\cdot\tfrac16 = \tfrac16(1+2+3+4+5+6) = \tfrac{21}{6} = 3.5.

Find the variance. First $E[X^2] = \tfrac16(1+4+9+16+25+36) = \tfrac{91}{6} \approx 15.167$ . Then by equation [5]:

\sigma^2 = 15.167 - (3.5)^2 = 15.167 - 12.25 = 2.9167.

Standard deviation: $\sigma = \sqrt{2.9167} \approx 1.708$ .

\boxed{\mu = 3.5 \text{ pips}, \quad \sigma^2 \approx 2.92 \text{ pips}^2, \quad \sigma \approx 1.71 \text{ pips}}

ExampleNormal Heights (Continuous)

Given: Adult male heights are approximately normal with $\mu = 175\text{ cm}$ and $\sigma = 7\text{ cm}$ .

Question 1: What fraction are between 168 cm and 182 cm? This is $\mu \pm \sigma$ , so by the empirical rule:

P(168 \le X \le 182) \approx 68\%.

Question 2: What fraction are taller than 189 cm? Note $189 = 175 + 2\sigma$ . The empirical rule gives 95% within $\mu \pm 2\sigma$ , leaving 5% in the two tails, so 2.5% above:

P(X > 189) \approx 2.5\%.

Question 3: Standardize 189 cm to a z-score:

z = \frac{x - \mu}{\sigma} = \frac{189 - 175}{7} = 2.0.

\boxed{68\% \text{ within } \pm\sigma; \quad P(X>189)\approx 2.5\%; \quad z = 2.0}

⚠️ Common mistake

"For a continuous variable, $f(x)$ is a probability." It is not — it is a density. A density can exceed 1 (e.g. a narrow uniform distribution on $[0, 0.5]$ has $f = 2$ ). Only areas under the curve are probabilities, and $P(X = c) = 0$ for any single exact value in a continuous distribution. Always integrate over an interval.

⚠️ Common mistake

"The mean is the most likely value." False in general. For a skewed distribution the mean, median, and mode differ. The mean is the balance point, the mode is the peak (most likely), and the median splits probability 50/50. They coincide only for symmetric distributions like the normal.

CounterexampleA High Density Is Not High Probability

Consider a uniform distribution on the interval $[0, 0.1]$ . To make the area equal 1, the density must be $f(x) = 10$ for $0 \le x \le 0.1$ . So $f = 10 \gg 1$ , yet no probability exceeds 1: the probability of landing in any sub-interval of width $w$ is just $10w \le 1$ . This demonstrates density and probability are genuinely different quantities.

connection

Probability distributions sit at the heart of statistics and connect outward in many directions. The Binomial Distribution counts successes in fixed independent trials and approximates the Normal Distribution for large $n$ . The Poisson Distribution models rare events in time or space and arises as a limit of the binomial. The Central Limit Theorem explains why the normal appears so universally: sums of independent variables converge to it. The whole framework rests on Probability Axioms (Kolmogorov's rules) and on Expected Value for summarizing distributions. Sampling distributions of statistics underpin Hypothesis Testing and Confidence Intervals.

🔬 Where this appears

⏳ History

- 1654 — Blaise Pascal and Pierre de Fermat correspond about gambling problems, founding the mathematical theory of probability.

- 1713 — Jacob Bernoulli's Ars Conjectandi introduces the binomial distribution and the Law of Large Numbers.

- 1733 — Abraham de Moivre derives the normal curve as an approximation to the binomial for large

n

.

- 1809 — Carl Friedrich Gauss formalizes the normal distribution in the theory of measurement errors.

- 1812 — Pierre-Simon Laplace publishes Théorie analytique des probabilités, sharpening the Central Limit Theorem.

- 1837 — Siméon Denis Poisson introduces the Poisson distribution for rare events.

- 1894 — Karl Pearson coins the term "standard deviation" and develops systematic families of distributions.

- 1908 — William Gosset ("Student") introduces the t-distribution for small samples in brewing quality control.

- 1933 — Andrey Kolmogorov axiomatizes probability theory, giving distributions a rigorous measure-theoretic foundation.

open-problem

Not every distribution is well-behaved. The Cauchy distribution has a perfectly valid bell shape yet no mean and no variance — the defining integrals diverge — so the Law of Large Numbers and the Central Limit Theorem fail for it. Such heavy-tailed distributions challenge naive intuition and appear in real data (financial crashes, earthquake magnitudes), where rare extreme events dominate. Modeling these requires care: the comforting empirical rule and even the existence of an average can break down. Deciding which distribution truly governs real data — and how to detect heavy tails from finite samples — remains an active practical and theoretical concern in statistics.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also