Bayes Theorem

⚡ Key Takeaways

Bayes' Theorem relates the conditional probability $P(A\mid B)$ to its "reverse" $P(B\mid A)$ via $P(A\mid B) = \dfrac{P(B\mid A)\,P(A)}{P(B)}$ . In words: it tells you how to update your belief in a hypothesis after seeing evidence. It applies anywhere uncertainty is quantified — medical diagnosis, machine learning, forensics, signal processing, and scientific inference. Named for Thomas Bayes (c. 1701–1761), published posthumously in 1763.

💡 The Core Insight

Evidence does not tell you $P(\text{hypothesis} \mid \text{evidence})$ directly — it gives you $P(\text{evidence} \mid \text{hypothesis})$ . Bayes' Theorem is the bridge that flips this direction, and the flip requires you to bring in the prior $P(\text{hypothesis})$ — your belief before the evidence. The posterior is proportional to likelihood times prior. Forgetting the prior (the base rate) is the single most common error in probabilistic reasoning.

DefinitionConditional Probability

For events $A$ and $B$ in a probability space with $P(B) > 0$ , the conditional probability of $A$ given $B$ is

1

\quad P(A\mid B) = \frac{P(A \cap B)}{P(B)}.

Derivation hint: This is the definition; it renormalizes the probability of the joint event $A\cap B$ to the reduced sample space in which $B$ is known to have occurred. Here $P(A\cap B)$ is the probability that both $A$ and $B$ occur, and $P(B)$ is the (positive) probability of the conditioning event.

TheoremBayes' Theorem

Let $A$ and $B$ be events with $P(A) > 0$ and $P(B) > 0$ . Then

2

\quad P(A\mid B) = \frac{P(B\mid A)\,P(A)}{P(B)}.

Derivation hint: Apply Equation [1] twice — once for $P(A\mid B)$ , once for $P(B\mid A)$ — and note both share the symmetric joint term $P(A\cap B) = P(B\cap A)$ . The terms are named:

$P(A\mid B)$ — the posterior: belief in $A$ after observing $B$ .
$P(B\mid A)$ — the likelihood: probability of evidence $B$ if $A$ holds.
$P(A)$ — the prior: belief in $A$ before observing $B$ .
$P(B)$ — the evidence (or marginal likelihood): total probability of observing $B$ .

DefinitionLaw of Total Probability and Expanded Form

If $\{A_1, A_2, \dots, A_n\}$ is a partition of the sample space (mutually exclusive, exhaustive, each with positive probability), the denominator can be expanded:

3

\quad P(B) = \sum_{i=1}^{n} P(B\mid A_i)\,P(A_i).

Derivation hint: Write $B = \bigcup_i (B\cap A_i)$ as a disjoint union over the partition, use countable additivity, then apply Equation [1] to each term. Substituting into Equation [2] gives the form used in practice:

4

\quad P(A_k\mid B) = \frac{P(B\mid A_k)\,P(A_k)}{\displaystyle\sum_{i=1}^{n} P(B\mid A_i)\,P(A_i)}.

Derivation hint: Substitute Equation [3] into the denominator of Equation [2] with $A = A_k$ . For the common binary case ( $A$ and its complement $A^c$ ):

5

\quad P(A\mid B) = \frac{P(B\mid A)\,P(A)}{P(B\mid A)\,P(A) + P(B\mid A^c)\,P(A^c)}.

Derivation hint: Equation [4] with the two-element partition $\{A, A^c\}$ .

DefinitionOdds Form

Dividing the posterior for $A$ by that for $A^c$ , the evidence $P(B)$ cancels, yielding the elegant odds form:

6

\quad \underbrace{\frac{P(A\mid B)}{P(A^c\mid B)}}_{\text{posterior odds}} = \underbrace{\frac{P(B\mid A)}{P(B\mid A^c)}}_{\text{likelihood ratio}} \times \underbrace{\frac{P(A)}{P(A^c)}}_{\text{prior odds}}.

Derivation hint: Take the ratio of Equation [2] for $A$ and for $A^c$ ; the common denominator $P(B)$ divides out. The likelihood ratio (or Bayes factor) is the multiplicative strength of the evidence — how many times more often $B$ occurs under $A$ than under $A^c$ .

ProofDerivation of Bayes' Theorem

We derive Equation [2] from the definition of conditional probability.

Step 1. By Equation [1], the conditional probability of $A$ given $B$ is

P(A\mid B) = \frac{P(A\cap B)}{P(B)}.

Step 2. By the same definition, swapping the roles of $A$ and $B$ ,

P(B\mid A) = \frac{P(B\cap A)}{P(A)}.

Step 3. The intersection is symmetric: $A\cap B = B\cap A$ , so $P(A\cap B) = P(B\cap A)$ . Solving Step 2 for this joint probability:

P(A\cap B) = P(B\mid A)\,P(A).

This is the multiplication rule.

Step 4. Substitute the result of Step 3 into Step 1:

P(A\mid B) = \frac{P(B\mid A)\,P(A)}{P(B)}.

This is Equation [2]. $\blacksquare$ The entire content of the theorem is the symmetry of the joint probability viewed through two different conditionings. The expanded forms [4]–[6] are obtained by rewriting $P(B)$ using the law of total probability or by taking ratios.

∎

mathematician · Thomas Bayes | c. 1701–1761

Thomas Bayes was an English Presbyterian minister and mathematician, a Fellow of the Royal Society. He never published his theorem in his lifetime; it appeared in "An Essay towards solving a Problem in the Doctrine of Chances", communicated to the Royal Society in 1763 by his friend Richard Price, two years after Bayes' death. Bayes' original problem was inverse: given a number of successes and failures in independent trials with unknown success probability $p$ , what can be said about $p$ ? He effectively solved a special case of inference for a binomial parameter using a uniform prior. The general formulation and its systematic use in inference are largely due to Pierre-Simon Laplace, who rediscovered and vastly extended the idea independently. Much of what we call "Bayesian" is really Laplacian.

🧠 Intuition

Think of probability as area. The whole sample space is a unit square. Event $A$ carves out a region; event $B$ carves out another; they overlap in $A\cap B$ . When you learn that $B$ happened, you erase everything outside $B$ — your world shrinks to the $B$ region. The question "how likely is $A$ now?" becomes: what fraction of the surviving $B$ -region also belongs to $A$ ? That fraction is $P(A\cap B)/P(B)$ — exactly Equation [1]. Bayes' Theorem is just the observation that you can compute that overlap two ways. You might naturally know "of all the $A$ -cases, what fraction are also $B$ " (the likelihood) plus "how big is $A$ to begin with" (the prior). Bayes assembles these into the overlap and divides by the size of $B$ to flip the conditioning. The crucial intuition is base rates: even if $B$ almost always accompanies $A$ (high likelihood), if $A$ is itself tiny (low prior), then most of region $B$ may come from the complement $A^c$ . A big slice of a rare event can be dwarfed by a tiny slice of a common one.

Figure

Updating belief: prior region, evidence narrows the world, posterior is the surviving overlap

ExampleMedical Test with a Rare Disease

Given: A disease affects 1 in 1000 people, so the prior is $P(D) = 0.001$ . A test has:

Sensitivity (true positive rate): $P(+\mid D) = 0.99$
False positive rate: $P(+\mid D^c) = 0.05$ (so specificity $= 0.95$ ).

A randomly screened person tests positive. Find $P(D\mid +)$ .

Step 1 — Prior of no disease: $P(D^c) = 1 - 0.001 = 0.999$ .

Step 2 — Evidence (total probability of a positive), Equation [3]:

P(+) = P(+\mid D)P(D) + P(+\mid D^c)P(D^c)

= (0.99)(0.001) + (0.05)(0.999) = 0.00099 + 0.04995 = 0.05094.

Step 3 — Bayes, Equation [5]:

P(D\mid +) = \frac{(0.99)(0.001)}{0.05094} = \frac{0.00099}{0.05094} \approx 0.01943.

\boxed{P(D\mid +) \approx 0.0194 \;=\; 1.94\%}

Despite a "99% accurate" test, a positive result means only about a 2% chance of actually having the disease — because false positives from the huge healthy population (0.04995) swamp the true positives (0.00099). This is the base-rate fallacy.

ExampleTwo Factories and a Defective Part — Odds Form

Given: A warehouse stocks bolts from two suppliers. Supplier 1 made 70% of the bolts with a 2% defect rate; Supplier 2 made 30% with an 8% defect rate. A randomly chosen bolt is defective. Find the probability it came from Supplier 2. Let $S_2$ = "from Supplier 2", $S_1 = S_2^c$ = "from Supplier 1", $B$ = "defective".

Step 1 — Priors and likelihoods:

P(S_2) = 0.30,\quad P(S_1) = 0.70,

P(B\mid S_2) = 0.08,\quad P(B\mid S_1) = 0.02.

Step 2 — Prior odds (in favor of $S_2$ ):

\frac{P(S_2)}{P(S_1)} = \frac{0.30}{0.70} = \frac{3}{7}.

Step 3 — Likelihood ratio (Equation [6]):

\frac{P(B\mid S_2)}{P(B\mid S_1)} = \frac{0.08}{0.02} = 4.

Step 4 — Posterior odds:

\frac{P(S_2\mid B)}{P(S_1\mid B)} = 4 \times \frac{3}{7} = \frac{12}{7}.

Step 5 — Convert odds to probability: odds $12{:}7$ give

P(S_2\mid B) = \frac{12}{12+7} = \frac{12}{19}.

\boxed{P(S_2\mid B) = \tfrac{12}{19} \approx 0.632 \;=\; 63.2\%}

Though Supplier 2 makes only 30% of bolts, its higher defect rate makes it the more likely source of any given defective bolt.

⚠️ The Prosecutor's Fallacy

Do not confuse the likelihood $P(B\mid A)$ with the posterior $P(A\mid B)$ — they are generally very different. In a courtroom, "the probability of this DNA match if the defendant is innocent is one in a million" ( $P(\text{match}\mid \text{innocent}) = 10^{-6}$ ) is not the same as "the probability the defendant is innocent given the match." With a database of millions of people, many innocent matches can occur. Equating these two reversed conditionals is the prosecutor's fallacy, and it has led to real wrongful convictions. Bayes' Theorem [2] shows the conversion requires the prior $P(A)$ — which the fallacy silently ignores.

CounterexampleHigh Accuracy ≠ High Posterior

A common belief: "If a test is 99% accurate and I test positive, I'm 99% likely to have the condition." The medical example above is a direct counterexample: with $P(+\mid D) = 0.99$ and a low base rate $P(D) = 0.001$ , the posterior is only 1.94%. The same "99% accurate" test gives very different posteriors depending entirely on the prior:

Prior $P(D)$	Posterior $P(D\mid +)$
0.001	1.9%
0.01	16.7%
0.10	68.7%
0.50	95.2%

Accuracy is a property of the test; the posterior is a property of the test plus the population. There is no posterior without a prior.

connection

Bayes' Theorem sits at the hub of probability and inference:

It is an immediate corollary of Conditional Probability and the Law of Total Probability.
It generalizes from events to densities, giving the foundation of Bayesian Inference, where one updates a Prior Distribution into a Posterior Distribution using a Likelihood Function.
In its repeated/sequential form it is the engine of Bayesian Updating and recursive estimators like the Kalman Filter and Hidden Markov Models.
The Naive Bayes Classifier applies it with a conditional-independence assumption for classification.
It contrasts philosophically with Frequentist Inference and connects to Maximum Likelihood Estimation (the MAP estimate reduces to the MLE under a flat prior).
The odds form links it to Information Theory via the log-likelihood ratio as evidence in bits.

🔬 Where this appears

⏳ History

- 1763 — Richard Price posthumously presents Bayes' Essay towards solving a Problem in the Doctrine of Chances to the Royal Society, containing the first version of the theorem for inferring a binomial parameter.

- 1774 — Pierre-Simon Laplace, independently, formulates the general principle of inverse probability and applies it broadly (population statistics, astronomy, jurisprudence).

- 1812 — Laplace's Théorie analytique des probabilités systematizes "inverse probability," the dominant inferential framework of the 19th century.

- 1920s–1930s — R. A. Fisher, Jerzy Neyman, and Egon Pearson develop frequentist methods (significance tests, confidence intervals), pushing Bayesian methods to the margins amid criticism of subjective priors.

- 1939–1961 — Harold Jeffreys revives objective Bayesianism in Theory of Probability, introducing non-informative ("Jeffreys") priors.

- 1950s–1960s — Bruno de Finetti and L. J. Savage ground subjective probability and Bayesianism in axioms of rational decision-making (coherence, exchangeability).

- 1953 / 1970 — The Metropolis and Hastings algorithms (MCMC) are developed, eventually making high-dimensional posteriors computable.

- 1980s–1990s — Cheap computing and the Gibbs sampler (Geman & Geman, 1984; Gelfand & Smith, 1990) trigger a Bayesian renaissance across statistics and machine learning.

- 2000s–present — Probabilistic programming (Stan, PyMC), Bayesian deep learning, and Bayesian optimization make the theorem a standard workhorse of modern data science.

open-problem · The Problem of the Prior

Bayes' Theorem is mathematically uncontroversial, but its use in inference raises deep, partly unresolved issues:

Choosing the prior. Where does $P(A)$ come from when no frequency data exist? "Objective" or "non-informative" priors (uniform, Jeffreys, reference priors) attempt to encode ignorance, but a prior uniform in one parameterization is non-uniform in another — there is no universally canonical ignorance prior. For continuous infinite-dimensional spaces, ignorance can even be incoherent.
Improper priors. Common non-informative priors (e.g. $p(\mu)\propto 1$ over all reals) do not integrate to 1; they sometimes yield proper posteriors and sometimes yield paradoxes or nonexistent posteriors.
Sensitivity and robustness. With small data, conclusions can depend heavily on the prior; quantifying and bounding this sensitivity (robust Bayes) is an active area.
Computation. Exact posteriors are intractable for most realistic models; MCMC and variational methods give approximations whose convergence and error are not always guaranteed.
Calibration vs. coherence. Bayesian updates are internally coherent but need not be well-calibrated against reality if the model or prior is misspecified — the theorem cannot rescue a wrong model.

These are not flaws in the theorem (an exact identity) but open questions about how to deploy it as a theory of inference.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also