Conditional Probability and Independence

⚡ Key Takeaways

Conditional probability measures the chance of an event $A$ given that another event $B$ has occurred, written $P(A\mid B)=P(A\cap B)/P(B)$ . Two events are independent when knowing one tells you nothing about the other, i.e. $P(A\cap B)=P(A)P(B)$ . Used everywhere from medical testing to gambling to machine learning. Formalized by Bayes, Laplace, and Kolmogorov between the 1760s and 1933.

💡 The Core Insight

Conditioning is renormalization. When you learn that $B$ occurred, you discard all outcomes outside $B$ and rescale the probabilities of what remains so they sum to 1 again. The event $A$ is then judged only by the portion of it that lives inside $B$ . Independence is the dividing line: if the conditional probability $P(A\mid B)$ equals the unconditional $P(A)$ , the information in $B$ is irrelevant to $A$ .

DefinitionConditional Probability

Let $(\Omega,\mathcal{F},P)$ be a probability space and let $A,B\subseteq\Omega$ be events with $P(B)>0$ . The conditional probability of $A$ given $B$ is

1

\quad P(A\mid B)=\frac{P(A\cap B)}{P(B)}.

Derivation hint: Restrict the sample space to $B$ and renormalize by dividing by $P(B)$ so that $P(B\mid B)=1$ . Symbols:

$\Omega$ — the sample space (all possible outcomes).
$A,B$ — events (subsets of $\Omega$ ).
$A\cap B$ — the event that both $A$ and $B$ occur.
$P(\cdot)$ — the probability measure, assigning numbers in $[0,1]$ .
$P(A\mid B)$ — probability of $A$ assuming $B$ is known to have occurred.

The quantity is undefined when $P(B)=0$ .

DefinitionIndependence of Two Events

Events $A$ and $B$ are independent if and only if

2

\quad P(A\cap B)=P(A)\,P(B).

Derivation hint: Substitute $P(A\mid B)=P(A)$ into equation [1] and clear the denominator; the symmetric product form drops the requirement $P(B)>0$ . Equivalently, when $P(B)>0$ , independence means $P(A\mid B)=P(A)$ , and when $P(A)>0$ , it means $P(B\mid A)=P(B)$ .

TheoremMultiplication Rule and Chain Rule

Rearranging [1] gives the multiplication rule:

3

\quad P(A\cap B)=P(B)\,P(A\mid B)=P(A)\,P(B\mid A).

Derivation hint: Multiply both sides of equation [1] by $P(B)$ ; the second form follows by symmetry of $A\cap B$ . For a sequence of events $A_1,\dots,A_n$ this generalizes to the chain rule:

4

\quad P(A_1\cap\cdots\cap A_n)=P(A_1)\,P(A_2\mid A_1)\,P(A_3\mid A_1\cap A_2)\cdots P(A_n\mid A_1\cap\cdots\cap A_{n-1}).

Derivation hint: Apply equation [3] repeatedly, conditioning each new event on the intersection of all preceding ones.

TheoremLaw of Total Probability and Bayes' Theorem

If $B_1,\dots,B_k$ partition $\Omega$ (mutually exclusive, exhaustive, each of positive probability), then for any event $A$ :

5

\quad P(A)=\sum_{i=1}^{k}P(B_i)\,P(A\mid B_i).

Derivation hint: Write $A=\bigcup_i (A\cap B_i)$ as a disjoint union, apply additivity, then equation [3] to each term. Combining [3] and [5] yields Bayes' theorem:

6

\quad P(B_j\mid A)=\frac{P(B_j)\,P(A\mid B_j)}{\sum_{i=1}^{k}P(B_i)\,P(A\mid B_i)}.

Derivation hint: Write $P(B_j\mid A)=P(A\cap B_j)/P(A)$ , expand the numerator with equation [3] and the denominator with equation [5].

ProofThe Conditional Measure Is a Genuine Probability

We verify that for fixed $B$ with $P(B)>0$ , the map $Q(A):=P(A\mid B)$ satisfies Kolmogorov's axioms. 1. Non-negativity. Since $P(A\cap B)\ge 0$ and $P(B)>0$ , the ratio $Q(A)=P(A\cap B)/P(B)\ge 0$ . 2. Normalization. Take $A=\Omega$ . Then $\Omega\cap B=B$ , so $Q(\Omega)=P(B)/P(B)=1$ . 3. Countable additivity. Let $A_1,A_2,\dots$ be pairwise disjoint. The sets $A_n\cap B$ are also pairwise disjoint, so

Q\!\left(\bigcup_n A_n\right)=\frac{P\!\left(\bigcup_n (A_n\cap B)\right)}{P(B)}=\frac{\sum_n P(A_n\cap B)}{P(B)}=\sum_n Q(A_n).

4. Conclusion. $Q$ is a probability measure on $\Omega$ concentrated on $B$ . Therefore conditioning is internally consistent: all theorems of probability apply unchanged inside the conditioned world. $\blacksquare$

∎

ProofEquivalence of the Two Definitions of Independence

Suppose $P(B)>0$ . 1. If $P(A\cap B)=P(A)P(B)$ (equation [2]), divide both sides by $P(B)$ : $P(A\cap B)/P(B)=P(A)$ , i.e. $P(A\mid B)=P(A)$ . 2. Conversely, if $P(A\mid B)=P(A)$ , multiply by $P(B)$ : $P(A\cap B)=P(A)P(B)$ . 3. The product form [2] is preferred because it is symmetric in $A$ and $B$ and remains meaningful even when $P(A)$ or $P(B)$ equals zero — in which case both sides are $0$ and independence holds trivially. $\blacksquare$

∎

mathematician · Thomas Bayes | c. 1701–1761

An English Presbyterian minister and amateur mathematician, Bayes never published his famous result during his lifetime. His essay An Essay towards Solving a Problem in the Doctrine of Chances was communicated to the Royal Society by Richard Price in 1763, two years after Bayes' death. It addressed an inverse problem: given observed successes, infer the underlying probability — the seed of Bayesian Inference. Pierre-Simon Laplace independently rediscovered and vastly generalized the idea, applying it to celestial mechanics, demography, and jurisprudence, and it is largely through Laplace that "Bayesian" reasoning entered scientific practice.

🧠 Intuition

Picture the sample space as a unit square of area 1. An event is a region; its probability is its area. Now learn that $B$ occurred: erase everything outside $B$ and blow up the region $B$ until it again has area 1. The new area of $A$ inside this magnified picture is exactly $P(A\mid B)$ — the fraction of $B$ that overlaps $A$ . Independence means $A$ slices through $B$ in the same proportion that it slices through the whole square. If $A$ takes up 30% of $\Omega$ , then under independence it also takes up 30% of $B$ , 30% of $B^c$ , and 30% of every other region — its "density" is uniform with respect to $B$ . Dependence simply means $A$ is concentrated more in some parts of $\Omega$ than others, so knowing where you are (inside $B$ or not) shifts the odds.

Figure

Conditioning as renormalization: the overlap of A with B becomes the whole new sample space

ExampleMedical Test — Bayes in Action

Given: A disease affects 1% of a population: $P(D)=0.01$ . A test has sensitivity $P(+\mid D)=0.99$ and false-positive rate $P(+\mid D^c)=0.05$ . A person tests positive. What is $P(D\mid +)$ ?

Step 1 — Total probability of a positive (eq. [5]):

P(+)=P(D)P(+\mid D)+P(D^c)P(+\mid D^c)=0.01(0.99)+0.99(0.05)=0.0099+0.0495=0.0594.

Step 2 — Apply Bayes (eq. [6]):

P(D\mid +)=\frac{P(D)P(+\mid D)}{P(+)}=\frac{0.0099}{0.0594}.

Step 3 — Compute: $0.0099/0.0594\approx 0.1667$ .

\boxed{P(D\mid +)\approx 0.167 = 16.7\%}

Despite a "99% accurate" test, a positive result means only about a 1-in-6 chance of disease, because healthy people vastly outnumber sick ones — the base-rate dominates.

ExampleDrawing Cards Without Replacement — Chain Rule

Given: Draw two cards from a standard 52-card deck without replacement. Find the probability both are aces, then check independence.

Step 1 — First ace: $P(A_1)=\dfrac{4}{52}=\dfrac{1}{13}$ .

Step 2 — Second ace given first (eq. [3]): $P(A_2\mid A_1)=\dfrac{3}{51}=\dfrac{1}{17}$ .

Step 3 — Multiply (chain rule, eq. [4]):

P(A_1\cap A_2)=\frac{1}{13}\cdot\frac{1}{17}=\frac{1}{221}\approx 0.00452.

\boxed{P(\text{both aces})=\dfrac{1}{221}\approx 0.452\%}

Independence check: If the draws were independent we'd need $P(A_1\cap A_2)=P(A_1)P(A_2)=\left(\tfrac{1}{13}\right)^2=\tfrac{1}{169}\ne\tfrac{1}{221}$ . Since $\tfrac{1}{221}<\tfrac{1}{169}$ , the events are dependent — removing one ace makes a second slightly less likely.

⚠️ Independence ≠ Mutual Exclusivity

These are opposite ideas, frequently confused. Mutually exclusive events cannot both happen: $P(A\cap B)=0$ . Independent events satisfy $P(A\cap B)=P(A)P(B)$ . If $A$ and $B$ both have positive probability and are mutually exclusive, then $P(A\cap B)=0\ne P(A)P(B)>0$ , so they are actually strongly dependent — knowing $A$ occurred guarantees $B$ did not. Never assume "they don't overlap" means "they're independent."

⚠️ The Conditioning Bar Is Not Symmetric

$P(A\mid B)$ and $P(B\mid A)$ are generally different numbers. Confusing them is the prosecutor's fallacy: the probability of innocence given the evidence is not the probability of the evidence given innocence. In the medical example, $P(+\mid D)=0.99$ but $P(D\mid +)=0.167$ — a factor-of-6 difference. Bayes' theorem (eq. [6]) is precisely the tool that converts one direction into the other.

CounterexamplePairwise Independence Does Not Imply Mutual Independence

Toss two fair coins. Define:

$A$ = first coin is Heads,
$B$ = second coin is Heads,
$C$ = the two coins match (both H or both T).

Each has probability $\tfrac12$ . Check pairs: $P(A\cap C)=P(\text{HH})=\tfrac14=P(A)P(C)$ , and similarly for the other two pairs — so the events are pairwise independent. But $P(A\cap B\cap C)=P(\text{HH})=\tfrac14$ , while $P(A)P(B)P(C)=\tfrac18$ . Since $\tfrac14\ne\tfrac18$ , the three are not mutually independent: knowing $A$ and $B$ determines $C$ completely. Mutual independence requires the product rule to hold for every subset of events, not just pairs.

connection

Conditional probability is the gateway to the rest of probability theory. The renormalization viewpoint extends to Conditional Expectation, where we condition a random variable rather than just an event. Independence of events generalizes to Independence of Random Variables and underlies the Law of Large Numbers and the Central Limit Theorem, both of which assume independent (or weakly dependent) samples. Bayes' theorem [eq. 6] is the engine of Bayesian Inference, while the law of total probability [eq. 5] is the discrete cousin of marginalization in Joint Probability Distributions. The notion of conditioning also seeds Markov Chains, where the future depends on the present but not the past.

🔬 Where this appears

⏳ History

- 1654 — Blaise Pascal and Pierre de Fermat correspond about gambling problems, founding mathematical probability and implicitly using conditional reasoning.

- 1657 — Christiaan Huygens publishes De ratiociniis in ludo aleae, the first printed probability treatise, introducing expectation.

- 1713 — Jacob Bernoulli's posthumous Ars Conjectandi proves the first law of large numbers, framing long-run frequency.

- 1763 — Thomas Bayes' essay, communicated by Richard Price, gives the inverse-probability rule that bears his name.

- 1774–1812 — Pierre-Simon Laplace develops and generalizes Bayesian methods, stating the principle of inverse probability and applying it across the sciences in Théorie analytique des probabilités.

- 1837 — Siméon Denis Poisson refines the law of large numbers and studies dependent trials.

- 1900s — Andrey Markov introduces dependent sequences (Markov chains), demonstrating rich behavior beyond independence.

- 1933 — Andrey Kolmogorov's Grundbegriffe der Wahrscheinlichkeitsrechnung axiomatizes probability measure-theoretically, giving conditional probability and independence their modern rigorous definitions.

- 1950s–present — Conditional independence becomes the structural backbone of Bayesian networks (Judea Pearl, 1980s) and modern statistical learning.

open-problem

The definition $P(A\mid B)=P(A\cap B)/P(B)$ breaks down when $P(B)=0$ — yet conditioning on probability-zero events is unavoidable when the conditioning variable is continuous (e.g. "given that $X=3.0$ exactly"). Resolving this requires measure-theoretic conditional expectation and the Radon–Nikodym theorem, where the conditional probability becomes a function defined only up to sets of measure zero. The Borel–Kolmogorov paradox shows that conditioning on the same zero-probability event ("a point on a sphere") can yield different answers depending on which limiting family of positive-probability sets you use — so naïve continuous conditioning is genuinely ambiguous. Philosophically, debates persist between frequentist and Bayesian interpretations of what $P(A\mid B)$ even means: a long-run frequency, or a rational degree of belief. These foundational questions remain live in statistics and the philosophy of probability.

Conditional Probability and Independence

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also