Conditional Probability, Bayes' Theorem, Independence

⚡ Key Takeaways

Conditional probability $P(A\mid B)$ measures the probability of event $A$ given that $B$ has occurred; it is the cornerstone of probabilistic reasoning. Bayes' theorem inverts conditionals, letting us update beliefs from evidence. Independence is the special case where conditioning changes nothing: $P(A\mid B)=P(A)$ . Applies wherever uncertainty is updated by information — statistics, machine learning, medicine, forensics. Formalized by Kolmogorov (1933), with Bayes' rule due to Thomas Bayes (posth. 1763) and Laplace.

💡 The Core Insight

Conditioning on $B$ is nothing more than renormalizing the probability measure to live inside $B$ . The whole machinery — Bayes, independence, the chain rule, total probability — follows mechanically from the single defining ratio $P(A\mid B)=P(A\cap B)/P(B)$ . Bayes' theorem is just this definition written twice and equated through the shared joint probability $P(A\cap B)$ .

DefinitionConditional Probability

Let $(\Omega,\mathcal{F},P)$ be a probability space and let $A,B\in\mathcal{F}$ with $P(B)>0$ . The conditional probability of $A$ given $B$ is

1

\quad P(A\mid B)=\frac{P(A\cap B)}{P(B)}.

Derivation hint: restrict the sample space to $B$ and renormalize so that $P(B\mid B)=1$ ; the ratio achieves exactly this. Symbols: $\Omega$ is the sample space; $\mathcal{F}$ the $\sigma$ -algebra of events; $P$ a probability measure; $A\cap B$ the joint event "both $A$ and $B$ "; $P(B)$ the prior weight of the conditioning event, required positive so the ratio is defined.

DefinitionThe Multiplication (Chain) Rule

Rearranging [1] gives the joint probability as a product:

2

\quad P(A\cap B)=P(A\mid B)\,P(B)=P(B\mid A)\,P(A).

Derivation hint: clear the denominator in [1], then write [1] with $A$ and $B$ swapped and equate the joints. Generalized to $n$ events:

3

\quad P\!\left(\bigcap_{k=1}^{n}A_k\right)=\prod_{k=1}^{n}P\!\left(A_k\;\middle|\;\bigcap_{j<k}A_j\right).

Derivation hint: telescope [2] by peeling off one conditioning event at a time, $P(A_1\cap A_2\cap A_3)=P(A_3\mid A_1\cap A_2)P(A_2\mid A_1)P(A_1)$ .

DefinitionIndependence

Events $A$ and $B$ are independent iff

4

\quad P(A\cap B)=P(A)\,P(B),

Derivation hint: substitute [4] into [1]; when $P(B)>0$ this is equivalent to $P(A\mid B)=P(A)$ , i.e. conditioning is informationally inert. A family $\{A_i\}_{i\in I}$ is mutually independent iff for every finite subset $S\subseteq I$ ,

5

\quad P\!\left(\bigcap_{i\in S}A_i\right)=\prod_{i\in S}P(A_i).

Derivation hint: pairwise independence checks only $|S|=2$ ; mutual independence demands the product rule for all subsets simultaneously — a strictly stronger condition.

TheoremLaw of Total Probability

If $\{B_1,\dots,B_n\}$ is a partition of $\Omega$ (disjoint, exhaustive, each $P(B_k)>0$ ), then for any event $A$ ,

6

\quad P(A)=\sum_{k=1}^{n}P(A\mid B_k)\,P(B_k).

Derivation hint: write $A=\bigcup_k (A\cap B_k)$ as a disjoint union, apply countable additivity, then [2] to each term.

TheoremBayes' Theorem

For events $A,B$ with $P(A),P(B)>0$ ,

7

\quad P(B\mid A)=\frac{P(A\mid B)\,P(B)}{P(A)}.

Derivation hint: equate the two products in [2] and solve for $P(B\mid A)$ . With a partition $\{B_1,\dots,B_n\}$ , expanding the denominator via [6]:

8

\quad P(B_k\mid A)=\frac{P(A\mid B_k)\,P(B_k)}{\displaystyle\sum_{j=1}^{n}P(A\mid B_j)\,P(B_j)}.

Derivation hint: apply [7] with $B=B_k$ , then replace $P(A)$ by the total-probability sum [6]. Here $P(B_k)$ is the prior, $P(A\mid B_k)$ the likelihood, $P(B_k\mid A)$ the posterior, and the denominator the evidence (marginal likelihood / normalizing constant).

ProofBayes' Theorem from the Multiplication Rule

1. By definition [1], assuming $P(A)>0$ and $P(B)>0$ :

P(B\mid A)=\frac{P(A\cap B)}{P(A)},\qquad P(A\mid B)=\frac{P(A\cap B)}{P(B)}.

2. The joint $P(A\cap B)$ is symmetric — it is the same number in both expressions. Solve the second for it:

P(A\cap B)=P(A\mid B)\,P(B).

3. Substitute into the first:

P(B\mid A)=\frac{P(A\mid B)\,P(B)}{P(A)},

which is [7]. $\quad\blacksquare$

∎

ProofLaw of Total Probability

1. Since $\{B_k\}$ partitions $\Omega$ , the sets $A\cap B_k$ are pairwise disjoint and their union is $A$ :

A=A\cap\Omega=A\cap\bigcup_k B_k=\bigcup_k(A\cap B_k).

2. By (countable) additivity of $P$ on disjoint sets:

P(A)=\sum_k P(A\cap B_k).

3. Apply the multiplication rule [2] to each term: $P(A\cap B_k)=P(A\mid B_k)P(B_k)$ , yielding [6]. $\quad\blacksquare$

∎

mathematician · Thomas Bayes | c. 1701–1761

A Presbyterian minister and Fellow of the Royal Society, Bayes left an unpublished essay solving the "inverse probability" problem: given an observed number of successes, what can be said about the underlying chance? His friend Richard Price edited and read it posthumously to the Royal Society in 1763 ("An Essay towards solving a Problem in the Doctrine of Chances"). Bayes treated only the uniform-prior case for a binomial parameter; the general theorem and its systematic use were developed independently and far more powerfully by Pierre-Simon Laplace from 1774 onward. The modern measure-theoretic foundation came from Kolmogorov in 1933.

🧠 Intuition

Picture probability as area on a unit-area dartboard $\Omega$ . Event $B$ is a region; conditioning on $B$ means throwing away the rest of the board and asking what fraction of the remaining region $B$ also lies in $A$ . That is why we divide by $P(B)$ : we are recomputing area as a fraction of the new, smaller universe.

Independence means $A$ occupies the same fraction of $B$ as it does of the whole board — knowing you landed in $B$ tells you nothing about whether you also landed in $A$ .

Bayes is the bookkeeping that lets you trade $P(A\mid B)$ for $P(B\mid A)$ once you know how big the two regions are individually. The deep point: the direction of a conditional matters enormously. "Probability of a rash given measles" and "probability of measles given a rash" are wildly different numbers, related only through the prevalence (prior) of measles.

Figure

Conditional probability as renormalized area:

P(A\mid B)

is the shaded overlap as a fraction of region

B

ExampleMedical Test and the Base-Rate Effect

Given: A disease has prevalence $P(D)=0.001$ (1 in 1000). A test has sensitivity $P(+\mid D)=0.99$ and specificity $P(-\mid D^c)=0.99$ , so the false-positive rate is $P(+\mid D^c)=0.01$ . A patient tests positive. Find $P(D\mid +)$ .

Steps: 1. Total probability of a positive test [6]:

P(+)=P(+\mid D)P(D)+P(+\mid D^c)P(D^c)

=(0.99)(0.001)+(0.01)(0.999)=0.00099+0.00999=0.01098.

2. Bayes' theorem [7]:

P(D\mid +)=\frac{P(+\mid D)P(D)}{P(+)}=\frac{0.00099}{0.01098}\approx 0.0902.

Answer:

\boxed{P(D\mid +)\approx 0.090 = 9.0\%}

Despite a "99% accurate" test, a positive result indicates disease only ~9% of the time — because true positives ( $\approx 1$ per 1000) are swamped by false positives ( $\approx 10$ per 1000). This is the base-rate fallacy.

ExampleMutual vs. Pairwise Independence

Given: Toss two fair coins. Let $A=\{$ first is heads $\}$ , $B=\{$ second is heads $\}$ , $C=\{$ the two coins match $\}$ . Each has probability $\tfrac12$ .

Steps: 1. Check pairwise. $P(A\cap B)=P(HH)=\tfrac14=P(A)P(B).$ ✓ $P(A\cap C)=P(HH)=\tfrac14=P(A)P(C).$ ✓ $P(B\cap C)=P(HH)=\tfrac14=P(B)P(C).$ ✓ — all three pairs independent. 2. Check mutual independence via [5] for $S=\{A,B,C\}$ :

P(A\cap B\cap C)=P(HH)=\tfrac14.

But $P(A)P(B)P(C)=\tfrac12\cdot\tfrac12\cdot\tfrac12=\tfrac18.$ 3. Since $\tfrac14\neq\tfrac18$ , the triple product rule fails.

Answer:

\boxed{A,B,C \text{ are pairwise independent but NOT mutually independent.}}

Indeed $C$ is fully determined by $A$ and $B$ together — knowing both fixes whether they match.

⚠️ Six Frequent Errors

1. Confusing the direction: $P(A\mid B)\neq P(B\mid A)$ in general (the "prosecutor's fallacy"). They differ by the factor $P(B)/P(A)$ . 2. Ignoring the base rate: treating $P(D\mid +)$ as if it equals the test's sensitivity $P(+\mid D)$ . 3. Confusing independent with mutually exclusive: if $A,B$ are disjoint with positive probability, they are dependent, since $P(A\mid B)=0\neq P(A)$ . 4. Assuming pairwise independence implies mutual independence (see counterexample above). 5. Conditioning on a null event: $P(\cdot\mid B)$ is undefined when $P(B)=0$ (needs regular conditional probability / measure theory to repair). 6. Forgetting that independence is relative to the measure $P$ : the same events can be independent under one $P$ and dependent under another.

CounterexampleMutually Exclusive Is the Opposite of Independent

Roll one fair die. Let $A=\{$ roll is $\le 3\}$ and $B=\{$ roll is $\ge 5\}$ . These are mutually exclusive: $A\cap B=\varnothing$ , so $P(A\cap B)=0$ . Yet $P(A)P(B)=\tfrac12\cdot\tfrac13=\tfrac16\neq 0$ . Knowing $B$ occurred makes $A$ impossible — maximal dependence. Students who hear " $A$ and $B$ can't both happen" and conclude "they don't influence each other" have it exactly backwards.

connection

Conditional probability is the gateway to nearly all of probability theory. It specializes to Conditional Expectation when we condition random variables rather than events, and to the Conditional Density $f_{X\mid Y}(x\mid y)=f_{X,Y}(x,y)/f_Y(y)$ in the continuous case. Independence of events generalizes to independence of Random Variables (factorization of joint distributions) and underlies the Law of Large Numbers and Central Limit Theorem, both of which assume i.i.d. samples. Bayes' theorem is the foundation of Bayesian Inference, the engine of Markov Chains (via conditional transition rules), and the backbone of Naive Bayes Classifiers. The chain rule [3] structures Probabilistic Graphical Models and Bayesian networks.

🔬 Where this appears

⏳ History

- 1654 — Pascal and Fermat correspond on the problem of points, implicitly using conditional reasoning about unfinished games.

- 1713 — Jacob Bernoulli's Ars Conjectandi (posthumous) states the law of large numbers, framing how evidence accumulates.

- 1763 — Richard Price reads Thomas Bayes' essay to the Royal Society, solving inverse probability for the binomial with a uniform prior.

- 1774 — Laplace independently formulates the general principle of inverse probability ("Mémoire sur la probabilité des causes").

- 1812 — Laplace's Théorie analytique des probabilités systematizes Bayesian methods and applies them to astronomy and demography.

- 1900s–1920s — Frequentists (Fisher, Neyman, Pearson) challenge "inverse probability," sidelining Bayesian methods for decades.

- 1933 — Kolmogorov's Grundbegriffe der Wahrscheinlichkeitsrechnung gives the axiomatic, measure-theoretic definition of conditional probability and independence.

- 1950s–60s — Revival of Bayesian statistics through Jeffreys, Savage, and de Finetti (subjective probability and exchangeability).

- 1990s–present — Computational power (MCMC, Gibbs sampling) makes Bayesian inference practical, fueling modern statistics and machine learning.

open-problem

Conditioning on null events. The elementary definition [1] breaks when $P(B)=0$ — yet we routinely condition on continuous events like $\{Y=y\}$ , each of measure zero. The resolution, regular conditional probability via the Radon–Nikodym theorem and the disintegration theorem, is subtle and (in pathological measurable spaces) not always guaranteed to exist nicely. The Borel–Kolmogorov paradox shows that "the probability conditional on a measure-zero event" can depend on how one approaches that event (which limiting family of positive-measure sets is used), so the naive limit is ambiguous. Foundationally, the meaning of subjective vs. objective probability, the choice of priors (objective/Jeffreys vs. subjective), and the philosophical status of the likelihood principle remain actively debated at the boundary between probability and statistics.

Conditional Probability, Bayes' Theorem, Independence

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also