Probability Rules

⚡ Key Takeaways

Probability Rules are the algebraic laws governing how probabilities of events combine. One-line statement: Probabilities lie between 0 and 1, sum to 1 over all outcomes, and combine via the addition rule (for "or"), the multiplication rule (for "and"), and the complement rule (for "not"). Where it applies: any situation involving uncertainty — dice, cards, weather, genetics, medical testing, insurance. Who/when: formalized by Pierre de Fermat and Blaise Pascal (1654), axiomatized by Andrey Kolmogorov (1933).

💡 The Core Insight

Probability is a measure — a way of assigning a "size" between 0 and 1 to sets of outcomes. Because outcomes are sets, the entire subject reduces to the arithmetic of overlapping and disjoint sets. "Or" corresponds to union and uses addition (with a correction for overlap); "and" corresponds to intersection and uses multiplication (adjusted for dependence); "not" corresponds to the complement and uses subtraction from 1. Master these three correspondences and you can solve almost any introductory problem.

DefinitionSample Space, Events, and Probability

The sample space $S$ is the set of all possible outcomes of an experiment. An event $A$ is any subset of $S$ . A probability function $P$ assigns to each event a number satisfying Kolmogorov's axioms:

1

\quad 0 \le P(A) \le 1

Derivation hint: An event cannot happen less often than never (0) or more often than always (1); this is an axiom, not derived.

2

\quad P(S) = 1

Derivation hint: Some outcome in the sample space must occur, so the certain event has probability 1 — an axiom.

3

\quad P(A \cup B) = P(A) + P(B) \quad \text{if } A \cap B = \varnothing

Derivation hint: For mutually exclusive (disjoint) events, the count of favourable outcomes simply adds; this is the additivity axiom.

Symbols: $S$ = sample space; $A, B$ = events (subsets of $S$ ); $P(A)$ = probability of $A$ ; $A \cup B$ = " $A$ or $B$ " (union); $A \cap B$ = " $A$ and $B$ " (intersection); $\varnothing$ = empty set; $A^c$ = complement of $A$ (everything not in $A$ ).

DefinitionEqually Likely Outcomes

When $S$ has $n$ equally likely outcomes and event $A$ contains $k$ of them, the classical definition gives:

4

\quad P(A) = \frac{k}{n} = \frac{\text{number of favourable outcomes}}{\text{total number of outcomes}}

Derivation hint: Each outcome carries equal probability $1/n$ by symmetry; adding $k$ of these via [3] gives $k/n$ .

TheoremThe Four Core Rules

From the axioms, the following derived rules hold for all events $A, B$ .

Complement Rule:

5

\quad P(A^c) = 1 - P(A)

Derivation hint: $A$ and $A^c$ are disjoint and together form $S$ ; apply [3] then [2].

General Addition Rule:

6

\quad P(A \cup B) = P(A) + P(B) - P(A \cap B)

Derivation hint: Adding $P(A)$ and $P(B)$ double-counts the overlap $A\cap B$ ; subtract it once.

Conditional Probability:

7

\quad P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \quad P(B) > 0

Derivation hint: Restricting to the world where $B$ occurred rescales probabilities by dividing by $P(B)$ .

General Multiplication Rule:

8

\quad P(A \cap B) = P(A)\,P(B \mid A) = P(B)\,P(A \mid B)

Derivation hint: Rearrange [7]; the chance of both is the chance of one times the chance of the other given the first. For independent events, $P(B\mid A) = P(B)$ , so [8] simplifies:

9

\quad P(A \cap B) = P(A)\,P(B) \quad \text{(independence)}

Derivation hint: Independence means knowing $A$ tells nothing about $B$ , so $P(B\mid A)=P(B)$ ; substitute into [8].

ProofDeriving the General Addition Rule from the Axioms

We prove [6] using only axioms [1]–[3].

Step 1. Partition $A \cup B$ into three disjoint pieces: outcomes in $A$ only, in $B$ only, and in both. Formally, write $A = (A \cap B^c) \cup (A \cap B)$ , a disjoint union.

Step 2. By additivity [3]:

P(A) = P(A \cap B^c) + P(A \cap B).

Step 3. Similarly decompose $A \cup B = (A \cap B^c) \cup (A \cap B) \cup (B \cap A^c)$ , three mutually disjoint pieces. By [3]:

P(A \cup B) = P(A \cap B^c) + P(A \cap B) + P(B \cap A^c).

Step 4. From Step 2, $P(A \cap B^c) = P(A) - P(A \cap B)$ . Likewise $P(B \cap A^c) = P(B) - P(A \cap B)$ .

Step 5. Substitute into Step 3:

P(A \cup B) = [P(A) - P(A\cap B)] + P(A\cap B) + [P(B) - P(A\cap B)].

Step 6. Simplify to obtain $P(A \cup B) = P(A) + P(B) - P(A \cap B)$ , which is [6]. $\blacksquare$

∎

mathematician · Andrey Kolmogorov | 1903–1987

The Russian mathematician Andrey Kolmogorov transformed probability from a collection of clever gambling tricks into a rigorous branch of mathematics. In his 1933 monograph Grundbegriffe der Wahrscheinlichkeitsrechnung ("Foundations of the Theory of Probability"), he showed that probability is just a special kind of measure theory — the mathematics of assigning sizes to sets. His three axioms (non-negativity, normalization to 1, and countable additivity) are exactly equations [1]–[3] above, and every rule in this article is a logical consequence of them. This unification let probability borrow the full power of real analysis, and it underpins modern statistics, quantum mechanics, and finance.

🧠 Intuition

Think of the sample space as a dartboard of total area 1. Each event is a region on the board, and its probability is just the fraction of the board it covers. Then:

Complement [5]: the chance of not hitting a region is the rest of the board — $1$ minus that region.
Addition [6]: to find the area covered by region $A$ or region $B$ , add their areas but subtract the overlap you counted twice.
Multiplication [8]: "and" shrinks things — the chance of landing in $A$ and then in a sub-region $B$ is a fraction of a fraction, so you multiply.

The key mental switch: "or" tends to make probabilities bigger (addition); "and" tends to make them smaller (multiplication).

Figure

Venn diagram of two overlapping events showing the addition rule

ExampleDrawing a Card: Addition Rule

Given: A standard 52-card deck. Draw one card. Let $A$ = "card is a King", $B$ = "card is a Heart". Find $P(\text{King or Heart})$ .

Step 1. Count favourable outcomes. There are 4 Kings, so $P(A) = 4/52$ . There are 13 Hearts, so $P(B) = 13/52$ .

Step 2. Find the overlap. The King of Hearts is both, so $P(A \cap B) = 1/52$ .

Step 3. Apply the addition rule [6]:

P(A \cup B) = \frac{4}{52} + \frac{13}{52} - \frac{1}{52} = \frac{16}{52}.

Step 4. Simplify: $16/52 = 4/13 \approx 0.308$ .

\boxed{P(\text{King or Heart}) = \dfrac{4}{13} \approx 0.308}

Note: had we forgotten to subtract the overlap, we'd wrongly get $17/52$ .

ExampleTwo Draws: Multiplication Rule With and Without Replacement

Given: A bag has 5 red and 3 blue marbles (8 total). Draw two marbles. Find $P(\text{both red})$ (a) with replacement, (b) without replacement.

Part (a) — with replacement (independent events). First draw: $P(\text{red}_1) = 5/8$ . After replacing, the second draw is identical: $P(\text{red}_2) = 5/8$ . By independence [9]:

P(\text{both red}) = \frac{5}{8}\times\frac{5}{8} = \frac{25}{64} \approx 0.391.

\boxed{P_{\text{with}} = \dfrac{25}{64} \approx 0.391}

Part (b) — without replacement (dependent events). First draw: $P(\text{red}_1) = 5/8$ . Now 4 red remain out of 7. So $P(\text{red}_2 \mid \text{red}_1) = 4/7$ . By the multiplication rule [8]:

P(\text{both red}) = \frac{5}{8}\times\frac{4}{7} = \frac{20}{56} = \frac{5}{14} \approx 0.357.

\boxed{P_{\text{without}} = \dfrac{5}{14} \approx 0.357}

The "without replacement" probability is lower because removing a red marble reduces the chance of a second red.

⚠️ Independent ≠ Mutually Exclusive

These two terms are routinely confused but are nearly opposite ideas.

Mutually exclusive events cannot both happen: $P(A \cap B) = 0$ . Use the simple addition rule.
Independent events do not affect each other: $P(A \cap B) = P(A)P(B)$ .

If two events with nonzero probability are mutually exclusive, they are necessarily dependent — because if $A$ happens, $B$ definitely cannot, so $A$ gives strong information about $B$ . Never assume an event combination is independent without checking; physical setup determines it.

CounterexampleThe Gambler's Fallacy

Claim (false): "A fair coin has landed heads five times in a row, so tails is now 'due' and more likely on the next flip."

Reality: Successive coin flips are independent [9]. The coin has no memory. The probability of tails on flip six is exactly $1/2$ , regardless of history:

P(\text{tails on flip 6} \mid \text{HHHHH}) = P(\text{tails}) = \tfrac12.

What is rare is getting six heads in a row from the start: $P = (1/2)^6 = 1/64$ . But once five heads have already occurred, that part is no longer uncertain — only the sixth flip remains, and it is a fresh $1/2$ . Confusing the probability of a whole sequence with the probability of the next step is the gambler's fallacy.

connection

Probability rules sit at the centre of a web of related topics. They generalize from Set Theory, since events are sets and the addition rule is exactly the inclusion–exclusion principle for set sizes. Conditional Probability (equation [7]) extends the multiplication rule and leads directly to Bayes' Theorem, the engine of modern statistical inference and machine learning. Counting favourable outcomes for equation [4] requires Combinatorics (permutations and combinations). When repeated independent trials are involved, the multiplication rule [9] gives rise to the Binomial Distribution, and averaging outcomes connects to Expected Value. Together these form the foundation of Descriptive Statistics and Inferential Statistics.

🔬 Where this appears

⏳ History

- 1564 (publ. 1663): Gerolamo Cardano writes Liber de Ludo Aleae, the first systematic treatise on gambling odds, anticipating equation [4].

- 1654: Blaise Pascal and Pierre de Fermat exchange letters solving the "problem of points" (how to divide stakes in an interrupted game), founding mathematical probability.

- 1657: Christiaan Huygens publishes De Ratiociniis in Ludo Aleae, the first printed probability textbook, introducing expected value.

- 1713: Jacob Bernoulli's Ars Conjectandi proves the Law of Large Numbers and formalizes the multiplication rule for independent trials.

- 1763: Thomas Bayes' essay on inverse probability is published posthumously, giving the conditional-probability formula [7] its inferential power.

- 1812: Pierre-Simon Laplace's Théorie analytique des probabilités consolidates the classical theory and the equally-likely definition [4].

- 1900: David Hilbert's sixth problem calls for the axiomatization of probability and physics.

- 1933: Andrey Kolmogorov publishes the measure-theoretic axioms [1]–[3], placing all probability rules on a rigorous footing.

open-problem

The classical rules assume a well-defined sample space with assignable probabilities — but specifying these is not always straightforward. Three live issues:

What does probability mean? The frequentist view (probability = long-run relative frequency) and the Bayesian view (probability = degree of belief) give the same algebra [1]–[9] but disagree philosophically about single, non-repeatable events ("the probability it will rain tomorrow"). This interpretation debate remains unresolved.
Infinite and continuous sample spaces: When outcomes are uncountable (e.g., a randomly chosen real number in $[0,1]$ ), individual outcomes have probability zero, and equation [4] fails. One needs measure theory and probability density functions — beyond the equally-likely counting model.
Ill-posed problems: The Bertrand paradox shows that "choose a random chord of a circle" yields different probabilities depending on the (unstated) method of randomization, revealing that "at random" must always be precisely defined.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also