From the free course Introductory Statistics on Lykke

Probability Topics

Key concepts: Independent Events · Mutually Exclusive · Conditional Probability · Complement Rule

The mathematical study of uncertainty and the rules governing the likelihood of events.

Probability Topics

Probability is the mathematical language of uncertainty. While descriptive statistics allows us to summarize the past, probability provides the theoretical framework required for inferential statistics—the process of using sample data to make assertions about a population. In technical terms, probability is a measure of the likelihood that an event will occur, mapped onto a scale from 0 (impossibility) to 1 (certainty).

The Foundations: Sample Spaces and Events

To analyze any probabilistic system, we must first define the Sample Space ($S$), which is the set of all possible outcomes. An Event is any subset of that sample space. The probability of an event $A$, denoted $P(A)$, is the sum of the probabilities of the individual outcomes within that event.

The Axioms of Probability:

  1. For any event $A$, $0 \le P(A) \le 1$.
  2. The sum of probabilities of all possible outcomes in $S$ is exactly 1.
  3. The probability of an event occurring is the sum of the probabilities of the outcomes that comprise it.

The Complement Rule

The Complement of an event $A$ (denoted as $A'$ or $A^c$) consists of all outcomes in the sample space that are not in $A$. The Complement Rule is one of the most powerful shortcuts in probability, particularly when calculating "at least one" scenarios.

$$P(A) + P(A^c) = 1 \implies P(A^c) = 1 - P(A)$$

Concept Notation Definition Logical Equivalent
Event $A$ A specific outcome or set of outcomes. "It happens."
Complement $A^c$ All outcomes in $S$ not in $A$. "It does not happen."
Intersection $A \cap B$ Outcomes common to both $A$ and $B$. "Both $A$ AND $B$."
Union $A \cup B$ Outcomes in $A$, $B$, or both. "Either $A$ OR $B$."

Conditional Probability: The Logic of Updated Information

Conditional Probability is the probability that event $A$ occurs, given that event $B$ has already occurred. This is written as $P(A|B)$, read as "the probability of $A$ given $B$."

Mathematically, conditional probability effectively restricts the sample space. Instead of looking at the entire universe of possibilities $S$, we ignore everything outside of $B$.

The Formula

$$P(A|B) = \frac{P(A \cap B)}{P(B)}$$ Note: This is only defined if $P(B) > 0$.

Why It Matters

Conditional probability is the engine behind Bayesian inference, spam filters, and medical diagnostics. It allows us to update our beliefs as new data arrives. For example, the probability that a person has a specific disease ($A$) changes significantly once we know they have tested positive ($B$).


Independent Events vs. Mutually Exclusive Events

One of the most common points of confusion in statistics is the distinction between Independence and Mutual Exclusivity. While both describe relationships between events, they represent fundamentally different structural properties.

Independent Events

Two events $A$ and $B$ are Independent if the occurrence of one does not change the probability of the other. In other words, knowing that $B$ happened provides zero information about whether $A$ will happen.

Formal Tests for Independence: Two events are independent if and only if:

  1. $P(A|B) = P(A)$
  2. $P(B|A) = P(B)$
  3. $P(A \cap B) = P(A) \cdot P(B)$

Mutually Exclusive (Disjoint) Events

Two events are Mutually Exclusive if they cannot happen at the same time. If $A$ happens, $B$ cannot happen. Their intersection is an empty set.

Formal Test for Mutual Exclusivity: $$P(A \cap B) = 0$$

Comparison Table

Feature Independent Events Mutually Exclusive Events
Definition One event doesn't affect the other's likelihood. The events cannot occur simultaneously.
Intersection $P(A \cap B) = P(A)P(B)$ $P(A \cap B) = 0$
Relationship Information-based (Knowledge of $B$ is useless). Occurrence-based (Occurrence of $B$ precludes $A$).
Can they be both? No (unless one event has probability 0). No.
Visual Representation Overlapping areas in a Venn diagram. Non-overlapping circles in a Venn diagram.

The Multiplication and Addition Rules

These rules allow us to combine probabilities to find the likelihood of complex compound events.

1. The Multiplication Rule (The "AND" Rule)

Used to find the probability of the intersection $P(A \cap B)$.

  • General Rule: $P(A \cap B) = P(B) \cdot P(A|B)$
  • If Independent: $P(A \cap B) = P(A) \cdot P(B)$

2. The Addition Rule (The "OR" Rule)

Used to find the probability of the union $P(A \cup B)$.

  • General Rule: $P(A \cup B) = P(A) + P(B) - P(A \cap B)$
  • If Mutually Exclusive: $P(A \cup B) = P(A) + P(B)$

Mathematical Derivation of the General Addition Rule

The subtraction of $P(A \cap B)$ is necessary because the intersection is counted twice—once in $P(A)$ and once in $P(B)$.

\begin{aligned}
\text{Let } S &= \text{Sample Space} \\
P(A \cup B) &= \frac{|A \cup B|}{|S|} \\
\text{By Principle of Inclusion-Exclusion:} \\
|A \cup B| &= |A| + |B| - |A \cap B| \\
\implies \frac{|A \cup B|}{|S|} &= \frac{|A|}{|S|} + \frac{|B|}{|S|} - \frac{|A \cap B|}{|S|} \\
P(A \cup B) &= P(A) + P(B) - P(A \cap B)
\end{aligned}

Tools for Visualization and Calculation

When dealing with multiple events, raw formulas can become cumbersome. Professional statisticians use three primary tools to organize their logic.

1. Contingency Tables (Two-Way Tables)

These are ideal for analyzing the relationship between two categorical variables. They allow for easy calculation of marginal, joint, and conditional probabilities.

Example: Product Defects by Shift

Defective (D) Non-Defective (N) Total
Day Shift (S1) 15 485 500
Night Shift (S2) 25 475 500
Total 40 960 1000

From this table:

  • $P(D) = 40/1000 = 0.04$
  • $P(D|S2) = 25/500 = 0.05$
  • Since $P(D|S2) \neq P(D)$, the shift and defect rate are not independent.

2. Tree Diagrams

Tree diagrams are best for sequential events or multi-stage experiments where the outcome of the first stage affects the probabilities of the second.

3. Venn Diagrams

Venn diagrams are the gold standard for visualizing the logical relationship (unions and intersections) between 2 or 3 events.


Common Pitfalls and Misconceptions

The Gambler’s Fallacy

The mistaken belief that if an event happens more frequently than normal during a given period, it will happen less frequently in the future (or vice versa). This is a failure to recognize Independence. If you flip a fair coin 10 times and get 10 heads, the probability of tails on the 11th flip is still exactly 0.5.

Confusing "Independent" with "Mutually Exclusive"

As noted in the comparison table, these are nearly opposites. If two events are mutually exclusive (and have non-zero probability), they cannot be independent. Why? Because if $A$ occurs, the probability of $B$ immediately drops to 0. The occurrence of $A$ gave you perfect information about $B$, which is the definition of dependence.

The Base Rate Fallacy

Ignoring the overall probability of an event (the "base rate") when presented with specific evidence. This is a common error in medical testing interpretation, where people focus on the test's accuracy but ignore how rare the disease is in the general population.


Summary of Formulas

Rule Formula Context
Complement $P(A^c) = 1 - P(A)$ Finding "not A"
Addition (General) $P(A \cup B) = P(A) + P(B) - P(A \cap B)$ Finding "A or B"
Addition (M.E.) $P(A \cup B) = P(A) + P(B)$ Finding "A or B" when they can't overlap
Multiplication (General) $P(A \cap B) = P(B) \cdot P(A\vert B)$ Finding "A and B"
Multiplication (Indep.) $P(A \cap B) = P(A) \cdot P(B)$ Finding "A and B" when they don't affect each other
Conditional $P(A\vert B) = \frac{P(A \cap B)}{P(B)}$ Finding "A given B"
Probability Topics - Introductory Statistics - diagram 1
Probability Topics - Introductory Statistics - diagram 1
Probability Topics - Introductory Statistics - diagram 2
Probability Topics - Introductory Statistics - diagram 2
Probability Topics - Introductory Statistics - diagram 3
Probability Topics - Introductory Statistics - diagram 3

Source Materials

Study Introductory Statistics on Lykke

This course page is free to open: signed out, you can read its first two sections, and signing in, on any plan, opens the rest. The free Lykke Chrome extension syncs your own Canvas courses, so ChatGPT, Claude and Muse can answer from your classes and deadlines. It only reads Canvas.

Add the free extensionSign in to open every section

All sections of Introductory Statistics · Browse more course pages