Slides 📊

4.2. Probability

Now that we’ve established the language of set theory, we can build upon this foundation to describe uncertainty using probability.

Road Map 🧭

Define probability as a function that maps events to numerical values representing likelihood.
Establish the axioms that make a probability measure well-defined.
Compare frequentist and Bayesian interpretations of probability.
Develop fundamental rules for calculating probabilities.

4.2.1. Probability as a Function

Probability is a function that maps events (sets) to real numbers in the interval \([0, 1]\):

\[P: \{\text{events}\} \rightarrow [0, 1].\]

For any event \(A\) in the sample space \(\Omega\), we denote its probability as \(P(A)\). The value of \(P(A)\) expresses how likely it is for \(A\) to occur.

4.2.2. Axioms of Probability

Not every function that maps events to numbers between 0 and 1 is a valid probability measure. To be considered a probability, the function must satisfy three fundamental axioms:

Axiom 1: Non-negativity

For any event \(A\) in the sample space \(\Omega\), its probability is always non-negative. That is,

\[P(A) \geq 0.\]

Axiom 2: Normalization

The probability of the sample space is 1.

\[P(\Omega) = 1\]

This axiom ensures that something from the sample space must occur when we perform our random experiment.

Axiom 3: Additivity

For any sequence of mutually exclusive events \(A_1, A_2, ...\) (that is, \(A_i \cap A_j = \emptyset\) for all \(i\neq j\)),

\[P\left(\bigcup_{i=1}^{n} A_i\right) = \sum_{i=1}^{n} P(A_i)\]

This axiom states that the probability of a union of mutually exclusive events equals the sum of their individual probabilities.

Additional properties

From these three axioms, we can derive several additional properties:

The probability of the empty set is zero. \(P(\emptyset) = 0\).
The probability of any event is at most 1. \(P(A) \leq 1\) for all events \(A\).
If \(A \subseteq B\), then \(P(A) \leq P(B)\).

You are encouraged to try proving these on your own.

Example 💡: Two Dice Probability

Suppose we roll a six-sided die followed by a four-sided die, and record the outcome as an ordered pair with the result from the six-sided die always listed first.

The sample space \(\Omega\) consists of all possible ordered pairs:

\[\Omega = \{(1,1), (1,2), (1,3), (1,4), (2,1), ..., (6,4)\}\]

There are 6 × 4 = 24 possible outcomes in total. Assuming the dice are fair, each outcome is equally likely with probability 1/24. Define \(A\) as the event that the sum equals 6, \(B\) as the event that the sum equals 10, and \(C\) as the event of rolling doubles.

\(A = \{(2,4), (3,3), (4,2), (5,1)\}\)
\(B = \{(6,4)\}\)
\(C = \{(1,1), (2,2), (3,3), (4,4)\}\)

Compute the probability of \(A\).

Using the third axiom of probability,

\[\begin{split}P(A) &= P((2,4)) + P((3,3)) + P((4,2)) + P((5,1)) \\ &= \frac{1}{24} + \frac{1}{24} + \frac{1}{24} + \frac{1}{24} = \frac{1}{6}\end{split}\]

Since all outcomes are equally likely, the probability of an event depends only on the number of outcomes it contains—also known as its cardinality. We denote the cardinality of a set \(A\) as \(|A|\).

\[\]
Compute \(P(A \cup C)\).

First, we find that the set \(A \cup C\) consists of \(\{(2,4), (3,3), (4,2), (5,1), (1,1), (2,2), (4,4)\}\). Then

\[P(A \cup C) = \frac{|A \cup B|}{|\Omega|} = \frac{7}{24}\]

4.2.3. Interpretations of Probability

There are different ways to interpret what probabilities are. The two major interpretations are the frequentist and Bayesian approaches.

Frequentist Interpretation

The frequentists define the probability of an event \(A\) as the relative frequency of its occurrence as the number of trials \(n\) goes to infinity. Mathematically,

\[P(A) = \lim_{n \to \infty} \frac{\text{Number of times A occurs}}{n}\]

This view approaches probability as an intrinsic property of the random process being studied. For example, the statement “a fair coin has 0.5 probability of landing heads” means that if we toss the coin infinitely many times, the proportion of heads would approach 0.5.

Bayesian Interpretation

The Bayesians view probability as a degree of belief that can be updated as new information becomes available. They always have a prior belief about an event’s probability which gets updated to form a posterior probability once new evidence emerges. In this perspective, probability represents a state of knowledge rather than an intrinsic property of the world.

While this course primarily uses the frequentist approach, some Bayesian concepts will appear later in the semester.

4.2.4. Basic Rules of Probability

Several rules help us calculate probabilities for complex events based on simpler ones.

A. Complement Rule

For any event A,

\[P(A') = 1 - P(A).\]

Why does this rule hold?

This rule follows from the Axioms of Probability. Recall that \(A \cup A' = \Omega\). Two equal events must have the same probability, so

\[P(A \cup A') = P(\Omega).\]

Since \(A\) and \(A'\) are disjoint, Axiom 3 says \(P(A \cup A') = P(A) + P(A')\). Also, Axiom 2 says \(P(\Omega)=1\). Using these, the equation above can be updated to

\[P(A) + P(A') = 1.\]

The complement rule is simply a rearrangement of this equation.

Diagram of the complement rule — Fig. 4.9 Graphical illustration of the complement rule

Example 💡: When is the Complement Rule Useful?

Continued from the first example of throwing two dice. Compute the probability that the sum of the two numbers is not equal to 10.

Approach 1: without using the complement rule

Name the event that the sum of the two numbers is not equal to 10.
List the elements in this event.
Add the probabilitities of each outcome in the event.

Approach 2: using the complement rule

Recognize that this event is the complement of \(B\). Therefore, we are essentially looking for \(P(B')\). Using the complement rule,

\[P(B') = 1-P(B) = 1 - \frac{1}{24} = \frac{23}{24}.\]

B. General Addition Rule

For any two events A and B,

\[P(A \cup B) = P(A) + P(B) - P(A \cap B).\]

Why does this rule hold?

Diagram of general addition rule — Fig. 4.10 Graphical illustration of the general addition rule

The key component of the general addition rule is the final subtraction of the intersection probability. If we simply add \(P(A)\) and \(P(B)\), we would count the outcomes in the intersection twice. Subtracting \(P(A ∩ B)\) corrects for this double-counting.

i. A Special Case: Mutually Exclusive Events

If \(A\) and \(B\) are mutually exclusive (\(A \cap B = \emptyset\)),

\[P(A \cup B) = P(A) + P(B).\]

This is a restatement of the third axiom of probability, but it can also be seen as a special case of the general addition rule. When the rule is applied to the disjoint events \(A\) and \(B\), the third term disappears because \(P(A \cap B) = P(\emptyset) = 0\).

Avoid the common mistake ‼️

Any special case formulas should only be used when the conditions for the special situation are fully met. When unsure, always begin with the general version.

ii. Extension to Multiple Events

This two-way general addition rule is a special case of the broader inclusion-exclusion principle, which provides a way to decompose the union probability of two or more events. For \(n\) events \(A_1, A_2, ..., A_n\), the inclusion-exclusion principle is constructed following the steps below:

Add the probabilities of individual events.

Subtract the probabilities of all pairwise intersections.

Add the probabilities of all triple intersections.

Subtract the probabilities of all quadruple intersections.

Continue this pattern, with the sign alternating for each step.

Applying the principle to three events \(A\), \(B\), and \(C\), the probability of their union is:

\[\begin{split}P(A \cup B \cup C) &= P(A) + P(B) + P(C) - P(A \cap B) - \\ &P(A \cap C) - P(B \cap C) + P(A \cap B \cap C)\end{split}\]

Example 💡: Two Dice Probability

Continued from the previous examples of throwing two dice.

Compute \(P(A\cup C)\) using the general addition rule, and confirm that the answer is the same as our previous approach without the rule.

\(A\cap C = \{(3,3)\}\). Using the rule and the fact that all outcomes are equally likely,

\[\begin{split}P(A\cup C) &= P(A) + P(C) - P(A\cap C) \\ &= \frac{|A|}{|\Omega|} + \frac{|C|}{|\Omega|} - \frac{|A\cap C|}{|\Omega|} \\ &= \frac{4}{24} + \frac{4}{24} - \frac{1}{24} = \frac{7}{24}.\end{split}\]
Compute the probability that the outcome is a double or the sum is equal to 10.

The probability can be written as \(P(B \cup C)\). Since the two events are mutually exclusive, we can use the special addition rule:

\[P(B \cup C) = P(B) + P(C) = \frac{1}{24} + \frac{4}{24} = \frac{5}{24}.\]

4.2.5. Bringing It All Together

Key Takeaways 📝

Probability is a function that maps events to values between 0 and 1, representing the likelihood of those events occurring.
A valid probability function must satisfy three axioms: non-negativity, normalization (sample space has probability 1), and additivity for mutually exclusive events.
The complement rule: \(P(A') = 1 - P(A)\). It isuseful for calculating probabilities of events defined by “at least one” or “none.”
The general addition rule: \(P(A \cup B) = P(A) + P(B) - P(A \cap B)\). It can be extended to cases with multiple events using inclusion-exclusion principle.

In the next chapter, we’ll build on these concepts to explore conditional probability and independence, which allow us to analyze how events influence each other.

4.2.6. Exercises

These exercises build your skills in applying probability axioms and rules to calculate probabilities of events.

Exercise 1: Equally Likely Outcomes

A quality control system randomly selects one circuit board from a batch for inspection. The batch contains boards labeled 1 through 20. Define the following events:

\(A\) = “Board number is a multiple of 3”
\(B\) = “Board number is greater than 12”
\(C\) = “Board number is odd”

Assuming each board is equally likely to be selected:

List the elements of \(A\), \(B\), and \(C\).
Calculate \(P(A)\), \(P(B)\), and \(P(C)\).
Find \(A \cap B\) and calculate \(P(A \cap B)\).
Calculate \(P(A \cup B)\) using the general addition rule.
Are events \(A\) and \(C\) mutually exclusive? Justify your answer and calculate \(P(A \cup C)\).

Exercise 2: Complement Rule Applications

A software testing team runs automated tests on a program. Each test can result in one of four outcomes: {Pass, Minor Bug, Major Bug, Crash}. Based on historical data:

\(P(\text{Pass}) = 0.72\)
\(P(\text{Minor Bug}) = 0.15\)
\(P(\text{Major Bug}) = 0.08\)
\(P(\text{Crash}) = 0.05\)

Verify that this is a valid probability distribution.
Calculate the probability that the test does NOT pass.
Let \(F\) = “test finds a defect” (Minor Bug, Major Bug, or Crash). Calculate \(P(F)\) using the complement rule.
Let \(S\) = “test reveals a serious issue” (Major Bug or Crash). Calculate \(P(S')\).
If 100 tests are run independently, approximately how many would you expect to find defects?

Exercise 3: General Addition Rule

In a computer science department survey of 200 students:

120 students know Python
85 students know Java
50 students know both Python and Java

Let \(P\) = “student knows Python” and \(J\) = “student knows Java.”

Calculate \(P(P)\), \(P(J)\), and \(P(P \cap J)\).
Calculate \(P(P \cup J)\) — the probability a randomly selected student knows at least one of the languages.
Calculate the probability that a student knows Python but NOT Java.
Calculate the probability that a student knows exactly one of the two languages.
Calculate the probability that a student knows neither language.

Exercise 4: Two Dice Problem

Two fair six-sided dice are rolled. Let the outcome be the ordered pair \((d_1, d_2)\) where \(d_1\) is the result of the first die and \(d_2\) is the result of the second die.

How many outcomes are in the sample space \(\Omega\)?
Define the event \(S_7\) = “the sum equals 7.” List the outcomes in \(S_7\) and calculate \(P(S_7)\).
Define the event \(D\) = “both dice show the same value” (doubles). List the outcomes in \(D\) and calculate \(P(D)\).
Are \(S_7\) and \(D\) mutually exclusive? Calculate \(P(S_7 \cup D)\).
Define \(E\) = “at least one die shows an even number.” Calculate \(P(E)\) using the complement rule.
Calculate the probability that the sum is at least 10.

Exercise 5: Inclusion-Exclusion for Three Events

A tech company surveyed 500 employees about which development tools they use:

280 use VS Code (\(V\))
220 use PyCharm (\(P\))
150 use Sublime (\(S\))
100 use both VS Code and PyCharm
70 use both VS Code and Sublime
50 use both PyCharm and Sublime
30 use all three tools

Calculate \(P(V)\), \(P(P)\), \(P(S)\), \(P(V \cap P)\), \(P(V \cap S)\), \(P(P \cap S)\), and \(P(V \cap P \cap S)\).
Using the inclusion-exclusion principle, calculate the probability that a randomly selected employee uses at least one of the three tools.
What is the probability that an employee uses none of these three tools?
What is the probability that an employee uses exactly one of the three tools?

Exercise 6: Verifying Probability Axioms

Determine whether each of the following could be a valid probability assignment. If not, identify which axiom is violated.

\(P(A) = 0.5, P(B) = 0.6, P(A \cap B) = 0.7\)
\(P(A) = 0.3, P(A') = 0.8\)
For sample space \(\Omega = \{1, 2, 3, 4\}\): \(P(\{1\}) = 0.2, P(\{2\}) = 0.3, P(\{3\}) = 0.3, P(\{4\}) = 0.2\)
\(P(A) = 1.2\) for some event \(A\)
For mutually exclusive events A and B: \(P(A) = 0.4, P(B) = 0.5, P(A \cup B) = 0.8\)
\(P(A) = 0.6, P(B) = 0.7, P(A \cup B) = 1.0\)

4.2.7. Additional Practice Problems

True/False Questions (1 point each)

If \(P(A) = 0.4\) and \(P(B) = 0.5\), then \(P(A \cup B)\) must equal 0.9.

Ⓣ or Ⓕ
The probability of any event is always between 0 and 1, inclusive.

Ⓣ or Ⓕ
If \(A\) and \(B\) are mutually exclusive, then \(P(A \cap B) = P(A) \cdot P(B)\).

Ⓣ or Ⓕ
For any event \(A\), \(P(A) + P(A') = 1\).

Ⓣ or Ⓕ
If \(A \subseteq B\), then \(P(A) \leq P(B)\).

Ⓣ or Ⓕ
The general addition rule \(P(A \cup B) = P(A) + P(B) - P(A \cap B)\) applies only when A and B are mutually exclusive.

Ⓣ or Ⓕ

Multiple Choice Questions (2 points each)

If \(P(A) = 0.35\) and \(P(B) = 0.60\), and A and B are mutually exclusive, what is \(P(A \cup B)\)?

Ⓐ 0.21

Ⓑ 0.95

Ⓒ 0.25

Ⓓ Cannot be determined
A fair six-sided die is rolled. What is the probability of rolling an even number or a number greater than 4?

Ⓐ 1/2

Ⓑ 2/3

Ⓒ 5/6

Ⓓ 1
In a batch of 100 items, 15 are defective. If one item is randomly selected, what is the probability that it is NOT defective?

Ⓐ 0.15

Ⓑ 0.75

Ⓒ 0.85

Ⓓ 0.90