Worksheet 3: Conditional Probability and Bayes’ Theorem

Learning Objectives 🎯

  • Understand conditional probability and its notation

  • Create and interpret tree diagrams for multi-stage experiments

  • Apply the multiplication rule and law of total probability

  • Use Bayes’ theorem to update probabilities with new information

  • Implement probability calculations in R

Introduction

Conditional probability allows us to understand how the likelihood of one event changes when we have additional information about another event. It does not assume one event causes another. Instead, it refines our understanding of probabilities by narrowing down the possibilities based on what we know.

Examples:

  • If you know it is cloudy, the likelihood of rain may increase because clouds are often associated with rain

  • If you know the card drawn from a deck is a spade, it impacts the probability that the card is a face card

For example, imagine you are selecting a gummy bear from one of several jars. The probability of selecting a red gummy depends on which jar you chose. If you know the jar you picked, you can narrow down the probability of getting a red gummy. Conditional probability helps us formalize this reasoning.

We write the conditional probability of drawing a red gummy from the i-th jar as:

\[P(\text{Red Gummy}|\text{Jar}_i)\]

Here, the vertical bar | means “given that.” It indicates that we are finding the probability of the event drawing a Red Gummy, given that the jar selected is Jari.

Part 1: Understanding Conditional Probability

Question 1: To understand the concept of conditional probability, consider three jars filled with gummy bears:

  • Jar₁ contains 30 red, 10 green, and 10 blue gummies

  • Jar₂ contains 20 red and 40 green gummies

  • Jar₃ contains 35 yellow gummies

Answer the following questions using formal probability statements and clearly show your work:

  1. Suppose you were handed Jar₃ and you randomly sample a single gummy bear. Compute the probability that the gummy bear you sampled would be red.

  2. Suppose instead you were handed Jar₂ and you randomly sample a single gummy bear. Compute the probability that the gummy bear you sampled would be red.

  3. Suppose you were handed Jar₁ and you randomly sample a single gummy bear. Compute the probability that the gummy bear you sampled would be either red or blue.

  4. Instead suppose you were not allowed to see the contents of the selected jar but are aware of the distribution of colors in each jar. You randomly sample one gummy, and it is yellow. Determine the probability that the gummy came from each of the three jars (Jar₁, Jar₂, or Jar₃).

R Code for Visualization:

# Define jar contents
jar1 <- c(red = 30, green = 10, blue = 10)
jar2 <- c(red = 20, green = 40)
jar3 <- c(yellow = 35)

# Create a visualization of jar contents
library(ggplot2)

# Prepare data for plotting
jar_data <- data.frame(
  jar = c(rep("Jar 1", 3), rep("Jar 2", 2), "Jar 3"),
  color = c(names(jar1), names(jar2), names(jar3)),
  count = c(jar1, jar2, jar3)
)

# Create bar plot
ggplot(jar_data, aes(x = jar, y = count, fill = color)) +
  geom_bar(stat = "identity", position = "stack") +
  scale_fill_manual(values = c("red" = "red", "green" = "green",
                               "blue" = "blue", "yellow" = "gold")) +
  labs(title = "Gummy Bear Distribution in Jars",
       x = "Jar", y = "Number of Gummies") +
  theme_minimal()

Part 2: Tree Diagrams and Sequential Sampling

A tree diagram is a visual representation of the general multiplication rule. It illustrates all possible outcomes of a sequence of events, where each branch of the tree represents a possible event, and probabilities are assigned to these branches. By following the branches of the tree, we can calculate the probabilities of different outcomes by multiplying the conditional probabilities along the paths.

Tree diagrams also rely on the principles of mutual exclusivity and exhaustiveness:

  • Mutual exclusivity ensures that each complete path through the tree represents a distinct and non-overlapping outcome

  • Exhaustiveness ensures that all possible outcomes are represented exactly once in the tree, so the probabilities of the mutually exclusive outcomes add up to 1

Together, these two properties ensure that the tree diagram forms a partition of the sample space, dividing it into distinct and complete subsets of outcomes.

Question 2: Considering the same jars of gummy bears, suppose you randomly select one jar, with each jar being “equally likely” to be chosen:

\[P(\text{Jar}_1) = P(\text{Jar}_2) = P(\text{Jar}_3) = \frac{1}{3}\]

Next you sample two gummies from the selected jar without replacement.

  1. Create a tree diagram to represent all possible outcomes of sampling two gummies. The tree diagram should convey all probabilities clearly, showing:

    • Unconditional probabilities (e.g., the probability of selecting each jar)

    • Conditional probabilities (e.g., the probabilities of drawing specific gummy colors on the first and second draws, accounting for the reduced contents of the jar after the first draw)

    • Intersection probabilities (e.g., the probabilities at the end of each path, representing the probability of a specific sequence of events occurring)

  2. Logically explain why your tree diagram satisfies both mutual exclusivity (each complete path represents a distinct, non-overlapping outcome) and exhaustiveness (all possible outcomes of the experiment are included in the diagram, and their probabilities sum to 1).

  3. Using your tree diagram, compute the following probabilities (try to maintain mathematical formalism):

    1. The probability of drawing two yellow gummies

    2. The probability of drawing two blue gummies

    3. The probability of drawing two green gummies given that you know the samples came from Jar₁

    4. The probability of drawing two green gummies given that you know the samples came from Jar₂

    5. Use the law of total probability to determine the probability of sampling two green gummies and relate it to the paths of the tree diagram

R Code for Tree Diagram Calculations:

# Function to calculate probability of drawing two gummies of same color
# without replacement from a specific jar
prob_two_same <- function(jar_contents, color) {
  total <- sum(jar_contents)
  if (!(color %in% names(jar_contents))) return(0)

  n_color <- jar_contents[color]
  if (n_color < 2) return(0)

  # P(first is color) * P(second is color | first was color)
  prob <- (n_color / total) * ((n_color - 1) / (total - 1))
  return(prob)
}

# Calculate specific probabilities
# P(two yellow gummies | Jar 3)
p_yy_given_jar3 <- prob_two_same(jar3, "yellow")
cat("P(YY|Jar3) =", p_yy_given_jar3, "\n")

# P(two blue gummies | each jar)
p_bb_given_jar1 <- prob_two_same(jar1, "blue")
p_bb_given_jar2 <- prob_two_same(jar2, "blue")  # Will be 0
p_bb_given_jar3 <- prob_two_same(jar3, "blue")  # Will be 0

# Law of total probability for two blue gummies
p_bb_total <- (1/3) * p_bb_given_jar1 + (1/3) * p_bb_given_jar2 + (1/3) * p_bb_given_jar3
cat("P(BB) =", p_bb_total, "\n")

Part 3: Bayes’ Theorem and Sequential Updating

Bayes’ Theorem provides a way to compute conditional probabilities when directly calculating them is difficult. In particular, it allows us to express a probability in terms of its reverse conditional, which is often easier to determine. It helps answer questions like:

“Given an observed outcome, how should we determine the probability of an event that may have led to it?”

Beyond this basic use, Bayes’ Theorem also provides a framework for updating probabilities when additional evidence is observed, refining our estimates step by step.

“Given an initial observation, how should we revise our probabilities after receiving additional evidence?”

The first application focuses on computing a conditional probability when its direct computation is complex, while the second extends this concept to sequential updating, where each new observation refines our understanding of the underlying probabilities.

Question 3: To understand the concept of Bayes formula, consider the same three jars filled with gummy bears:

  • Jar₁ contains 30 red, 10 green, and 10 blue gummies

  • Jar₂ contains 20 red and 40 green gummies

  • Jar₃ contains 35 yellow gummies

Suppose you randomly select one jar, with each jar being “equally likely” to be chosen. Next, you draw one gummy bear without looking at the contents of the jar and observe that it is red.

  1. Compute the probability that the red gummy bear came from each jar (Jar₁, Jar₂, or Jar₃).

  2. Now assume you draw a second gummy bear from the same jar, and it is also red. Compute the updated probabilities that the jar you sampled from was from Jar₁, Jar₂, or Jar₃. Your tree diagram may be helpful in answering this question.

R Implementation of Bayes’ Theorem:

# Bayes' theorem implementation
bayes_update <- function(prior, likelihoods) {
  # Calculate posterior probabilities
  joint <- prior * likelihoods
  posterior <- joint / sum(joint)
  return(posterior)
}

# Initial setup
prior_probs <- c(1/3, 1/3, 1/3)  # P(Jar_i)

# Likelihoods of observing red from each jar
p_red_given_jar <- c(
  jar1["red"] / sum(jar1),      # P(Red|Jar1) = 30/50
  jar2["red"] / sum(jar2),      # P(Red|Jar2) = 20/60
  0                             # P(Red|Jar3) = 0/35
)

# First update: observed one red gummy
posterior_1 <- bayes_update(prior_probs, p_red_given_jar)
names(posterior_1) <- c("Jar1", "Jar2", "Jar3")

cat("After observing one red gummy:\n")
print(round(posterior_1, 4))

# For second update, calculate P(Red|Jar_i, first was red)
# This accounts for sampling without replacement
p_red2_given_jar_red1 <- c(
  (jar1["red"] - 1) / (sum(jar1) - 1),   # 29/49 for Jar1
  (jar2["red"] - 1) / (sum(jar2) - 1),   # 19/59 for Jar2
  0                                       # Still 0 for Jar3
)

# Second update: observed another red gummy
posterior_2 <- bayes_update(posterior_1, p_red2_given_jar_red1)

cat("\nAfter observing two red gummies:\n")
print(round(posterior_2, 4))

# Visualization of belief updating
update_data <- data.frame(
  jar = rep(c("Jar1", "Jar2", "Jar3"), 3),
  stage = rep(c("Prior", "After 1 Red", "After 2 Red"), each = 3),
  probability = c(prior_probs, posterior_1, posterior_2)
)

ggplot(update_data, aes(x = jar, y = probability, fill = stage)) +
  geom_bar(stat = "identity", position = "dodge") +
  labs(title = "Bayesian Updating with Sequential Observations",
       x = "Jar", y = "Probability", fill = "Stage") +
  theme_minimal() +
  ylim(0, 1)

Key Takeaways

Summary 📝

  • Conditional probability P(A|B) represents the probability of A given that B has occurred

  • Tree diagrams visualize multi-stage experiments and help calculate complex probabilities

  • The multiplication rule states that P(A ∩ B) = P(A) × P(B|A)

  • The law of total probability allows us to find P(B) by summing over all possible ways B can occur

  • Bayes’ theorem provides a way to reverse conditional probabilities: P(A|B) = P(B|A) × P(A) / P(B)

  • Bayes’ theorem can be applied sequentially to update beliefs as new evidence arrives