5.2. Joint Probability Mass Functions

Many real-world scenarios involve multiple random quantities that interact with each other. To analyze such situations, we need to understand how random variables behave together. Joint probability mass functions provide the mathematical foundation for analyzing multiple discrete random variables simultaneously.

Road Map 🧭

• Define joint probability mass functions for multiple discrete random variables.

• Explore tabular and functional representations of joint PMFs.

• Understand how to derive marginal distributions from joint distributions.

• Identify when random variables are independent based on their joint PMF.

5.2.1. Joint Probability Mass Functions

When dealing with a single discrete random variable, we used a probability mass function to specify the probabilities associated with each possible value. We now extend this concept to multiple random variables.

Definition

The Joint Probability Mass Function (joint PMF) for two discrete random variables \(X\) and \(Y\) is denoted by \(p_{X,Y}\), and it gives the probability that \(X\) equals some value \(x\) and simultaneously \(Y\) equals some value \(y\):

\[p_{X,Y}(x,y) = P(\{X = x\} \cap \{Y = y\}).\]

Concisely, we also write \(p_{X,Y}(x,y) = P(X=x, Y=y).\)

This definition extends naturally to more than two random variables. For example, the joint PMF for three random variables \(X\), \(Y\), and \(Z\) would be denoted as \(p_{X,Y,Z}(x,y,z)\).

Support

The support of a joint PMF is the set of all pairs (x,y) for which the PMF assigns a positive probability:

\[\text{supp}(X,Y) = \{(x,y) \mid p_{X,Y}(x,y) > 0\}.\]

Representation of Joint PMFs

Joint probability mass functions can be represented in several ways:

Tabular Form

For two discrete random variables with finite supports, we can represent the joint PMF as a table. Each cell contains the probability that X = x and Y = y for the corresponding row and column values.

Example💡: Joint PMF in tabular Form

Consider rolling a fair four-sided die and a fair six-sided die which are indepedent. Let X represent the outcome of the four-sided die and Y represent the outcome of the six-sided die.

Joint PMF \(p_{X,Y}(x,y)\) for the fair 4-sided and 6-sided dice
\(x \backslash y\)	1	2	3	4	5	6
1	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)
2	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)
3	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)
4	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)

Since the dice are fair and independent, each combination has the same probability: 1/24 (because there are 4 × 6 = 24 possible outcomes).

Functional Form

For certain pairs of random variables, it is possible to express their joint PMF as a mathematical formula involving \(x\) and \(y.\)

Example💡: Joint PMF in functional form

For the dice example, we can express the joint PMF concisely as:

\[p_{X,Y}(x,y) = \frac{1}{24}\]

for \(x \in \{1, 2, 3, 4\}\) and \(y \in \{1, 2, 3, 4, 5, 6\}.\)

Validity of a Joint PMF

Like single-variable PMFs, joint PMFs must satisfy the follwing two axioms:

1. Non-negativity

   For all values of \(x\) and \(y\), \(0 \leq p_{X,Y}(x,y) \leq 1.\)

2. Total probability of 1

   The sum of all probabilities in the joint PMF equals 1:

\[\sum_{(x,y) \in \text{supp}(X,Y)} p_{X,Y}(x,y) = 1.\]

5.2.2. Marginal Distributions

Definition

A marginal PMF refers to the individual probability mass function of a random variable that forms part of a joint PMF with other variables.

Deriving Marginal Distributions from a Joint PMF

One of the most important operations we can perform with a joint PMF is to derive the marginal PMFs for each random variable.

To find the marginal PMF \(p_X(x)\), we sum the joint PMF for each fixed value \(x\) over all possible values of \(Y\):

\[p_X(x) = \sum_{y: p_Y(y) > 0} p_{X,Y}(x,y)\]

Similarly, to find the marginal PMF \(p_Y(y)\), we sum the joint PMF for each fixed value \(y\) over all possible values of \(X\):

\[p_Y(y) = \sum_{x: p_X(x) > 0} p_{X,Y}(x,y)\]

In tabular form, the marginal PMF values are computed as row-wise or column-wise sums of the joint PMF and are often recorded in the margins of the table–hence the name marginal PMF.

Example💡: Calculating marginal PMFs from a Joint PMF

Let’s calculate the marginal distributions for the independent fair dice example.

Marginal PMFs from the Joint PMF of for the fair 4-sided and 6-sided dice
\(x \backslash y\)	1	2	3	4	5	6	\(p_X(x)\)
1	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{6}{24}=\tfrac14\)
2	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{6}{24}=\tfrac14\)
3	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{6}{24}=\tfrac14\)
4	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{1}{24}\)	\(\tfrac{6}{24}=\tfrac14\)
\(p_Y(y)\)	\(\tfrac{4}{24} =\tfrac16\)	\(\tfrac{4}{24} =\tfrac16\)	\(\tfrac{4}{24} =\tfrac16\)	\(\tfrac{4}{24} =\tfrac16\)	\(\tfrac{4}{24} =\tfrac16\)	\(\tfrac{4}{24} =\tfrac16\)

5.2.3. Independence of Random Variables

Definition

Two discrete random variables \(X\) and \(Y\) are independent if and only if their joint PMF factors as the product of their marginal PMFs for all values in the support:

\[p_{X,Y}(x,y) = p_X(x) p_Y(y) \text{ for all } (x,y) \in \text{supp}(X,Y)\]

What does it mean?

Independence of random variables \(X\) and \(Y\) means that knowing the value of one provides no information about the value of the other.

With respect to the previously learned concept of independence of two events, this means that any event written in terms of \(X\) is independent of any event expressed in terms of \(Y\).

Example💡: Independence of two dice shown mathematically

In our dice example, \(X\) and \(Y\) are independent because

\[p_{X,Y}(x,y) = \frac{1}{24} = \frac{1}{4} \times \frac{1}{6} = p_X(x) \times p_Y(y)\]

for all values of \(x\) and \(y\) in the support.

Be cautious 🛑

Independence is an important property that often simplifies probability calculations. However, its convenient properties should only be used when the idependence of \(X\) and \(Y\) is provided or shown mathematically.

Example💡: When the Dice Constrain Each Other

So far we have relied on independence to keep our calculations simple. But real-world mechanisms often couple random quantities, forcing their outcomes to move together.

Two ordinary six-sided dice are altered so that any roll whose sum is less than 3 or greater than 9 is physically impossible.

Let \(X\) represent the outcome of the first die and \(Y\) the outcome of the second die. The rule \(3 \le X+Y \le 9\) prunes the sample space, but among the allowed pairs, every combination is still equally likely. The table below shows the pruned outcomes as ❌ as well as the probabilities of the possible pairs. Since there are 29 unpruned entries in the table, each possible pair \((x,y)\) gets \(p_{X,Y} (x,y)= 1/29.\)

Joint and marginal PMFs of two dice constraining each other
\(x \backslash y\)	1	2	3	4	5	6	\(p_X(x)\)
1	❌	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{5}{29}\)
2	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{6}{29}\)
3	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{6}{29}\)
4	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	❌	\(\tfrac{5}{29}\)
5	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	❌	❌	\(\tfrac{4}{29}\)
6	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	\(\tfrac{1}{29}\)	❌	❌	❌	\(\tfrac{3}{29}\)
\(p_Y(y)\)	\(\tfrac{5}{29}\)	\(\tfrac{6}{29}\)	\(\tfrac{6}{29}\)	\(\tfrac{5}{29}\)	\(\tfrac{4}{29}\)	\(\tfrac{3}{29}\)

Both dice are now biased toward lower numbers—a direct result of our sum constraint.

Let us now prove or disprove the independence of \(X\) and \(Y\). If we suspect that they are not independent, it suffices to show that the equation \(p_{X,Y}(x,y) = p_X(x)p_Y(y)\) fails for any single pair. Pick the pair \((x=6,\; y=1)\):

\[p_{X,Y}(6,1) \;=\; \frac{1}{29} \quad\text{but}\quad p_X(6)\,p_Y(1) \;=\; \frac{3}{29} \times \frac{5}{29} \;=\; \frac{15}{29^{2}} \;\neq\; \frac{1}{29}.\]

Since the requirement for independence is \(p_{X,Y}(x,y) = p_X(x)p_Y(y)\) for all possible pairs, \(X\) and \(Y\) has already failed the criterion. Therefore, they are dependent.

A joint distribution can encode constraints (a bounded sum in the previous example) that never appear in the marginals alone. Whenever the joint PMF doesn’t factor, dependence is at play.

5.2.4. Bringing It All Together

Key Takeaways 📝

1. A joint probability mass function specifies the probability of two or more discrete random variables taking on specific values simultaneously.

2. Joint PMFs must satisfy the basic probability axioms: non-negativity and summing to 1 over the entire support.

3. Marginal distributions can be derived from a joint PMF by summing over all possible values of the other variable(s).

4. Random variables are independent if and only if their joint PMF equals the product of their marginal PMFs for all values in the support.

5. When random variables are dependent, their joint distribution contains important information about how they relate to each other that isn’t captured by their marginal distributions alone.

Exercises

1. Marginal Distributions: Given the joint PMF below, find the marginal PMFs for X and Y.

Joint PMF \(p_{X,Y}(x,y)\)
\(x \backslash y\)	1	2	3
1	0.1	0.2	0.1
2	0.2	0.3	0.1

1. Conditional Probability: Using the joint PMF from problem 2, calculate:

   1. P(X = 1 | Y = 2)

   2. P(Y = 3 | X = 2)

2. Testing Independence: Determine whether the random variables X and Y with the following joint PMF are independent.

Joint PMF \(p_{X,Y}(x,y)\)
\(x \backslash y\)	0	1
0	0.3	0.2
1	0.1	0.4

1. Joint PMF Construction: Two fair six-sided dice are rolled. Let X be the minimum of the two values and Y be the maximum.

   1. Construct the joint PMF of X and Y.

   2. Are X and Y independent? Explain why or why not.