13.3. Model Diagnostics and Statistical Inference

Having developed several point estimates related to simple linear regression, we now face the critical stage of uncertainty quantification using confidence regions and hypothesis tests. Before conducting any statistical inference, however, we must first verify that our model assumptions are reasonable.

Road Map 🧭

Understand why all assumption-checking procedures involve the residuals.
List the appropriate graphcial tools to assess each linear regression assumption, and systematically search for signs of assumption violation in each plot.
Perform inference on the overall model utility and on the model parameters.

Slides 📊

Download Chapter 13 slides (PPTX)

13.3.1. Preliminaries for Model Diagnostics

Review of Simple Linear Regression Assumptions

The simple linear regression model

(13.9)\[Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i\]

requires four key assumptions:

For each given $x_i$, $Y_i$ is a size-1 simple random sample from the distribution of all possible responses, $Y|X=x_i$.
The association between the explanatory and response variables is linear on average.
The error terms are normally distributed: $\varepsilon_i \stackrel{iid}{\sim} N(0, \sigma^2) \quad \text{for } i = 1, 2, \ldots, n.$
The error terms have constant variance $\sigma^2$ across all values of $X$.

For detailed discussion on the assumptions, revisit Section 11.5.2.

Overview of Residuals and Their Properties

Note that $\varepsilon_i$’s are the only random component of the current linear regression model, and therefore, all model assumptions are essentially assumptions about the behavior of the error terms.

Also recall the mathematical definition of residuals:

\[e_i = y_i - \hat{y}_i, \text{ for each } i = 1,2,\cdots, n.\]

Residuals can be viewed as the observed counterparts of the true error terms $\varepsilon_i$ because:

\[\varepsilon_i = y_i - (\beta_0 + \beta_1 x_i) \approx y_i - (\hat{\beta}_0 + \hat{\beta}_1 x_i) = y_i - \hat{y}_i = e_i.\]

Since we do not have access to the true realizations of $\varepsilon_i$, the residuals instead play a key role in model diagnostics.

13.3.2. Scatter Plot and Residual Plot for Linearity and Constant Variance

We have used scatter plots for the preliminary assessment of the association between two quantitative variables. We introduce the residual plot as an additional tool for model diagnosis; it is simply a scatter plot of $(x_i, e_i)$ for $i=1,\cdots, n$.

Fig. 13.17 Comparison of scatter plot and residual plot

A residual plot can be viewed as the original scatter plot rotated so that the regression line becomes horizontal. This removes the visual bias of the tilted trend line and highlights the deviations from the fitted line.

Scatter plots and residual plots are typically used together to assess two assumptions: linearity and constant variance.

Characteristics of Ideal Scatter and Residual Plots

When the linearity and constant variance assumptions hold, the points on both the scatter plot and residual plot form a random scatter around their respective summary lines, with a roughly constant spread across the $x$-axis.

Fig. 13.18 A residual plot showing no signs of assumption violation

Signs of Assumption Violation on Scatter and Resiual Plots

1. Violation of the Linearity Assumption

If residuals systematically fall below the zero line in certain regions of $x$ values and above the line in others, this suggests the true relationship is non-linear.

Fig. 13.19 Resiual plots with linearity assumption violated

2. Violation of the Constant Variance Assumption

If the spread of points around the trend line is inconsistent, the data likely violate the constant variance assumption.

Fig. 13.20 Resiual plots with constant variance assumption violated

Hourglass Pattern: The spread is larger for extreme values of $X$ than in the middle range.
Cone Pattern: As $X$ increases, the residual errors become larger.

Other patterns of non-constant variance can also occur. Even without systematic patterns, an inconsistent spread alone is a significant sign of assumption violation.

3. Outliers

Recall that scatter plots are also used to identify potential outliers and influential points. Residual plots can serve the same purpose.

Fig. 13.21 Resiual plots with outliers; left plot shows a $y$-outlier, and right shows an $x$-outlier that potentially has high influence on the model.

Violations May or May not Occur Simultaneously

While Fig. 13.19 show signs of non-linearity, it does not violate the constant variance assumption since the bandwidth of data points around their true summary curve (quadratic association) is reasonably constant.

Likewise, Fig. 13.20 shows no sign of non-linearity, as the patterns of spread above and below the zero line are roughly symmetric.

Be aware that different types of violations may occur simultaneously or separately, and be prepared to distinguish one case from another on the graphs.

Exercise: draw a residual plot that shows violation of lineary and constant variance assumptions simultaneously.

13.3.3. Histogram and Normal Probability Plot of Residuals for Normality

To verify normality of the error terms, we construct a histogram and a normal probability plot of the residuals.

These plots are used in the same way as in any prior context of normality assessment. We check whether the points follow a straight line on the normal probability plot and whether the histogram is bell-shaped.

13.3.4. Summary of Diagnostic Tools and Comprehensive Eaxmples

We have introduced four graphical tools to assess the assumptions of linear regression. See the table below for a summary:

Assumption	Graphical Tools
SRS	None (must be ensured through experimental design)
Linearity	Scatter plot and residual plot
Constant variance	Scatter plot and residual plot
Normality of errors	Histogram and QQ plot of residuals

Example 💡: Comprehensive Diagnostic Exercise

For each of the following sets of graphical representations, assess whether the data set meets the assumptions required for valid linear regression.

Example 1

First example of comprehensive model diagnostics — Fig. 13.23 Example 1 of graphical diagnosis

Scatter plot and residual plot show a systematic curved pattern.
If the correct summary curve is used (quadratic form), then the size of their vertical spread would not vary by the $x$-values. So non-constant variance is not a concern.
Histogram and normal probability plots look reasonable.

This data set violates the linearity assumption.

Example 2

Second example of comprehensive model diagnostics — Fig. 13.24 Example 2 of graphical diagnosis

The dots follow a linear pattern according to both the scatter plot and the residual plot—they form mirror images below and above their summary lines.
The spread of the points are narrower in the two ends than in the middle region. This raises a concern for non-constant variance.
The histogram and the QQ plot indicate that the error distribution has heavier tails than the blue curve. The error distribution may not be normal.

This data set violates the constant-variance assumption and is suspected to have non-normal error terms.

When Assumptions Are Violated

If diagnostic procedures reveal serious assumption violations, we should not proceed with linear regression analysis. Instead, we should take an appropriate measure to mitigate the effect of the violation or use a different analysis method.

Linearity Violations:

Consider transformations of variables (log, square root, etc.)
Fit non-linear models (beyond course scope)
Use piecewise or segmented regression for different regions

Constant Variance Violations:

Variable transformations may help stabilize variance
Weighted least squares methods (beyond course scope)
Robust regression techniques

Normality Violations:

Often less critical for large sample sizes due to Central Limit Theorem
Bootstrap methods for inference (beyond course scope)
Non-parametric alternatives

Independence Violations:

Time series methods for temporally correlated observations
Mixed effects models for clustered data

13.3.5. The F-Test for Model Utility

Once we have verified that our model assumptions are reasonably satisfied, we proceed with statistical inference. The first question we typically ask is: “Does the simple linear regression model provide useful information about the relationship between the explanatory and response variables?”

We organize our answer to this question into the usual four-step hypothesis test. We refer to the ANOVA table constructed in Chapter 13.2.

Step 1: Parameter of Interest

The focus of this test is on the overall usefulness of the linear relationship rather than on specific parameters. Therefore, it is okay to skip explicit parameter definition.

Step 2: Hypotheses

$H_0:$ There is no linear association between $X$ and $Y$.
$H_0:$ There is a linear association between $X$ and $Y$.

Important: Always state the hypotheses using the experimental context, replacing “X” and “Y” with the actual variable names and providing sufficient background information.

Step 3: Test Statistic and P-value

Recall that MSR is a scaled measurement of the variability attributed to the model structure, while MSE is a scaled measure of random error. Therefore, if MSR is substantially greater than MSE, we can consider the model to be statistically significant.

The corresponding test statistic is:

\[F_{TS} = \frac{\text{MSR}}{\text{MSE}},\]

with degrees of freedom $df_1 = 1$ and $df_2 = n-2$.

A large observed value of $F_{TS}$ is in favor of the alternative hypothesis that the model is significant. Therefore,

\[\text{p-value} = P(F_{1, n-2} > f_{TS}).\]

Step 4: Decision and Conclusion

If p-value ≤ $\alpha$:

Reject $H_0$. At the $\alpha$ significance level, we have sufficient evidence to conclude that there is a linear association between [explanatory variable] and [response variable] in [context].”
If p-value > $\alpha$:

Fail to reject $H_0$. At the $\alpha$ significance level, we do not have sufficient evidence to conclude that there is a linear association between [explanatory variable] and [response variable] in [context].”

Example 💡: Blood Pressure Study, Continued

Assuming that the assumptions of linear regression have been verified, perform the $F$-test for model utility using $\alpha = 0.05.$ Use the partially filled ANOVA table from the previous lesson:

Source	df	Sum of Squares	Mean Square	F-statistic	p-value
Regression	1	$555.7126$	$555.7126$	?	?
Error	$9$	$382.8267$	$42.5363$
Total	$10$	$938.5393$

Step 1: Parameter Definition

This step is skipped for model utility tests.

Step 2: Hypotheses:

$H_0$: There is no linear association between patient age and change in blood pressure.
$H_a$: There is a linear association between patient age and change in blood pressure.

Step 3: Computation

\[f_{TS} = \frac{\text{MSR}}{\text{MSE}} = \frac{555.7126}{42.5363} = 13.06\]

The test statistic has two degrees of freedom: $df_1=1$ and $df_2=9$. The $p$-value is:

\[P(F_{1,9} > 13.06) = 0.0055.\]

Step 4: Conclusion

At $\alpha = 0.05$, we reject $H_0$ and conclude that there is sufficient evidence of a linear association between patient age and change in blood pressure.

13.3.6. Distributional Properties of the Slope and Intercept Estimators

To develop inference procedures for $\beta_0$ and $\beta_1$, we need to understand the statistical properties of their estimators, $b_0$ and $b_1$. This draws distinction from our focus on $b_0$ and $b_1$ as estimates so far, yielding one set of realized values based on a single data. We would now like to study their behaviors across many different datasets.

1. Slope and Intecept Estimators Are Linear Combinations of the Responses

The first step in constructing inference for regression parameters is to recognize that their estimators are linear combinations of the response variables, $Y_i, i=1,\cdots, n$. Capital $Y$ is used throughout this section to emphasize that we are discussing responses as random variables, not their observed values. The values of the explanatory variable are still considered given and fixed.

We first show that the slope estimator $b_1$ is a linear combination of $Y_i, i=1,\cdots, n$, starting with its definition:

(13.10)\[\begin{split}b_1 &= \frac{\sum_{i=1}^n x_iY_i - n\bar{x}\bar{Y}}{S_{XX}} \\ &= \frac{\sum_{i=1}^n x_iY_i - \sum_{i=1}^n\bar{x}Y_i}{S_{XX}}\\ &= \sum_{i=1}^n \left(\frac{x_i - \bar{x}}{S_{XX}}\right) Y_i\end{split}\]

Each coefficient to the responses above only consists of non-random quantities involving the explanatory variable.

For the intercept estimator, we borrow the result of Eq. (13.10).

\[\begin{split}b_0 &= \bar{Y} -b_1\bar{x} \\ &= \sum_{i=1}^n \frac{1}{n}Y_i - \bar{x}\sum_{i=1}^n \left(\frac{x_i - \bar{x}}{S_{XX}}\right) Y_i\\ &= \sum_{i=1}^n \left(\frac{1}{n} - \frac{\bar{x}}{S_{XX}}(x_i - \bar{x})\right) Y_i\\\end{split}\]

Again, the coefficient of each term above consists of non-random quantities involving only the sample size and the explanatory values.

‼️ Key Observation

Recall that each $Y_i$ is normally disributed given an observed $x_i$. Since both $b_1$ and $b_0$ are linear combinations of normal random variables, they must also be normally distributed. We now proceed to state their expectations and variances.

2. The Expectation and Variance of the Parameter Estimators

The Slope Estimator

\[\begin{split}&E[b_1] = \beta_1\\ &\text{Var}(b_1) = \frac{\sigma^2}{S_{XX}}\end{split}\]

The Intercept Estimator

\[\begin{split}&E[b_0] = \beta_0\\ &\text{Var}(b_0) = \sigma^2\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{XX}}\right)\end{split}\]

Key Insights:

Both $b_1$ and $b_0$ are unbiased estimators of $\beta_1$ and $\beta_0$, respectively.
The slope variance depends only on the error variance and the spread of the explanatory values.
The intercept variance includes additional uncertainty when $\bar{x} \neq 0$.

Derive the Results as an Independent Exercise

The expectations and variances of both estimators can be derived using general properties and some algebraic manipulation. You are encouraged to verify these results as an independent exercise.

3. The Complete Distribution of the Estimators

Provided that all model assumptions hold,

\[b_1 \sim N\left(\beta_1, \frac{\sigma^2}{S_{XX}}\right)\]

\[b_0 \sim N\left(\beta_0, \sigma^2 \left(\frac{1}{n} + \frac{\bar{x}^2}{S_{XX}}\right)\right)\]

4. Estimated Standard Errors and t-Distribution

Since $\sigma^2$ is unknown, we replace it with its estimate $s^2 = MSE$ to obtain the estimated standard errors:

\[\widehat{SE}(b_1) = \sqrt{\frac{MSE}{S_{XX}}}\]

\[\widehat{SE}(b_0) = \sqrt{MSE\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{XX}}\right)}\]

Studentization of each estimator then gives us a $t$-distributed random variable, both with $df=n-2$:

\[\frac{b_1 - \beta_1}{\widehat{SE}(b_1)} \sim t_{n-2}\]

\[\frac{b_0 - \beta_0}{\widehat{SE}(b_0)} \sim t_{n-2}\]

We use these as the foundation for confidence regions and hypothesis tests on the true values of the regression parameters, $\beta_1$ and $\beta_0$.

13.3.7. Confidence Regions for Parameters

Recall the general form of a $t$-confidence region:

\[\text{estimate} \pm t_{crit} \widehat{SE}(\text{estimate})\]

The table below applies the general form to all possible parameter-side combination:

Side	$\beta_0$	$\beta_1$
CI	\[b_0 \pm t_{\alpha/2, n-2}\sqrt{MSE\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{XX}}\right)}\]	\[b_1 \pm t_{\alpha/2, n-2} \sqrt{\frac{MSE}{S_{XX}}}\]
UCB	\[b_0 + t_{\alpha, n-2}\sqrt{MSE\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{XX}}\right)}\]	\[b_1 + t_{\alpha, n-2} \sqrt{\frac{MSE}{S_{XX}}}\]
LCB	\[b_0 - t_{\alpha, n-2}\sqrt{MSE\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{XX}}\right)}\]	\[b_1 - t_{\alpha, n-2} \sqrt{\frac{MSE}{S_{XX}}}\]

13.3.8. Hypothesis Testing for the Slope Parameter

Step 1: Parameter of Interest

Define $\beta_1$ as the slope of the true regression line relating [explanatory variable] to [response variable]. Make sure to use experiment-specific variable names and context.

Step 2: Hypotheses

It is possible to construct a one-sided or two-sided hypothesis test for the slope parameter against any null value $\beta_{10}$.

\[\begin{split}&H_0: \beta_{1} (=, \leq, \geq) \beta_{10}\\ &H_1: \beta_{1} (\neq, >, <) \beta_{10}\end{split}\]

The most common hypothesis is the two-sided variant with the null value $\beta_{10} = 0$.

\[\begin{split}&H_0: \beta_1 = 0\\ &H_a: \beta_1 \neq 0\end{split}\]

Step 3: Test Statistic and P-value

\[T_{ts} = \frac{b_1 - \beta_{10}}{\widehat{SE}(b_1)}\]

has $t$-distribution with $df = n-2$. Once an observed value $t_{TS}$ is obtained, the $p$-value computation should align with the sidedness of the hypothesis.

Two-sided: $p$-value $= 2P(T_{n-2} > |t_{TS}|)$
Upper-tailed: $p$-value $= P(T_{n-2} > t_{TS})$
Lower-tailed: $p$-value $= P(T_{n-2} < t_{TS})$

Step 4: Decision and Conclusion

Compare p-value to $\alpha$ and draw conclusions about the slope parameter in context.

Hypothesis Tests for the Intercept

Hypothesis tests on $\beta_0$ can be constructed in a similar manner. They are not explored in detail because:

The procedure only differs by the form of the standard error estimate, and
The intercept often does not have any practial significance.

Example 💡: Blood Pressure Study, Continued

For the blood pressure vs age data set, perform a hypothesis test to determine if the slope parameter is non-zero. Then, construct a corresponding confidence region.

Step 1: Define the parameter of interest

We are interested in $\beta_1$, the true change in blood pressure per year increase in patient age.

Step 2: State the hypotheses

\[\begin{split}&H_0: \beta_1 = 0\\ &H_a: \beta_1 \neq 0\end{split}\]

Step 3: Computation

The test statistic is:

\[t_{TS} = \frac{b_1-0}{\sqrt{\text{MSE}/S_{xx}}}\frac{-0.526 - 0}{0.146} = -3.61.\]

The $p$-value is:

\[2P(T_9 > 3.61) = 0.0055.\]

Step 4: Conclusion

$p$-value $=0.0055 < \alpha$. The null hypothesis is rejected. At $\alpha=0.05$, there is enough evidence to conclude that the slope of the true linear association between age and change in blood pressure is different than $0$.

95% Confidence Interval:

The confidence region corresponding to a two-sided hypothesis test is a confidence interval. Use R to compute the critical value:

qt(0.025, df=9, lower.tail=FALSE)
#2.262

Then putting the components together,

\[-0.526 \pm (2.262)(0.146) = -0.526 \pm 0.330 = (-0.856, -0.196).\]

We are 95% confident that each additional year of age is associated with a decrease in blood pressure between 0.196 and 0.856 mm Hg after treatment.

13.3.9. Special Case: Equivalence of F-test and t-test

For simple linear regresssion, the hypothesis test on $H_a: \beta_1 \neq 0$ is equivalent to the $F$-test for model utility. Intuitively, this makes sense because, in order for the model to meaningfully describe the response variable, the explanatory variable must have a significant association with it.

Mathematically, this equivalence is explained through the special equality:

\[T_{ts}^2 = F_{TS}\]

To see why this is true, recall that $b_1 = \frac{S_{XY}}{S_{XX}}$ and $\text{MSR} = b_1\text{S}_{XY}$. Using these,

\[T_{ts}^2 = \frac{b_1^2}{\text{MSE}/S_{XX}} = \frac{b_1 \frac{S_{XY}}{S_{XX}} S_{XX}}{MSE} = \frac{MSR}{MSE}\]

This implies that both tests provide identical p-values and conclusions.

The Equivalence Only Holds for Simple LR

The two tests serve different purposes in general linear regression with multiple predictors:

The $F$-test assesses whether at least one of the predictors provide useful information about the response variable.
The $t$-test on a slope parameter assesses whether the single corresponding predictor is useful.

Example 💡: Blood Pressure Study, Continued

For the blood pressure dataset, verify the equivalence of the model utility $F$-test and the two-tailed $t$-test on the slope parameter against $\beta_{10}=0$.

From the two previous examples,

$t_{TS}^2 = (-3.61)^2 = 13.03$
$f_{TS}=13.06$
There is a small difference due to rounding, but both tests give $p$-value $= 0.0055$. ✔

13.3.10. Bring It All Tgether

Key Takeaways 📝

Diagnostic plots are essential for verifying model assumptions before conducting hypothesis tests or constructing confidence intervals.
Four key diagnostic tools work together: scatter plots, residual plots, histograms of residuals, and QQ plots provide complementary information about different aspects of model adequacy.
The model utility F-test assesses whether the linear relationship explains a significant portion of the variability in the response variable.
Inference on the slope and intercept adapts familiar principles used in previous inference procedures on single parameters.

In simple linear regression, the model utility F-test is equivalent to the $t$-test on $H_a: \beta_1 \neq 0$.

Exercises

Diagnostic Interpretation: For each residual plot pattern described, identify the assumption violation and potential consequences:
1. Residuals form a cone shape, spreading out as X increases.
2. Residuals show a clear curved (U-shaped) pattern.
3. Residuals appear randomly scattered around zero with constant spread.
4. Most residuals are near zero with a few extremely large positive and negative values.
5. Residuals show alternating positive and negative values in sequence.
ANOVA Table Completion: Given the following partial ANOVA table for a regression with n = 15, complete all missing values:

Source

df

Sum of Squares

Mean Square

F-statistic

Regression

?

240

?

?

Error

?

?

18

Total

?

474
Parameter Inference: A study of house prices yields the regression equation

\[\text{Price} = 45,000 + 120(\text{Size}),\]

where Price is in dollars and Size is in square feet. With
- $n = 20$
- $MSE = 50,000,000$, and
- $S_{XX} = 2500$,
1. Calculate the standard error of the slope.
2. Construct a 95% confidence interval for the slope.
3. Test $H_0: β_1 = 100$ vs $H_a: β_1 \neq 100$ at $\alpha = 0.05$.
4. Interpret the slope coefficient and its interval in context.
F-test vs t-test: Using the house price data from Exercise 3:
1. Conduct the $F$-test for model utility.
2. Conduct the $t$-test for $H_0: β_1 = 0$ vs $H_a: β_1 \neq 0$.
3. Verify that $f_{TS} = t^2_{TS}$ and explain why this relationship holds.
4. Discuss when you might prefer one test over the other.
Critical Evaluation: A researcher reports: “The regression has $R^2 = 0.95$, so the model is excellent and all assumptions are satisfied.”
1. What’s wrong with this reasoning?
2. What additional information would you need to evaluate the model?
3. Describe how a high $R^2$ could coexist with serious assumption violations.
4. What would you recommend the researcher do?
Design Considerations: Explain how each factor affects the precision of slope estimation:
1. Increasing the sample size $n$
2. Increasing the range of $X$ values observed
3. Reducing the error variance $σ^2s$
Confidence Interval Interpretation: For each confidence interval interpretation, identify whether it’s correct or incorrect and explain:
1. “There’s a 95% chance that the true slope lies in this interval.”
2. “95% of sample slopes will fall in this interval.”
3. “If we repeated this study many times, 95% of the intervals would contain the true slope.”
4. “We’re 95% confident about the slope value for this specific dataset.”
Hypothesis Testing Scenarios: For each research scenario, formulate appropriate hypotheses:
1. Is there any linear relationship between study hours and test scores?
2. Is the slope of salary vs. experience at least $2000 per year?
3. Is the relationship between temperature and ice cream sales negative?
4. Is the effect of fertilizer on plant growth different than 5 cm per gram?
Comprehensive Case Study: A medical researcher studies the relationship between patient age ($X$) and recovery time in days ($Y$) for a surgical procedure. With $n = 25$ patients, the analysis yields:
- Fitted model: $Ŷ = 8.5 + 0.3X$
- $\text{SSR} = 156$, $\text{SSE} = 234$, $\text{SST} = 390$
- $S_{XX} = 1200$
1. Discuss what diagnostic plots you would need to see.
2. Provide the complete ANOVA table and $R^2$.
3. Perform the $F$-test for model utility.
4. Compute the 95% confidence interval for the slope.
5. Test whether the slope exceeds $0.25$ days per year.
6. Provide a practical interpretation of all results.