Chapter 4: Resampling Methods

Resampling methods represent one of the most profound shifts in statistical practice since the advent of electronic computing. Rather than relying on closed-form sampling distributions derived under idealized assumptions, resampling techniques use the observed data itself to approximate the variability of statistical estimators. The bootstrap, introduced by Bradley Efron in 1979, embodies a deceptively simple idea: treat the sample as a proxy for the population, resample from it repeatedly, and let the empirical distribution of recomputed statistics stand in for the unknown theoretical sampling distribution.

This chapter develops the complete theory and practice of resampling methods. We begin with the fundamental problem that motivates these techniques: the sampling distribution of a statistic is rarely available in closed form, and asymptotic approximations may be inadequate for finite samples or complex statistics. The plug-in principle provides the conceptual foundation—estimate the population distribution \(F\) with the empirical distribution \(\hat{F}_n\), then propagate this estimate through the statistic of interest. The nonparametric bootstrap operationalizes this idea via Monte Carlo simulation, generating thousands of resampled datasets to approximate standard errors, construct confidence intervals, and test hypotheses.

We examine multiple confidence interval constructions—percentile, basic, studentized, and the bias-corrected and accelerated (BCa) method—each with different theoretical properties and practical trade-offs. The jackknife, the bootstrap’s deterministic precursor, provides complementary tools for variance and bias estimation. For regression problems, we develop specialized resampling schemes: pairs bootstrap for heteroskedastic data, residual bootstrap under homoskedasticity, and wild bootstrap for preserving heteroskedasticity structure. Complex data structures—stratified samples, clustered observations, and time series—require careful adaptation of the resampling mechanism, from within-stratum resampling to block bootstrap methods that preserve temporal dependence.

The chapter also addresses the parametric bootstrap, which leverages model assumptions to improve efficiency when those assumptions hold, and develops the connection between bootstrap inference and permutation tests for hypothesis testing. We conclude with cross-validation methods that use resampling for prediction assessment and model selection, bridging to the machine learning applications in later chapters.

Throughout, we emphasize both mathematical rigor and computational practice. Every method is accompanied by complete Python implementations, diagnostic tools for detecting when resampling fails, and decision frameworks for selecting among alternative approaches. By the chapter’s end, you will command a versatile toolkit for distribution-free inference that complements the parametric methods of Chapter 3 and sets the stage for Bayesian computation in Chapter 5.

Learning Objectives: Upon completion of this chapter, students will be able to:

Foundational Understanding

  • Define the sampling distribution problem and explain why closed-form solutions are rarely available for complex statistics

  • Derive the plug-in principle from the empirical distribution function and prove the Glivenko-Cantelli uniform convergence theorem

  • Distinguish between statistical uncertainty (finite sample) and Monte Carlo uncertainty (finite resamples) in bootstrap inference

  • Analyze when asymptotic approximations fail and resampling methods provide superior finite-sample inference

Bootstrap Methods

  • Implement the nonparametric bootstrap algorithm for arbitrary statistics with proper seed management

  • Compute bootstrap standard errors and bias estimates with appropriate Monte Carlo sample sizes

  • Construct confidence intervals using percentile, basic, studentized, BC, and BCa methods

  • Derive the BCa adjustment formula and compute bias-correction \(z_0\) and acceleration \(a\) parameters

  • Evaluate coverage accuracy of different interval methods using Edgeworth expansion results

Jackknife Methods

  • Apply leave-one-out jackknife for standard error and bias estimation

  • Compute pseudo-values and understand their connection to influence functions

  • Compare jackknife and bootstrap for smooth versus non-smooth statistics

  • Identify when jackknife fails (non-smooth statistics) and appropriate alternatives

Regression and Complex Designs

  • Select among pairs, residual, and wild bootstrap based on design and heteroskedasticity

  • Implement stratified and cluster bootstrap for survey data with appropriate resampling units

  • Apply moving block, circular block, and stationary bootstrap for time series data

  • Choose block length using data-driven methods and assess sensitivity

Parametric and Testing Methods

  • Implement parametric bootstrap when model assumptions are justified

  • Evaluate trade-offs between parametric and nonparametric bootstrap under model misspecification

  • Construct bootstrap hypothesis tests by resampling under the null hypothesis

  • Compare bootstrap tests to permutation tests and identify when each is appropriate

Cross-Validation and Model Selection

  • Implement leave-one-out and K-fold cross-validation for prediction assessment

  • Design nested cross-validation schemes avoiding optimistic bias in model selection

  • Apply the .632 and .632+ bootstrap estimators for prediction error

  • Connect cross-validation to information criteria (AIC, BIC) for model comparison

Diagnostics and Practice

  • Diagnose bootstrap distribution pathologies including multimodality, heavy tails, and boundary effects

  • Select the number of bootstrap replicates \(B\) based on target precision and computational budget

  • Identify bootstrap failure modes (extreme statistics, boundary parameters, small samples) and apply remedies

  • Report resampling results with appropriate uncertainty quantification and reproducibility information

Sections