.. _chapter3:

Chapter 3: Parametric Inference and Likelihood Methods
======================================================

.. contents:: Chapter Contents
   :local:
   :depth: 2

This chapter marks a fundamental shift in perspective: from **generating** random samples (Chapter 2) to **learning** from observed data. Where Monte Carlo methods asked "given a distribution, how do we simulate from it?", parametric inference asks the inverse question: "given data, what distribution generated it?" This reversal—from known model to unknown parameters—is the central problem of statistical inference.

We adopt the frequentist framework, treating parameters as fixed but unknown quantities with randomness arising solely from the sampling process. An estimator's quality is judged by its behavior across hypothetical repeated samples from the same population. This perspective leads to concepts like bias, variance, consistency, and efficiency—properties that characterize how estimators perform "on average" over many realizations of the sampling mechanism.

The chapter opens with **exponential families**, a unifying framework that encompasses most distributions encountered in practice. Understanding this structure reveals why certain distributions admit elegant sufficient statistics, why maximum likelihood estimation takes a particularly simple form, and why generalized linear models work. We then develop **maximum likelihood estimation**—the workhorse of parametric inference—with both analytical solutions and numerical optimization algorithms. The theory of **sampling variability** formalizes how estimators behave across repeated samples, leading to standard errors, confidence intervals, and hypothesis tests.

The **plug-in principle** provides a simple but powerful approach: estimate population quantities by substituting sample analogs. This idea, formalized through empirical distribution functions and influence functions, connects classical estimation to the bootstrap methods of Chapter 4. We then turn to **linear models**, developing least squares estimation, the Gauss-Markov theorem, and comprehensive diagnostics. The chapter culminates with **generalized linear models**, which extend regression to non-normal responses—logistic regression for binary outcomes, Poisson regression for counts, and beyond.

Throughout, we emphasize the computational aspects of inference: numerical optimization for MLEs, matrix computations for linear models, and iteratively reweighted least squares for GLMs. The theory developed here provides the foundation for Bayesian inference (Chapter 5), where parameters themselves become random variables with prior distributions updated by data.

**Learning Objectives:** Upon completion of this chapter, students will be able to:

**Exponential Families and Likelihood**

* **Recognize** exponential family distributions and convert standard parameterizations to canonical form
* **Extract** moments from the log-partition function using differentiation rather than integration
* **Apply** the Neyman-Fisher factorization theorem to identify sufficient statistics
* **Construct** conjugate priors for exponential family likelihoods

**Maximum Likelihood Estimation**

* **Derive** maximum likelihood estimators analytically for common distributions
* **Implement** numerical MLE via Newton-Raphson, Fisher scoring, and gradient-based optimization
* **State** and apply asymptotic properties: consistency, normality, efficiency, and invariance
* **Construct** likelihood ratio, Wald, and score tests for parametric hypotheses

**Sampling Variability and Estimator Properties**

* **Analyze** estimator properties including bias, variance, mean squared error, and consistency
* **Apply** the delta method to derive standard errors for transformed parameters
* **Distinguish** exact sampling distributions from asymptotic approximations
* **Implement** robust standard errors when model assumptions may be violated

**Plug-in Methods**

* **Apply** the plug-in principle to estimate population functionals
* **State** the Glivenko-Cantelli theorem and its implications for empirical distribution functions
* **Compute** influence functions for common statistics and use them for variance estimation
* **Connect** plug-in estimation to bootstrap methodology (developed in Chapter 4)

**Linear Models**

* **Derive** ordinary least squares estimators via calculus and projection geometry
* **State** and prove the Gauss-Markov theorem establishing OLS as BLUE
* **Implement** comprehensive residual diagnostics and influence measures
* **Apply** robust standard errors (HC0-HC3) when homoskedasticity fails
* **Use** regularization methods (ridge, LASSO, elastic net) for high-dimensional settings

**Generalized Linear Models**

* **Specify** GLMs through their three components: random, systematic, and link function
* **Implement** the IRLS algorithm as Fisher scoring for exponential family responses
* **Diagnose** model fit using deviance residuals, overdispersion tests, and goodness-of-fit measures
* **Handle** special cases including separation in logistic regression and overdispersion in count models

.. toctree::
   :maxdepth: 2
   :caption: Sections

   ch3_1-exponential-families
   ch3_2-maximum-likelihood-estimation
   ch3_3-sampling-variability
   ch3_4-linear-models
   ch3_5-generalized-linear-models
   ch3_6-chapter-summary