STAT 418: Computational Methods in Data Science
Course Content
Part I: Foundations of Probability and Computation
Chapter 1: Statistical Paradigms and Core Concepts
Section 1.1 Paradigms of Probability and Statistical Inference
The Mathematical Foundation: Kolmogorov’s Axioms
Interpretations of Probability
Statistical Inference Paradigms
Historical and Philosophical Debates
Bringing It All Together
Looking Ahead: Our Course Focus
Practice Problems
Chapter 2: Monte Carlo Simulation
Section 2.1 Monte Carlo Fundamentals
The Historical Development of Monte Carlo Methods
The Core Principle: Expectation as Integration
Theoretical Foundations
Variance Estimation and Confidence Intervals
Worked Examples
Comparison with Deterministic Methods
Sample Size Determination
Convergence Diagnostics and Monitoring
Practical Considerations
Chapter 2.1 Exercises: Monte Carlo Fundamentals Mastery
Bringing It All Together
Transition to What Follows
Section 2.2 Uniform Random Variates (Optional)
Why Uniform? The Universal Currency of Randomness
The Paradox of Computational Randomness
Chaotic Dynamical Systems: An Instructive Failure
Linear Congruential Generators
Shift-Register Generators
The KISS Generator: Combining Strategies
Modern Generators: Mersenne Twister and PCG
Statistical Testing of Random Number Generators
Practical Considerations
Chapter 2.2 Exercises: Uniform Random Variates Mastery
Bringing It All Together
Transition to What Follows
Section 2.3 Inverse CDF Method
Mathematical Foundations
Continuous Distributions with Closed-Form Inverses
Numerical Inversion
Discrete Distributions
Mixed Distributions
Practical Considerations
Chapter 2.3 Exercises: Inverse CDF Method Mastery
Lessons from the Exercises
Bringing It All Together
Transition to What Follows
Section 2.4 Transformation Methods (Optional)
Why Transformation Methods?
The Box–Muller Transform
The Polar (Marsaglia) Method
Method Comparison: Box–Muller vs Polar vs Ziggurat
The Ziggurat Algorithm
The CLT Approximation (Historical)
Distributions Derived from the Normal
Multivariate Normal Generation
Implementation Guidance
Chapter 2.4 Exercises: Transformation Methods Mastery
Bringing It All Together
Section 2.5 Rejection Sampling
The Dartboard Intuition
The Accept-Reject Algorithm
Efficiency Analysis
Choosing the Proposal Distribution
Python Implementation
The Squeeze Principle
Geometric Example: Sampling from the Unit Disk
Worked Examples
Limitations and the Curse of Dimensionality
Connections to Other Methods
Practical Considerations
Chapter 2.5 Exercises: Rejection Sampling Mastery
Bringing It All Together
References
Section 2.6 Variance Reduction Methods
The Variance Reduction Paradigm
Importance Sampling
Control Variates
Antithetic Variates
Stratified Sampling
Common Random Numbers
Conditional Monte Carlo (Rao–Blackwellization)
Combining Variance Reduction Techniques
Practical Considerations
Bringing It All Together
Chapter 2.6 Exercises: Variance Reduction Mastery
Section 2.7 Chapter 2 Summary
The Complete Monte Carlo Workflow
Method Selection Guide
Quick Reference Tables
Common Pitfalls Checklist
Connections to Later Chapters
Learning Outcomes Checklist
Further Reading: Optimization and Missing Data
Final Perspective
Part II: Frequentist Inference
Chapter 3: Parametric Inference and Likelihood Methods
Section 3.1 Exponential Families
Historical Origins: From Scattered Results to Unified Theory
The Canonical Exponential Family
Converting Familiar Distributions
The Log-Partition Function: A Moment-Generating Machine
Sufficiency: Capturing All Parameter Information
Minimal Sufficiency and Completeness
Conjugate Priors and Bayesian Inference
Exponential Dispersion Models and GLMs
Python Implementation
Practical Considerations
Chapter 3.1 Exercises: Exponential Families Mastery
Bringing It All Together
Section 3.2 Maximum Likelihood Estimation
The Likelihood Function
The Score Function
Fisher Information
Closed-Form Maximum Likelihood Estimators
Numerical Optimization for MLE
Asymptotic Properties of MLEs
The Cramér-Rao Lower Bound
The Invariance Property
Likelihood-Based Hypothesis Testing
Confidence Intervals from Likelihood
Practical Considerations
Connection to Bayesian Inference
Chapter 3.2 Exercises: Maximum Likelihood Estimation Mastery
Bringing It All Together
Section 3.3 Sampling Variability and Variance Estimation
Statistical Estimators and Their Properties
Sampling Distributions
The Delta Method
The Plug-in Principle
Variance Estimation Methods
Applications and Worked Examples
Practical Considerations
Bringing It All Together
Exercises
Section 3.4 Linear Models
Matrix Calculus Foundations
The Linear Model
Ordinary Least Squares: The Calculus Approach
Ordinary Least Squares: The Geometric Approach
Properties of the OLS Estimator
The Gauss-Markov Theorem
Estimating the Error Variance
Distributional Results Under Normality
Diagnostics and Model Checking
Bringing It All Together
Numerical Stability: QR Decomposition
Model Selection and Information Criteria
Regularization: Ridge and LASSO
Chapter 3.4 Exercises: Linear Models Mastery
Section 3.5 Generalized Linear Models
Historical Context: Unification of Regression Methods
The GLM Framework: Three Components
Score Equations and Fisher Information
Iteratively Reweighted Least Squares
Logistic Regression: Binary Outcomes
Poisson Regression: Count Data
Gamma Regression: Positive Continuous Data
Inference in GLMs: The Testing Triad
Model Diagnostics
Model Comparison and Selection
Quasi-Likelihood and Robust Inference
Practical Considerations
Bringing It All Together
Further Reading
Chapter 3.5 Exercises: Generalized Linear Models Mastery
Section 3.6 Chapter 3 Summary
The Parametric Inference Pipeline
The Five Pillars of Chapter 3
How the Pillars Connect
Method Selection Guide
Quick Reference: Core Formulas
Connections to Future Material
Practical Guidance
Final Perspective
Chapter 4: Resampling Methods
Section 4.1 The Sampling Distribution Problem
The Fundamental Target: Sampling Distributions
Historical Development: The Quest for Sampling Distributions
Three Routes to the Sampling Distribution
When Asymptotics Fail: Motivating the Bootstrap
The Plug-In Principle: Theoretical Foundation
Computational Perspective: Bootstrap as Monte Carlo
Practical Considerations
Bringing It All Together
Chapter 4.1 Exercises
Section 4.2 The Empirical Distribution and Plug-in Principle
The Empirical Cumulative Distribution Function
Convergence of the Empirical CDF
Parameters as Statistical Functionals
The Plug-in Principle
When the Plug-in Principle Fails
The Bootstrap Idea in One Sentence
Computational Implementation
Bringing It All Together
Section 4.2 Exercises: ECDF and Plug-in Mastery
Section 4.3 The Nonparametric Bootstrap
The Bootstrap Principle
Bootstrap Standard Errors
Bootstrap Bias Estimation
Bootstrap Confidence Intervals
Bootstrap for Regression
Bootstrap Diagnostics
When Bootstrap Fails
Practical Considerations
Bringing It All Together
Exercises
Section 4.4: The Parametric Bootstrap
The Parametric Bootstrap Principle
Location-Scale Families
Parametric Bootstrap for Regression
Confidence Intervals
Model Checking and Validation
When Parametric Bootstrap Fails
Parametric vs. Nonparametric: A Decision Framework
Practical Considerations
Bringing It All Together
Section 4.5: Jackknife Methods (Optional)
Historical Context and Motivation
The Delete-1 Jackknife
Jackknife Bias Estimation
The Delete-
\(d\)
Jackknife
Jackknife versus Bootstrap
The Infinitesimal Jackknife
Practical Considerations
Bringing It All Together
Exercises
Section 4.6 Bootstrap Hypothesis Testing and Permutation Tests
From Confidence Intervals to Hypothesis Tests
The Bootstrap Hypothesis Testing Framework
Permutation Tests: Exact Tests Under Exchangeability
Testing Equality of Distributions
Bootstrap Tests for Regression
Bootstrap vs Classical Tests
Permutation vs Bootstrap: Choosing the Right Approach
Multiple Testing with Bootstrap
Practical Considerations
Bringing It All Together
Exercises
Section 4.7 Bootstrap Confidence Intervals: Advanced Methods (Optional)
Why Advanced Methods?
The Studentized (Bootstrap-t) Interval
Bias-Corrected (BC) Intervals
Bias-Corrected and Accelerated (BCa) Intervals
Choosing B and Assessing Monte Carlo Error
Diagnostics for Advanced Bootstrap Methods
Method Selection Guide
Bringing It All Together
Chapter 4.7 Exercises: Bootstrap Confidence Interval Mastery
Section 4.8: Cross-Validation Methods (Optional)
Historical Context: From Jackknife to Cross-Validation
Leave-One-Out Cross-Validation
K-Fold Cross-Validation
Generalized Cross-Validation
Nested Cross-Validation
Bootstrap Prediction Error Estimation
Connection to Information Criteria
The Variance Estimation Problem
Cross-Validation for Structured Data
Computational Considerations
Practical Considerations
Bringing It All Together
Exercises
Section 4.9 Chapter 4 Summary
The Resampling Philosophy
The Complete Resampling Workflow
The Eight Pillars of Chapter 4
Method Selection Guide
Quick Reference Tables
Common Pitfalls Checklist
Connections to Other Chapters
Learning Outcomes Checklist
Practical Guidance
Further Reading: Advanced Resampling Topics
Final Perspective
Part III: Bayesian Inference
Chapter 5: Bayesian Inference
Section 5.1 Foundations of Bayesian Inference
Historical Development: From Bayes’ Essay to the MCMC Revolution
The Bayesian Workflow
Bayes’ Theorem for Parameters
Discrete Parameter Spaces
Continuous Parameters: Grid Approximation
The Posterior as Complete Inference
The Likelihood Principle Revisited
Practical Considerations
Bringing It All Together
Chapter 5.1 Exercises
Section 5.2 Prior Specification and Conjugate Analysis
The Role of the Prior
Beta-Binomial Conjugate Analysis
Normal-Normal Model: Known Variance
Normal-Inverse-Gamma: Unknown Mean and Variance
Poisson-Gamma Model
Multinomial-Dirichlet Model
Beyond Conjugacy: The Full Landscape of Prior Specification
Bayesian vs. Frequentist Synthesis and the Limits of Conjugacy
Practical Considerations
Bringing It All Together
Exercises
Section 5.3 Posterior Inference: Credible Intervals and Hypothesis Assessment
Credible Intervals: Probability Statements About Parameters
Posterior Probability and Directional Hypothesis Assessment
Posterior Predictive Intervals
Communicating Bayesian Results
Practical Considerations
Bringing It All Together
Exercises
Section 5.4 Markov Chains: The Mathematical Foundation of MCMC
From Grid Approximation to Markov Chains
Markov Chains
Stationary Distributions and Detailed Balance
The Ergodic Theorem: Why MCMC Averages Converge
The MCMC Estimator, Effective Sample Size, and
\(\hat{R}\)
Python: Simulating Convergence and Diagnosing Chains
Mixing, Thinning, and Practical Considerations
Bringing It All Together
Exercises
Section 5.5 MCMC Algorithms: Metropolis-Hastings and Gibbs Sampling
The Metropolis-Hastings Algorithm
The Gibbs Sampler
Hamiltonian Monte Carlo: A Brief Introduction
Practical Workflow
Bringing It All Together
Exercises
Section 5.6 Probabilistic Programming with PyMC
How PyMC Works: From Model Spec to Gradient
The Diagnostic Toolkit
Full Worked Example 1: Logistic Regression
Full Worked Example 2: Poisson Regression
Scale Parameters: Half-Normal and Half-Cauchy Priors
Derived Quantities with
pm.Deterministic
Practical Workflow Checklist
Bringing It All Together
Exercises
Section 5.7 Bayesian Model Comparison
The Predictive Accuracy Target
WAIC
PSIS-LOO Cross-Validation
Implementation: Comparing Two Models
Bayes Factors
Bringing It All Together
Exercises
Section 5.8 Hierarchical Models and Partial Pooling (Optional)
The Pooling Problem
The Mathematical Structure
Worked Example 1: Eight Schools (Normal Likelihood)
Worked Example 2: NBA Team Scoring – Does Shrinkage Pay Off?
Worked Example 3: Dirichlet-Multinomial (Categorical Likelihood)
LOO Model Comparison: Three Pooling Strategies
When to Use Hierarchical Models
Practical Considerations
Bringing It All Together
Exercises
References
Section 5.9 Chapter 5 Summary
The Bayesian Workflow
Section-by-Section Synthesis
Decision Guide: Which Bayesian Tool When
Quick Reference: Core Formulas
Common Pitfalls
Connections Across the Chapter
Connections to Earlier Chapters
Part IV: Large Language Models in Data Science
Chapter 6: LLMs in Data Science Workflows
Section 6.1 LLM Foundations: Architecture, Training, and Deployment
From Statistical Models to Neural Language Models
The Transformer Architecture
Pre-Training, Fine-Tuning, and In-Context Learning
Model Families and the Landscape
Open vs. Closed Models: Trade-offs for Data Scientists
Deployment Options
Getting Started with GenAI Studio
Chapter 6.1 Exercises: LLM Foundations
Transition to What Follows
Section 6.2 Embeddings and Feature Extraction
From Sparse to Dense Representations
Generating Embeddings with GenAI Studio
Similarity Search
What an Embedding Encodes
Classification with Embeddings
Integration with Statistical Models
Chapter 6.2 Exercises: Embeddings and Feature Extraction
Transition to What Follows
Section 6.3 Text Preprocessing for LLM Pipelines
Tokenization: How Models See Text
Context Windows and Token Limits
Chunking Strategies
Text Normalization and Cleaning
Building a Complete Preprocessing Pipeline
Chapter 6.3 Exercises: Text Preprocessing
Transition to What Follows
Section 6.4 LLM-Assisted Data Annotation
The Annotation Bottleneck
LLM as Annotator
Annotation Tasks for Data Science
Quality Control and Validation
When LLM Annotation Works and When It Does Not
Chapter 6.4 Exercises: Data Annotation
Transition to What Follows
Section 6.5 Retrieval-Augmented Generation
Why RAG?
Building RAG with GenAI Studio
Building RAG from Scratch
Chunking Strategies for RAG
Evaluating RAG Systems
RAG Failure Modes and Mitigations
Chapter 6.5 Exercises: Retrieval-Augmented Generation
Transition to What Follows
Section 6.6 Prompt Engineering for Data Science
Prompts as Code
Systematic Prompt Design
Few-Shot Prompting
Chain-of-Thought Reasoning
Advanced Techniques
Debugging and Iterating on Prompts
Data Science Prompt Patterns
Chapter 6.6 Exercises: Prompt Engineering
Transition to What Follows
Section 6.7 Tool Use
Why Tool Use?
What Tool Use Is
Declaring a Tool with
@tool
A Single Tool Call, End to End
When to Use Tool Use
Risks and Safeguards
Where Tool Use Ends and Agents Begin
Chapter 6.7 Exercises: Tool Use
Transition to What Follows
Section 6.8 Reliability and Evaluation
The Reliability Challenge
Consistency Assessment
Can You Trust a Confidence Number?
LLM-as-Judge
Uncertainty Quantification
Evaluation Protocols
Chapter 6.8 Exercises: Reliability and Evaluation
Transition to What Follows
Section 6.9 Responsible AI Practices
Privacy and Data Protection
Bias in LLM Outputs
Transparency and Disclosure
Ethical Frameworks
Appropriate Use in Data Science
Building a Personal AI Use Policy
Chapter 6.9 Exercises: Responsible AI
Transition to What Follows
Section 6.10 Chapter Summary
Section-by-Section Recap
GenAI Studio Quick Reference
Connections to Earlier Chapters
Purdue AI Working Competency Mapping
Common Pitfalls Checklist
End-to-End Example
Learning Outcomes Checklist
Further Reading
Final Perspective
Appendices
Appendices
Appendix A: Calculus Review
From Formulas to Statistical Reasoning
Univariate Differentiation
Basic Rules
Logarithmic Differentiation
Higher Derivatives and the Second Derivative Test
Common Derivatives Reference
Integration
Definite Integrals as Expectations
Integration by Parts
Change of Variables — Univariate
Improper Integrals and Convergence
Taylor Series and Approximation
Taylor’s Theorem
First-Order Taylor: The Delta Method Connection
Second-Order Taylor: Quadratic Likelihood Approximation
Convergence of Taylor Series
Leibniz Rule and Differentiation Under the Integral
Statement and Regularity Conditions
Application: The Score Has Mean Zero
Application: Fisher Information Equivalence
Multivariate Differential Calculus
Partial Derivatives and the Gradient
The Hessian Matrix
The Multivariate Chain Rule
Second-Order Conditions
Convexity and Optimization
Convex Sets and Functions
Hessian Characterization of Convexity
Convexity and Exponential Families
Strict Convexity and Unique Optimization
Multivariate Taylor Expansion
The Multivariate Taylor Formula
The Wald Statistic
Multiple Integrals and Change of Variables
Multiple Integrals
The Jacobian Determinant
Worked Example: Polar Coordinates
Connections: From Calculus to Computation
Practice Problems
Appendix B: Linear Algebra for Data Science
From Notation to Intuition
Vectors and Inner Products
Inner Product and Norm
Orthogonality
Matrix Operations and Properties
Multiplication, Transpose, and Inverse
Trace
Determinant
Rank
Special Matrix Structures
Symmetric Matrices
Diagonal and Identity Matrices
Idempotent Matrices
Positive Definite and Positive Semi-Definite Matrices
Stochastic Matrices
Block Matrices
Block Partitioning
Block Multiplication
Block Transpose and Block Diagonal
The Schur Complement
Sherman-Morrison-Woodbury Formula
Column Space, Projections, and Least Squares
Column Space and Null Space
Orthogonal Projection
The Hat Matrix
Leverage
Matrix Decompositions
Eigendecomposition (Spectral Decomposition)
Cholesky Decomposition
QR Decomposition
Singular Value Decomposition (SVD)
Decomposition Summary
Quadratic Forms and Covariance
Quadratic Forms
Covariance Matrices
Matrix Calculus
Gradients of Scalar Functions
The Hessian Matrix
Derivative of Log-Determinant
Jacobian and Change of Variables
Numerical Considerations
Condition Number and Stability
Avoiding
\(\mathbf{X}^\top\mathbf{X}\)
Near-Singular Matrices
Key Takeaways
Looking Ahead
Practice Problems
Appendix C: Numerical Analysis Review
From Exact to Approximate
Floating-Point Arithmetic
IEEE 754 Basics
Catastrophic Cancellation
Log-Space Arithmetic
Root Finding
Fixed-Point Iteration and Contraction Mapping
Bisection
Newton’s Method for Root Finding
Numerical Optimization
Optimization as Fixed-Point Iteration
Gradient Descent
Newton’s Method for Optimization
Quasi-Newton Methods
Numerical Differentiation
Forward and Central Differences
The Step Size Dilemma
The Complex-Step Method
Numerical Integration
Error Analysis for Quadrature
Adaptive Quadrature
Monte Carlo Error Rate
Convergence and Stopping Criteria
Asymptotic (Big-O) Notation
Convergence Orders
Stopping Criteria
Diagnosing Non-Convergence
Connections: From Numerical Analysis to Computation
Practice Problems
Appendix D: Probability Distributions — Theory and Computation
From Abstract Foundations to Concrete Tools
The Moment of Discovery: De Moivre’s Insight
This Appendix’s Philosophy: Theory Meets Computation
Structure of This Appendix
The Python Ecosystem for Probability
Practical Example: Complete Distribution Analysis
Why Study Named Distributions?
Discrete Distributions
Bernoulli Distribution
Binomial Distribution
Poisson Distribution
Geometric Distribution
Negative Binomial Distribution
Continuous Distributions
Uniform Distribution
Normal (Gaussian) Distribution
Exponential Distribution
Gamma Distribution
Beta Distribution
Additional Important Distributions
Student’s t-Distribution
Chi-Square Distribution
F-Distribution
Summary and Practical Guidelines
Choosing the Right Distribution
Key Relationships Between Distributions
Python Tools Summary
Parameterization Notes
Common Pitfalls and Best Practices
Conclusion
Practice Problems
Appendix E: Statistical Inference Review
Point Estimation
Estimators and Their Properties
Uniformly Minimum Variance Unbiased Estimators
Common Estimators
Method of Moments
Sampling Distributions
Confidence Intervals
Constructing Intervals from Pivotal Quantities
Margin of Error and Sample Size Determination
Duality of Confidence Intervals and Hypothesis Tests
Common Misinterpretations
Computational Verification: Coverage
When Coverage Breaks Down: The Wald Proportion Interval
Hypothesis Testing
The Neyman-Pearson Framework
Error Types and Power
P-Values
Power Analysis
The Neyman-Pearson Lemma
Uniformly Most Powerful Tests
Common Hypothesis Tests
Multiple Testing
Sufficiency and Information
Sufficient Statistics
The Fisher-Neyman Factorization Theorem
Examples of Sufficient Statistics
Minimal Sufficiency
Completeness
Ancillary Statistics and Basu’s Theorem
Fisher Information
Two Equivalent Forms of Fisher Information
The Cramér-Rao Lower Bound
Efficiency of Estimators
The Likelihood Function
Likelihood versus Probability
Maximum Likelihood Estimation
The Score Function and Information
MLE Properties
Profile Likelihood
Computational Illustration: Likelihood Anatomy
Asymptotic Theory
Modes of Convergence
The Weak Law of Large Numbers
The Central Limit Theorem
Slutsky’s Theorem
The Continuous Mapping Theorem
The Delta Method
Asymptotic Theory of the MLE
Connections: From Review to Computation
Practice Problems
Appendix F: Python Random Generation
From Mathematical Distributions to Computational Samples
The Python Ecosystem at a Glance
Understanding Pseudo-Random Number Generation
The Nature of Pseudo-Randomness
What Makes a Good PRNG?
The Mersenne Twister
NumPy’s PCG64
The Standard Library:
random
Module
Generating Random Numbers
Random Operations on Sequences
Distribution Generators
Controlling Randomness: Seeds and State
NumPy: Fast Vectorized Random Sampling
Why NumPy Is the Default for Scientific Work
The Modern Generator API
Performance Comparison
Univariate Distributions
Multivariate Distributions
NumPy Sampling Utilities
Parallel Random Number Generation
SciPy Stats: The Complete Statistical Toolkit
Why SciPy Is the “Next Stop” After NumPy
The Frozen Distribution Pattern
The Unified Interface
Parameterization: The Most Common Error Source
Complete Analysis Example
Bringing It All Together: Library Selection Guide
Looking Ahead: From Random Numbers to Monte Carlo Methods
Practice Problem
STAT 418: Computational Methods in Data Science
Index
Index