Virtual Humans Lecture
Contents:
Lecture 01.1 – Historical Body Models
Introduction
Early Origins: Simplified Primitives and Kinematic Skeletons (1970s–1980s)
Advances in the 1990s: Superquadrics, Differentiable Fitting, and Physical Models
The Impact of 3D Scanning and Data: From Anthropometry to Statistical Models (1990s–2000s)
SCAPE and the Emergence of Pose-Aware Models (Mid-2000s)
Consolidation in the 2010s: SMPL and Integration with Learning-Based Methods
Deep Learning and Neural Implicit Models (Late 2010s–Present)
Timeline Summary of Milestones
Conclusion
Lecture 01.2 – Introduction to Human Models
1. Historical Context of Human Body Modeling
Early Developments
Mid-20th Century Approaches
Data-Driven Revolution (1990s-2000s)
2. Mathematical Foundations of Human Body Models
The SMPL Model
PCA-Based Statistical Shape Modeling
Kinematic Modeling
3. Applications of Human Body Models
Computer Animation and Visual Effects
Virtual Humans and Avatars
Biomechanics and Ergonomics
Human-Computer Interaction (HCI)
Computer Vision and AI
Education and Training
4. Challenges and Future Directions
Computational Efficiency
Accuracy and Detail
Generalization
Clothing and Accessories
Emerging Approaches
5. Conclusion
Lecture 01.3 – Introduction to Human Models (Overview)
1. Historical Context
Early Scientific Studies
Mid-20th Century to Digital Era
21st Century Advances
2. Mathematical Foundations
Parametric Body Models
Implicit Surface Representations
Kinematic Modeling
3. Image Formation and Rendering
Camera Models
Shading and Visibility
Differentiable Rendering
4. Surface Representation Methods
Explicit Mesh Models
Implicit Function Models
5. Motion Capture and Behavior Synthesis
Capturing Human Motion
Behavior Synthesis
6. Clothing Modeling
Physically-Based Simulation
Data-Driven Approaches
Implicit Clothing Models
7. Human-Object Interaction
Physics-Based Methods
Learning-Based Approaches
Hybrid Systems
8. Applications
Entertainment and Media
Healthcare and Biomechanics
Engineering and Design
Human-Computer Interaction
Scientific Research
9. Challenges and Future Directions
Scalability and Generalization
Higher-Fidelity Dynamics
Data and Labeling Constraints
Physics and Learning Integration
Semantic and Cognitive Aspects
Realism vs. Controllability
Lecture 02.1 – Image Formation
1. Historical Developments in Image Formation
Ancient and Medieval Optics – Camera Obscura
Renaissance Perspective and Geometry
Early Cameras and Photographic Imaging
Modern Developments
2. The Pinhole Camera Model
Coordinate Setup
Proof by Similar Triangles
Numerical Example
Inadequacy of a Simple Pinhole
3. Camera Intrinsics and the Projection Matrix
Extrinsic Parameters
Full Projection Example
4. Image Distortions & Correction
5. Properties of Perspective Projection
6. Advanced Theoretical Extensions
Light Field Imaging and Plenoptic Cameras
Non-Conventional Imaging Techniques
7. Applications in Modern Vision and Graphics
Computer Vision and 3D Reconstruction
Medical Imaging
Photorealistic Rendering in Computer Graphics
8. Python Example: Simulating Image Formation
Lecture 02.2 – Rotations and Kinematic Chains
1. Representations of 3D Rotations
A) Rotation Matrices
B) Euler Angles
C) Quaternions
2. Lie Algebra
\(so(3)\)
and Exponential Map
3. Rodrigues’ Rotation Formula
4. Kinematic Chains: Forward & Inverse Kinematics
Comparison of Rotation Representations
Lecture 03.1 – Surface Representations
1. Mathematical Foundations of Surface Representations
A) Parametric Surfaces
B) Implicit Surfaces
C) Explicit Surfaces
2. Surface Differential Properties
A) Surface Normals
B) Fundamental Forms and Curvature
C) Geodesics
3. Discrete Surface Representations
A) Polygon Meshes
B) Point Clouds
C) Signed Distance Fields (SDF)
4. Advanced Surface Representations
A) Bézier Curves and Surfaces
B) B-Splines and NURBS
C) Subdivision Surfaces
D) Level Sets
E) Neural Implicit Representations
5. Comparative Analysis and Applications
A) Computational Efficiency and Storage
B) Practical Applications
C) Operations Complexity
6. Implementation Examples
A) Basic Mesh Processing (Python)
B) Implicit Surface Utilities
C) Bézier Curve Implementation
D) Curvature Estimation on Meshes
7. Advanced Topics and Future Directions
A) Multi-Resolution Representations
B) Machine Learning for Geometry
C) Dynamic Surfaces
D) Non-Manifold Geometries
Lecture 03.2 – Procrustes Alignment
Introduction
Goal: Learning a Model of Pose and Shape
The Challenge of Registration
Surface Representation: Mesh
The Procrustes Alignment Problem: Mathematical Formulation
Rigid Transformations
Procrustes Alignment Solution
Decoupling Translation by Centroid Alignment
Optimal Rotation via SVD
Reflection Adjustment
Optimal Scale (Optional)
Complete Mathematical Derivation
Translation Derivation
Rotation Derivation
Scale Derivation
Summary of Procrustes Alignment Algorithm
Python Implementation Example
Practical Applications
Interactive Visualization Ideas
Lecture 4.1: Iterative Closest Point
Introduction to Shape Alignment and Registration
The Registration Problem
Review: Procrustes Analysis
Problem: Unknown Correspondences
The Iterative Closest Point (ICP) Algorithm
Basic ICP Algorithm
Computational Considerations
Closest Point Computation
Convergence and Local Minima
Point-to-Point vs. Point-to-Plane ICP
Point-to-Point ICP
Point-to-Plane ICP
Gradient-based ICP for Non-Rigid Registration
Gradient-based ICP Algorithm
Advantages of Gradient-based ICP
Computing Gradients
Improving ICP’s Robustness
Data Association Direction
Robust Cost Functions
Trimmed ICP
RANSAC-based Approaches
Additional Information
Point-to-Surface Distance
ICP Variants and Extensions
Generalized ICP (GICP)
EM-ICP and Probabilistic Approaches
Coherent Point Drift (CPD)
Multi-Scale Approaches
Applications of ICP
Implementing ICP
Efficient Python Implementation
Practical Tips
Conclusion
Lecture 04.2 - Body Models: Vertex-Based Models and SMPL
1. Body Models as Parameterized Functions
2. Rotations, Articulation, and Pose Representation
2.1 Rotation Representation
2.2 Kinematic Chain
3. Linear Blend Skinning and its Limitations
3.1 Linear Blend Skinning
3.2 Problems with Standard LBS
3.3 Blend Shapes for Correcting LBS Artifacts
4. The SMPL Body Model
4.1 SMPL Philosophy
4.2 SMPL Model Architecture
4.2.1 Shape Blend Shapes
4.2.2 Pose Blend Shapes
4.2.3 Joint Regression
4.3 Model Training
5. Comparison with SCAPE
5.1 The SCAPE Model
5.2 Different Approaches to Deformation
5.3 Performance Comparison
5.4 Other Advantages
6. Alignment Techniques: Procrustes Analysis and ICP
6.1 Procrustes Analysis
6.2 Iterative Closest Point (ICP)
6.3 Fitting SMPL to Scans
7. Image Formation and the Pinhole Camera Model
7.1 The Pinhole Camera Model
7.2 Lens Distortion
8. Extensions and Advanced Applications
8.1 Dynamic Soft Tissue Modeling
8.2 Specialized Extensions
8.3 Deep Learning for Model Fitting
8.4 Probabilistic Approaches
8.5 Hybrid Models
Conclusion
Lecture 5.1 - Training a Body Model and Fitting SMPL to Scans
Introduction
Body Models Based on Triangle Deformations
SCAPE and BlendSCAPE Models
Triangle Deformation Process
Comparison: SMPL vs. SCAPE/BlendSCAPE
Training a Body Model from Registrations
The Challenge of Raw Scan Data
Training from Registrations
Obtaining Registrations: Fitting SMPL to Scans
Non-Rigid Registration Process
Iterative Closest Point (ICP) Review
Registration Objective Formulation
Point-to-Surface Distance
Multi-Stage Optimization Strategy
Joint Registration and Model Training
Co-Registration Approach
Summary
Lecture 05.2 - 3D Registration: From Classical ICP to Modern Methods
Introduction
1. Rigid Registration and the ICP Algorithm
The Iterative Closest Point (ICP) Algorithm
Convergence Analysis and Failure Modes
2. Classical Non-Rigid Registration
Thin Plate Spline Robust Point Matching (TPS-RPM)
Coherent Point Drift (CPD)
Other Non-Rigid Methods
3. Parametric Models and the SMPL Body Model
Model Structure
Shape Blend Shapes (Identity Variation)
Pose Blend Shapes (Pose-Dependent Deformation)
Linear Blend Skinning (LBS) for Articulation
Learning SMPL
Using SMPL for Registration
4. Modeling Clothing and Fine Details: SMPL+D
Why SMPL+D?
How Displacements Are Applied
Limitations
Applications
5. Survey of 3D Registration Methods: From ICP to Deep Learning
5.1 Early Pioneering Works (1990s) – Foundational Rigid Registration
5.2 The 2000s – Robust and Non-Rigid Registration Emerges
5.3 2010s – Template-based and Parametric Model Registration
5.4 2020s – Learning-Based Parametric Registration and Hybrid Approaches
Summary
Lecture 06.1 - Fitting the SMPL Model to Images via Optimization
Introduction
Mathematical Background: Pinhole Camera and Projections
Perspective Projection
Weak-Perspective Projection
Camera Extrinsics vs. Model Pose
2D Keypoints and Projection
The SMPL Model as a Differentiable Function of Shape and Pose
Shape Blend Shapes
Pose Blend Shapes
Joint Positions
Linear Blend Skinning
Differentiability of SMPL
Fitting SMPL to Images via Optimization (SMPLify)
Objective Function
Combined Objective
Optimization Strategy
Optimization Algorithms
Automatic Differentiation and Jacobians
Result of SMPLify
Historical Progression and Method Comparisons
SMPLify (Bogo et al. 2016)
SMPLify-X (2019)
Lecture 06.2 - Learning-Based Fitting of the SMPL Model to Images
Introduction
Foundations of Learning-Based SMPL Estimation
Integrating SMPL into Neural Networks
Projection Functions
Loss Functions
Statistical Priors and Adversarial Losses
Early Regression Approaches: HMR and NBF
Human Mesh Recovery (HMR)
Neural Body Fitting (NBF)
Evolving Architectures: Hybrid and Improved Regression Methods
SPIN: Optimization in the Training Loop
PyMAF: Pyramidal Mesh Alignment Feedback
CLIFF: Using Full-Frame Context for Camera Orientation
PIXIE: Whole-Body Regression with Part Experts
Temporal Methods: From Single Images to Video Sequences
VIBE: Adversarial Motion Prior with GRUs
TCMR: Temporally Consistent Mesh Recovery
MotionBERT: Transformer-Based Motion Representations
Comparison of Learning-Based Methods
Supervision and Data
Network Architecture
Objective Functions
Pose and Shape Priors
Performance
Runtime
Strengths and Weaknesses
Conclusion
Lecture 07.1: Fitting SMPL to IMU Data Using Optimization-Based Methods
Introduction
Classical IMU-Based Pose Estimation: A Historical Perspective
Attitude and Heading Reference Systems
Kalman Filter Approaches
Early Sparse-Sensor Approaches
Model-Based Optimization Methods
Learning-Based Methods
Inertial Sensor Fundamentals and Orientation Representations
Gravity Alignment and Drift Correction
Orientation Representations
Sensor Calibration
Optimization-Based SMPL Fitting with IMU Data
Kinematic Model and Sensor Prediction
Regularization and Prior Terms
Gradient and Jacobian Computation
Pseudocode: SMPL Pose Estimation from IMU Sequence
IMU-Based Human Pose Datasets and Resources
DIP-IMU Dataset (2018)
TotalCapture (2017)
AMASS (2019)
Other Datasets and Resources
Lecture 07.2: Fitting SMPL to IMU Data Using Learning-Based Methods
Introduction
Optimization-Based vs. Learning-Based Approaches
Learning-Based IMU-to-Pose Estimation: Historical Overview of Key Models
Deep Inertial Poser (DIP, 2018)
TransPose (2021)
Transformer Inertial Poser (TIP, 2022)
Physics/Physical Inertial Poser (PIP, 2022)
Other Notable Models and Developments
Problem Formulation and Learning Task Definition
Input and Output Representations
Learning Objective and Loss Functions
Temporal Modeling Approaches
Supervised vs. Semi-Supervised Training; Synthetic Data
Model Architectures and Design Considerations
Encoding IMU Measurements
Network Structures
Training Pipeline and Pseudocode
Datasets, Benchmarks, and Resources
DIP-IMU Dataset (2018)
TotalCapture (2017)
AMASS (2019)
Other Datasets and Resources
Challenges and Outlook
Conclusion
Lecture 08.1: Vertex-Based Clothing Modeling for Virtual Humans
Introduction
Clothing Representation as Vertex Displacements
Registration of Clothed Human Scans
Estimating Body Shape Under Clothing: The BUFF Method
Multi-Layer Clothing Capture: The ClothCap Method
Data Term
Boundary Term
Boundary Smoothness
Laplacian Smoothness
Applications of Multi-Layer Registration
Advantages and Limitations
Conclusion
Lecture 09.1: Neural Implicit and Point-Based Representations for Clothed Human Modeling
Introduction
Background: Explicit vs. Implicit vs. Point-Based Representations
Mesh-Based Models
Point-Based Models
Neural Implicit Representations
Neural Radiance Fields (NeRFs) for Humans
Hybrid Approaches
Expressiveness and Topology
Differentiability and Learning
Data Efficiency and Performance
Neural Implicit Function Foundations
Signed Distance Field (SDF)
Occupancy Field (Indicator Function)
Volume Radiance Field
Articulated Deformation Fields for Implicit Models
Backward Warping (Inverse Skinning Field)
Forward Warping (Forward Skinning Field)
Pose-Dependent Deformation (Secondary Motion)
Generative Implicit Models for Clothed Bodies
SMPLicit: Topology-Aware Clothed Human Model
imGHUM: Implicit Generative Human Model
Pose-Dependent Implicit Models and Animatable Avatars
NASA: Neural Articulated Shape Approximation (ECCV 2020)
SCANimate: Weakly-Supervised Skinned Avatar Networks (CVPR 2021)
Neural-GIF: Neural Generalized Implicit Functions (ICCV 2021)
SNARF: Skinned Neural Articulated Implicit Shapes (ICCV 2021)
POP: The Power of Points (ICCV 2021) – Point-Based Modeling
Other Noteworthy Methods
Blueprint Algorithms for Key Methods
SNARF (Training Procedure): Differentiable Forward Skinning for Implicit Surfaces
POP (Training & Fitting Pipeline): Point-Based Model for Pose-Dependent Clothing
SMPLicit (Inference/Fitting Workflow): Generative Implicit Garment Model conditioned on SMPL
Historical Perspective and Future Outlook
Challenges and Future Directions
Conclusion
Neural Radiance Fields: A Historical and Theoretical Overview
Introduction
Foundations: 3D Scene Representation and Reconstruction Techniques
Voxel Grids
Point Clouds
Mesh-Based Surfaces
Light Fields and Volumetric Rendering
Emergence of Neural Radiance Fields (NeRF)
Core Idea
Training Procedure
NeRF Architecture
Hierarchical Sampling
Original Results
Theoretical and Mathematical Analysis of NeRF
Volume Rendering Formulation in NeRF
Positional Encoding and Neural Network Architecture
Loss Function and Optimization
Major Advancements and Extensions of NeRF
Anti-Aliasing and Unbounded Scenes: mip-NeRF and NeRF++
Efficiency Improvements: Instant NeRF and PlenOctrees
Dynamic and Deformable NeRFs (D-NeRF, Nerfies, NSFF, etc.)
Neural Radiance Fields for Human Modeling (with SMPL and Body Models)
Other Notable Extensions
Comparison with Other 3D Representations
Vs. Polygonal Meshes
Vs. Point Clouds / 3D Splatting
Vs. Voxel Grids and Volumetric Methods
Vs. Multi-Plane Images (MPIs) / Light Fields
Accuracy and Fidelity
Applicability
Datasets for NeRF Training and Evaluation
Blender Synthetic NeRF Dataset
Local Light Field Fusion (LLFF) Real Forward-Facing Dataset
Tanks and Temples
DTU Dataset
Human3.6M
ZJU-MoCap Dataset
People-Snapshot
Synthetic dynamic scenes
Other datasets
Conclusion
3D Gaussian Splatting: A Basic Introduction
Introduction
Foundations
What is a 3D Scene?
3D Scene Representation
Computer Graphics Fundamentals
Rasterization and Ray Tracing
Alpha Blending and Compositing
The Evolution of Novel View Synthesis
Image-Based Rendering
Structure-from-Motion and Multi-View Stereo
Point-Based Rendering
Neural Rendering
Accelerated Neural Fields
3D Gaussian Splatting: A Convergence of Approaches
Point-Based Rendering
Point Clouds and Their Challenges
The Concept of Splatting
Elliptical Weighted Average (EWA) Filtering
Differentiable Point-Based Rendering
3D Gaussian Splatting: Core Principles
Key Insight: Unifying Points and Volumes
3D Gaussians as Scene Primitives
The Volumetric Rendering Equation
Alpha Compositing with Gaussians
Mathematical Formulation of 3D Gaussian Splatting
Projecting 3D Gaussians to 2D
Parameterization of 3D Gaussians
View-Dependent Appearance
Differentiable Rendering Equations
Training and Optimization
Photometric Loss
Initial Point Cloud
Optimization Process
Differentiable Splatting Pipeline
Adaptive Density Control
Implementation and Real-Time Rendering
Tile-Based Rendering
Fast Sorting Strategies
GPU-Accelerated Rasterization
Memory Considerations
Comparison with Other Methods
NeRF vs. 3D Gaussian Splatting
Voxel-Based Representations vs. Gaussians
Traditional Point-Based Rendering vs. Gaussian Splatting
Applications and Extensions
Static Scene Reconstruction
Dynamic Scene Capture
Avatar Creation and Animation
Integration with Neural Rendering
Large-Scale Scene Rendering
Bézier Gaussian Triangles (BG-Triangle) for Sharper Rendering
Representation
Performance
Human Reconstruction with Gaussian Splatting and Priors
EG-HumanNeRF: Efficient Generalizable Human NeRF
GPS-Gaussian: Pixel-Wise Gaussian Splatting for Humans
Generalizable Human Gaussians (GHG) with SMPL
Dynamic Scene Reconstruction with 4D Gaussian Splatting
4D Gaussian Splatting (4DGS)
Speed and Memory Enhancements (4DGS-1K and MEGA)
Applications to MoCap and 4D Human Rendering
Implementation Details and Real-Time Performance
Data Structures
Rasterization & Shaders
GPU Memory and Throughput
Differentiable Rendering Implementation
Training vs. Inference Compute
Accuracy vs. Speed trade-offs
Benchmarks and Comparative Evaluation
Static Scene Comparison
Human Novel-View Comparison
Dynamic Scene Comparison
Comparative Summary Table
Datasets and Resources
Synthetic NeRF Dataset (Blender Scenes)
LLFF (Local Light Field Fusion)
Tanks and Temples
Multi-Object 360 (CO3D - Common Objects in 3D)
AMASS (Archive of Motion Capture as Surface Shapes)
CAPE (Clothed Auto Person Encoding)
THuman / THuman2.0
RenderPeople
Open-Source Implementations
Future Directions
Improved Compression Techniques for Memory Efficiency
Handling Dynamic and Deformable Scenes
Advanced Material Modeling for Realistic Rendering
Hybrid Approaches Integrating Neural Fields and Explicit Representations
Scalability for Large-Scale Scenes (City-Level and Beyond)
Real-Time Applications in AR/VR and Gaming
Integration with Existing Graphics Pipelines
Learning from Limited Data
Conclusion
Glossary
References
Lecture 01.1 (Historical Body Models)
Lecture 01.2 (Introduction to Human Models)
Lecture 01.3 (Introduction to Human Models Continued)
Lecture 02.1 (Image Formation)
Lecture 02.2 (Rotations & Kinematic Chains)
Lecture 03.1 (Surface Representations)
Lecture 03.2 (Procrustes Alignment)
Lecture 04.1 (Iterative Closest Points)
Lecture 04.2 (Body Models)
Lecture 05.1 (Body Model Training)
Lecture 05.2 (3D Registration)
Lecture 06.1 (Fitting SMPL to Images)
Lecture 06.1 (Optimization-Based Fitting of SMPL to Images)
Lecture 06.2 (Learning-Based Fitting of SMPL to Images)
Lecture 07.1 (Fitting SMPL to IMU Optimization)
Lecture 07.2 (Fitting SMPL to IMU Learning)
Classic and Optimization-Based Methods
Learning-Based Methods
Datasets and Resources
Relevant Software and Libraries
Lecture 08.1: References for Vertex-Based Clothing Modeling for Virtual Humans
Body Models and Shape Estimation
Alternative Representations
Datasets
Software and Libraries
Lecture 09.1: References for Neural Implicit and Point-Based Representations for Clothed Human Modeling
Point-Based Models
Datasets and Resources
Related Software and Libraries
Review Papers and Tutorials
Neural Radiance Fields (NERF)
Gaussian Splatting
Virtual Humans Lecture
Index
Index
A
|
C
|
D
|
E
|
I
|
J
|
L
|
M
|
N
|
P
|
Q
|
R
|
S
|
T
|
V
A
Alpha Blending
Alpha Compositing
Anisotropic Gaussian
C
Covariance Matrix
D
Densification
Differentiable Rendering
E
EWA (Elliptical Weighted Average)
I
Implicit Representation
J
Jacobian
L
L1 Loss
Light Field
Linear Blend Skinning (LBS)
M
Multi-View Stereo (MVS)
N
NeRF (Neural Radiance Field)
Novel View Synthesis
P
Parametric Model
Point Cloud
Pruning
Q
Quaternion
R
Radiance Field
Rasterization
Ray Marching
Ray Tracing
S
SMPL (Skinned Multi-Person Linear Model)
Spherical Harmonics
Splatting
SSIM (Structural Similarity Index Measure)
Structure-from-Motion (SfM)
T
Tile-Based Rendering
Transmittance
V
Volume Rendering
Voxel Grid