Virtual Humans Lecture

Contents:

  • Lecture 01.1 – Historical Body Models
    • Introduction
    • Early Origins: Simplified Primitives and Kinematic Skeletons (1970s–1980s)
    • Advances in the 1990s: Superquadrics, Differentiable Fitting, and Physical Models
    • The Impact of 3D Scanning and Data: From Anthropometry to Statistical Models (1990s–2000s)
    • SCAPE and the Emergence of Pose-Aware Models (Mid-2000s)
    • Consolidation in the 2010s: SMPL and Integration with Learning-Based Methods
    • Deep Learning and Neural Implicit Models (Late 2010s–Present)
    • Timeline Summary of Milestones
    • Conclusion
  • Lecture 01.2 – Introduction to Human Models
    • 1. Historical Context of Human Body Modeling
      • Early Developments
      • Mid-20th Century Approaches
      • Data-Driven Revolution (1990s-2000s)
    • 2. Mathematical Foundations of Human Body Models
      • The SMPL Model
      • PCA-Based Statistical Shape Modeling
      • Kinematic Modeling
    • 3. Applications of Human Body Models
      • Computer Animation and Visual Effects
      • Virtual Humans and Avatars
      • Biomechanics and Ergonomics
      • Human-Computer Interaction (HCI)
      • Computer Vision and AI
      • Education and Training
    • 4. Challenges and Future Directions
      • Computational Efficiency
      • Accuracy and Detail
      • Generalization
      • Clothing and Accessories
      • Emerging Approaches
    • 5. Conclusion
  • Lecture 01.3 – Introduction to Human Models (Overview)
    • 1. Historical Context
      • Early Scientific Studies
      • Mid-20th Century to Digital Era
      • 21st Century Advances
    • 2. Mathematical Foundations
      • Parametric Body Models
      • Implicit Surface Representations
      • Kinematic Modeling
    • 3. Image Formation and Rendering
      • Camera Models
      • Shading and Visibility
      • Differentiable Rendering
    • 4. Surface Representation Methods
      • Explicit Mesh Models
      • Implicit Function Models
    • 5. Motion Capture and Behavior Synthesis
      • Capturing Human Motion
      • Behavior Synthesis
    • 6. Clothing Modeling
      • Physically-Based Simulation
      • Data-Driven Approaches
      • Implicit Clothing Models
    • 7. Human-Object Interaction
      • Physics-Based Methods
      • Learning-Based Approaches
      • Hybrid Systems
    • 8. Applications
      • Entertainment and Media
      • Healthcare and Biomechanics
      • Engineering and Design
      • Human-Computer Interaction
      • Scientific Research
    • 9. Challenges and Future Directions
      • Scalability and Generalization
      • Higher-Fidelity Dynamics
      • Data and Labeling Constraints
      • Physics and Learning Integration
      • Semantic and Cognitive Aspects
      • Realism vs. Controllability
  • Lecture 02.1 – Image Formation
    • 1. Historical Developments in Image Formation
      • Ancient and Medieval Optics – Camera Obscura
      • Renaissance Perspective and Geometry
      • Early Cameras and Photographic Imaging
      • Modern Developments
    • 2. The Pinhole Camera Model
      • Coordinate Setup
      • Proof by Similar Triangles
      • Numerical Example
      • Inadequacy of a Simple Pinhole
    • 3. Camera Intrinsics and the Projection Matrix
      • Extrinsic Parameters
      • Full Projection Example
    • 4. Image Distortions & Correction
    • 5. Properties of Perspective Projection
    • 6. Advanced Theoretical Extensions
      • Light Field Imaging and Plenoptic Cameras
      • Non-Conventional Imaging Techniques
    • 7. Applications in Modern Vision and Graphics
      • Computer Vision and 3D Reconstruction
      • Medical Imaging
      • Photorealistic Rendering in Computer Graphics
    • 8. Python Example: Simulating Image Formation
  • Lecture 02.2 – Rotations and Kinematic Chains
    • 1. Representations of 3D Rotations
      • A) Rotation Matrices
      • B) Euler Angles
      • C) Quaternions
    • 2. Lie Algebra \(so(3)\) and Exponential Map
    • 3. Rodrigues’ Rotation Formula
    • 4. Kinematic Chains: Forward & Inverse Kinematics
    • Comparison of Rotation Representations
  • Lecture 03.1 – Surface Representations
    • 1. Mathematical Foundations of Surface Representations
      • A) Parametric Surfaces
      • B) Implicit Surfaces
      • C) Explicit Surfaces
    • 2. Surface Differential Properties
      • A) Surface Normals
      • B) Fundamental Forms and Curvature
      • C) Geodesics
    • 3. Discrete Surface Representations
      • A) Polygon Meshes
      • B) Point Clouds
      • C) Signed Distance Fields (SDF)
    • 4. Advanced Surface Representations
      • A) Bézier Curves and Surfaces
      • B) B-Splines and NURBS
      • C) Subdivision Surfaces
      • D) Level Sets
      • E) Neural Implicit Representations
    • 5. Comparative Analysis and Applications
      • A) Computational Efficiency and Storage
      • B) Practical Applications
      • C) Operations Complexity
    • 6. Implementation Examples
      • A) Basic Mesh Processing (Python)
      • B) Implicit Surface Utilities
      • C) Bézier Curve Implementation
      • D) Curvature Estimation on Meshes
    • 7. Advanced Topics and Future Directions
      • A) Multi-Resolution Representations
      • B) Machine Learning for Geometry
      • C) Dynamic Surfaces
      • D) Non-Manifold Geometries
  • Lecture 03.2 – Procrustes Alignment
    • Introduction
    • Goal: Learning a Model of Pose and Shape
    • The Challenge of Registration
    • Surface Representation: Mesh
    • The Procrustes Alignment Problem: Mathematical Formulation
    • Rigid Transformations
    • Procrustes Alignment Solution
      • Decoupling Translation by Centroid Alignment
      • Optimal Rotation via SVD
        • Reflection Adjustment
      • Optimal Scale (Optional)
    • Complete Mathematical Derivation
      • Translation Derivation
      • Rotation Derivation
      • Scale Derivation
    • Summary of Procrustes Alignment Algorithm
    • Python Implementation Example
    • Practical Applications
    • Interactive Visualization Ideas
  • Lecture 4.1: Iterative Closest Point
    • Introduction to Shape Alignment and Registration
      • The Registration Problem
    • Review: Procrustes Analysis
    • Problem: Unknown Correspondences
    • The Iterative Closest Point (ICP) Algorithm
      • Basic ICP Algorithm
      • Computational Considerations
        • Closest Point Computation
        • Convergence and Local Minima
    • Point-to-Point vs. Point-to-Plane ICP
      • Point-to-Point ICP
      • Point-to-Plane ICP
    • Gradient-based ICP for Non-Rigid Registration
      • Gradient-based ICP Algorithm
      • Advantages of Gradient-based ICP
      • Computing Gradients
    • Improving ICP’s Robustness
      • Data Association Direction
      • Robust Cost Functions
      • Trimmed ICP
      • RANSAC-based Approaches
      • Additional Information
    • Point-to-Surface Distance
    • ICP Variants and Extensions
      • Generalized ICP (GICP)
      • EM-ICP and Probabilistic Approaches
      • Coherent Point Drift (CPD)
      • Multi-Scale Approaches
    • Applications of ICP
    • Implementing ICP
      • Efficient Python Implementation
      • Practical Tips
    • Conclusion
  • Lecture 04.2 - Body Models: Vertex-Based Models and SMPL
    • 1. Body Models as Parameterized Functions
    • 2. Rotations, Articulation, and Pose Representation
      • 2.1 Rotation Representation
      • 2.2 Kinematic Chain
    • 3. Linear Blend Skinning and its Limitations
      • 3.1 Linear Blend Skinning
      • 3.2 Problems with Standard LBS
      • 3.3 Blend Shapes for Correcting LBS Artifacts
    • 4. The SMPL Body Model
      • 4.1 SMPL Philosophy
      • 4.2 SMPL Model Architecture
      • 4.2.1 Shape Blend Shapes
      • 4.2.2 Pose Blend Shapes
      • 4.2.3 Joint Regression
      • 4.3 Model Training
    • 5. Comparison with SCAPE
      • 5.1 The SCAPE Model
      • 5.2 Different Approaches to Deformation
      • 5.3 Performance Comparison
      • 5.4 Other Advantages
    • 6. Alignment Techniques: Procrustes Analysis and ICP
      • 6.1 Procrustes Analysis
      • 6.2 Iterative Closest Point (ICP)
      • 6.3 Fitting SMPL to Scans
    • 7. Image Formation and the Pinhole Camera Model
      • 7.1 The Pinhole Camera Model
      • 7.2 Lens Distortion
    • 8. Extensions and Advanced Applications
      • 8.1 Dynamic Soft Tissue Modeling
      • 8.2 Specialized Extensions
      • 8.3 Deep Learning for Model Fitting
      • 8.4 Probabilistic Approaches
      • 8.5 Hybrid Models
    • Conclusion
  • Lecture 5.1 - Training a Body Model and Fitting SMPL to Scans
    • Introduction
    • Body Models Based on Triangle Deformations
      • SCAPE and BlendSCAPE Models
      • Triangle Deformation Process
      • Comparison: SMPL vs. SCAPE/BlendSCAPE
    • Training a Body Model from Registrations
      • The Challenge of Raw Scan Data
      • Training from Registrations
    • Obtaining Registrations: Fitting SMPL to Scans
      • Non-Rigid Registration Process
      • Iterative Closest Point (ICP) Review
      • Registration Objective Formulation
      • Point-to-Surface Distance
      • Multi-Stage Optimization Strategy
    • Joint Registration and Model Training
      • Co-Registration Approach
    • Summary
  • Lecture 05.2 - 3D Registration: From Classical ICP to Modern Methods
    • Introduction
    • 1. Rigid Registration and the ICP Algorithm
      • The Iterative Closest Point (ICP) Algorithm
      • Convergence Analysis and Failure Modes
    • 2. Classical Non-Rigid Registration
      • Thin Plate Spline Robust Point Matching (TPS-RPM)
      • Coherent Point Drift (CPD)
      • Other Non-Rigid Methods
    • 3. Parametric Models and the SMPL Body Model
      • Model Structure
      • Shape Blend Shapes (Identity Variation)
      • Pose Blend Shapes (Pose-Dependent Deformation)
      • Linear Blend Skinning (LBS) for Articulation
      • Learning SMPL
      • Using SMPL for Registration
    • 4. Modeling Clothing and Fine Details: SMPL+D
      • Why SMPL+D?
      • How Displacements Are Applied
      • Limitations
      • Applications
    • 5. Survey of 3D Registration Methods: From ICP to Deep Learning
      • 5.1 Early Pioneering Works (1990s) – Foundational Rigid Registration
      • 5.2 The 2000s – Robust and Non-Rigid Registration Emerges
      • 5.3 2010s – Template-based and Parametric Model Registration
      • 5.4 2020s – Learning-Based Parametric Registration and Hybrid Approaches
    • Summary
  • Lecture 06.1 - Fitting the SMPL Model to Images via Optimization
    • Introduction
    • Mathematical Background: Pinhole Camera and Projections
      • Perspective Projection
      • Weak-Perspective Projection
      • Camera Extrinsics vs. Model Pose
      • 2D Keypoints and Projection
    • The SMPL Model as a Differentiable Function of Shape and Pose
      • Shape Blend Shapes
      • Pose Blend Shapes
      • Joint Positions
      • Linear Blend Skinning
      • Differentiability of SMPL
    • Fitting SMPL to Images via Optimization (SMPLify)
      • Objective Function
      • Combined Objective
      • Optimization Strategy
      • Optimization Algorithms
      • Automatic Differentiation and Jacobians
      • Result of SMPLify
    • Historical Progression and Method Comparisons
      • SMPLify (Bogo et al. 2016)
      • SMPLify-X (2019)
  • Lecture 06.2 - Learning-Based Fitting of the SMPL Model to Images
    • Introduction
    • Foundations of Learning-Based SMPL Estimation
      • Integrating SMPL into Neural Networks
      • Projection Functions
      • Loss Functions
      • Statistical Priors and Adversarial Losses
    • Early Regression Approaches: HMR and NBF
      • Human Mesh Recovery (HMR)
      • Neural Body Fitting (NBF)
    • Evolving Architectures: Hybrid and Improved Regression Methods
      • SPIN: Optimization in the Training Loop
      • PyMAF: Pyramidal Mesh Alignment Feedback
      • CLIFF: Using Full-Frame Context for Camera Orientation
      • PIXIE: Whole-Body Regression with Part Experts
    • Temporal Methods: From Single Images to Video Sequences
      • VIBE: Adversarial Motion Prior with GRUs
      • TCMR: Temporally Consistent Mesh Recovery
      • MotionBERT: Transformer-Based Motion Representations
    • Comparison of Learning-Based Methods
      • Supervision and Data
      • Network Architecture
      • Objective Functions
      • Pose and Shape Priors
      • Performance
      • Runtime
      • Strengths and Weaknesses
    • Conclusion
  • Lecture 07.1: Fitting SMPL to IMU Data Using Optimization-Based Methods
    • Introduction
    • Classical IMU-Based Pose Estimation: A Historical Perspective
      • Attitude and Heading Reference Systems
      • Kalman Filter Approaches
      • Early Sparse-Sensor Approaches
      • Model-Based Optimization Methods
      • Learning-Based Methods
    • Inertial Sensor Fundamentals and Orientation Representations
      • Gravity Alignment and Drift Correction
      • Orientation Representations
      • Sensor Calibration
    • Optimization-Based SMPL Fitting with IMU Data
      • Kinematic Model and Sensor Prediction
      • Regularization and Prior Terms
      • Gradient and Jacobian Computation
    • Pseudocode: SMPL Pose Estimation from IMU Sequence
    • IMU-Based Human Pose Datasets and Resources
      • DIP-IMU Dataset (2018)
      • TotalCapture (2017)
      • AMASS (2019)
      • Other Datasets and Resources
  • Lecture 07.2: Fitting SMPL to IMU Data Using Learning-Based Methods
    • Introduction
    • Optimization-Based vs. Learning-Based Approaches
    • Learning-Based IMU-to-Pose Estimation: Historical Overview of Key Models
      • Deep Inertial Poser (DIP, 2018)
      • TransPose (2021)
      • Transformer Inertial Poser (TIP, 2022)
      • Physics/Physical Inertial Poser (PIP, 2022)
      • Other Notable Models and Developments
    • Problem Formulation and Learning Task Definition
      • Input and Output Representations
      • Learning Objective and Loss Functions
      • Temporal Modeling Approaches
      • Supervised vs. Semi-Supervised Training; Synthetic Data
    • Model Architectures and Design Considerations
      • Encoding IMU Measurements
      • Network Structures
    • Training Pipeline and Pseudocode
    • Datasets, Benchmarks, and Resources
      • DIP-IMU Dataset (2018)
      • TotalCapture (2017)
      • AMASS (2019)
      • Other Datasets and Resources
    • Challenges and Outlook
    • Conclusion
  • Lecture 08.1: Vertex-Based Clothing Modeling for Virtual Humans
    • Introduction
    • Clothing Representation as Vertex Displacements
    • Registration of Clothed Human Scans
    • Estimating Body Shape Under Clothing: The BUFF Method
    • Multi-Layer Clothing Capture: The ClothCap Method
      • Data Term
      • Boundary Term
      • Boundary Smoothness
      • Laplacian Smoothness
    • Applications of Multi-Layer Registration
      • Advantages and Limitations
    • Conclusion
  • Lecture 09.1: Neural Implicit and Point-Based Representations for Clothed Human Modeling
    • Introduction
    • Background: Explicit vs. Implicit vs. Point-Based Representations
      • Mesh-Based Models
      • Point-Based Models
      • Neural Implicit Representations
      • Neural Radiance Fields (NeRFs) for Humans
      • Hybrid Approaches
    • Expressiveness and Topology
    • Differentiability and Learning
    • Data Efficiency and Performance
    • Neural Implicit Function Foundations
      • Signed Distance Field (SDF)
      • Occupancy Field (Indicator Function)
      • Volume Radiance Field
    • Articulated Deformation Fields for Implicit Models
      • Backward Warping (Inverse Skinning Field)
      • Forward Warping (Forward Skinning Field)
      • Pose-Dependent Deformation (Secondary Motion)
    • Generative Implicit Models for Clothed Bodies
      • SMPLicit: Topology-Aware Clothed Human Model
      • imGHUM: Implicit Generative Human Model
    • Pose-Dependent Implicit Models and Animatable Avatars
      • NASA: Neural Articulated Shape Approximation (ECCV 2020)
      • SCANimate: Weakly-Supervised Skinned Avatar Networks (CVPR 2021)
      • Neural-GIF: Neural Generalized Implicit Functions (ICCV 2021)
      • SNARF: Skinned Neural Articulated Implicit Shapes (ICCV 2021)
      • POP: The Power of Points (ICCV 2021) – Point-Based Modeling
      • Other Noteworthy Methods
    • Blueprint Algorithms for Key Methods
      • SNARF (Training Procedure): Differentiable Forward Skinning for Implicit Surfaces
      • POP (Training & Fitting Pipeline): Point-Based Model for Pose-Dependent Clothing
      • SMPLicit (Inference/Fitting Workflow): Generative Implicit Garment Model conditioned on SMPL
    • Historical Perspective and Future Outlook
      • Challenges and Future Directions
    • Conclusion
  • Neural Radiance Fields: A Historical and Theoretical Overview
    • Introduction
    • Foundations: 3D Scene Representation and Reconstruction Techniques
      • Voxel Grids
      • Point Clouds
      • Mesh-Based Surfaces
      • Light Fields and Volumetric Rendering
    • Emergence of Neural Radiance Fields (NeRF)
      • Core Idea
      • Training Procedure
      • NeRF Architecture
      • Hierarchical Sampling
      • Original Results
    • Theoretical and Mathematical Analysis of NeRF
      • Volume Rendering Formulation in NeRF
      • Positional Encoding and Neural Network Architecture
      • Loss Function and Optimization
    • Major Advancements and Extensions of NeRF
      • Anti-Aliasing and Unbounded Scenes: mip-NeRF and NeRF++
      • Efficiency Improvements: Instant NeRF and PlenOctrees
      • Dynamic and Deformable NeRFs (D-NeRF, Nerfies, NSFF, etc.)
      • Neural Radiance Fields for Human Modeling (with SMPL and Body Models)
      • Other Notable Extensions
    • Comparison with Other 3D Representations
      • Vs. Polygonal Meshes
      • Vs. Point Clouds / 3D Splatting
      • Vs. Voxel Grids and Volumetric Methods
      • Vs. Multi-Plane Images (MPIs) / Light Fields
      • Accuracy and Fidelity
      • Applicability
    • Datasets for NeRF Training and Evaluation
      • Blender Synthetic NeRF Dataset
      • Local Light Field Fusion (LLFF) Real Forward-Facing Dataset
      • Tanks and Temples
      • DTU Dataset
      • Human3.6M
      • ZJU-MoCap Dataset
      • People-Snapshot
      • Synthetic dynamic scenes
      • Other datasets
    • Conclusion
  • 3D Gaussian Splatting: A Basic Introduction
    • Introduction
    • Foundations
      • What is a 3D Scene?
      • 3D Scene Representation
      • Computer Graphics Fundamentals
      • Rasterization and Ray Tracing
      • Alpha Blending and Compositing
    • The Evolution of Novel View Synthesis
      • Image-Based Rendering
      • Structure-from-Motion and Multi-View Stereo
      • Point-Based Rendering
      • Neural Rendering
      • Accelerated Neural Fields
    • 3D Gaussian Splatting: A Convergence of Approaches
      • Point-Based Rendering
      • Point Clouds and Their Challenges
      • The Concept of Splatting
      • Elliptical Weighted Average (EWA) Filtering
      • Differentiable Point-Based Rendering
      • 3D Gaussian Splatting: Core Principles
      • Key Insight: Unifying Points and Volumes
      • 3D Gaussians as Scene Primitives
      • The Volumetric Rendering Equation
      • Alpha Compositing with Gaussians
    • Mathematical Formulation of 3D Gaussian Splatting
      • Projecting 3D Gaussians to 2D
      • Parameterization of 3D Gaussians
      • View-Dependent Appearance
      • Differentiable Rendering Equations
    • Training and Optimization
      • Photometric Loss
      • Initial Point Cloud
    • Optimization Process
      • Differentiable Splatting Pipeline
      • Adaptive Density Control
    • Implementation and Real-Time Rendering
      • Tile-Based Rendering
      • Fast Sorting Strategies
      • GPU-Accelerated Rasterization
      • Memory Considerations
    • Comparison with Other Methods
    • NeRF vs. 3D Gaussian Splatting
    • Voxel-Based Representations vs. Gaussians
    • Traditional Point-Based Rendering vs. Gaussian Splatting
    • Applications and Extensions
    • Static Scene Reconstruction
    • Dynamic Scene Capture
    • Avatar Creation and Animation
    • Integration with Neural Rendering
    • Large-Scale Scene Rendering
    • Bézier Gaussian Triangles (BG-Triangle) for Sharper Rendering
      • Representation
      • Performance
    • Human Reconstruction with Gaussian Splatting and Priors
      • EG-HumanNeRF: Efficient Generalizable Human NeRF
      • GPS-Gaussian: Pixel-Wise Gaussian Splatting for Humans
      • Generalizable Human Gaussians (GHG) with SMPL
    • Dynamic Scene Reconstruction with 4D Gaussian Splatting
      • 4D Gaussian Splatting (4DGS)
      • Speed and Memory Enhancements (4DGS-1K and MEGA)
      • Applications to MoCap and 4D Human Rendering
    • Implementation Details and Real-Time Performance
      • Data Structures
      • Rasterization & Shaders
      • GPU Memory and Throughput
      • Differentiable Rendering Implementation
      • Training vs. Inference Compute
      • Accuracy vs. Speed trade-offs
    • Benchmarks and Comparative Evaluation
      • Static Scene Comparison
      • Human Novel-View Comparison
      • Dynamic Scene Comparison
      • Comparative Summary Table
    • Datasets and Resources
      • Synthetic NeRF Dataset (Blender Scenes)
      • LLFF (Local Light Field Fusion)
      • Tanks and Temples
      • Multi-Object 360 (CO3D - Common Objects in 3D)
      • AMASS (Archive of Motion Capture as Surface Shapes)
      • CAPE (Clothed Auto Person Encoding)
      • THuman / THuman2.0
      • RenderPeople
      • Open-Source Implementations
    • Future Directions
      • Improved Compression Techniques for Memory Efficiency
      • Handling Dynamic and Deformable Scenes
      • Advanced Material Modeling for Realistic Rendering
      • Hybrid Approaches Integrating Neural Fields and Explicit Representations
      • Scalability for Large-Scale Scenes (City-Level and Beyond)
      • Real-Time Applications in AR/VR and Gaming
      • Integration with Existing Graphics Pipelines
      • Learning from Limited Data
    • Conclusion
    • Glossary
  • References
    • Lecture 01.1 (Historical Body Models)
    • Lecture 01.2 (Introduction to Human Models)
    • Lecture 01.3 (Introduction to Human Models Continued)
    • Lecture 02.1 (Image Formation)
    • Lecture 02.2 (Rotations & Kinematic Chains)
    • Lecture 03.1 (Surface Representations)
    • Lecture 03.2 (Procrustes Alignment)
    • Lecture 04.1 (Iterative Closest Points)
    • Lecture 04.2 (Body Models)
    • Lecture 05.1 (Body Model Training)
    • Lecture 05.2 (3D Registration)
    • Lecture 06.1 (Fitting SMPL to Images)
    • Lecture 06.1 (Optimization-Based Fitting of SMPL to Images)
    • Lecture 06.2 (Learning-Based Fitting of SMPL to Images)
    • Lecture 07.1 (Fitting SMPL to IMU Optimization)
    • Lecture 07.2 (Fitting SMPL to IMU Learning)
      • Classic and Optimization-Based Methods
      • Learning-Based Methods
    • Datasets and Resources
    • Relevant Software and Libraries
    • Lecture 08.1: References for Vertex-Based Clothing Modeling for Virtual Humans
    • Body Models and Shape Estimation
    • Alternative Representations
    • Datasets
    • Software and Libraries
    • Lecture 09.1: References for Neural Implicit and Point-Based Representations for Clothed Human Modeling
    • Point-Based Models
    • Datasets and Resources
    • Related Software and Libraries
    • Review Papers and Tutorials
    • Neural Radiance Fields (NERF)
    • Gaussian Splatting
Virtual Humans Lecture
  • Search

© Copyright .

Built with Sphinx using a theme provided by Read the Docs.