An in-depth description of the state-of-the-art of 3D shape analysis techniques and their applications

This book discusses the different topics that come under the title of "3D shape analysis". It covers the theoretical foundations and the major solutions that have been presented in the literature. It also establishes links between solutions proposed by different communities that studied 3D shape, such as mathematics and statistics, medical imaging, computer vision, and computer graphics.

The first part of 3D Shape Analysis: Fundamentals, Theory, and Applications provides a review of the background concepts such as methods for the acquisition and representation of 3D geometries, and the fundamentals of geometry and topology. It specifically covers stereo matching, structured light, and intrinsic vs. extrinsic properties of shape. Parts 2 and 3 present a range of mathematical and algorithmic tools (which are used for e.g., global descriptors, keypoint detectors, local feature descriptors, and algorithms) that are commonly used for the detection, registration, recognition, classification, and retrieval of 3D objects. Both also place strong emphasis on recent techniques motivated by the spread of commodity devices for 3D acquisition. Part 4 demonstrates the use of these techniques in a selection of 3D shape analysis applications. It covers 3D face recognition, object recognition in 3D scenes, and 3D shape retrieval. It also discusses examples of semantic applications and cross domain 3D retrieval, i.e. how to retrieve 3D models using various types of modalities, e.g. sketches and/or images. The book concludes with a summary of the main ideas and discussions of the future trends.

3D Shape Analysis: Fundamentals, Theory, and Applications is an excellent reference for graduate students, researchers, and professionals in different fields of mathematics, computer science, and engineering. It is also ideal for courses in computer vision and computer graphics, as well as for those seeking 3D industrial/commercial solutions.

Preface xv

Acknowledgments xvii

1 Introduction 1

1.1 Motivation 1

1.2 The 3D Shape Analysis Problem 2

1.3 About This Book 5

1.4 Notation 9

Part I Foundations 11

2 Basic Elements of 3D Geometry and Topology 13

2.1 Elements of Differential Geometry 13

2.1.1 Parametric Curves 13

2.1.2 Continuous Surfaces 15

2.1.2.1 Differential Properties of Surfaces 17

2.1.2.1.1 First Fundamental Form 17

2.1.2.1.2 Second Fundamental Form and Shape Operator 18

2.1.2.2 Curvatures 19

2.1.2.3 Laplace and Laplace–Beltrami Operators 21

2.1.3 Manifolds, Metrics, and Geodesics 22

2.1.4 Discrete Surfaces 24

2.1.4.1 Representations of Discrete Surfaces 24

2.1.4.2 Mesh Data Structures 28

2.1.4.3 Discretization of the Differential Properties of Surfaces 29

2.2 Shape, Shape Transformations, and Deformations 30

2.2.1 Shape-Preserving Transformations 31

2.2.1.1 Normalization for Translation 32

2.2.1.2 Normalization for Scale 32

2.2.1.3 Normalization for Rotation 32

2.2.1.3.1 Rotation Normalization Using Principal Component Analysis (PCA) 33

2.2.1.3.2 Rotation Normalization Using Planar Reflection Symmetry Analysis 34

2.2.2 Shape Deformations 35

2.2.3 Bending 35

2.2.4 Stretching 37

2.3 Summary and Further Reading 38

3 3D Acquisition and Preprocessing 41

3.1 Introduction 41

3.2 3D Acquisition 41

3.2.1 Contact 3D Acquisition 43

3.2.1.1 Coordinate Measuring Machine (CMM) 43

3.2.1.2 Arm-Based 3D Scanner 44

3.2.2 Noncontact 3D Acquisition 44

3.2.2.1 Time-of-Flight 44

3.2.2.1.1 Pulse-Based TOF 44

3.2.2.1.2 Phase Shift-Based TOF 45

3.2.2.2 Triangulation 45

3.2.2.3 Stereo 47

3.2.2.4 Structured Light 50

3.2.2.4.1 Temporal Coded Patterns 51

3.2.2.4.2 Spatial Coded Patterns 52

3.2.2.4.3 Direct Coded Patterns 55

3.2.2.5 Shape from X 55

3.3 Preprocessing 3D Models 56

3.3.1 Surface Smoothing and Fairing 57

3.3.1.1 Laplacian Smoothing 57

3.3.1.2 Taubin Smoothing 58

3.3.1.3 Curvature Flow Smoothing 58

3.3.2 Spherical Parameterization of 3D Surfaces 58

3.4 Summary and Further Reading 62

Part II 3D Shape Descriptors 65

4 Global Shape Descriptors 67

4.1 Introduction 67

4.2 Distribution-Based Descriptors 69

4.2.1 Point Sampling 69

4.2.2 Geometric Features 70

4.2.2.1 Geometric Attributes 70

4.2.2.2 Differential Attributes 71

4.2.3 Signature Construction and Comparison 72

4.3 View-Based 3D Shape Descriptors 73

4.3.1 The Light Field Descriptors (LFD) 74

4.3.2 Feature Extraction 75

4.3.3 Properties 76

4.4 Spherical Function-Based Descriptors 77

4.4.1 Spherical Shape Functions 78

4.4.2 Comparing Spherical Functions 80

4.4.2.1 Spherical Harmonic Descriptors 80

4.4.2.2 SphericalWavelet Transforms 81

4.4.2.2.1 Wavelet Coefficients as a Shape Descriptor 82

4.4.2.2.2 SphericalWavelet Energy Signatures 82

4.5 Deep Neural Network-Based 3D Descriptors 83

4.5.1 CNN-Based Image Descriptors 84

4.5.2 Multiview CNN for 3D Shapes 85

4.5.2.1 Network Architecture 86

4.5.2.2 View Aggregation using CNN 86

4.5.3 Volumetric CNN 87

4.6 Summary and Further Reading 89

5 Local Shape Descriptors 93

5.1 Introduction 93

5.2 Challenges and Criteria 94

5.2.1 Challenges 94

5.2.2 Criteria for 3D Keypoint Detection 95

5.2.3 Criteria for Local Feature Description 96

5.3 3D Keypoint Detection 96

5.3.1 Fixed-Scale Keypoint Detection 97

5.3.1.1 Curvature-Based Methods 97

5.3.1.1.1 Local Surface Patch (LSP) 98

5.3.1.2 Other Surface Variation-Based Methods 98

5.3.1.2.1 Matei’s Method 99

5.3.1.2.2 Intrinsic Shape Signatures (ISS) 99

5.3.1.2.3 Harris 3D 99

5.3.2 Adaptive-Scale Keypoint Detection 101

5.3.2.1 Extrinsic Scale-Space Based Methods 101

5.3.2.1.1 3D Shape Filtering 101

5.3.2.1.2 Multiscale Surface Variation 104

5.3.2.2 Intrinsic Scale-Space Based Methods 106

5.3.2.2.1 Scale-Space on 2D Parameterized Images 106

5.3.2.2.2 Scale-Space on 3D Shapes 109

5.3.2.2.3 Scale-Space on Transformed Domains 112

5.4 Local Feature Description 113

5.4.1 Signature-Based Methods 114

5.4.1.1 Splash 114

5.4.1.2 Point Signature 115

5.4.2 Histogram Based Methods 115

5.4.2.1 Histogram of Spatial Distributions 115

5.4.2.1.1 Spin Images 116

5.4.2.1.2 3D Shape Context 117

5.4.2.1.3 Intrinsic Shape Signature (ISS) 118

5.4.2.1.4 Rotational Projection Statistics (RoPS) 118

5.4.2.2 Histogram of Geometric Attributes 122

5.4.2.2.1 Point Feature Histograms (PFH) 122

5.4.2.2.2 Fast Point Feature Histograms (FPFH) 123

5.4.2.2.3 Signature of Histograms of Orientations (SHOT) 123

5.4.2.3 Histogram of Oriented Gradients 124

5.4.3 Covariance-Based Methods 124

5.5 Feature Aggregation Using Bag of Feature Techniques 126

5.5.1 Dictionary Construction 127

5.5.1.1 Feature Extraction 127

5.5.1.2 Codebook Construction 127

5.5.2 Coding and Pooling Schemes 128

5.5.2.1 Sparse Coding 128

5.5.2.2 Fisher Vectors 129

5.5.3 Vector of Locally Aggregated Descriptors (VLAD) 129

5.5.4 Vector of Locally Aggregated Tensors (VLAT) 130

5.6 Summary and Further Reading 131

5.6.1 Summary of 3D Keypoint Detection 131

5.6.2 Summary of Local Feature Description 132

5.6.3 Summary of Feature Aggregation 133

Part III 3D Correspondence and Registration 135

6 Rigid Registration 137

6.1 Introduction 137

6.2 Coarse Registration 138

6.2.1 Point Correspondence-Based Registration 138

6.2.1.1 The Typical Pipeline 139

6.2.1.2 Transformation Estimation from a Group of Correspondences 139

6.2.1.3 Transformation Estimation fromThree Correspondences 140

6.2.1.4 Transformation Estimation from Two Correspondences 141

6.2.1.5 Transformation Estimation from One Correspondence 142

6.2.2 Line-Based Registration 143

6.2.2.1 Line Matching Method 143

6.2.2.2 Line Clustering Method 144

6.2.2.2.1 Rotation Estimation 145

6.2.2.2.2 Translation Estimation 146

6.2.3 Surface-Based Registration 146

6.2.3.1 Principal Component Analysis (PCA) 146

6.2.3.2 RANSAC-Based DARCES 147

6.2.3.3 Four-Points Congruent Sets (4PCS) 149

6.2.3.3.1 Affine Invariants of 4-Points Set 149

6.2.3.3.2 Congruent 4-Points Extraction 151

6.2.3.3.3 The 4PCS Algorithm 151

6.3 Fine Registration 152

6.3.1 Iterative Closest Point (ICP) 153

6.3.1.1 Closest Point Search 153

6.3.1.2 Transformation Estimation 153

6.3.1.3 Summary of the ICP Method 154

6.3.2 ICP Variants 155

6.3.2.1 Point Selection 155

6.3.2.2 Point Matching 156

6.3.2.3 Point PairWeighting 156

6.3.2.4 Point Pair Rejection 156

6.3.2.5 Error Metrics 157

6.3.3 Coherent Point Drift 157

6.4 Summary and Further Reading 160

7 Nonrigid Registration 161

7.1 Introduction 161

7.2 Problem Formulation 162

7.3 Mathematical Tools 165

7.3.1 The Space of Diffeomorphisms 165

7.3.2 Parameterizing Spaces 166

7.4 Isometric Correspondence and Registration 168

7.4.1 Möbius Voting 168

7.4.2 Examples 170

7.5 Nonisometric (Elastic) Correspondence and Registration 171

7.5.1 Surface Deformation Models 171

7.5.1.1 Linear Deformation Model 171

7.5.1.2 Elastic Deformation Models 172

7.5.2 Square-Root Normal Fields (SRNF) Representation 173

7.5.3 Numerical Inversion of SRNF Maps 174

7.5.3.1 SRNF Inversion Algorithm 176

7.5.4 Correspondence 177

7.5.4.1 Optimization Over SO(3) 178

7.5.4.2 Optimization Over Γ 178

7.5.4.3 Differential of 𝜙 [184] 179

7.5.4.4 Initialization of the Gradient [184] 179

7.5.5 Elastic Registration and Geodesics 181

7.5.6 Coregistration 181

7.6 Summary and Further Reading 184

8 Semantic Correspondences 187

8.1 Introduction 187

8.2 Mathematical Formulation 188

8.3 Graph Representation 191

8.3.1 Characterizing the Local Geometry and the Spatial Relations 191

8.3.1.1 Unary Descriptors 192

8.3.1.2 Binary Descriptors 192

8.3.2 Cross Mesh Pairing of Patches 192

8.4 Energy Functions for Semantic Labeling 194

8.4.1 The Data Term 194

8.4.2 Smoothness Terms 194

8.4.2.1 Smoothness Constraints 194

8.4.2.2 Geometric Compatibility 195

8.4.2.3 Label Compatibility 196

8.4.3 The Intermesh Term 196

8.5 Semantic Labeling 196

8.5.1 Unsupervised Clustering 197

8.5.2 Learning the Labeling Likelihood 199

8.5.2.1 GentleBoost Classifier 199

8.5.2.2 Training GentleBoost Classifiers 200

8.5.3 Learning the Remaining Parameters 201

8.5.4 Optimization Using Graph Cuts 202

8.6 Examples 202

8.7 Summary and Further Reading 204

Part IV Applications 207

9 Examples of 3D Semantic Applications 209

9.1 Introduction 209

9.2 Semantics: Shape or Status 209

9.3 Semantics: Class or Identity 212

9.4 Semantics: Behavior 216

9.5 Semantics: Position 219

9.6 Summary and Further Reading 221

10 3D Face Recognition 223

10.1 Introduction 223

10.2 3D Face Recognition Tasks, Challenges and Datasets 224

10.2.1 3D Face Verification 224

10.2.2 3D Face Identification 225

10.2.3 3D Face Recognition Challenges 225

10.2.3.1 Intrinsic Transformations 225

10.2.3.2 Acquisition Conditions 226

10.2.3.3 Data Acquisition 226

10.2.3.4 Computation Time 227

10.2.4 3D Face Datasets 227

10.3 3D Face Recognition Methods 228

10.3.1 Holistic Approaches 232

10.3.1.1 Eigenfaces and Fisherfaces 232

10.3.1.1.1 Eigenfaces 232

10.3.1.1.2 Fisherfaces 233

10.3.1.2 Iterative Closest Point 234

10.3.1.3 Hausdorff Distance 234

10.3.1.4 Canonical Form 234

10.3.2 Local Feature-Based Matching 235

10.3.2.1 Keypoint-Based Methods 235

10.3.2.1.1 Landmark-Based Methods 235

10.3.2.1.2 SIFT-Like Keypoints 236

10.3.2.2 Curve-Based Features 237

10.3.2.3 Patch-Based Features 238

10.3.2.4 Other Features 239

10.4 Summary 239

11 Object Recognition in 3D Scenes 241

11.1 Introduction 241

11.2 Surface Registration-Based Object Recognition Methods 241

11.2.1 Feature Matching 242

11.2.2 Hypothesis Generation 242

11.2.2.1 Geometric Consistency-Based Hypothesis Generation 243

11.2.2.2 Pose Clustering-Based Hypothesis Generation 244

11.2.2.3 Constrained Interpretation Tree-Based Hypothesis Generation 244

11.2.2.4 RANdom SAmple Consensus-Based Hypothesis Generation 245

11.2.2.5 GameTheory-Based Hypothesis Generation 246

11.2.2.5.1 Preliminary on Game Theory 246

11.2.2.5.2 Matching Game for Transformation Hypothesis Generation 247

11.2.2.6 Generalized Hough Transform-Based Hypothesis Generation 248

11.2.3 Hypothesis Verification 249

11.2.3.1 Individual Verification 249

11.2.3.2 Global Verification 251

11.3 Machine Learning-Based Object Recognition Methods 255

11.3.1 Hough Forest-Based 3D Object Detection 255

11.3.1.1 3D Local Patch Extraction 255

11.3.1.2 3D Local Patch Representation 256

11.3.1.3 Hough Forest Training and Testing 256

11.3.1.3.1 Offline Training 256

11.3.1.3.2 Online detection 258

11.3.2 Deep Learning-Based 3D Object Recognition 260

11.3.2.1 Hand-crafted Feature-Based Methods 262

11.3.2.2 2D View-Based Methods 262

11.3.2.3 3D Voxel-Based Methods 263

11.3.2.4 3D Point Cloud-Based Methods 265

11.4 Summary and Further Reading 265

12 3D Shape Retrieval 267

12.1 Introduction 267

12.2 Benchmarks and Evaluation Criteria 270

12.2.1 3D Datasets and Benchmarks 270

12.2.2 Performance Evaluation Metrics 271

12.2.2.1 Precision 272

12.2.2.2 Recall 272

12.2.2.3 Precision-Recall Curves 273

12.2.2.4 F- and E-Measures 273

12.2.2.5 Area under Curve (AUC) or Average Precision (AP) 273

12.2.2.6 Mean Average Precision (mAP) 274

12.2.2.7 Cumulated Gain-Based Measure 274

12.2.2.8 Nearest Neighbor (NN), First-Tier (FT), and Second-Tier (ST) 275

12.3 Similarity Measures 275

12.3.1 Dissimilarity Measures 275

12.3.2 Hashing and Hamming Distance 277

12.3.3 Manifold Ranking 278

12.4 3D Shape Retrieval Algorithms 280

12.4.1 Using Handcrafted Features 280

12.4.2 Deep Learning-Based Methods 282

12.5 Summary and Further Reading 284

13 Cross-domain Retrieval 285

13.1 Introduction 285

13.2 Challenges and Datasets 287

13.2.1 Datasets 288

13.2.2 Training Data Synthesis 289

13.2.2.1 Photo Synthesis from 3D Models 289

13.2.2.2 2D Sketch Synthesis from 3D Models 290

13.3 Siamese Network for Cross-domain Retrieval 290

13.4 3D Shape-centric Deep CNN 292

13.4.1 Embedding Space Construction 293

13.4.1.1 Principal Component Analysis 295

13.4.1.2 Multi-dimensional Scaling 296

13.4.1.3 Kernel-Based Analysis 296

13.4.2 Learning Shapes from Synthesized Data 298

13.4.3 Image and Sketch Projection 298

13.5 Summary and Further Reading 300

14 Conclusions and Perspectives 301

References 303

Index 337