Face analysis is essential for a large number of applications such as human-computer interaction or multimedia (e.g. content indexing and retrieval). Although many approaches are under investigation, performance under uncontrolled conditions is still not satisfactory. The variations that impact facial appearance (e.g. pose, expression, illumination, occlusion, motion blur) make it a difficult problem to solve. This book describes the progress towards this goal, from a core building block – landmark detection – to the higher level of micro and macro expression recognition. Specifically, the book addresses the modeling of temporal information to coincide with the dynamic nature of the face. It also includes a benchmark of recent solutions along with details about the acquisition of a dataset for such tasks.

Table of Contents:
Preface xi Romain BELMONTE and Benjamin ALLAERT Part 1. Facial Landmark Detection 1 Introduction to Part 1 3 Romain BELMONTE, Pierre TIRILLY, IoanMarius BILASCO, Nacim IHADDADENE and Chaabane DJERABA Chapter 1. Facial Landmark Detection 13 Romain BELMONTE, Pierre TIRILLY, IoanMarius BILASCO, Nacim IHADDADENE and Chaabane DJERABA 1.1. Facial landmark detection in still images 14 1.1.1.Generativeapproaches 14 1.1.2.Discriminative approaches 18 1.1.3.Deep learningapproaches 24 1.1.4.Handlingchallenges 34 1.1.5.Summary 40 1.2.Extendingfacial landmarkdetectionto videos 41 1.2.1.Trackingby detection 41 1.2.2.Box, landmarkand pose tracking 43 1.2.3.Adaptive approaches 45 1.2.4. Joint approaches 46 1.2.5. Temporal constrained approaches 47 1.2.6.Summary 49 1.3.Discussion 50 1.4.References 52 Chapter 2. Effectiveness of Facial Landmark Detection 67 Romain BELMONTE, Pierre TIRILLY, IoanMarius BILASCO, Nacim IHADDADENE and Chaabane DJERABA 2.1.Overview 68 2.2.Datasets and evaluationmetrics 69 2.2.1. Image and videodatasets 69 2.2.2. Face preprocessing and data augmentation 73 2.2.3.Evaluationmetrics 75 2.2.4.Summary 77 2.3. Image andvideobenchmarks 77 2.3.1. Compiled results on 300W 77 2.3.2. Compiled results on 300VW 79 2.4.Cross-dataset benchmark 80 2.4.1.Evaluationprotocol 80 2.4.2.Comparisonof selected approaches 82 2.5.Discussion 86 2.6.References 88 Chapter 3. Facial Landmark Detection with Spatio-temporal Modeling 93 Romain BELMONTE, Pierre TIRILLY, IoanMarius BILASCO, Nacim IHADDADENE and Chaabane DJERABA 3.1.Overview 94 3.2.Spatio-temporalmodelingreview 95 3.2.1.Hand-craftedapproaches 95 3.2.2.Deep learningapproaches 97 3.2.3.Summary 103 3.3.Architecturedesign 104 3.3.1. Coordinate regression networks 104 3.3.2.Heatmapregressionnetworks 106 3.4.Experiments 107 3.4.1.Datasets andevaluationprotocols 107 3.4.2. Implementationdetails 108 3.4.3.EvaluationonSNaP-2DFe 109 3.4.4. Evaluation on 300VW 111 3.4.5.Comparisonwith existingmodels 112 3.4.6. Qualitative results 112 3.4.7.Propertiesof the networks 114 3.5.Design investigations 114 3.5.1.Encoder-decoder 115 3.5.2. Complementarity between spatial and temporal information 117 3.5.3. Complementarity between local and global motion 119 3.6.Discussion 122 3.7.References 123 Conclusion to Part 1 133 Romain BELMONTE, Pierre TIRILLY, IoanMarius BILASCO, Nacim IHADDADENE and Chaabane DJERABA Part 2. Facial Expression Analysis 147 Introduction to Part 2 149 Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA Chapter 4. Extraction of Facial Features 157 Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA 4.1. Introduction 157 4.2.Face detection 158 4.2.1.Point-of-interestdetectionalgorithms 160 4.2.2.Face alignment approaches 162 4.2.3.Synthesis 166 4.3.Face normalization 166 4.3.1.Dealingwith headpose variations 167 4.3.2.Dealingwith facial occlusions 170 4.3.3.Synthesis 172 4.4.Extractionof visual features 172 4.4.1.Facial appearancefeatures 172 4.4.2.Facial geometric features 174 4.4.3. Facial dynamics features 175 4.4.4.Facial segmentationmodels 177 4.4.5.Synthesis 179 4.5. Learning methods 179 4.5.1.Classification versus regression 180 4.5.2.Fusionmodel 182 4.5.3.Synthesis 184 4.6.Conclusion 185 4.7.References 186 Chapter 5. Facial Expression Modeling 191 Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA 5.1. Introduction 191 5.2.Modelingof the affective state 192 5.2.1.Categoricalmodeling 192 5.2.2.Dimensionalmodeling 194 5.2.3.Synthesis 196 5.3. The challenges of facial expression recognition 197 5.3.1. The variation of the intensity of the expressions 197 5.3.2.Variationof facialmovement 199 5.3.3.Synthesis 200 5.4.The learningdatabases 201 5.4.1. Improvementof learningdata 201 5.4.2. Comparison of learning databases 203 5.4.3.Synthesis 205 5.5. Invariance to facial expression intensities 206 5.5.1.Macro-expression 206 5.5.2.Micro-expression 208 5.5.3.Synthesis 209 5.6. Invarianceto facialmovements 211 5.6.1. Pose variations (PV) and large displacements (LD) 211 5.6.2.Synthesis 214 5.7.Conclusion 215 5.8.References 216 Chapter 6. Facial Motion Characteristics 223 Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA 6.1. Introduction 223 6.2.Characteristics of the facialmovement 225 6.2.1. Local constraint of magnitude and direction 226 6.2.2. Local constraint of the motion distribution 228 6.2.3.Motionpropagationconstraint 230 6.3.LMP 232 6.3.1. Local consistency of the movement 233 6.3.2.Consistencyof local distribution 236 6.3.3. Coherence in the propagationof themovement 238 6.4.Conclusion 241 6.5.References 242 Chapter 7. Micro- and Macro-Expression Analysis 243 Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA 7.1. Introduction 243 7.2. Definition of a facial segmentation model 244 7.3.Feature vector construction 247 7.3.1.Motionfeaturesvector 247 7.3.2.Geometric featuresvector 248 7.3.3.Features fusion 249 7.4. Recognition process 250 7.5. Evaluation on micro- and macro-expressions 251 7.5.1.Learningdatabases 252 7.5.2. Micro-expression recognition 253 7.5.3. Macro-expressions recognition 255 7.5.4. Synthesis of experiments on micro- and macro-expressions 258 7.6. Same expression with different intensities 260 7.6.1.Data preparation 260 7.6.2.Fractional time analysis 263 7.6.3.Analysis on a different time frame 264 7.6.4. Synthesis of experiments on activation segments 265 7.7.Conclusion 265 7.8.References 266 Chapter 8. Towards Adaptation to Head Pose Variations 271 Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA 8.1. Introduction 271 8.2.Learningdatabase challenges 273 8.3. Innovative acquisition system (SNaP-2DFe) 274 8.4. Evaluation of face normalization methods 276 8.4.1. Does the normalization preserve the facial geometry? 277 8.4.2. Does normalization preserve facial expressions? 280 8.5.Conclusion 283 8.6.References 284 Conclusion to Part 2 287 Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA List of Authors 293 Index 295