Many problems in the sciences and engineering can be rephrased as optimization problems on matrix search spaces endowed with a so-called manifold structure. This book shows how to exploit the special structure of such problems to develop efficient numerical algorithms. It places careful emphasis on both the numerical formulation of the algorithm and its differential geometric abstraction--illustrating how good algorithms draw equally from the insights of differential geometry, optimization, and numerical analysis. Two more theoretical chapters provide readers with the background in differential geometry necessary to algorithmic development. In the other chapters, several well-known optimization methods such as steepest descent and conjugate gradients are generalized to abstract manifolds. The book provides a generic development of each of these methods, building upon the material of the geometric chapters. It then guides readers through the calculations that turn these geometrically formulated methods into concrete numerical algorithms. The state-of-the-art algorithms given as examples are competitive with the best existing algorithms for a selection of eigenspace problems in numerical linear algebra. Optimization Algorithms on Matrix Manifolds offers techniques with broad applications in linear algebra, signal processing, data mining, computer vision, and statistical analysis. It can serve as a graduate-level textbook and will be of interest to applied mathematicians, engineers, and computer scientists.

Table of Contents:
List of Algorithms xi Foreword, by Paul Van Dooren xiii Notation Conventions xv Chapter 1. Introduction 1 Chapter 2. Motivation and Applications 5 2.1 A case study: the eigenvalue problem 5 2.1.1 The eigenvalue problem as an optimization problem 7 2.1.2 Some benefits of an optimization framework 9 2.2 Research problems 10 2.2.1 Singular value problem 10 2.2.2 Matrix approximations 12 2.2.3 Independent component analysis 13 2.2.4 Pose estimation and motion recovery 14 2.3 Notes and references 16 Chapter 3. Matrix Manifolds: First-Order Geometry 17 3.1 Manifolds 18 3.1.1 Definitions: charts, atlases, manifolds 18 3.1.2 The topology of a manifold* 20 3.1.3 How to recognize a manifold 21 3.1.4 Vector spaces as manifolds 22 3.1.5 The manifolds Rn x p and Rn x p 22 3.1.6 Product manifolds 23 3.2 Differentiable functions 24 3.2.1 Immersions and submersions 24 3.3 Embedded submanifolds 25 3.3.1 General theory 25 3.3.2 The Stiefel manifold 26 3.4 Quotient manifolds 27 3.4.1 Theory of quotient manifolds 27 3.4.2 Functions on quotient manifolds 29 3.4.3 The real projective space RPn x 1 30 3.4.4 The Grassmann manifold Grass(p, n) 30 3.5 Tangent vectors and differential maps 32 3.5.1 Tangent vectors 33 3.5.2 Tangent vectors to a vector space 35 3.5.3 Tangent bundle 36 3.5.4 Vector fields 36 3.5.5 Tangent vectors as derivations? 37 3.5.6 Differential of a mapping 38 3.5.7 Tangent vectors to embedded submanifolds 39 3.5.8 Tangent vectors to quotient manifolds 42 3.6 Riemannian metric, distance, and gradients 45 3.6.1 Riemannian submanifolds 47 3.6.2 Riemannian quotient manifolds 48 3.7 Notes and references 51 Chapter 4. Line-Search Algorithms on Manifolds 54 4.1 Retractions 54 4.1.1 Retractions on embedded submanifolds 56 4.1.2 Retractions on quotient manifolds 59 4.1.3 Retractions and local coordinates* 61 4.2 Line-search methods 62 4.3 Convergence analysis 63 4.3.1 Convergence on manifolds 63 4.3.2 A topological curiosity* 64 4.3.3 Convergence of line-search methods 65 4.4 Stability of fixed points 66 4.5 Speed of convergence 68 4.5.1 Order of convergence 68 4.5.2 Rate of convergence of line-search methods* 70 4.6 Rayleigh quotient minimization on the sphere 73 4.6.1 Cost function and gradient calculation 74 4.6.2 Critical points of the Rayleigh quotient 74 4.6.3 Armijo line search 76 4.6.4 Exact line search 78 4.6.5 Accelerated line search: locally optimal conjugate gradient 78 4.6.6 Links with the power method and inverse iteration 78 4.7 Refining eigenvector estimates 80 4.8 Brockett cost function on the Stiefel manifold 80 4.8.1 Cost function and search direction 80 4.8.2 Critical points 81 4.9 Rayleigh quotient minimization on the Grassmann manifold 83 4.9.1 Cost function and gradient calculation 83 4.9.2 Line-search algorithm 85 4.10 Notes and references 86 Chapter 5. Matrix Manifolds: Second-Order Geometry 91 5.1 Newton's method in Rn 91 5.2 Affine connections 93 5.3 Riemannian connection 96 5.3.1 Symmetric connections 96 5.3.2 Definition of the Riemannian connection 97 5.3.3 Riemannian connection on Riemannian submanifolds 98 5.3.4 Riemannian connection on quotient manifolds 100 5.4 Geodesics, exponential mapping, and parallel translation 101 5.5 Riemannian Hessian operator 104 5.6 Second covariant derivative* 108 5.7 Notes and references 110 Chapter 6. Newton's Method 111 6.1 Newton's method on manifolds 111 6.2 Riemannian Newton method for real-valued functions 113 6.3 Local convergence 114 6.3.1 Calculus approach to local convergence analysis 117 6.4 Rayleigh quotient algorithms 118 6.4.1 Rayleigh quotient on the sphere 118 6.4.2 Rayleigh quotient on the Grassmann manifold 120 6.4.3 Generalized eigenvalue problem 121 6.4.4 The nonsymmetric eigenvalue problem 125 6.4.5 Newton with subspace acceleration: Jacobi-Davidson 126 6.5 Analysis of Rayleigh quotient algorithms 128 6.5.1 Convergence analysis 128 6.5.2 Numerical implementation 129 6.6 Notes and references 131 Chapter 7. Trust-Region Methods 136 7.1 Models 137 7.1.1 Models in Rn 137 7.1.2 Models in general Euclidean spaces 137 7.1.3 Models on Riemannian manifolds 138 7.2 Trust-region methods 140 7.2.1 Trust-region methods in Rn 140 7.2.2 Trust-region methods on Riemannian manifolds 140 7.3 Computing a trust-region step 141 7.3.1 Computing a nearly exact solution 142 7.3.2 Improving on the Cauchy point 143 7.4 Convergence analysis 145 7.4.1 Global convergence 145 7.4.2 Local convergence 152 7.4.3 Discussion 158 7.5 Applications 159 7.5.1 Checklist 159 7.5.2 Symmetric eigenvalue decomposition 160 7.5.3 Computing an extreme eigenspace 161 7.6 Notes and references 165 Chapter 8. A Constellation of Superlinear Algorithms 168 8.1 Vector transport 168 8.1.1 Vector transport and affine connections 170 8.1.2 Vector transport by differentiated retraction 172 8.1.3 Vector transport on Riemannian submanifolds 174 8.1.4 Vector transport on quotient manifolds 174 8.2 Approximate Newton methods 175 8.2.1 Finite difference approximations 176 8.2.2 Secant methods 178 8.3 Conjugate gradients 180 8.3.1 Application: Rayleigh quotient minimization 183 8.4 Least-square methods 184 8.4.1 Gauss-Newton methods 186 8.4.2 Levenberg-Marquardt methods 187 8.5 Notes and references 188 A. Elements of Linear Algebra, Topology, and Calculus 189 A.1 Linear algebra 189 A.2 Topology 191 A.3 Functions 193 A.4 Asymptotic notation 194 A.5 Derivatives 195 A.6 Taylor's formula 198 Bibliography 201 Index 221