Uncover common project management myths to improve project success
How to Measure Anything in Project Management explains why popular methods for measurement in project management are flawed and describes how to conduct measurements that better inform decisions, reduce project risks, and improve the chance of project success. The authors argue that anything that matters to project management at all is measurable and that these measurements address many of the problems in project management. The authors leverage an exclusive survey on the state-of-the-art of measuring projects, new case studies of things that are seemingly hard to measure and a database, collected by Oxford Global Projects, of thousands of projects in software development, construction, energy, and many other fields, including some of the biggest projects in history. The book is accompanied by a set of useful spreadsheet-based "power tools" that support the more technical aspects of quantifying project risk, forecasting outcomes, and conducting seemingly difficult measurements. In this book, readers will learn:
- Why many of the methods they have been taught to use are little more than a type of “analysis placebo”
- Why many popular methods lead to extreme overconfidence in estimates
- How some of the most important measurements a project could conduct are currently rarely used
How to Measure Anything in Project Management earns a well-deserved spot on the bookshelves of managers, executives, auditors, controllers, and consultants seeking to improve project performance through superior measurement methodology.
Table of Contents:
Foreword xv
Preface xix
Acknowledgments xxi
About the Authors xxiii
Chapter 1 A World-scale Risk and a World-scale Opportunity 1
The Size of Projects 2
The Size of Project Problems 4
Efforts to Fix Projects: The Emergence of Project Management 5
A Path Forward: The Meta Project 8
Notes 10
Chapter 2 A Measurement Primer for Project Management 13
The Concept of Measurement 14
A Definition of Measurement 15
Measurement and Probabilities for Practical Decision-making 16
Are Scales Really Measurements? 18
The Object of Measurement 21
What Do You See When You See More of It? 21
Why Do You Care? 23
The Methods of Measurement 25
Statistical Significance: What’s the Significance? 26
Small Samples Tell You More Than You Think 28
Other Sources of Measurement Aversion 30
The Cost Objection 30
Measurements Change What Is Being Measured 31
Statistics Can Prove Anything 32
Ethical Objections to Measurement 33
Notes 34
Chapter 3 How We Know What Works 35
Skepticism for Project Managers 36
The Analysis Placebo 36
The Problem of Feedback and Learning 38
How to Test Methods 40
Controlled Experiments and Component Testing 40
Evaluating Sources 41
The Performance of Quantitative Methods 43
Experts Versus Algorithms 43
The Exsupero Ursus Fallacy: Algorithm Aversion 44
Potential Reasons for Exsupero Ursus 45
Improving the Human Expert 47
Calibrating the Expert 48
The Expert Consistency Component 49
Collaboration on Estimates 50
The Decomposition Component 52
A Summary of Research on Other Project Planning and Management Methods 54
Reference Class Forecasting 54
Various Project Management Methods 55
The Performance of Monte Carlo Simulations 58
Notes 60
Chapter 4 The Project Decision Model: The Reason for Measurements 63
Two Types of Project Measurements 64
Proto-purpose Discovery Measurements 64
Decision-driven Measurements 66
Unproductive Incentives vs. Measurements 69
Decisions Before: Thinking Slow 70
Exploration vs. Exploitation 71
Tracking the Outside World 73
Choosing How to Run the Project 74
How Models Indicate What to Measure 77
The Expected Value of Information: A Simple Introduction 77
The Measurement Inversion: Measuring the Wrong Things 79
The Value of Imperfect Measurements 80
An Aspirational Model 82
The Rise of Digital Twins 83
Digital Twins in Project Management 84
A Practical Path Forward 87
Notes 88
Chapter 5 Project Uncertainty and Risk: A Primer 91
Basic Concepts and Definitions 92
Uncertainty as a Probability Distribution 93
Risk: A Special Case of Uncertainty 96
The Problem with Current Methods 98
Why Risk “Scores” Don’t Work 99
How the Risk Matrix Makes Scores Worse 101
A Quantitative Risk Model: Starting Very Simple 105
The One-for-One Substitution 106
Monte Carlo Mechanics: A Brief Introduction 108
Supporting Decisions 111
A Return on Mitigation 112
How Much Risk Do You Tolerate? 113
Risk Versus Return: The Powerful Theory of Utility 115
Simple Tools for Measuring Uncertainty and Risk 117
A First Estimate of a Discrete Probability 118
A First Estimate of a Continuous Probability 119
Final Clarifications 120
Case Examples for What Probability Means 121
Uncertainty Versus Risk Versus Opportunity 123
Epistemic Versus Aleatory Uncertainty 124
Even More Ordinal Scales 125
Risk as Governance or Compliance 125
The Problem of “Black Swans” 126
Some Items That Aren’t Really Risks 127
More Improvements to Come 128
Notes 129
Chapter 6 Calibrated Subjective Probability Estimates 131
Introduction to Subjective Probability 132
Calibration Exercise 135
The Calibration Exercises 136
Evaluating Performance and Typical Results 137
Improving Calibration 140
The Equivalent Bet 141
More Techniques 142
More Advanced Calibration Topics to Come 144
The Effects of Calibration 146
Conceptual Obstacles to Calibration 149
Conflating Uncertainty with Knowing Nothing 149
Hypotheses That Contradict the Data 152
Objections Based on the Philosophical Debate in Statistics 153
Notes 155
Chapter 7 Cost and Schedule Measurements 157
The Big Plan Versus Iteration: Meta-measurements of Common Estimation Methods 158
Top-down Estimations: Reference Class Forecasting 162
Bottom-up Forecasting with Monte Carlo 165
A Deterministic View of Tasks 165
Probability Distributions for Project Tasks 167
Correlations 168
Multiple Prerequisites and Stochastic Critical Paths 170
Parade of Trades 171
Comparing Top Down and Bottom Up: Case Examples 174
The Swedish Nuclear Waste Program 175
High-speed Rail 176
How to Improve the Models 181
The Granularity of the Monte Carlo Model 182
Distributions and Biases 182
Correlations 183
Improving the RCF with Monte Carlo 184
Notes 185
Chapter 8 Betting on Benefits 187
Meta-measurements of Benefits 189
How Much Should Benefits Be to Justify a Project? 190
Why This May Be Optimistic 192
Why Measuring Benefits Is Rare 195
Fermi Decompositions for Benefits 196
Introduction to Fermi 197
Some Example Decompositions 199
Monetizing Benefits 201
Forecasts of Monetary Impacts 201
Preferences 202
Quantifying Preferences 203
The Use of Scores and Multiple Objectives 205
An Example of Challenging Benefit Measurement: Biodiversity 206
Measuring What Matters in Projects 206
A (Slightly) More Realistic Information Value Calculation 207
The High Information Values for Projects 209
Getting Started on Measuring What Matters 211
Considering Risk and Return 213
A Risk Neutral Decision-maker for Projects 214
Adding Utility Theory to Projects 215
Some Alternatives within Utility Math 217
Are Executives Too Risk Averse for Projects? 219
A Framework and Its Consequences 221
Findings from Quantitative Analysis of Past Projects 223
How and When, Not Just Whether 223
Benefits Are Not Just for Project Approval Decisions 224
Notes 225
Chapter 9 Measuring Progress 227
The Progress Problem 227
Simple Progress, Simple Interventions 228
Earned Value Management 229
EVM Basics 230
The XRL Example 231
Recovery vs. Performance 233
Forecasting with EVM 235
Progress in Information Projects 237
Waterfall 237
Agile and Measurement in Other Software Development Methods 237
Summarizing Software Metric Difficulties 239
Four Stories and Lessons 240
Interfaces in a Global Bank Transformation 240
An Energy Project Front End 241
Construction Constraints 243
Testing as Software Checkpoints 245
Lessons 246
The Remaining Project Simulation 247
Conditional Reference Class Forecasting (CRCF) 247
The Bottom-up Simulation for the Remaining Project 251
Further Considerations for the RPA 252
Notes 254
Chapter 10 More Measurement Methods Made Easy 257
Intuition for the Habitually Scientific 258
A Jelly Bean Example 258
A Little Probability Theory 260
Consequences of Probability Theory 262
Myths Exposed by Probability Theory 262
Significant Points About Statistical Significance 265
Basic Sampling Methods 266
The “Mathless” Table for Medians 269
Estimating a Population Proportion 270
Project Cancellation Rates as a Function of Duration 274
Measuring Population Size 274
Measuring Some Things by Knowing Other Things 276
Controlled Experiments 277
Regression 277
More Advanced Methods of Regression and Classification 283
Estimating the Whole Distribution 285
Summarizing Methods 289
Brainstorming a Measurement Approach 289
Data Gathering Methods 291
A Review of Methods in This Chapter 292
Notes on Surveys 293
Notes 296
Chapter 11 The Meta-project: Implementing Better Project Measurements 297
Start with the End in Mind: The Continuous Improvement Process 299
Measure What Matters 299
(Real) Skepticism and Meta-measurements 301
Measuring and Forecasting the Outside World 302
AI: The Most Important Project Ecosystem Measurement? 304
More Thinking, Fewer Projects, Bigger Wins 307
Start Your Meta-project 307
Examples of Meta-projects Deliverables: Continuous Improvement 308
Develop an Initial Team 309
Assess the Current State of the Project Portfolio 310
Considerations for the Meta-project Plan 312
The Pilot Project 312
Scaling to the Final Deliverable 314
Organizational Challenges 315
Resistance to Change 315
Addressing Organizational Objections to Measurement 316
The Politics of Measurement 318
Notes 319
Chapter 12 A Call to Action for the Industry 321
Call for Action for Project Software Vendors 321
Put Decisions at the Center 322
Deal in Uncertainties 324
Build the User-buyer-builder Federation 325
Be the Vendor That Measures Its Performance 325
Be Forward-looking 326
Call for Action for the Standard-setting Bodies 327
Call to Action for Consultants, Researchers, and Advisory Firms 329
Big Future Projects 331
A Mars Mission 331
Stopping Hurricanes 332
The Meta-Project 333
Notes 333
Appendix 1 Analysis of Survey Responses on Project Management Practices 335
Introduction and data overview 335
Success Metrics: Cost and Schedule Overrun Ratios 337
Overview of Project Management Practices Reported in the Survey 339
Project Management Methodologies 339
Cost and Schedule Estimation Methods 339
Uncertainty and Risk Assessment Tools 340
Certifications 341
Results 341
Project Management Methodologies 341
Cost and Schedule Estimation Methods 343
Uncertainty and Risk Assessment Tools 343
Certifications 343
Interpreting the (Mostly) Statistically Insignificant Results 344
Appendix 2 Reference Class Data on Project Cost, Schedule, and Benefit Overruns 345
Relevance of the Data and Reference Class Forecasting 346
Using Historical Data to Improve Estimates – An Example 347
Notes 351
Appendix 3 Selected Distributions 353
Uniform 354
Beta 355
Beta PERT 356
Triangular 357
Binary 358
Normal 359
Lognormal 360
Power Law 361
Truncated Power Law 362
Quantile-parameterized 363
Gamma Poisson 365
Stochastic Information Packet 366
Appendix 4 Chapter 6 Calibration Question Answers 369
Answers to Confidence Interval Questions 369
Answers to True/False Questions 371
Appendix 5 Measuring Biodiversity 373
The Benefits of Biodiversity 373
Measuring Biodiversity 375
Notes 376
Index 377
