How to Find and Fix the Killer Software Bugs that Evade Conventional Testing
In Exploratory Software Testing, renowned software testing expert James Whittaker reveals the real causes of today’s most serious, well-hidden software bugs--and introduces powerful new “exploratory” techniques for finding and correcting them.
Drawing on nearly two decades of experience working at the cutting edge of testing with Google, Microsoft, and other top software organizations, Whittaker introduces innovative new processes for manual testing that are repeatable, prescriptive, teachable, and extremely effective. Whittaker defines both in-the-small techniques for individual testers and in-the-large techniques to supercharge test teams. He also introduces a hybrid strategy for injecting exploratory concepts into traditional scripted testing. You’ll learn when to use each, and how to use them all successfully.
Concise, entertaining, and actionable, this book introduces robust techniques that have been used extensively by real testers on shipping software, illuminating their actual experiences with these techniques, and the results they’ve achieved. Writing for testers, QA specialists, developers, program managers, and architects alike, Whittaker answers crucial questions such as:
• Why do some bugs remain invisible to automated testing--and how can I uncover them?
• What techniques will help me consistently discover and eliminate “show stopper” bugs?
• How do I make manual testing more effective--and less boring and unpleasant?
• What’s the most effective high-level test strategy for each project?
• Which inputs should I test when I can’t test them all?
• Which test cases will provide the best feature coverage?
• How can I get better results by combining exploratory testing with traditional script or scenario-based testing?
• How do I reflect feedback from the development process, such as code changes?
Table of Contents:
Foreword by Alan Page xv
Preface xvii
Chapter 1 The Case for Software Quality 1
The Magic of Software 1
The Failure of Software 4
Conclusion 9
Exercises 9
Chapter 2 The Case for Manual Testing 11
The Origin of Software Bugs 11
Preventing and Detecting Bugs 12
Manual Testing 14
Conclusion 19
Exercises 20
Chapter 3 Exploratory Testing in the Small 21
So You Want to Test Software? 21
Testing Is About Varying Things 23
User Input 23
What You Need to Know About User Input 24
How to Test User Input 25
State 32
What You Need to Know About Software State 32
How to Test Software State 33
Code Paths 35
User Data 36
Environment 36
Conclusion 37
Exercises 38
Chapter 4 Exploratory Testing in the Large 39
Exploring Software 39
The Tourist Metaphor 41
“Touring” Tests 43
Tours of the Business District 45
Tours Through the Historical District 51
Tours Through the Entertainment District 52
Tours Through the Tourist District 55
Tours Through the Hotel District 58
Tours Through the Seedy District 60
Putting the Tours to Use 62
Conclusion 63
Exercises 64
Chapter 5 Hybrid Exploratory Testing Techniques 65
Scenarios and Exploration 65
Applying Scenario-Based Exploratory Testing 67
Introducing Variation Through Scenario Operators 68
Inserting Steps 68
Removing Steps 69
Replacing Steps 70
Repeating Steps 70
Data Substitution 70
Environment Substitution 71
Introducing Variation Through Tours 72
The Money Tour 73
The Landmark Tour 73
The Intellectual Tour 73
The Back Alley Tour 73
The Obsessive-Compulsive Tour 73
The All-Nighter Tour 74
The Saboteur 74
The Collector’s Tour 74
The Supermodel Tour 74
The Supporting Actor Tour 74
The Rained-Out Tour 75
The Tour-Crasher Tour 75
Conclusion 75
Exercises 76
Chapter 6 Exploratory Testing in Practice 77
The Touring Test 77
Touring the Dynamics AX Client 78
Useful Tours for Exploration 79
The Collector’s Tour and Bugs as Souvenirs 81
Tour Tips 84
Using Tours to Find Bugs 86
Testing a Test Case Management Solution 86
The Rained-Out Tour 87
The Saboteur 88
The FedEx Tour 89
The TOGOF Tour 90
The Practice of Tours in Windows Mobile Devices 90
My Approach/Philosophy to Testing 91
Interesting Bugs Found Using Tours 92
Example of the Saboteur 94
Example of the Supermodel Tour 94
The Practice of Tours in Windows Media Player 97
Windows Media Player 97
The Garbage Collector’s Tour 97
The Supermodel Tour 100
The Intellectual Tour 100
The Intellectual Tour: Boundary Subtour 102
The Parking Lot Tour and the Practice of Tours in Visual Studio Team System Test Edition 103
Tours in Sprints 103
Parking Lot Tour 105
Test Planning and Managing with Tours 106
Defining the Landscape 106
Planning with Tours 107
Letting the Tours Run 109
Analysis of Tour Results 109
Making the Call: Milestone/Release 110
In Practice 110
Conclusion 111
Exercises 111
Chapter 7 Touring and Testing’s Primary Pain Points 113
The Five Pain Points of Software Testing 113
Aimlessness 114
Define What Needs to Be Tested 115
Determine When to Test 115
Determine How to Test 116
Repetitiveness 116
Know What Testing Has Already Occurred 117
Understand When to Inject Variation 117
Transiency 118
Monotony 119
Memorylessness 120
Conclusion 121
Exercises 122
Chapter 8 The Future of Software Testing 123
Welcome to the Future 123
The Heads-Up Display for Testers 124
“Testipedia” 126
Test Case Reuse 127
Test Atoms and Test Molecules 128
Virtualization of Test Assets 129
Visualization 129
Testing in the Future 132
Post-Release Testing 134
Conclusion 134
Exercises 135
Appendix A Building a Successful Career in Testing 137
How Did You Get into Testing? 137
Back to the Future 138
The Ascent 139
The Summit 140
The Descent 142
Appendix B A Selection of JW’s Professorial “Blog” 143
Teach Me Something 143
Software’s Ten Commandments 143
1. Thou Shalt Pummel Thine App with Multitudes of Input 145
2. Thou Shalt Covet Thy Neighbor’s Apps 145
3. Thou Shalt Seek Thee Out the Wise Oracle 146
4. Thou Shalt Not Worship Irreproducible Failures 146
5. Thou Shalt Honor Thy Model and Automation 146
6. Thou Shalt Hold Thy Developers Sins Against Them 147
7. Thou Shalt Revel in App Murder (Celebrate the BSOD) 147
8. Thou Shalt Keep Holy the Sabbath (Release) 148
9. Thou Shalt Covet Thy Developer’s Source Code 148
Testing Error Code 149
Will the Real Professional Testers Please Step Forward 151
The Common Denominators I Found Are (In No Particular Order) 152
My Advice Can Be Summarized as Follows 53
Strike Three, Time for a New Batter 154
Formal Methods 154
Tools 155
Process Improvement 156
The Fourth Proposal 156
Software Testing as an Art, a Craft and a Discipline 157
Restoring Respect to the Software Industry 160
The Well-Intentioned but Off-Target Past 160
Moving On to Better Ideas 161
A Process for Analyzing Security Holes and Quality Problems 161
Appendix C An Annotated Transcript of JW’s Microsoft Blog 165
Into the Blogoshere 165
July 2008 166
Before We Begin 166
PEST (Pub Exploration and Software Testing) 167
Measuring Testers 168
Prevention Versus Cure (Part 1) 169
Users and Johns 170
Ode to the Manual Tester 171
Prevention Versus Cure (Part 2) 173
Hail Europe! 174
The Poetry of Testing 175
Prevention Versus Cure (Part 3) 176
Back to Testing 177
August 2008 178
Prevention Versus Cure (Part 4) 179
If Microsoft Is So Good at Testing, Why Does Your Software Still Suck? 180
Prevention Versus Cure (Part 5) 183
Freestyle Exploratory Testing 183
Scenario-Based Exploratory Testing 183
Strategy-Based Exploratory Testing 184
Feedback-Based Exploratory Testing 184
The Future of Testing (Part 1) 184
The Future of Testing (Part 2) 186
September 2008 188
On Certification 188
The Future of Testing (Part 3) 189
The Future of Testing (Part 4) 191
The Future of Testing (Part 5) 192
October 2008 193
The Future of Testing (Part 6) 194
The Future of Testing (Part 7) 195
The Future of Testing (Part 8) 196
Speaking of Google 198
Manual Versus Automated Testing Again 198
November 2008 199
Software Tester Wanted 200
Keeping Testers in Test 200
December 2008 201
Google Versus Microsoft and the Dev:Test Ratio Debate 201
January 2009 202
The Zune Issue 203
Exploratory Testing Explained 204
Test Case Reuse 205
More About Test Case Reuse 206
I’m Back 207
Of Moles and Tainted Peanuts 208
Index 211
