Use Visual Studio 2010’s Breakthrough Testing Tools to Improve Quality Throughout the Entire Software Lifecycle
Together, Visual Studio 2010 Ultimate, Visual Studio Test Professional 2010, Lab Management 2010, and Team Foundation Server offer Microsoft developers the most sophisticated, well-integrated testing solution they’ve ever had. Now, Microsoft MVP and VS testing guru Jeff Levinson shows exactly how to use Microsoft’s new tools to save time, reduce costs, and improve quality throughout the entire development lifecycle.
Jeff demonstrates how Microsoft’s new tools can help you finally overcome long-standing communication, coordination, and management challenges. You’ll discover how to perform first-rate functional testing; quickly create and execute tests and record the results with log files and video; and create bugs directly from tests, ensuring reproducibility and eliminating wasted time. Levinson offers in-depth coverage of Microsoft’s powerful new testing metrics, helping you ensure traceability all the way from requirements through finished software.
Coverage includes
• Planning your tests using Microsoft Test Manager (MTM)
• Creating test settings, structuring test cases, and managing the testing process
• Executing manual tests with Microsoft Test Manager and Test Runner
• Filing and resolving bugs, and customizing your bug reporting process
• Automating test cases and linking automated tests with requirements
• Executing automated test cases through both Visual Studio and Microsoft Test Manager
• Integrating automated testing into the build process
• Using Microsoft’s Lab Management virtualization platform to test applications, snapshot environments, and reproduce bugs
• Implementing detailed metrics for evaluating quality and identifying improvements
Whether you’re a developer, tester, manager, or analyst, this book can help you significantly improve the way you work and the results you deliver—both as an individual right now, and as a team member throughout your entire project.
Table of Contents:
Foreword xvii
Preface xxi
Acknowledgments xxix
About the Author xxxi
1 State of Testing 1
Software Testing Challenges 1
The Need for Testers 3
A Different Approach 5
Fixing Communication 5
Increasing Project Visibility 6
What Are the Tools Designed to Do? 7
Metrics 10
Citations 12
2 Software Quality and Testing Overview 13
Software Quality 13
Requirements 14
Business Value 14
Expectations 15
Nonfunctional Requirements 15
Where Do You Build Quality? 17
Process and Quality 19
Software Testing 19
The Testing Mindset 20
Software Testing Strategies 21
Types of Software Testing 22
Test Management 27
After the Product Is Shipped or Deployed 27
3 Planning Your Testing 29
Microsoft Test Manager 30
Test Plans 36
Properties 38
Contents 43
Adding Suites and Test Cases to Your Plan 46
Testing Configurations 48
Managing Test Configurations 49
Assigning Test Configurations 51
Assigning Testers 53
Test Case Planning Workflow 55
Analysis and Initial Design 56
Construction 61
User Acceptance Testing 62
Common Scenarios 64
Scheduling and Tracking Test Case Creation and Execution 64
Feature Driven Development 65
Moving from One Iteration to Another 67
Handling Different Test Configurations 68
4 Executing Manual Tests 71
Using the Test Case Work Item Type 72
Shared Steps 75
Data Driven Test Cases (Test Parameters) 77
Running Your First Tests 79
Test Runner 80
Examining Test Results 92
Test Run Results 93
Detailed Test Results 95
Exploratory Testing with MTM 101
5 Resolving Bugs 107
A Bug’s Life 107
Customer Reported Bug 110
Test Team Reported Bug 110
Triaging the Bug 110
Reactivations 111
Bug Differences and Modifications 111
The Generated Bug 116
How a Developer Uses IntelliTrace 120
Fixing the Bug 122
Associated Changesets 124
Associated Work Items 124
Impacted Tests 125
Setting the Build Quality 125
Assigning a New Build 127
Verifying That the Bug Is Fixed 129
Dealing with Impacted Tests 131
6 Automating Test Cases 135
To Automate or Not to Automate 136
The Automated Testing Framework 139
Creating an Automated Test from a Manual Test 141
Examining a Generated Web Application Coded UI Test 142
Adding Validations 157
Adding Additional Recorded Steps 164
Parameterized Coded UI Tests 166
Handling Issues Due to Inconsistency 168
Resolving the Data Inconsistency 169
Handling Dynamic Values 172
Other Tips 177
Combining Multiple Tests 178
Associating Coded UI Tests and Test Cases 178
7 Executing Automated Test Cases 183
Executing Automated Tests Through Visual Studio 183
Local Execution 184
Local Execution with Remote Collection 184
Remote Execution 185
Executing Automated Tests from the Command Line 190
Executing Automated Tests in MTM 191
Creating an Automated Build 191
Setting Up the Physical Environment 193
Running a Coded UI Test Through MTM 196
Executing Automated Tests with Team Build 202
Automated Testing Gotchas 205
Custom Dialogs 205
Cleaning Up Your Tests 207
8 Lab Management 209
Managing Virtual Environments Through MTM 210
Finishing Virtual Environment Configuration 217
Automated Test Settings 221
Lab Management Workflow 222
Executing a Lab Build 231
Running Automated Tests Through MTM 233
Manual Tests in a Virtual Environment 234
9 Reporting and Metrics 239
Understanding the Reporting Structure 240
Built-In Reports 242
Bug Status 244
Bug Trends 245
Reactivations 246
Build Quality Indicators 246
Build Success over Time 248
Build Summary 249
Stories Overview 250
Test Case Readiness 251
Test Plan Progress 252
Excel Services Reports (Dashboards) 253
Reporting with Microsoft Excel 254
Creating a Generated Report 255
The Testing Measures 256
Metrics 268
What to Measure 271
First-Time Defect Rate 273
Bug Reactivations 276
General Bug Counts 277
Index 283
