BUSINESS EXPERIMENTS with R A unique text that simplifies experimental business design and is dedicated to the R language Business Experiments with R offers a guide to, and explores the fundamentals of experimental business designs. The book fills a gap in the literature to provide a text on the topic of business statistics that addresses issues such as small samples, lack of normality, and data confounding. The author—a noted expert on the topic—puts the focus on the A/B tests (and their variants) that are widely used in industry, but not typically covered in business statistics textbooks. The text contains the tools needed to design and analyze two-treatment experiments (i.e., A/B tests) to answer business questions. The author highlights the strategic and technical issues involved in designing experiments that will truly affect organizations. The book then builds on the foundation in Part I and expands the multivariable testing. Since today’s companies are using experiments to solve a broad range of problems, Business Experiments with R is an essential resource for any business student. This important text: Presents the key ideas that business students need to know about experiments Offers a series of examples, focusing on a specific business question Helps develop the ability to frame ill-defined problems and determine what data and analysis would provide information about that problem Written for students of general business, marketing, and business analytics, Business Experiments with R is an important text that helps to answer business questions by highlighting the strategic and technical issues involved in designing experiments that will truly affect organizations.

Table of Contents:
Preface xiii Acknowledgments xvii Bruce McCullough xix About the Companion Website xxi 1 Why Experiment? 1 1.1 Case: Life Expectancy and Newspapers 2 Exercises 6 1.2 Case: Credit Card Defaults 7 1.2.1 Lurking Variables 9 1.2.2 Sample Selection Bias 11 Exercises 13 1.3 Case: Salk Polio Vaccine Trials 14 Exercises 17 1.4 What Is a Business Experiment? 17 1.4.1 Four Steps of an Experiment 21 1.4.2 Big Three of Causality 22 1.4.3 Most Experiments Fail 23 Exercises 24 1.5 Improving Website Designs 24 Exercises 30 1.6 A Brief History of Experiments 31 1.7 Chapter Exercises 34 1.8 Learning More 34 1.9 Statistics Refresher 38 2 Analyzing A/B Tests: Basics 43 2.1 Case: Improving Response to Sales Calls (Two-Sample Test of Means) 44 2.1.1 Initial Analysis and Visualization 45 2.1.2 Confidence Interval for Difference Between Means 47 2.1.3 Reporting Results 52 2.1.4 Hypothesis Test for Comparing Means 52 2.1.5 Power and Sample Size for Tests of Difference of Means 56 2.1.6 Considering Costs 60 Exercises 62 2.2 Case: Email Response Test (Two-Sample Test of Proportions) 64 2.2.1 Confidence Interval and Hypothesis Test for Comparing Two Proportions 66 2.2.2 Better Confidence Intervals for Comparing Two Proportions 66 2.2.3 Power and Sample Size for Tests of Difference of Two Proportions 68 Exercises 70 2.3 Case: Comparing Landing Pages (Two-Sample Test of Means, Again) 71 Exercises 74 2.4 Case: Display Ad Clickthrough Rate 75 2.4.1 Beta-Binomial Model 75 2.4.2 Comparing Two Proportions Using the Beta-Binomial Model 78 Exercises 80 2.5 Case: Hotel Ad Test 81 2.5.1 Tips on Presenting Experimental Findings 83 Exercises 84 2.6 Chapter Exercises 84 2.7 Learning More 86 3 Designing A/B Tests with Large Samples 91 3.1 The Average Treatment Effect 92 Exercises 93 3.2 Internal and External Validity 93 3.2.1 Threats to Internal Validity 93 3.2.2 Threats to External Validity 95 Exercises 96 3.3 Designing Conclusive Experiments 96 Exercises 102 3.4 The Lady Tasting Tea 103 Exercises 103 3.5 Testing a New Checkout Button 104 Exercises 104 3.6 Chapter Exercises 104 3.7 Learning More 104 4 Analyzing A/B Tests: Advanced Techniques 107 4.1 Case: Audio/Video Test Reprise (One-Sided Tests) 108 4.1.1 One-Sided Confidence Intervals 109 4.1.2 One-Sided Power 112 Exercises 113 4.2 Case: Typing Test (Paired t-Test) 114 4.2.1 Matched Pairs 115 Exercises 119 4.3 A/B/n Tests 121 Exercises 126 4.4 Minimum Detectable Effect 126 Exercises 128 4.5 Subgroup Analysis 129 4.5.1 Deficiencies of Subgroup Analysis 132 4.5.2 Subgroup Analysis of Bank Data 133 Exercises 135 4.6 Simpson’s Paradox 136 4.6.1 Sex Discrimination at UC Berkeley 137 4.6.2 Do You Want Kidney Stone Treatment A or Treatment B? 138 4.6.3 When the Subgroup Is Misleading 140 Exercises 143 4.7 Test and Roll 143 Exercises 145 4.8 Chapter Exercises 146 4.9 Learning More 146 4.10 Appendix on One-Sided CIs, Tests, and Sample Sizes 151 5 Designing Tests with Small Samples 159 5.1 Case: Call Center Scripts (ANOVA) 160 5.1.1 Blocking 161 Exercises 165 5.2 Case: Facebook Geo-Testing (Latin Square Design) 166 5.2.1 More on Latin Square Designs 169 5.2.2 Latin Squares and Degrees of Freedom 172 Exercises 175 5.3 Dealing with Covariate Imbalance 177 5.3.1 Matching 179 5.3.2 Rerandomization 180 5.3.3 Propensity Score 181 5.3.4 Optimal Matching 184 5.3.5 Sophisticated Matching: Selling Slushies 184 Exercises 185 5.4 Chapter Exercises 187 5.5 Learning More 188 6 Analyzing Designs via Regression 193 6.1 Experiments and Linear Regression 193 Exercises 198 6.2 Dummies, Effect Coding, and Orthogonality 198 Exercises 203 6.3 Case: Loan Experiment Revisited (Interactions) 203 6.3.1 Interactions 203 6.3.2 Loan Experiment 208 Exercises 215 6.4 Case: Direct Mail (Three-Way Interactions) 215 Exercises 224 6.5 Pretreatment Covariates in Regression 224 Exercises 225 6.6 Chapter Exercises 226 6.7 Learning More 228 6.8 Appendix: The Covariance Matrix of the Regression Coefficients 233 7 Two-Level Full Factorial Experiments 237 7.1 Case: The Postcard Example 238 Exercises 247 7.2 Case: Email Campaign 247 Exercises 250 7.3 The Determinant of a Matrix 252 Exercises 257 7.4 Aliasing 258 Exercises 264 7.5 Blocking (Again) 265 Exercises 269 7.6 Mee’s Blunders 269 7.7 Chapter Exercises 270 7.8 Learning More 271 7.9 Appendix on aliasMatrix and colorMap 273 8 Two-Level Screening Designs 279 8.1 Preliminaries 280 Exercises 286 8.2 Case: Puncture Resistance (Small Screening Experiment) 287 Exercises 288 8.3 Case: College Giving (Big Screening Experiment) 289 Exercises 292 8.4 How to Set Up a Screening Experiment 294 Exercises 295 8.5 Creating a Screening Design 295 Exercises 298 8.6 Chapter Exercises 299 8.7 Learning More 300 9 Custom Design of Experiments 305 9.1 Case: Selling Used Cars at Auction I (Small Custom Screening) 306 9.1.1 Create the Design 307 9.1.2 Evaluate the Design 312 9.1.3 Use the Design 316 Exercises 319 9.2 Case: Selling Used Cars at Auction II (Custom Experiment) 319 Exercises 322 9.3 Custom Experiment with Blocking 322 Exercises 324 9.4 Custom Screening Experiments 326 Exercises 331 9.5 More Than Two Levels 332 Exercises 337 9.6 Chapter Exercises 338 9.7 Learning More 338 10 Epilogue 341 10.1 The Sequential Nature of Experimentation 342 10.2 Approaches to Sequential Experimentation 345 References 347 Index 357