A nuts-and-bolts introduction to program evaluation for the nonprofit and public sectors
Evaluation Essentials: Methods for Conducting Sound Research, 2nd Edition, is a substantial update to the popular overview of the evaluation process. Virtually every form of externally funded nonprofit activity must be periodically evaluated, and trained evaluators are in higher demand than ever before. This book offers a step-by-step introduction to the process and methods of program evaluation, with over 40 examples from public policy, public health, non-profit management, social work, arts management, education, international assistance, and labor.
Evaluation Essentials covers updates in the field of evaluation, including contribution analyses, as well as current best practices for forming evaluation questions, applying program theory, performing literature reviews, collecting data, creating outcome measures, designing and conducting surveys, grant writing, and much more. This Second Edition also includes an expanded international context, examining evaluation in international organizations. The process outlined in this book is also applicable to policy evaluation and the evaluation of organizational performance, adding additional value to this timely update.
- Gain a foundational knowledge of the principles and practices of program evaluation
- Design evaluations and conduct research, including quantitative and qualitative analysis
- Generate data that can be used to demonstrate a program's impact to funders and stakeholders
- Learn from examples drawn from a broad range of nonprofit organizations
This definitive guide to evaluation will appeal to professionals in fields as diverse as education, policy sciences, public administration, sociology, health, and beyond.
Table of Contents:
Preface ix
Acknowledgments x
One: Introduction 1
Learning Objectives 1
Introduction 2
Size and Importance of Nonprofit Sector 3
The Evaluation Framework 5
Structure of Book 9
Summary 9
Key Terms 10
Discussion Questions 10
Two: Describing the Initiative 11
Learning Objectives 11
Introduction 12
Reasons to Rigorously Describe the Program 13
The Logistics of Describing the Program 14
Common Mistakes Made Describing Initiative and Programs 16
The Program Is Alive and So Is Its Description 18
Theory of Change 19
The Program Logic Model 25
Analyzing the Program Logic Model 34
Challenges of Multisite Programs 35
Program Implementation Model 36
Examples of Program Descriptions 37
Summary 67
Key Terms 68
Discussion Questions 68
Three: Laying the Evaluation Ground Work 69
Learning Objectives 69
Evaluation Approaches 70
Framing Evaluation Questions 72
Insincere Reasons for Evaluation 75
High Stakes Evaluation 75
Who Will do the Evaluation? 76
External Evaluators 77
Internal Evaluators 79
Independence of the Evaluator 79
External Versus Internal Evaluators 80
Confidentiality and Ownership of Evaluations 81
Endorsement and Action on Recommendations 82
Evaluation Policy 83
The Evaluation Report 84
Summary 86
Key Terms 86
Discussion Questions 86
Four: Causation 87
Learning Objectives 87
Introduction 88
Necessary and Sufficient 88
Setting Cutoff Points and Causal Interpretation 91
Intervening Variables 95
Types of Causal Effects 97
Lagged Effects 98
Permanency of Effects 98
Functional Form of Impact 99
Spectacular Causes and Effects 100
Summary 101
Key Terms 102
Discussion Questions 102
Five: the Prisms of Validity 103
Learning Objectives 103
Introduction 104
Statistical Conclusion Validity 105
Types of Erroneous Conclusions: Type 1 and Type 2 Errors 106
Threats to Statistical Conclusion Validity 107
Internal Validity 111
Threats to Internal Validity 111
Summary 118
Construct Validity 119
Threats to Construct Validity 120
Summary 122
External Validity 122
Considerations for Determining the External Validity of Studies 122
Summary 123
Conclusions 124
Key Terms 124
Discussion Questions 124
Six: Attributing Outcomes to the Program: Quasi-Experimental Design 126
Learning Objectives 126
Introduction 127
Quasi-Experimental Notation 127
Frequently used Designs that do not Show Causation 128
One-Group Posttest-Only 128
Posttest-Only with Nonequivalent Groups 129
Participants’ Pretest–Posttest 130
Designs that Generally Permit Causal Inferences 131
Untreated Control Group Design with Pretest and Posttest 131
Delayed Treatment Control Group 137
Different Samples from the Same Populations Design 140
Nonequivalent Observations Drawn from One Group 141
Equivalent Groups Using Switched Measures 142
Cohort Designs 144
Time Series Designs 146
Summary 148
Key Terms 149
Discussion Questions 149
Seven: Collecting Data 150
Learning Objectives 150
Introduction 151
Informal and Loosely Structured Interviews 152
Focus Groups 154
Surveys 160
Survey Instrument Design 161
Target Populations and Sampling 169
Informal Sampling Approaches 170
Formal Sampling Approaches 171
Best Practices in Administering Surveys to a Purposive Sample 172
Follow-up for Nonresponse 173
Secondary Data Sources 173
Summary 174
Key Terms 174
Discussion Questions 175
Eight: Conclusions 176
Learning Objectives 176
Using Evaluation Tools to Write Grant Proposals 177
Hiring Evaluation Consultants 177
Writing Recommendations 180
The Tone of Evaluation Reports 180
Finalization Process of Evaluation Reports 181
Conclusions 182
Discussion Questions 182
Glossary 183
Index 188
