5 - Using Propensity Score Methods to Empower the Story of Program Evaluation

Abstract Information: Statistical causal inference is always challenging in evaluation practice because evaluators often use quasi-experiments or observational data to estimate treatment effects when randomized designs are not feasible. Unfortunately, the lack of random assignment to conditions makes it challenging for evaluators to ensure that they are providing valid evaluation results. Propensity score methods (PSM), which are used to improve covariate balance, are well-established methods for improving causal inferences. Therefore, this workshop will help evaluators learn ways to strengthen the validity of their evaluations through statistical adjustments to treatment effects when random assignment is not feasible. The workshop will include a review of problems when using quasi-experimental designs, an introduction to why and when to use PSM, hands-on activities, and demonstrations of how to use PSM. More specifically, we will cover basic theories and principles through examples, a step-by-step demonstration using real-world data, and practices on how to conduct PS matching and treatment effect estimation using R. We will provide attendees with exercise sheets for activities, flash drives containing R code, and sample data sets. Attendees are encouraged to bring laptops to follow demonstrations at their convenience but these are not required.

Relevance Statement: Evaluators often use non-randomized studies when experimental designs are not feasible. Unfortunately, internal validity is often threatened in quasi-experiments and observational studies due to selection bias (Rosenbaumn & Rubin, 1983). To tackle this problem, propensity score methods (PSM) are used to reduce selection bias by balancing the distributions of covariates between the compared groups to approximate the characteristics of experimental designs (Rosenbaumn & Rubin, 1983; 1985). Although PS methods have been gaining popularity in social science and behavioral research with regard to observational studies, many evaluators do not have a clear idea of how to use PSM to determine program effects. Therefore, the purpose of this workshop is to teach evaluators how to improve the accuracy of outcome evaluations when participants are not randomly assigned to programs.

The workshop will cover: (a) basic theories for using propensity score methods; (b) propensity score estimation approaches; (c) considerations for selecting the best adjustment methods; and (d) how to use R (a free program) to conduct propensity score methods (including matching, subclassification, and weighting), assess covariate balance, estimate the adjusted program effect, and conduct sensitivity analysis. During the many years that we have offered these professional development workshops on PSM, we have found that evaluators have increased their use of PSM and federal grant funding agencies have requested that evaluators use PSM when random assignment is not possible to evaluate program effects. Therefore, the demand for workshops and instruction on how to implement propensity score methods to enhance evaluation practices has also increased.

Dr. Clark and Dr. Bai have been providing instruction on propensity score methods for more than 15 years through workshops, professional development courses, webinars, and academic courses. In addition to several papers and presentations on the use and study of propensity scores, we have also published books on propensity score methods (Bai & Clark, 2019; Pan & Bai, 2015). Many of the workshops we have given have been for AEA attendees. Based on the feedback from previous AEA workshop attendees, we have continued to improve our workshop by using more time for hands-on activities and demonstrations. To keep up with changes in technology, we have focused our instruction on using R and including PSM other than matching. We believe that the need to improve causal estimates in evaluation, our experience, and our curriculum will make this a valuable workshop for program evaluators.

References

Bai, H., & Clark, M. H. (2019). Propensity score methods and applications. Sage.

Pan, W., & Bai, H. (2015). Propensity score analysis: Fundamentals and developments. Guilford.

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55.

Rosenbaum, P. R., & Rubin, D. B. (1985). Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician, 39(1), 33-38.