Costs, Effectiveness, Benefits, and Economics
Brian Yates, PhD
Professor
American University
Washington, District of Columbia, United States
Brian Yates, PhD
Professor
American University
Washington, District of Columbia, United States
Location: Room 301
Abstract Information: Following a quick introduction to cost-inclusive evaluation (CIE), three stories explore failure, success, and surprises happen when planning, implementing, and reporting cost-inclusive evaluations for a residential youth program, national policy revision, and consumer-operated psychological services. Participants are asked to engage in interactive meta-evaluation following each story, i.e., to discuss whether these events made the evaluations failures or successes, or some combination of both. - 45-minute presentation (Expert Lecture) - 10-minute definition, basic logic model, quick examples of Cost-Inclusive Evaluation, followed by … - 3 stories - 5-minute storytelling - 2-minute 1-question meta-analytic poll of participants: *Was this cost-inclusive evaluation a* … - failure? - success? - both (yes)? - each story segment closed with up to 5 minutes of Q&A with participants: why did you consider this a failure, a success, or something else? The three stories: 1. “That’s amazing! But what will you do your dissertation on?”, 2. Speaking truth to power by including costs, effectiveness, and benefits in Congressional testimony, and 3. You know you’re having an effect when your handouts get shredded and your slides get removed before distribution!
Relevance Statement: Including costs, and monetary as well as nonmonetary outcomes in an evaluation increases the stakes for evaluator and evaluand. Traditional approaches to evaluation, while interdisciplinary, usually balk at adding monetary measures of resources used and outcomes discovered. But much can be gained by this more complete form of evaluation, along with some surprises less common in traditional evaluation.
> The first story contrasts the enthusiasm with which program advocates and policy makers greet inclusion of costs and benefits in evaluation, with the emphasis on maintaining boundaries between cost-inclusive evaluation and one of its cousins, really rigorous research. For the evaluator in this story, initial success was met with apparent failure: a proposal for a cost-inclusive dissertation had to be dropped, an entirely new proposal written, and new resources accessed. Eventually, however, the doctorate earned in this dissertation opened doors to five decades (and counting!) of cost-inclusive evaluation in a wide variety of programs. Users of cost-inclusive evaluations, sometimes including the evaluand, often use simple and potentially simplistic rules for regulatory and funding decisions, to their eventual detriment.
> The second story illustrates how cost-inclusive evaluation findings can be used by program advocates to overturn policy that prevents competition in the marketplace of health services. For some advocates, and most policy makers, this seemed to be a successful application of findings from cost-inclusive evaluations that allowed less costly providers (Clinical Psychology Ph.D.s) to receive independent reimbursement for services that proved as effective and beneficial as services provided by psychiatrist M.Ds. The result fueled decades of growth in the less expensive health service, more recently challenged with similar evidence-based arguments from equally effective and even less costly providers, i.e., MAs in Counseling Psychology in addition to Masters in Psychiatric Social Work. The immediacy and apparent validity of judgements in cost-inclusive evaluations can obscure nuance and contextual specificity of evaluation findings. This can result in burying the evaluation, defunding the program, or both.
> The third story illustrates the danger with which findings of cost-inclusive evaluation are often regarded; threats so severe that even tentative, interim findings judged unfavorable to evaluands may be repressed to the point of destroying data visualizations of findings. A funder unexpectedly dropped in on a meeting of researchers and consumers in the midst of a four-year eight-site evaluation of adding consumer-operated services to traditional mental health services. Initial findings of the cost-inclusive evaluation were not particularly favorable, but were couched in multiple qualifications. Nonetheless, handouts from the cost-inclusive evaluation were gathered for disposal by the principal investigator of the entire multi-site evaluation. The cost-inclusive evaluation clearly had an impact, even early in the evaluation. Judging whether it was a failure or a success, or both, promises to bring insight into participants’ definition of evaluation, its uses, its dangers, and its future.