From Beginning to End: Engaging Stakeholders in Program Evaluation in a Rural, Historically Marginalized Community

Abstract Information: The U Care Connections program was founded in 2015 in Robeson County, home to the largest Native American tribe east of the Mississippi River and regularly ranked the least healthy county in North Carolina. U Care volunteers provide non-emergency patient transport to appointments, ensuring care continuity among patients who might otherwise miss chemotherapy or dialysis appointments. The U Care team was funded to engage the NC Healthcare Foundation (NCHF) for a program evaluation articulating the impact of U Care to spur sustained funding in Robeson County and beyond. Each step of the way, our program evaluation was closely linked to stakeholder needs. Multiple factors influenced our robust stakeholder engagement. Yarbrough’s Program Evaluation Standards (PES) provide a useful framework for explaining how we kept U Care stakeholders at the center of our program evaluation. In our presentation, we will use the PES to describe our program evaluation approach. Presentation attendees will be able to list and compare a subset of the PES, identify how to apply the PES outside the education sector in which it was originally designed, and discuss how the U Care evaluation evolved to prioritize the culture and assets of a historically marginalized community like Robeson County.

Relevance Statement: Introduction The integrity of a program evaluation is especially important in a historically marginalized community. The U Care Connections program was founded in 2015 in Robeson County, home to the largest Native American tribe east of the Mississippi River and regularly ranked the least healthy county in North Carolina. U Care reduces the harm of physical and social isolation by connecting patients with serious illness to reliable volunteer drivers, many of whom have experienced serious illness themselves. U Care volunteers provide non-emergency patient transport to appointments, ensuring care continuity among patients who might otherwise miss chemotherapy or dialysis appointments. The U Care team was funded to engage the NC Healthcare Foundation (NCHF) for a program evaluation to help demonstrate and articulate the impact of U Care, thus increasing the odds of the grant-funded program becoming a sustained service in Robeson County and beyond. Evaluation theory used Interest in stakeholder engagement for program evaluation tends to be more robust during development. At the point of implementing and disseminating program evaluation results, however, support for stakeholder engagement typically wanes. (i) Our program evaluation instead was closely linked, each step, to stakeholder needs. This meant that the traditional written evaluation report was less relevant than a documentary-style video produced by a consultant hired by the program lead, and a toolkit designed to showcase program impact and successful strategies for others considering implementing the model. Multiple factors influenced our robust stakeholder engagement. Yarbrough’s Program Evaluation Standards (PES) (ii, iii)provide a useful framework for explaining how we kept U Care stakeholders at the center of our program evaluation. Examples of some of the most relevant standards are contextual viability, responsive and inclusive orientation, and formal agreements. Relevance This proposal is relevant to the field of program evaluation for a demographic and a policy reason. First, program evaluation needs to become more focused on diverse populations and perspectives as the US becomes increasingly diverse. Second, a growing interest among state Medicaid offices in reimbursing for social determinants of health, such as transportation, makes it important for evaluators to demonstrate what grassroots efforts are effective. Attendees who view this presentation will be able to list and compare a subset of the PES, identify how to apply the PES outside the education sector in which it was originally designed, and discuss how the U Care evaluation evolved to continually prioritize the culture and assets of a historically marginalized community such as Robeson County. References i Concannon, T.W., Fuster, M., Saunders, T. et al. A Systematic Review of Stakeholder Engagement in Comparative Effectiveness and Patient-Centered Outcomes Research. J GEN INTERN MED 29, 1692–1701 (2014). https://doi.org/10.1007/s11606-014-2878-x ii Joint Committee on Standards for Educational Evaluation. (2018). Checklist of the program evaluation standards statements. Retrieved from https://wmich.edu/evaluation/checklists iii Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.