Mixed Methods Evaluation
Mariana Enriquez, PhD
Program Evaluation Consultant
Independent Consultant
Denver, Colorado, United States
Mariana Enriquez, PhD
Program Evaluation Consultant
Independent Consultant
Denver, Colorado, United States
Clea Vannet, MPH
Researcher
American Institutes for Research, United States
Clea Vannet, MPH
Researcher
American Institutes for Research, United States
Raquel Sanchez, Ph.D.
Senior Researcher
American Institutes for Research, United States
Raquel Sanchez, Ph.D.
Senior Researcher
American Institutes for Research, United States
Miriam Estrada, M.A.
Independent Program Evaluator
Independent Consultant, United States
Miriam Estrada, M.A.
Independent Program Evaluator
Independent Consultant, United States
Location: Room 201
Abstract Information: Conceived and implemented by Transformative Alliances, LLC, an anti-oppression and equity consulting group, the Community Leaders in Health Equity program (CLHE) was designed to increase awareness and understanding of systems of oppression, issues surrounding health equity, and the social determinants of health in diverse communities in Colorado. A team of evaluators from the American Institutes for Research (AIR) in collaboration with two local evaluators, developed a Communities of Practice framework to evaluate CLHE. The mixed methods evaluation included field observations, feedback surveys, focus groups, social network analysis (SNA), and participants’ self-reflections. Throughout the evaluation, participants’ voices were the common thread that helped make sense of the data and, most importantly, of the impact of CLHE on the participants and their communities. This multi-paper will include presentations of 1) the evolution of the field observations from highly structured to a narrative approach focused on impact on the program participants; 2) the qualitative data collected through focus groups and participants’ self-reflections; 3) the quantitative survey data and SNA; and 4) how the power of the participants’ voices became the guidepost that focused the overall evaluation of CLHE.
Relevance Statement: This multi-paper presentation will illustrate the evolution of the CLHE program evaluation. Each paper focuses on a different data source and how the approach to data collection and analysis was adapted to better reflect the participants’ experiences of the program over time. Program participants included members of traditionally underserved communities such as grassroots community members, people of color, immigrants, women, LGBTQ+ people, undocumented people, people with disabilities, and low-income people, from seven regions of Colorado. The mixed methods evaluation approach allowed the local evaluators to frequently engage in the program activities as participant observers, developing respectful and trusting relationships with the participants and the implementation team, which allowed the collection of the personal stories and inclusion of the diverse participants’ voices. The first paper describes the program observation data and analysis. Originally planned as a systematic and highly structured observation of behaviors and activities, this component of the evaluation was modified to open-ended field notes addressing indicators relevant to the intervention and reported in narrative form. This approach documented how participants experienced the program and demonstrated changes in their knowledge, attitudes and self-efficacy reflecting their transformation into change agents in their communities. The second paper describes the data collected through focus groups and self-reflections, as well as the primary findings from their analyses. These data collection methods offered participants a variety of formats to share their experience in and impact of the program in their lives, their families, and communities. A third paper introduces the communities of practice framework used to provide a structure to the development of the interactions between participants within and across regions and documented through the evaluators’ field notes and a social network analysis (SNA). This paper will include highlights of the data collected via a quantitative survey that collected participants’ self-report of their changes in knowledge and leadership skills. A fourth paper will describe how the participants’ voices gave meaning to the diversity of data collected, and how the power of their stories became the meaningful component of the evaluation. This last paper will include a description of a more participatory approach incorporated to engage participants in the interpretation of the evaluation results, an approach that showed limited success. The presentations in this multi-paper session will be especially relevant for beginners and intermediate evaluation practitioners by sharing with them the adaptations that were put into practice to adjust not only to the program but also to the context created by the Covid-19 pandemic, and the needs and interests of the funder, the implementation team and the participants themselves. Challenges and lessons learned also will be shared during the presentation.
References: Communities of Practice framework: https://wenger-trayner.com/introduction-to-communities-of-practice/ Mertens, D. M. (2011). Mixed Methods as Tools for Social Change. Journal of Mixed Methods Research, 5(3), 195–197. https://doi.org/10.1177/1558689811418095
Presenter: Raquel Sanchez, Ph.D. – American Institutes for Research
Presenter: Mariana Enriquez, PhD – Independent Consultant
Presenter: Clea Vannet, MPH – American Institutes for Research
Presenter: Miriam Estrada, M.A. – Independent Consultant