Use and Influence of Evaluation
Jackie Siegel, MPH (he/him/his)
Project Associate
Rosov Consulting, Canada
Allison Magagnosc, MA (she/her/hers)
Senior Project Associate
Rosov Consulting, United States
Location: White River Ballroom A
Abstract Information: Making equity central to evaluation practices is a topic of paramount importance to professionals in our field, and it requires thinking critically about each phase of the evaluation process. Our research and evaluation professional services firm has recently focused on the question of whether and how to ask survey respondents about how they identify (such as gender and race). We wanted to determine the extent to which identity information helps us tell a more nuanced story for our clients, and reflect on our practices as a team to ensure that we collect such information with sensitivity, intent, and transparency. During our session, we will outline how we systematically reviewed our data collection practices and engaged our whole company in internal discussions. After sharing our findings and lessons learned, we will invite our attendees to reflect on the implications for their work.
Relevance Statement: From the U.S. Census to program evaluation surveys, questions about identity, such as gender and race, are routinely asked to help shed light on who is being served by programs and if there are disparate outcomes for subgroups of program participants. But if not deployed and analyzed intentionally, collecting that information can be an othering experience for participants and perpetuate inequality. With this awareness, our research and evaluation firm undertook a process to critically review our data collection procedures, focusing on how we ask survey respondents about their identity. Our presentation will outline how we systematically reviewed our data collection practices and designed a team approach to analyze and develop a style guide for collecting identity information. We reviewed all the surveys our firm administered in 2021 and documented the language used for collecting information about identity. Then, we purposefully selected a subset of reports that included various methodologies, study participants, and client types. We determined how—if at all—identity data were used in the analysis, and what, if any, conclusions were drawn based on participants’ characteristics. As a team, we had critical conversations throughout this process about how we can ensure respondents feel represented across the data collection and reporting phases of our evaluation projects. This critical analysis helped us understand the stories that lie behind the numbers. Our internal reflections enabled us to shape our research practices so as to build narratives for our clients that were more representative and authentic. We learned how data collection practices in which respondents see themselves reflected in the questions ensure they are centered in the story we want to tell. We believe that our modified internal practices build the relationship between respondent and researcher and invite respondents to want to share their unique story. Our work, and subsequent proposal, adds to knowledge in the evaluation field by providing a concrete example of how to undertake an internal review of existing data collection practices, and how to reflect on the findings to tell a more nuanced narrative about data and the people who share their stories with us. While every project has unique research goals and contexts, the questions we posed about our internal practices can help others to reflect on how and why they include questions about identity in their research. We hope that sharing our experiences and insights will inspire others to critically examine their own data collection practices and to continually strive for more equitable and inclusive evaluation and storytelling.