Use and Influence of Evaluation
Leslie Fierro, PhD, MPH (she/her/hers)
Sydney Duder Professor in Program Evaluation
McGill University, United States
Alysson Akiko Oakley, PhD (she/her/hers)
Vice President
Pact, United States
Todd Franke, PhD
Professor
Department of Social Welfare, UCLA, United States
Bianca Montrosse-Moorhead, PhD
Associate Professor
University of Connecticut, Connecticut, United States
Sarah Mason, PhD
Director, Center for Research Evaluation
The University of Mississippi, United States
Laura Peck, PhD (she/her/hers)
Principal Scientist
Abt Associates, New York, United States
Jori Hall, PhD
Professor
University of Illinois at Chicago, Georgia, United States
Isabelle Bourgeois, PhD
Associate Professor
University of Ottawa, United States
Michael Harnar, PhD (he/him/his)
Assistant Professor
Western Michigan University, Michigan, United States
Mark Abrahams, PhD
Senior Consultant
Southern Hemisphere, United States
Mita Marra, PhD
Associate Professor
University of Naples “Federico II”, United States
Location: Room 309/310
Abstract Information: The publishing system serves a vital function in a profession – it is a primary means for sharing insights with our colleagues in an effort to advance our field and collective knowledge. However, publishing is prohibitive for the majority of professionals. Whether due to language barriers, capacity via formal training, time to focus outside a fulltime job, resources, mentoring or access to general support, published works are dominated by those who are rewarded for publishing as part of their job and have the time, capacity, and resources. As a result, the valuable experiences of many are being lost. This is particularly problematic for the field of evaluation, which is a practice-driven field. If our practice is not being shared, our written word does not tell the real story. This Think Tank aims to generate new ideas for how to expand the diversity of voices in publishing while at the same time maintaining the responsibility of publishing to advance a sector. What should these standards be? Who gets to decide? Is peer review or feedback a vital aspect? What form does constructive criticism play when, as a profession, we have moved to different definitions of rigor and overall greater inclusivity of perspectives? The findings and recommendations of this Think Tank will be used to advocate for greater inclusion in evaluation publishing.
Relevance Statement: There are many avenues for evaluators to share their stories –AEA conferences, AEA TIG gatherings, webinars, social media, grey literature, peer reviewed journals, and more. Publishing in journals is a means through which we not only share knowledge but create a lasting, formalized archive of what we have learned and how we continue to build upon what is already known. Ultimately these records tell the “story” of our discipline, indicate what is important to us as a field, and often “drive” what type of knowledge we continue to share and create. Scholars turn to our journals to understand where “gaps” are in the existing literature and how topics have already been studied to prioritize their own research, and practitioners consult journals to understand approaches to evaluation that have worked for others and new techniques or methods that may be fruitful to use in their practice. Given the central role journals play in our field, it is particularly important that we ask ourselves what is included and excluded from the peer-reviewed literature and why? Peer-reviewed publishing is situated in a complex ecosystem that includes, but is not limited to, individual authors and their feelings about and readiness to publish, workplaces that differentially incentivize (or disincentivize) time for publishing, and private publishing companies that aim to maximize revenue. It is important that we pause to consider the various drivers that affect what gets published by whom, and what steps we can take to create a more equitable system that better captures our collective knowledge and experiences. We might start with examining factors that contribute to individual authors deciding whether to submit work. How many of us wished to share our story through publishing, but have been unable to for a variety of reasons? Maybe we have not had the time, do not know how to write a paper or are intimidated to try. Many of us do not have formal training, or only learned English – the dominant publishing language – later in life. Still more are buffeted by life and work obligations that make publishing a distant dream. We might also consider the current “system” of publishing. What do we look for when reviewing manuscripts? What types of articles are available to us? What is the role of open vs. closed-access journals? This think tank idea emerged from editors of evaluation’s top journals across the globe as part of a collective movement to increase the voices in evaluation publications. Far from distant experts in ivory towers, these editors worry about the trajectory of a future world that does not work to represent the diversity of voices and experiences in evaluation and want to create systems and incentives that enable more evaluators to share their stories. While they sit in powerful positions, they cannot make decisions that impact answers to critical questions like “what is good evaluation quality?” that set the standards for an industry, without understanding the range of perspectives. This think tank will start the conversation more formally in AEA.