Use and Influence of Evaluation
Adam Trowbridge, MIDP
Senior Evaluation Advisor
U.S. Department of State & USAID, United States
Keith Ives, MIDP (he/him/his)
CEO
Causal Design
Wheat Ridge, Colorado, United States
Location: White River Ballroom A
Abstract Information: Impact evaluation methods have dominated discussions in the field of international development evaluation in recent years - but the discourse has largely missed shifts in the practice of how impact evaluations are conducted and their potential to reshape the way in which we implement humanitarian and development programs.
These approaches, which causally establish the impact of an intervention on the outcomes of interest using either experimental or quasi-experimental methods, have increasingly become a critical research tool to investigate potential policy interventions and programs. What is well known is how over the past two decades, these methods have been widely used to measure the overall impact of a program. Lesser known is how they have been used to test different approaches, innovate, learn, and tweak these programs during implementation to improve their cost-effectiveness.
This panel, featuring evaluators from USAID and Causal Design, will narrate the story of how impact evaluation methods have come to dominate the discourse, and what that story has gotten wrong. Misconceptions persist that impact evaluations must be complex, expensive investments that only provide implementers with a measurement of high-level outcomes years in the future. This story neglects the potential that these methods have to change the way we learn and adapt while implementing humanitarian and development programs. Join us as we help turn the page on the community’s understanding of impact evaluation - and move it from an academic tool for knowledge generation to an applied resource for decision making and program optimization.
Keith B. Ives is the CEO of Causal Design where their team have intentionally worked to change the paradigm on impact evaluation and emphasize the applied part of applied economics. His background in both the military and humanitarian operations have informed his perspective and focus on decision-making data. Keith routinely advises to projects, donors, and national ministries on decision making data tools - and how to (and when not to) leverage impact evaluations. Causal Design is currently implementing IE’s on humanitarian activities for USAID in Syria, Nigeria, Colombia, Ethiopia, and Madagascar.
Adam Trowbridge is a Senior Evaluation Advisor with the Office of Foreign Assistance at the Department of State and USAID where he works to increase the rigor and utility of evaluations. He was previously a Senior Monitoring and Evaluation Advisor with USAID’s Bureau for Humanitarian Assistance where he designed a program to improve the evidence base for BHA programming by addressing barriers to the use of impact evaluation in humanitarian contexts, and an Economist with USAID’s Office of Economic Policy where he conducted cost-benefit and cost-effectiveness analyses.
Relevance Statement: Impact evaluations are established as a mainstream tool for policy research in international development. These studies, and their results, have driven conversations and sparked debates in the conferences and hallways of every donor and implementing partner. However, the story of IEs is largely misunderstood - where most stakeholders continue to see the tool for its form at introduction rather than where it is today. These misreads fuel an antagonistic narrative against the role of causal inference in evaluation and have resulted in a backlash against IEs and their practitioners.
The 'randomistas' generally refers to a group of academic economists who began utilizing the “gold standard” of research in the hard sciences (the randomized controlled trials) to issues of global poverty. The development of this informal but interwoven thought collective has evolved over the past twenty years and is most visibly observed through the creation and expansion of organizations such as J-PAL at MIT, IPA at Yale, Georgetown University’s Initiative on Innovation Development and Evaluation, the World Bank’s DIME unit, CGD, and the International Initiative for Impact Evaluation (3ie).
At the center are a concentration of incredibly accomplished academics (E.g., Banerjee, Duflo, Kremer, Karlen, Blattman, Udry; Niehaus, Shapiro, and Glennerster) who’ve collectively garnered a Nobel Prize, senior government positions, executive roles as think tanks, and forever changed the landscape of what “evidence” means in international development. Their successes have not been without controversy though - with detractors publishing papers and articles calling to curb the widespread influence of the randomistas. In addition to well developed critiques of the movement - there is an expanding community of evaluators who campaign against IEs without a modern understanding of the tool. In particular, and despite development’s traditional fervor for private sector innovations, very few actors are exploring how econometrics and causal inference are being deployed outside of the poverty research!
Two decades of academic-led impact evaluation in the sector have neglected to highlight the potential of IEs to focus on programmatic questions of optimization (rather than “did it work”) and cost-efficiency (turning our eyes back to outputs rather than outcomes). In addition to the viability of quasi-experimental studies (which are not done by many academic evaluators) there is an enormous potential for (the more than 100 year old) A/B testing method which doesn’t require a control group. These types of studies (which are also impact evaluations) are being utilized by the private sector in manufacturing, retail, and online while we browse.
New firms, IE practitioners, and even donors are working to introduce and deploy these nimble or “IE 2.0” applications to the sector - but many are struggling to find traction or enough voice to shift the narrative. The first chapter in the story of IEs is over, and like our approaches to poverty alleviation, we must turn the page and continue to evolve into the future. Our session will narrate the history and help set the stage for the next act.
Presenter: Adam Trowbridge, MIDP – U.S. Department of State & USAID
Presenter: Keith B. Ives, MIDP (he/him/his) – Causal Design