Mixed Methods Evaluation
Genzo Yamamoto, PhD
Director, Knowledge Management
Opportunity International, United States
Genzo Yamamoto, PhD
Director, Knowledge Management
Opportunity International, United States
Lise Masselus, n/a
Researcher
Leibnitz Institute for Economic Research, United States
Luis Fernando Sanabria
COO
Fundacion Paraguaya
Asuncion, Paraguay
Genzo Yamamoto, PhD
Director, Knowledge Management
Opportunity International, United States
Nathan Fiala, PhD
Professor
University of Connecticut, Connecticut, United States
Michael Spingler
CEO
VisionFund, United States
Location: White River Ballroom H
Abstract Information: The evaluation community commonly recognizes the importance of both qualitative and quantitative methodologies in evaluation. But the two are not always used together. This panel considers the example of RCT research on microfinance to look at the stories that are related both at micro- and macro- levels in such studies and presents additional data, their related stories, and consider their implications. The macro level story may already be partially understood. Twenty years ago, overly enthusiastic voices had presented microfinance as a panacea for poverty eradication. But then, a series of RCT studies seemed to debunk this myth, leaving the impression that microfinance doesn’t work. And yet, recent studies indicate that this counter-narrative is as flawed as much as the previous though now ostensibly supported by a scientific methodology. Even as such macro-level narratives require attention, equal attention must be paid to the contextual and programmatic details. Numerous RCT write-ups produce the obligatory quantitative formulas and significance tables. But some papers reveal that researchers have not been willing to delve into the details of the programs and its contexts. Using a recent multi-year, multi-country effort to measure the impact of microfinance combined with training in Paraguay, Myanmar, and Uganda, this panel considers the original RCTs and their reliability, the complexities of the situation on the ground, how this next set of studies worked to avoid the flaws of the former, the lessons learned in the process, and how this might help us shape a more reliable narrative regarding the effectiveness of microfinance. Although construed as a panel, we will review the earlier RCTs, consider findings from current RCTs, have leaders present the key characteristics of their programs, and explore the complexities for an interactive discussion between field operators and the researchers. Ultimately, this will be an integrated conversation, not a series of separate statements by different parties. In general– • Dr. Genzo Yamamoto will facilitate the panel. • Dr. Nathan Fiala (Univ. of Connecticut) will review past and present RCT findings and the various factors that need to be taken into account in the design process. • Mr. Luis Fernando Sanabria (Fundación Paraguaya) will detail the characteristics of their program and what the hardest aspects of participating in the study was even as they tried to maintain normal operations. • Mr. Michael Spingler (VisionFund-Myanmar) will detail the characteristics of their program and what the hardest aspects of participating in the study was even as they tried to maintain normal operations.
Relevance Statement: • Randomized control trials continue to be an important impact evaluation methodology and may become even more the case. • Even as some quantitative methodologies offer alternatives to RCT such as Propensity Score Matching (PSM), RCTs remain a powerful methodology. • Often seen as an expensive and, therefore, relatively rare project, with new ways of capturing data digitally in various interventions, there are cases where the methodology can be conducted more quickly and less expensively than previously. • At the same time, review of RCTs have sometimes shown that RCT write-ups can often hide a multitude of weaknesses and insufficient care taken by researchers. • Conceptual strengths of the methodology aside, it is important to review areas where such studies can do injustice to the realities on the ground. • We believe this would be of interest to researchers, interested organizations that may conduct such studies of programs in the future, as well as consumers of such studies.