Integrating Technology into Evaluation
pete york, MSSA
Principal and Chief Data Scientist
BCT Partners
Garnet Valley, Pennsylvania, United States
Linda Raftree, BA in Anthropology
Founder
MERL Tech, New York, United States
Michael Bamberger, Ph.D
Senior research fellow
International Initiative for Impact Evaluation (3ie)
beaverton, Oregon, United States
.jpg "Randal Pinkett, PhD, MBA (he/him/his) photo")
Randal Pinkett, PhD, MBA (he/him/his)
Co-founder, Chairman and CEO
BCT Partners
Newark, New Jersey, United States
Location: Grand Ballroom 7
Abstract Information: The panel examines the causes and consequences of bias when evaluations are based on big data and data analytics. The session will begin with the presentation of a framework for identifying the main types of biases in big data. Three presenters will then tell stories about (a) the use of big data in different evaluation contexts: (b) using a data-driven Diversity, Equity and Inclusion (DEI) approach to navigating the bias terrain and (c) MERL Tech’s Natural Language Processing Community of Practice (a group of academics, data scientists, and evaluators from NGOs, UN and bilateral organizations). Each presentation will describe the benefits of using big data, the kinds of bias that can be introduced, and the consequences of the bias in terms of under-representation of vulnerable groups, as well as the operational and methodological consequences. The focus on bias is important, because some advocates claim that big data is more “objective” than conventional evaluation methods because sources of human bias are excluded. While big data and data analytics are powerful tools, the presentations will show that this claim of greater objectivity is overstated. Following the presentations, the panelists will discuss common themes and issues across sectors and ways to address each of the sources of bias.
Relevance Statement: In recent years there has been a dramatic increase in application of big data and data science in in the design, implementation and evaluation of development programs. This is driven by the increasing availability of powerful new tools for data collection and analysis, and the extensive media coverage of Artificial Intelligence and generative programs such as ChatGPT. These approaches are already changing the nature of evaluation, the questions that can be addressed, the kinds of analysis that can be done, and how the evaluations are used. While it is undeniable that these new tools and approaches will greatly enhance the practice of evaluation, many of these techniques are unfamiliar to evaluators, and an area of concern is the claim made by some proponents of big data that these methods are more “objective” than conventional evaluations because they avoid human error. This has meant that some of these techniques have been assessed less critically than conventional evaluations. This session has the following goals: ● To show the great potential benefits of the use of big and data analytics in program design and evaluation ● To provide a framework for identifying the potential sources of bias in big data ● To discuss the methodological, policy and equity consequences of these biases and ● To suggest ways to identify and address the different sources of bias. It should be stressed that we fully acknowledge the tremendous power of big data, but in order to fully benefit from this new evaluation paradigm, it is important to understand and address some of the issues arising from how these new types of data are generated and used, and in particular to recognize that humans (with all of their socio-cultural perceptions and beliefs, professional and political orientations) must be involved at all stages of the process. Our framework identifies 4 categories of bias: ● Human bias: humans make critical inputs at all stages of the evaluation, and many of these are affected by deep-seated socio-cultural and psychological factors of which most people are not fully aware. ● Who is at the table?: multiple sociocultural, technical, organizational and political factors determine who is involved in decision-making at each stage of the evaluation, and whose perspectives are (and are not) considered. The concerns of people who are not consulted are frequently ignored or misunderstood. ● Technological and methodological bias The original form of these data may be of low value and may require conversion before analysis. As a result, the use of big data in research may be subject to biases, such as data quality, selection, confirmation, algorithmic, overfit, and measurement biases, resulting in inappropriate conclusions. ● Organizational and political factors: Bias mat be introduced through: using different sectoral (e.g. health, education, employment) or different academic perspectives; pressures to not criticize the agency or the government; or through pressures to reduce the costs of data collection and analysis.
Presenter: pete york, MSSA – BCT Partners
Presenter: Michael Bamberger, Ph.D – International Initiative for Impact Evaluation (3ie)
Presenter: Randal Pinkett, PhD, MBA (he/him/his) – BCT Partners
Presenter: Linda Raftree, BA in Anthropology – MERL Tech
Presenter: miriam sarwana, PhD in Social/ Health Psychology – BCT Partners