Integrating Technology into Evaluation
Kerry Bruce, DrPH (she/her/hers)
CEO
North End Consulting, United States
Swapnil Shekhar, Indian Institute of Forest Management (he/him/his)
Co-Founder and Director
Sambodhi Research and Communications
Noida, Uttar Pradesh, India
Michael Bamberger, Ph.D
Senior research fellow
International Initiative for Impact Evaluation (3ie)
beaverton, Oregon, United States
Kecia Bertermann, n/a (she/her/hers)
Director, Learning and Impact
Luminate, United States
Location: Room 201
Abstract Information: The panel will review experiences with the process of assisting development partners to implement new information technologies. Most AEA presentations on information technology have tended to focus on the methodology and the benefits of the new kinds of data collection and analysis, but have given relatively little attention to the challenges during the process of implementation. The process of introducing new technology can be quite disruptive (in both positive and negative ways) and the process can sometimes affect the originally intended outcomes once the technology is operating. Some new technologies can also introduce unanticipated biases due to issues of data quality and sample bias, or the socio-cultural and political orientations of the different groups involved. Three presenters will tell stories about their experiences with the implementation of different kinds of information technologies and will reflect on the lessons learned. It is hoped that the presentations and discussion will provide insights and reflections for the many agencies that are involved with supporting local partners with the transition to potentially beneficial new information technologies. This session is a collaboration between the International and Cross-Cultural (ICCE) and Integrating Technology into Evaluation (ITE) TIGs. The presentation on "How to keep up with the pace of change for technology in evaluation" was submitted by the ICCE, and the proposals on experiences strengthening the Indian government’s use of survey and administrative data, and using big data to strengthen humanitarian programs were submitted by the ITE.
Relevance Statement: Previous AEA conferences have included a number of sessions describing the application of new information technologies by different kinds of development agencies. In most cases the focus has been on the methodology and findings, but they have usually not devoted much time to the process of introducing and implementing the new technologies. However, the introduction of any new technology, particularly information technologies tends to be disruptive. People have to learn new skills and often change how their work is organized, some staff feel threatened while others are very enthusiastic. Often the workload may increase, at least for some time, and during the transition period it may be more difficult for staff to produce the kinds of information and reports that the agency is mandated to provide. While in some cases the transition to the new technologies may be relatively rapid and smooth, in other cases, for example, when introducing a new agency-wide M&E system, the disruption can last for much longer. The panelists will tell stories about their experiences with the introduction of a range of new information technologies, what tended to go relatively smoothly and what kinds of issues were challenging, and what lessons were learned. Another set of questions ask how the implementation process affected, and in some cases changed the structure, content and use of the new technologies. Sometimes this process can have beneficial effects as the discussions and testing may identify issues concerning the quality and accessibility of the required data, and may result in enhanced data quality. These discussions may also result in consultations with stakeholders not originally involved, for example, community groups, vulnerable populations, civil society or academia. However, in other cases, elements of the original design may be eliminated due to opposition from some groups who may feel threatened. There is also an increasing awareness that new technologies such as big data, Artificial Intelligence or more recently generative technologies such as ChatGPT, might introduce unintended biases. Presenters have an extensive experience with the introduction and use of new technologies in a number of different organizations so that they can reflect on general patterns and lessons that they have found. Each presenter will: (1) Describe the kinds of organizations with whom they have worked on new technologies and the benefits that these technologies have brought, (2) The experiences of introducing new technologies, what worked well and what were the challenges, (3) How the implementation process affected the use and benefits of the new technologies and (4) Lessons learned It is hoped that the presentations and discussion will provide insights and reflections for the many agencies that are involved with supporting local partners with the transition to potentially beneficial new information technologies.
Presenter: Kecia Bertermann, n/a (she/her/hers) – Luminate
Presenter: Swapnil Shekhar, Indian Institute of Forest Management (he/him/his) – Sambodhi Research and Communications
Presenter: Kerry Bruce, DrPH (she/her/hers) – North End Consulting
Presenter: Michael Bamberger, Ph.D – International Initiative for Impact Evaluation (3ie)