Abstract Information: In conversations with others, we typically focus on what people say instead of paying attention to what they do not say. This may also happen when we look at evaluation data. At first glance, we may think we understand the story our data are telling us; however, our findings may shift if we carefully examine data to clean it and address missing data. In this Ignite Session, we will discuss our experience with missing data for the evaluation of Bridge to Employment (BTE) in an international context. BTE started in 1992 and was designed to inspire young people (ages 14-18) from underserved communities to stay in school, excel academically, and elevate their career aspirations. Funded by Johnson & Johnson and managed by FHI 360, the program has expanded over the years to reach more than 100 sites across the globe. As evaluators based in the US with approximately 20-25 sites per year all over the world, our research team has been limited to the data that local site coordinators gather and provide to us. In our case, this meant that sites submitted their data in various shapes and forms. Some sites were faithful to the survey format and gathered good data; others, however, rearranged the order of questions, reworded or omitted them. In some instances, there were no equivalents for these reworded questions, rendering them useless since they were site-specific. However, this was not even our biggest challenge. After looking more closely at the data, we quickly realized there were a large number of missing fields. The majority of sites had more than 50% missing data—some had logical reasons and some were random. This required our team to systematically review each variable to determine missingness. After much deliberation, we decided it was necessary to impute the missing data on the outcome variables of interest. We used multiple imputation by chained equation 10 times until we got stable estimates and the values were closest, statistically speaking, to the values that the students would have answered, with KNN=5 as weight. Otherwise, we would have had a very small sample for each outcome variable of interest. The BTE program is three years long, and we expect that some participants may leave the program as the years progress. We recognize that attrition is a common issue in RCTs and many evaluations. Hence, we planned to account for the effect of attrition. But what happens when students attrite at different rates? This can lead to bias in the results and problematic interpretations. Thus, it is important to state the evaluation limitations so that the audience understands that some results may be misleading and tell a different story. In this Ignite Session, we will highlight these challenges in the Bridge to Employment evaluation and quickly show how we dealt with them in a methodological and systematic manner.
Relevance Statement: The session reflects the conference theme of The Power of Story in that as evaluators, we can tell powerful stories through data, and we have the responsibility to tell the right story. Without the careful attention, discussion and decision-making process that we undertook for the BTE evaluation, we potentially could have told a powerful yet inaccurate story. The AEA Guiding Principle of Systemic Inquiry include adhering to the highest technical standards appropriate to the methods and making clear the limitations of the evaluation and its results. This session will illustrate how we first worked to ensure that we had a clear understanding of our data and then how we communicated to our clients and stakeholders about the data and its limitations. The AEA Guiding Principle of Integrity underscores the importance of communicating truthfully and openly with clients and stakeholders about data and its limitations, as well as accurately and transparently representing evaluation procedures, data and findings. While our procedures to address missing data complicated our presentations to the client and stakeholders, we included a description of our methods to allow transparency and facilitate their understanding of the analysis. In this session, we’ll share the slides we used to communicate this as concisely as possible. We believe that these real-world examples will be helpful to evaluators who are curious to learn more about methods to address missing data and/or how to best communicate those procedures to clients and stakeholders. This session holds relevance to the AEA evaluator competencies as well, especially to Methodology and Context. For Methodology, the competencies include analyzing data using credible, feasible and culturally appropriate methods and identifying strengths and limitations of the evaluation design and methods. While we allow the flexibility for BTE site evaluators to tailor our surveys in a culturally relevant manner, we acknowledge the complexities that arise with this approach. Our team drew on its quantitative data analysis expertise to use the most appropriate methods to address the resulting missing data. And as mentioned earlier, we identify the limitations of the data in our reports and presentations. For Context, the competencies include recognizing elements such as site/location/ environment, participants/stakeholders, organization/structure, culture/diversity, history/traditions, values/beliefs, politics/economics, power/privilege, and other characteristics. Over the years, in nearly 100 locations, we’ve had sites that represent various different sizes, socioeconomic status, languages, customs and education systems. Our interactions with site coordinators, evaluators and program partners have taken into account each individual community and its culture. The competencies mention describing the program and its functioning in broader contexts and communicating evaluation processes and results in timely, appropriate and effective ways. This has been an evolving process for our research team that we continue to grow. We will design this session to provide evaluators with a look into a large, international evaluation—providing real-world approaches to analysis as well as communication.
