Creating Surveys that Reflect Real Youth Stories: Using Cognitive Interviews and Other Youth-Focused Steps to Get Better Teen Pregnancy Prevention Survey Data

Abstract Information: As teen pregnancy prevention (TPP) program evaluators, we are often faced with the task of making decisions about the merit of programs using inadequate data. There are many reasons TPP surveys fail to provide top-quality data – inaccurate recall of behaviors, refusal to disclose behaviors, items that aren’t sensitive to change. Thus we may end up with items that don’t tell us what we want to know or data that are completely missing or not accurate. For our current federally-funded TPP program evaluation, we worked closely with youth to create a better survey and design a better evaluation. We spent months involving youth in cognitive interviews and discussions with our evaluation team. This resulted in new survey items, changes to our survey and tracking procedures, and an evaluation design that is more youth informed. In this session, we will review the history of our TPP program evaluation frustrations, describe how we involved youth in survey and design improvement, and discuss the changes we made in an attempt to improve our TPP data.

Relevance Statement: Credible evaluations rely on good, accurate, and appropriate measurement. However, for many evaluations of adolescent health promotion and risk reduction programs, the measures are less than perfect. This is especially true for HIV/STD/pregnancy prevention programs, that seek to measure behaviors that must rely on self-report. Still, we move forward with these measures, and make decisions, without discussing the potential impact of our imperfect measures. Meanwhile, programs for especially vulnerable populations may especially bear the burden of such unrealistic standards resulting in potentially effective programs being classified as not effective in the very populations of youth who may most need them. At the same time, there has been a movement to involve the recipients of evaluations as lived experience experts when designing the evaluation. It’s important to hear their stories to ensure that the data we collect accurately reflect their stories. Cognitive interviews are one way to improve survey quality as is involving these individuals with lived experience as evaluation consultants. Our session will provide tips on involving youth as experts in survey and evaluation design as well as highlight the survey and evaluation changes made based on this work with youth.