Director of Impact Social Contract, LLC, United States
Abstract Information: This session explores lessons learned from applying the secret shopper methodology in two (2), cross-border social impact initiatives. The first initiative was an assessment of youth-friendly sexual and reproductive health services in the Mexican states of Oaxaca, Chiapas, Tabasco, Guerrero, and Campeche, centered on the experiences and work of youth participants in the organizations’ local networks. This work included three rounds of secret user visits and was conducted between 2014 and 2016 by the Mexican research consulting firm Investigación en Salud y Demografía, S.C. (Insad), in collaboration with the nonprofit organizations Catholics for Choice Mexico and Equidad de Género: Ciudadanía, Trabajo y Familia (Equidad). The second initiative involved a round of visits during October through November (2022) and was conducted by Social Contract LLC. The primary goal of this work was to better understand the service delivery experience of impacted users across the state of Delaware, as part of a large-scale improvement effort by the client (the Delaware Department of Health and Social Services). In both cases, insights from the secret user visits allowed for more informed decision-making and perhaps more importantly, provided a space for participants to learn about the specific service workflows and to share their unique experiences from observational and direct engagement with service staff.
Relevance Statement: Secret shopping, also known as mystery shopping, is one qualitative, participatory method that has been employed in a variety of scenarios including healthcare delivery and library settings (Jacob et al., 2016; Rankin et al., 2022). Originating in the retail and hospitality industries, the methodology is typically used to evaluate a particular topic (e.g., customer service) by having trained people act as patrons or clients and then rating their experience based on given criteria. Prior work shows that the secret shopping method has been useful in gathering rich and nuanced data that may complement quantitative data such as statistics collected at service points and may help identify areas for improvement (Douglas, 2015; Jacob et al., 2016; Jeon, 2021). The secret shopper methodology provides a systematic way of collecting, analyzing and using data--which is a core tenet of evaluation. Additionally, the participatory nature of the methodology allows for collaborative work between evaluators and secret users. We believe that this methodology can enhance any evaluation practice, especially those that involve particularly complex topics or those that require deep collaborations with the community.
