87 - 10 years of Evaluating the Food Safety Program

Abstract Information: The regulation and oversight of foodservice and retail food establishments is performed by trained individuals who are typically employees of state and local health departments. Training on numerous standards provide a framework to accommodate traditional and emerging approaches to food safety. For the past ten years, the U.S, Food and Drug Administration (FDA) has partnered with the National Association of County and City Health Officials (NACCHO) to fund the Voluntary National Retail Food Regulatory Program Standards Mentorship Program. This program provides state, local, territorial, and tribal (SLTT) agencies with funding to participate in a peer mentorship program aimed to improve achievement of conformance with one or more of the standards. Mentee agencies enrolled in the training program are paired with mentors who have a record of success achieving conformance with the standards. Mentors share resources, guidance, and strategies to achieving conformance with the standards. NACCHO provides participants with opportunities to connect with peers, technical support, conferences, and opportunities to engage with local, state, and federal representatives in their states. The overall goals of the Mentorship Program are to increase enrollment in the training program, provide participating agencies with resources and tools to progress towards conformance, and provide opportunities for discussions and capacity building of participants. For ten years, annual program evaluation activities assessed progress and achievements of participants, especially mentees. Sources of data included training standards reporting, pre- and post-assessment surveys, and narrative reports. Summaries and final reports of qualitative data were examined using thematic analysis. Objectives of the final 10-year evaluation were to assess the effectiveness of the Mentorship Program in achieving its goals and determine whether the program improved conformance with the standards. Specifically, the 10-year review aimed to inform the following questions: 1. Did the program increase the capacity of mentees to implement the standards? 2. Did the program increase the capacity of mentees in other aspects of retail food safety? 3. Did the program support the creation of a growing network of retail food safety practitioners and resources? 4. Was the mentorship program well-implemented, and what lessons can be learned to improve implementation in the future? Findings showed that the Mentorship Program laid the groundwork for aspects of standards that mentees described as complex. Experienced mentors helped mentees navigate the conformance process, while NACCHO staff provided valuable tools, resources, encouragement, and opportunities for networking. Mentees valued resources and used them to work towards conformance and to improve other elements of their food safety programs such as increased infrastructure for food safety and strengthened relationships with relevant stakeholders. The Mentorship Program provided tools for mentees to grow their confidence with the framework, and positive experiences with the framework increased mentees’ motivation to continue pursuing conformance of standards. Mentees also used the momentum from the mentorship program to apply for additional funding to sustain the program for future years. Finally, mentees leveraged participation in the program as a talking point with decisionmakers and other external partners to generate buy-in for improved food safety practices and standards.

Relevance Statement: Ten years’ worth of program and evaluation data is a big deal at NACCHO. The majority of funding for programmatic work is on an annual basis, leaving us never really sure that a program will continue for a second year or longer. Measuring even short-term outcomes can be a challenge and most of the time we are limited to measuring intentions to change rather than changes in actual knowledge or behavior. Intermediate and long-term outcome measurement is practically non-existent. It is extremely rare that we are able to track a program’s progress throughout the years. When such an occurrence does happen, we are thrilled and want to make the most of it. In addition, there are many logistical obstacles we must face and overcome to collect ten years of data. The first hurdle is data consistency throughout the years. Programs change and the evaluation needs to adapt to those changes. While we try to collect the same core set of measures from year to year, it isn’t always a perfect one-to-one match. Program and evaluation staff come and go, and the administration and collection of information may be affected by slight nuances of individual differences. Some years, participants in the program are more receptive to participating in evaluation than in other years, i.e., response rate vary from year to year. As in-house evaluators at NACCHO, we are an internal services team to a greater programs team who implement the interventions. While there are advantages to being an internal evaluator (e.g., intimate knowledge of the program and the politics of the organization), there are accompanying disadvantages, the largest being that often we are not “at the table” for the initial program planning phases and are called into our roles too late in the game. Interventions may be underway before we step into the program, so things like pre/post measures are difficult to use. Sometimes program staff feel they know best what is needed for an evaluation and do things on their own without consulting us. Their methods are not always up to par resulting in substandard and inconsistent data to work with. Extensive data cleaning is required for both quantitative and qualitative measures. Nonetheless, we make the best of what we have and try to come up with broad conclusions so that we can address things such as the merit of this program for the past ten years. Did we move the needle? Our evaluation results say yes we did.