192 - Moving beyond the evaluation plan: Telling the story of a complex program

Abstract Information: Traditional evaluation plans are key to helping programs plan and tell their story effectively. However, complex, long-term plans require regular reassessment and revision. Indiana’s commercial tobacco control program (TPC), which is complex and multi-faceted with numerous partners, has a strong evaluation history and ongoing evaluation activities. Therefore, they needed more than a traditional evaluation plan. This paper will describe the process we used to move beyond the initially proposed plan to develop a series of evaluation deliverables for TPC and strategies for ongoing use. Developing an approach that tailored the deliverables to the client needs, we conducted a comprehensive study to assess whether the current evaluation activities were aligned with their strategic goals and met the learning objectives of the program. We documented data sources related to program components, facilitated a series of qualitative data collection sessions with program leadership, conducted a document review, and integrated best practices from the field of commercial tobacco control. These were used to identify strengths and gaps in current evaluation activities and build recommendations for current and future evaluations that aligned with Indiana’s five-year Commercial Tobacco Control Strategic Plan. The final evaluation products included a matrix of program activities, evaluation data sources and a report summarizing strengths, opportunities, and recommendations, and a detailed plan for implementing the recommendations. In this session, we will discuss our approach and methods for developing the evaluation products and describe the plan to support ongoing use of the recommendations in the statewide commercial tobacco control program.

Relevance Statement: Following the principles of utilization focused evaluation1, the current study focused on what evaluation products would be most useful to our partner, TPC’s, complex program with existing evaluation activities. Instead of the initially planned traditional evaluation plan, we facilitated information gathering, produced useful materials and recommendations, and determined an approach to implement them in a way that would be conducive to ongoing impact. Given the program’s complexity and the many evaluation activities, we focused on mapping the key ongoing activities and their related data sources into a matrix by collecting information through document review and qualitative methods. This important step in the new partnership provided both TPC and us with a clear story of their program and existing evaluation. After completing the first step to outline their primary activities, data sources, and active evaluations in conjunction with TPC, we began to assess the ongoing and planned evaluations to identify where changes were needed. We also wanted to better understand how they aligned with the program’s larger strategic goals and to suggest future activities, considering the context of program resources and circumstances into our recommendations. To develop recommendations for future and ongoing work that aligned with their strategic plan and best practices in the field, we used on analysis of the existing TPC documents, including the TPC strategic plan, information from qualitative sessions with TPC leadership, review of CDC best practices2 and promising practices from other state commercial tobacco control programs. To facilitate use, we suggested scheduled check-ins to adjust existing components and plan future components in response to ongoing changes in the program and the commercial tobacco control landscape. These included annual review and modification of the activity and evaluation matrix and recommendations, as well as discussion and integration of the recommendations during individual evaluation project planning. We also proposed future studies to assess the evaluation plan from the community’s perspective and integrate their story. In this session, we will share 1) the process used to determine the partner’s needs, 2) methodology, tools, and techniques used, 3) the evaluation outcomes and products developed, and 4) facilitators and barriers to the approach. With evaluation needs and program landscapes constantly evolving, this evaluation provides a valuable example of how to adapt a planned strategy into a process and product that is equally thorough and useful to the partner. It can help inform evaluators who are asking questions like: How can I make an evaluation plan useful and actionable? How can I be more strategic with my evaluation plan? How do you reassess an existing evaluation plan? Our session relates to competencies in the domains of Professional Practice, Methodology and Context and guiding principles of Systemic Inquiry and Competency. 1Patton, M.Q. (2008) Utilization-Focused Evaluation: 4th edition. Thousand Oaks, Ca: Sage Publications. 2Centers for Disease Control and Prevention. (2014). Best Practices for Comprehensive Tobacco Control Programs. Atlanta: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health.