10 - No More Crappy Surveys: Best Practices in Survey Planning, Design, Analysis, and Reporting

Abstract Information: Instructors JoAnna Hillman, Janelle Gowgiel, Christiana Reene, and Mary Davis are on a mission to free the world from crappy surveys. Through this session, we want you to become a “No More Crappy Surveys” ambassador. Whether you’ve only designed one survey, or you have experience deploying many surveys, this session will provide a complete overview of surveys from planning to reporting and wrap up, with hands-on activities to put your learning into practice. You’ll learn effective survey practices, including starting with evaluation questions, developing effective survey questions that gather useable data, implementing survey logic (branching, skip, display), organizing your survey, and optimizing your response rate. Additionally, the session will cover best practices in quantitative and qualitative data analysis, data visualization, reporting, and action planning for informed decision making with partners. The session will help you identify common issues that derail survey effectiveness and help you avoid these pitfalls while providing examples of best practices. Participants are encouraged to bring a sample survey for peer review and discussion throughout the workshop. Presenters will discuss specific cases encountered in their evaluation work, sharing their successes and lessons learned when dealing with difficult clients, partners, and situations. Participants will apply the skills learned during the session to a case study covering different scenarios followed by a group report-out. Finally, participants will collaboratively summarize learnings with an interactive activity and a question-and-answer session at the end of the session.

Relevance Statement: Following two sold out sessions at the 2022 Summer Evaluation Institute, presenters JoAnna Hillman, Janelle Gowgiel, and Christiana Reene (and adding in presenter Mary Davis) are back with a longer, more interactive, and more in-depth version of No More Crappy Surveys, responding to the AEA community’s ongoing interest in learning survey design topics. Offering a session to continue strengthening survey planning, design, analysis, and reporting is of demonstrated interest to the AEA community, as indicated via survey findings from the 2022 Summer Evaluation Institute. The event was very well-received among the 119 attendees, with the majority of the 86 respondents to our post-session evaluation (72% response rate) indicating that they would recommend the session to a friend or colleague (Net Promoter Score of 74.42 on a scale of –100 to 100). Attendees also reported increases in their confidence conducting surveys after the training (as compared to before the training), with 96.5% of trainees ranking their confidence as either a 3 or 4 (on a scale from 0 to 4) after the session. The majority of qualitative feedback and suggestions for improvement indicated that participants hoped for more time to spend on different activities. This suggests that attendees would be receptive to a longer and more in-depth session, such as a full day professional development workshop. In addition to the highly successful training at the Summer Evaluation Institute, the presenters are well-versed in presenting at the American Evaluation Association conferences, other conferences, webinars, graduate level classes, and more. As facilitators and trainers, we believe in helping others do evaluation better—and doing so in a fun and engaging way. We are “personable, engaging, and knowledgeable” presenters, we have designed this workshop to effectively appeal to an adult learner population. This approach includes providing opportunities for active engagement, direct practice of skills learned, and reinforcement of content. Two peer-review and collaborative opportunities will be included in the session, allowing small groups to apply the skills covered in the session to their own surveys and projects. Additionally, a small group collaborative case study will encourage participants to apply what they’ve learned to realistic case examples. Finally, the session will conclude with a cumulative review using a game-show style format, challenging participants to recall learnings from the whole day. This session will address and strengthen skills and confidence in all five AEA Competency Domains: Professional Practice, Methodology, Context, Planning & Management, and Interpersonal. The session covers content including reflecting on past evaluations to improve professional practice, creating evaluation questions, determining appropriate methods to engaging stakeholders/partners throughout the process, planning and documenting surveys, and facilitating shared decision making (just to name a few). Participants in this session will leave with a deeper understanding of survey methodology, an increased knowledge of survey best practices, and more confidence in their ability to plan, design, conduct, analyze, and report on surveys.