134 - Effective Use of Surveys in Evaluation: Designing and Planning Surveys to Collect Valid Data and Address Data and Time Constraints

Abstract Information: Conducting surveys is a common method for evaluators to gather data. Careful design of instruments and plans for collecting survey data is critical for evaluators who often work with limited time and resources. This poster session will provide participants with deeper knowledge about designing and conducting surveys. The session is geared toward evaluators, not social scientists. Evaluators face constraints that are not as common for researchers; similarly, evaluators typically only need to generalize to smaller populations such as program beneficiaries. Hands-on activities will illustrate and reinforce key learner outcomes for this session. The three presenters have extensive experience with surveys, including teaching survey methods, conducting research on sampling and instrument design, and training clients to develop and plan their own surveys. In addition to covering overall principles for designing and implementing surveys, the session will introduce established design guidelines evaluators should follow when developing surveys that include (i) pilot testing of both instruments and processes, (ii) survey content, (iii) visualization, and (iv) common mistakes to be avoided. This poster session will be interdisciplinary, and examples will be drawn from various fields of work. The session will help evaluators expand and strengthen the tools they need to access outcomes and impacts, and support them in presenting valid facts to improve their evaluation stories. Participants will also learn about how to protect their surveys from multiple sources of bias. Finally, participants will learn how early decisions related to survey design and plans can greatly aid or undermine the evaluation of projects. Developing high-quality surveys align with the American Evaluation Association guiding principles related to systematic inquiry and conducting data-based inquiries that are thorough, methodical, and contextually relevant. It is our hope that this capacity-building session will enlighten participants regarding the importance of robust survey design and the tricks that empower novel evaluators and survey practitioners to produce high-quality surveys.

Relevance Statement: Relevance and importance to the field of evaluation, Surveys are a widely used data collection method in evaluation due to their ease of use and relatively lower cost than many other data collection methods. A well-designed survey should yield valid data that can address a wide range of data and information needs for program evaluations including: (i) information about program staff and persons impacted by the program; (ii) data specific to outcomes; and (iii) data specific to areas for improvement and steps that can be taken to strengthen programs. While well-designed surveys can serve evaluators, poorly designed surveys have the potential to undermine evaluations by introducing bias or collecting invalid information that misinforms decisions or conclusions. Poorly designed surveys can also negatively impact evaluations by consuming resources and yielding little useable data. These are a few reasons why evaluators need to learn how to design valid and reliable tools following standard survey research guidelines. Implications for evaluation theory or practice This skill-building session will teach the audience to build robust surveys following the main survey research guidelines. We will focus on content and visual guidelines for survey design. Content guidelines will cover question construction, the most appropriate survey mode for various types of questions, and elements that should not be part of your questions and answer options. Visual guidelines will cover individual questions' visualization and the overall visualization of the survey in terms of pages or interface, depending on survey modes. Visual elements of a survey are critical in promoting clarity and encouraging respondents to complete the survey. We will demonstrate the survey guideline outlined in Dillman et al., 2014, and other survey research books. After attending this session, participants will be able to distinguish a well-articulated survey and will be able to construct higher-quality surveys. This poster session will be helpful for new evaluators and evaluators who may lack depth in survey methods and measurement. Surveys are widely used around the world because of their ease of distribution, relatively low cost and because they facilitate the collection of both quantitative and qualitative data. Knowing how to build reliable and valid surveys is extremely important for evaluators and significantly impacts their ability to measure program outcomes accurately. Reflecting quality standards in evaluation theory, methods, and/or practice. This proposal promotes the quality standards for survey design as one of the most used evaluation tools, which in turn stimulates the high quality of evaluation for collecting evaluative data. Strong evaluations rely on four central pillars, (i) evaluation planning, (ii) robust evaluation designs, (iii) valid and reliable tools, (iv) and robust analysis and data presentation. We hope the audience will sense the importance of maintaining high standards when building their surveys.