Multiple Data Sources, Different Audiences: Telling a Coherent Story about a Science Curriculum Field Test

Abstract Information: Presenting evaluation findings effectively often requires making sense of and weaving together multiple, rich data sources to tell a coherent story without overwhelming the audience. This paper examines challenges and lessons learned during the field test of a new open-access high school science curriculum. The field test data team examined teachers’ experiences in professional learning sessions and implementing the curriculum, and students’ learning and experiences with the curriculum, with a focus on equitable science learning. We discuss examples of how we wove together data from various sources in the large mixed-methods dataset created from the field test to craft a meaningful story about how the curriculum worked for teachers and students. We also describe decisions made about how to present findings to different stakeholder audiences and explore the utility of data dashboards and slide deck presentations, rather than formal reports, to share findings that meet the needs of multiple audiences.

Relevance Statement: Presenting evaluation findings effectively often requires making sense of and weaving together multiple, rich data sources to tell a coherent story without overwhelming the audience. This paper examines the field test of an open-access high school science curriculum and how findings from teachers and students were presented to different stakeholder audiences over time. We discuss challenges and lessons learned by the data team. The curriculum consists of three, one-year long courses (Biology, Chemistry, Physics) with 5-6 units per course, and utilizes a student-driven, inquiry-based approach aligned with science standards recently enacted by many states. Each of the 16 units was field tested by approximately 100 teachers across 10 states over two years. Professional learning for each unit focused on unit content and pedagogy to prepare teachers to enact the instructional model. The goals of the field test were to understand the effectiveness of the professional learning and the implementation, effectiveness, and utility of the curriculum, and provide data and recommendations to 1) curriculum developers and professional learning designers to inform revisions to materials, and 2) a steering committee of state education leaders from each of the participating states to inform their recommendations about development and revision processes that address their states’ needs. The evaluation examined teachers’ experiences in professional learning sessions and implementing the curriculum, and students’ learning and experiences with the curriculum, with a focus on equity. We collected and analyzed quantitative and qualitative data for each unit, including teacher surveys, teacher interviews, student surveys, and student work samples. A key challenge was making sense of the large amount of data and weaving it into a complete, coherent narrative without overwhelming the audience. One effective approach was focusing on how data from different sources complemented each other. For example, we utilized findings from interviews (conducted with a sample of three teachers per unit) to provide deeper context to findings from teacher surveys. To balance providing comprehensive data with telling a digestible story, we created data dashboards containing all quantitative survey data and slide decks highlighting noteworthy findings and comparisons across courses, states, and units. Another challenge was efficiently presenting findings to two audiences with different needs. We addressed this by using dashboards to show survey results in interactive figures, allowing viewers to slice data by course or state as needed. This helped curriculum developers easily view data for each course, while state leaders could view data for their state. We also tailored findings in slide decks to each audience. For example, presentations for curriculum developers might focus on challenges with specific instructional supports, while presentations to state leaders might focus on how units helped teachers meet standards. Additionally, as curriculum developers and professional learning designers made adjustments over time, we attended to how those efforts impacted teacher and student experience and highlighted key changes for each audience. This presentation will discuss key recommendations for how to craft a meaningful story for a specific audience from a large, diverse dataset so that stakeholders have relevant data needed to make decisions.