174 - Stories from the Field: Measurement Successes, Challenges, and Lessons Learned in K-8 Computer Science Evaluations

Abstract Information: Schools and extracurricular programs are increasingly incorporating computer science into their curriculum and instruction, but how can we know and tell the story of the impact this shift is having on student learning and attitudes toward computer science? Our evaluation designs and project findings are necessarily influenced by contextual factors, such as the grade level of participants, type of learning environments, and coding dosage within the programs themselves. Therefore, it can be tricky to compare results among different groups of students, even when using shared measures. This roundtable will discuss the evaluation of several computer science education projects for K-8 students: two different classroom curricula that integrate coding into core academic subjects, a statewide robotics competition, and an afterschool/summer enrichment program. Across these projects, evaluators have used different approaches, including traditional pre/post surveys, retrospective pre/post surveys, and rubrics, to measure coding attitudes and computational thinking skills, with mixed success. In this session, we will present successes and challenges from these projects, then allow participants to share their own stories of measuring computer science and computational thinking for K-8 students. Conversation questions include: How can we ensure reliable measurement of computer science skills? Which methods are best suited for telling the story of computer science in young students? What should we do when our data stories contradict each other or seem unclear? How can AEA evaluators capture K-8 computer science stories that will impact assessment and outcomes in the future?

Relevance Statement: Coding has been described as “the new literacy:” an essential skill for all 21st-century learners to master. To that end, it is widely accepted that computer science instruction needs to begin at a young age, supported through both in-class and extracurricular instruction. It can be difficult to evaluate the intended outcomes of such efforts. Some measurement tools focus on attitudinal constructs like coding interest, coding confidence, and perceptions of coders. Others focus on the assessment of technical skills in algorithm design and computational thinking. The use of shared measures across different projects can help evaluators to develop a common vocabulary, compare results, and begin to understand the most effective components and characteristics of a successful computer science education program for K-8 students. However, when validated and/or standardized instruments do not align with what students are learning in a particular program, it is often necessary for evaluators to create and use project-specific instruments (rubrics, observations, authentic assessment, etc.), which may or may not be usable in other contexts. Because computer science education is a rapidly evolving field, there is limited evidence about what works or doesn’t work in specific contexts, especially among young learners. Therefore, it is critical to share our experiences so that future evaluations remain relevant and continue to advance the field. Our storytelling of project successes and challenges will include discussions of how contextual factors, such as dosage and setting, influenced our choices and results. Dosage refers to how often students engage with computer science principles and learning. The setting can affect both the dosage and the depth of learning opportunities for students; for example, students who are exposed to computer science principles in a classroom setting once a month may not have the same expected outcomes as those who participate in an intensive afterschool program. Failure to take these contextual factors into account can lead to unreliable results or an inability to find an impact. This session will raise awareness about computer science measurement tools and explore the criteria that evaluators can (should?) use when selecting and/or designing evaluation instruments for a given project. Contextual, project-specific factors affect evaluators’ choices at all stages of the process, from the evaluation design to implementation, analysis, and reporting. In keeping with the AEA Guiding Principle of Systematic Inquiry, we believe it is important for evaluators to adhere to rigorous technical standards by, for example, selecting validated instruments where possible and using best practices in analysis and reporting. However, evaluators should both be aware of and be prepared to clearly communicate the strengths and limitations of any methodological choices they make. The roundtable format of this session will allow all attendees to participate in the conversation, reflect on their own experiences, and share their stories. Attendees will leave this session prepared to make more informed decisions about measuring outcomes in computer science education for K-8 students.