The Challenging Story of Assessing Mathematics Teachers' Growth: Evaluating the North Carolina Master Mathematics Teacher Fellows Program

Abstract Information: Thanks to a Noyce METS grant, the North Carolina Mathematics Master Teacher Fellows program has been helping a group of high-school mathematics teachers in high-needs schools across North Carolina obtain master's degrees in mathematics education and National Board of Professional Teaching Standards (NBPTS) certification, and participate in professional development and leadership training since 2019. To assess the participants' growth during the program, the evaluator has used an annual survey, classroom observations using the Instructional Quality Assessment (IQA )protocol, interviews, and observations of the annual summer institute. The data collected indicates changes in participants' beliefs regarding mathematics instruction, in how much emphasis they place on specific student objectives during their instruction, and in their participation in leadership activities/roles in their schools and districts. There were also some changes in instructional practices, although not as much as had been anticipated. The evaluator noted a discrepancy between the instructional practices participants reported using and the instructional practices actually used during classroom observations. Several possible reasons for this inconsistency were explored and the evaluator made recommendations to the project team to try to resolve this disconnect. Not all of the recommendations were adopted by the team, but they decided to maintain the same data collection tools and methods, resulting in the possibility this disconnect would persist. The evaluator then adjusted the classroom observation procedures to include communication with the participants regarding the specific instructional practices that the protocol described in an effort to resolve the issue, with some success.

Relevance Statement: This proposal describes the evaluation of a five-year program designed to improve mathematics teachers' instructional skills and leadership skills. The evaluation plan, instruments used for data collection, and adjustments to the evaluation throughout the program will be of interest to evaluators as they represent an effort to assess teachers’ progress in instructional practices while responding to the COVID-19 pandemic and to the project leaders’ inability to implement all of the changes recommended by the evaluator. The evaluation plan was created in collaboration with the project leaders and was designed to capture changes to the program participants during the program (Program Evaluation Standards U3 Negotiated Purposes, U5 Relevant Information, P2 Formal Agreements). The data collection tools and procedures were negotiated at the start of the program and their continued use was agreed upon annually. This proposal also discusses a specific situation in which the continued use of a specific classroom observation protocol without any corresponding education of the participants created the possibility of an evaluation mismatch (Program Evaluation Standard U8, Concern for Consequences and Influence). The Instructional Quality Assessment (IQA) (Boston, M. 2018) was used at the request of the project leaders at the start of the grant. The IQA is an observational protocol that guides the observer to focus on the mathematical tasks of the lesson, the class conversation that occurs regarding the tasks, the questions asked by the teacher, the responses given by the students, and the connections the teacher and students make among the statements made by students. The evaluator was trained to use the IQA, then asked to implement it during classroom observations. During the second year of the program, 2020-2021, the Fellows recorded lessons for observation due to the remote instruction they were conducting during the pandemic. This did not allow the evaluator to observe any classroom discussions or student interactions, requiring an adjustment to the evaluation plan. In the annual evaluation reports, beginning with the first report in 2020 and continuing through the following years, the evaluator recommended using the IQA to inform the professional development provided to the Fellows and/or to share the IQA’s rubrics with them, or to discontinue its use. Unfortunately, the project leaders were unable to fit these recommendations into the time they had with the Fellows, though they still wanted the IQA to be used as it provided information that was valuable to them. Their meetings with the Fellows were packed with information and activities to help them complete their master’s degrees, achieve NBPTS certification, and take on leadership roles in their schools. To address the unintended consequences of using a specific observational tool whose expectations were not clearly communicated to the participants, the evaluator adjusted her approach to classroom observations to include communication regarding the desired instructional practices with the participants prior to the observations (Program Evaluation Standard E2 Internal Metaevaluation; Guiding Principle C Integrity). Reference: Boston, M.D., Candela, A.G. The Instructional Quality Assessment as a tool for reflecting on instructional practice. ZDM Mathematics Education 50, 427–444 (2018). https://doi-org.prox.lib.ncsu.edu/10.1007/s11858-018-0916-6