146 - Feedback as a Formative Evaluation Approach to Enhance Fidelity to Evaluation and Maximize Evaluation Use in a Multi-institutional Project

Abstract Information: A multi-institutional mentoring project was created in 2020 (M.E.N.T.O.R., 2020). This initiative is a collaborative capacity building partnership between 1862 and 1890 land-grant universities (LGUs), in which faculty leadership teams (FLT) (at each of these LGUs) develop campus-based mentoring programs. By doing this, these teams can provide mentoring support to university students, especially underrepresented minorities (URMs) in the food, agricultural, natural resources, and human (FANH) sciences disciplines. The long-term impact of the project is to help students (as participants) be academically, socially, and professionally successful. Over the past two years, 11 LGUs have participated in this project. A core team (project PIs and evaluator) was created to coordinate the project activities and evaluate progress made and outcomes achieved. To implement project activities, participating FLTs were asked to submit and present mini-grant mentoring proposals outlining mentoring activities in their respective institutions. These proposals were reviewed and evaluated by the project leadership team to be approved for funding to receive $10,000. To facilitate the implementation of the project to achieve its goals, several (6) mentoring webinars were conducted, which included topics on: inclusion, diversity, mentoring, program development, administrative support, and program evaluation. The webinars were well received, and the evaluation data revealed that the webinars were immensely helpful in implementing project activities. During the second year, COVID-19 hit and the impact on implementing project activities was huge and significant and limited the abilities of project directors to implement activities in a timely manner. In several instances, they had to delay the project activities, adjust the implementation timeline, use different approaches than what was planned, and postpone collection of evaluation data. Despite all these constraints posed by COVID-19, the project directors continued to work on their respective projects, implemented the activities to the best extent possible, adjusted or revised project activities and measure progress to document outcomes of their respective projects. In February of 2022, a Zoom meeting of all FLT project directors and/or their representatives was organized to take stock of the situation and assess progress made thus far, and challenges faced, and the opportunities for remainder of the project and beyond.The purpose of this paper was to assess progress made in the M.E.N.T.O.R. project since it started, the impact of COVID-19 on project implementation, next steps, and challenges and opportunities moving forward. Specifically, project core team was interested in listening to each of the FLT project directors and assess progress thus far to determine ways to move forward with a goal of not only accomplishing project goals, but also help minimize the challenges they may face during the remainder of the project.

Relevance Statement: We used feedback research as an approach to assist the 11 LGUs to improve project effectiveness (Hattie & Temperly, 2007: Rog & Bickman, 1984). According to Heritage (2014), feedback can serve as a learning tool to participants and improve their own skills in implementing and evaluating their respective projects. Additionally, this approach was used to gather information to determine the need for evaluation and to increase the value and usefulness of evaluation in documenting outcomes of individual projects as well as the overall goals of the M.E.N.T.O.R. project. We believe that using feedback from LGUs can provide valuable information for them to improve project outcomes. Five questions were asked: (1) project information including the FLT, (2) progress made thus far (SU 21 and FA21), (3) planned activities in SP22, SU22, and FA22, (4) scholarly outputs produced, and (5) challenges and opportunities. Another question, “need for assistance” was also asked. Webinar participants submitted this information in an Excel spreadsheet as well as sent prepared notes following the webinar. Project evaluator took detailed notes and summarized the information. Overall, the FLTs were doing what they all could do to move the project forward. Based on the information (verbal and written) provided, some were making excellent progress, while others were slowly getting into implementing activities to meet their respective project goals. Regarding progress made, few FLTs created their mentoring programs, recruited URM students, and on-boarded and mobilized faculty to help with their projects. Some FLTs have collected baseline data, conducted interviews with participants, and developed resources for inclusive mentoring practices among students. Others offered seminars and mentoring workshops, worked with college administrators to seek funding match. This year, the FLTs are working hard (with the ease of COVID-19) to accomplish what they planned to do at the start of the project. Example of activities include expanding the program to 4-H youth, building relationships with 1890 institutions, hosting mentor-mentee exchange for better understanding of mentoring concepts, meeting with higher-level administrators to include mentoring in promotion and tenure documents, inclusion of mentoring concepts into career development and workforce development initiatives. Others are analyzing data for presentation and/or publications in the coming months and anticipate scholarly outputs from FLTs this year. Project team members documented several challenges which were mostly institutional. For example, option of in-person versus virtual classes making it difficult to organize project activities, difficulty in paring faculty and mentee groups, working with budget and financial issues, faculty time commitment, dealing with challenges of URM students coming into R1 institutions. FLTs are implementing their campus-based mentoring programs despite the delays caused by COVID-19 and were able to make progress in developing and implementing their mentoring program using collaborative approaches to build capacities in interested students and faculty. The FLTs were able to refine their plans as they learned more about mentoring and how they could effectively implement a mentoring program on their campuses. Moreover, FLTs requested assistance to evaluate their mentoring programs, including data collection, data analysis, and preparing publications and scholarly outputs.