Project Monitoring and Evaluation: Application of Data Analytics for Interview Response Grading

Abstract Information: This study explores if an automatic text analysis tool, LIWC-22, can effectively assist evaluators in monitoring and formatively evaluating the qualitative data collection process using the example of a grant-funded research project. LIWC has been utilized in various fields. However, no existing study embeds LIWC into a formative evaluation to understand the features of what key stakeholders value in the qualitative data they collect. Following Robinson et al. (2013), LIWC is used to analyze three common questions from 75 interviews about the experiences of underrepresented mathematics doctoral students. LIWC output variables that are highly correlated with team members’ quality ratings of data transcripts will be used. A Principal Component Analysis (PCA) will be conducted to identify the factor structure of the ratings involving the significant LIWC variables. This can assist the evaluator in understanding what linguistic features are more commonly present in the most valued transcripts. Ultimately, the aim is to guide evaluators in identifying practical suggestions for research teams to modify protocols or retrain interviewers.

Relevance Statement: Typical grant evaluations will have formative components for project improvement. In this study, we will use Linguistic Inquiry and Word Count (LIWC) to assist in the formative evaluation of a qualitative research project. We will identify the underlying linguistic features based on ratings of interview quality as defined by the research team. The study will demonstrate how evaluators can leverage data analytic tools to perform formative evaluations efficiently. Practically, this study will help evaluators design plans with transparent and efficient procedures. It also illustrates how evaluators can begin harnessing data analytics to create tailored dictionaries for expanded use. Loud and Mayne (2014) indicate that organizations are interested in supporting activities that can add value to themselves. This is echoed in the RoE literature that evaluators seek new tools and strategies to efficiently accomplish their tasks. For example, Azzam and Harman (2016) examined the use of Amazon’s Mechanical Turk (MTurk) website for crowd-sourcing to rate and code interview transcripts, noting consistency and value in the process. For a qualitative research project, the research team and interviewers function as key stakeholders in managing and carrying out the project's mission and incorporating formative evaluation findings. Robinson et al. (2013) provide valuable insight into using LIWC for prediction with factor analysis. Such a strategy aligns well with evaluation scenarios where qualitative inquiry is the evaluand, or qualitative data is a central component of decision-making. Using a qualitative grant-funded research project, we test the use of LIWC to formatively evaluate data collection practices and devise practical suggestions for enhancing the ability of the research team to collect the type of data they desire. LIWC is an automatic text analysis program that quantitively examines qualitative data (Boyd et al., 2022). LIWC outputs variables have been applied to analyze thousands of spoken and written text forms in the social sciences. Moreover, its dimensions have demonstrated good reliability and validity evidence through years of work and research (Zhao et al., 2016). However, LIWC has not been strategically embedded into formal evaluations. Therefore, it is essential to examine its value in evaluation. We consider the following research questions to guide this study: 1. To what extent is the program collecting good interview data? 2. Which LIWC variables predict project team members' ratings of good interview data? 3. Using LIWC output variables, what is the factor structure of team members' ratings of good interview data? 4. What factors are most closely associated with improved ratings of interviews, and how much variability in the ratings do they explain? The data analysis procedure will be (1) Having six members from the research team rate the interviews; (2) establishing inter-rater reliability; (3) conducting a correlation analysis on the LIWC output variables; (4) performing a principal component analysis to identify underlying factors; (5) conducting a multiple linear regression analysis on the factor scores to check which ones are predictive of the research team's ratings.