Ratings and comments given for formative R&D project evaluation – Do they depend on the attributes of external evaluators? And how?

Abstract Information:
In this study, we analysed data accumulated for NEDO’s past national R&D projects and found the relationships between the evaluated ratings / comments and personal charateristics of evaluators.
(Methods)
We analyzed the results of terminal evaluations of projects done between 2005 and 2021 (197 projects with 1,325 cumulative evaluators) regarding each evaluator’s scores and free comments. Charateristics of the evaluators include their institution, department and position, position in the evaluation subcommittee, and field of expertise.
We conducted a series of statistical analyses to see the bias in the terminal evaluation results influenced by the charateristics of the evaluators.
Text-mining analyses of the comments showed the relationship between the tendency to use specific words in the comments and the attributes of evaluators.
(Result1: Terminal evaluation - Qualitative analysis - Characteristics vs ratings)
Significant trends were found mainly for the two aspects of (2) Project Management and (4) Prospects for Practical Application.
(Result2: Terminal evaluation - Quantitative analysis - Correlation between aspects)
Examining the correlations between individual evaluators’ ratings of several aspects yielded exciting results. The correlation coefficients between aspects were smaller when the evaluator's affiliation category was “Management and humanities” or “Consulting and finance” than when the evaluator was “University (science and technology)” or “Company (science and technology)”.
(Result3: Terminal evaluation - New analysis via viewpoints)
Differences in evaluators’ characteristics do not directly influence the differences in ratings and comment. There are differences in the tendency of the selected viewpoints according to evaluators’ characteristics and in the reflection on the ratings for each viewpoint. Combining the two trends results in a correlation between evaluators’ characteristics and ratings and comments.
(Result4: Analysis of questions and answers in the ex-post survey)
An analysis of the relationship between the various question items in the ex-post survey showed some significant correlations. For example, two question items, "Involvement of expected users/customers in research and development" and "Progress towards commercialisation", had relevance in their answers.
(Result5: Relationship between the closing evaluation and the ex-post survey)
The relationship between the results of the terminal evaluation and the responses to the ex-post survey questionnaire was analysed with a focus on the stage of R&D progress. An attempt was made to analyse this in detail by allocating the values of the rating deviations to the evaluation viewpoint elements. By calculating and comparing the mean and variance of the rating deviations for each stage of R&D progress in the ex-post survey responses, it was possible to identify the main evaluation viewpoint of the projects that led to successful commercialisation or discontinuation. This leads to the usefulness of specifying the critical factors in the ex-post survey, e.g., the success rate of commercialisation, to be emphasised in advance at the time of the terminal evaluation.
Relevance Statement
New Energy and Industrial Technology Development Organization (NEDO) is one of Japan's largest public research and development management organizations. NEDO has established an evaluation system for assessing its projects: ex-ante evaluation, mid-term evaluation, terminal evaluation, and ex-post survey. NEDO conducts a terminal evaluation at the end of each project to evaluate the degree of achievement and collect constructive comments. Designated external expert members of the ad-hoc subcommittees for the project carry out all terminal evaluations. There are four aspects in the terminal evaluation: position and significance of the project, project management conditions, R&D technical achievement, and prospects for future practical application.
This study aims to measure the fairness of our evaluation methodology and obtain hints to optimize the composition of each evaluating subcommittee. While our reports before this study only treated the trends in the AVERAGE ratings of all members of each subcommittee (averaged scores and edited comments), this study, for the first time, extracted the characteristics of the individual evaluation results that we could not have revealed by using averaged data only.
The results showed the selection of the evaluation subcommittee members in the past had enough fairness and validity, together with policy implications for establishing more formative evaluation processes by fully utilizing comment texts as stories by the evaluators.
The present evaluation system was designed mainly as a straightforward, summative evaluation for accountability and has been running for almost 20 years, primarily paying attention to the selection of subcommittee members to ensure fair evaluation results.
However, this balanced and impartial evaluation culture may have been a drawback for the formative feedback using the evaluation results. Specifically, this is because the quantitatively evaluated rating was set by averaging the subcommittee members' scores, and the members' comments were edited to form the consensus of the subcommittee.
This study focused on the selection of subcommittee members and the relationship between the attributes of committee members and evaluation outputs to understand the tendency quantitatively and explore the possibility of realizing more formative evaluation using a more robust and impartial way to choose evaluators.
Based on the results of this study, we simulated the difference in evaluation results by the composition of the evaluation subcommittee. The result showed that including the humanities and social sciences committee members, the natural science members for technology evaluation, and the industrial field specialists for business plans would lead to a more formative but still balanced evaluation. The Industrial Structure Council of Japan's Ministry of Economy, Trade and Industry (METI) stated in 2022 that "In terms of the evaluation system, it is necessary to consider the participation of experts in the humanities and social sciences with a view to the social implementation of the results." This policy shifts the evaluation items and criteria for the national projects, such as whether the R&D genuinely contributes to solving social problems. The results of this research predicted the validity of this policy.