Proposal of how to create logic models at the time of evaluation – Does it serve as material for conversations with project participants?

Abstract Information: In this study, we propose our revision of the evaluation system and how logic models served as material for conversations with project participants. (Motivation) We found outcomes are sometimes separated from any outputs, and this fact may affect the success rate of projects. To increase the success rate of projects, it would be helpful to define intermediate outcomes more precisely at the project planning stage. Thus, we applied logic models to the ex-ante evaluation in FY2017. In recent years, various environmental changes have progressed rapidly, and both technology and social issues have become increasingly complex. We believe it is important to incorporate the perspective of reviewing logic models not only at the time of project planning but also regularly to show externally that project managers are responding to changes in the environment and to build consensus with project participants. (Methods) We revised our evaluation system and introduced logic models into the midterm and the terminal evaluation in FY2023.  (Conclusion) We had discussions with the Ministry of Economy, Trade and Industry (METI), which has jurisdiction over NEDO and established the revise-system of logic models. This new system will begin in FY2023. We would like to discuss these new schemes with all members based on the results of the introduction and feedback from those who have experienced them.

Relevance Statement: As an R&D funding agency under the Ministry of Economy, Trade and Industry in Japan, New Energy and Industrial Technology Development Organization (NEDO) has a mission to accelerate the commercialization of R&D results by private firms. To maximize the effect of funding, we continuously check our project management details from the viewpoint of evaluation. Using a plan-do-check-act cycle that incorporates evaluation results in the planning process, NEDO endeavors to improve its management to carry out appropriate technology development that achieves excellent development results. NEDO carries out an ex-ante evaluation, a midterm evaluation, and a terminal evaluation after project completion. In addition to these evaluations of ongoing projects, follow-up monitoring of project participants is carried out using surveys and other information collection techniques for six years after the project ends. This monitoring is used to determine the utilization of NEDO project results and to analyze key issues for transferring them to practical application. Before this study, logic models were created only in the ex-ante evaluation. In this study, we revised our evaluation system and introduced logic models into the midterm and the terminal evaluation. We would like to find out how logic models served as material for conversations with project participants.