69 - Federal Managers Telling Stories through Responses to the Federal Managers' Survey Administered by the US Government Accountability Office

Abstract Information: In keeping with AEA’s Conference theme “The Power of Story” the Center for Evaluation Methods and Issues (CEMI) is a center in the United States Government Accountability’s (GAO) Applied Research and Methods (ARM) Team that shares systematically acquired data on a range of evaluation issues vis-a-vis other forms of evidence. Like much of GAO's work, CEMI has used testimonial evidence to provide leadership and insights about how the field and practice of evaluation can continuously improve evidence quality. Evaluation managers in the federal government are periodically surveyed about their experiences with evidence to inform decision making. Recently, CEMI partnered with the Strategic Issues (SI) mission team to gather insights from federal managers (i.e., GS13, GS14, GS15, and Senior Executive Series equivalents). The CEMI team revised a survey (the Federal Managers Survey) that has been administered in the past to track how federal managers are using various types of evidence in order to comply with the Government Performance and Results Modernization Act (GPRAMA 2010) and the Foundations For Evidence Policymaking Act of 2018 as well as other related statutory and guidance. CEMI staff revised the program evaluation section in the most recent administration of the survey to gauge managers’ use of evaluation evidence and to identify several correlates with evaluation use: Familiarity with Evaluations, Evaluation Roles Served, Evaluation Quality, Human Resource Capacity for Evaluation, and Facilitating Factors. Evaluation use was operationalized with these other components to create a profile of managers' self-reported experiences with evaluation use. We collected data between July 2020 and December 2020. We surveyed 3993 federal managers from a population of 150,447 civilian managers at 24 major federal agencies to ask for their views on the availability and use of performance information, practices that can promote the use of this information, and their agencies’ data-driven reviews. The results of that online survey includes managers’ perspectives on how these various factors correlate with use. In this presentation, members of CEMI will discuss the constructs used to predict use. We will discuss the consideration of factors used to operationalize the key components of use and the use of evaluation. We contend that relying on this testimonial evidence, is akin to these managers sharing stories about their experiences with evaluation and how these stories can help evaluators better understand and improve the quality of evidence used in federal decision-making

Relevance Statement: The Center for Evaluation Methods and Issues (CEMI) in the Applied Research and Methods Team (ARM) at the United States Government Accountability Office (US GAO) has produced numerous studies that highlight the role that evaluation evidence plays in federal decision-making. Located in the United States Government Accountability’s (GAO) Applied Research and Methods (ARM) Team, CEMI has established its its leadership in evaluation related issues. GPRAMA, the Evidence Act, and other associated statutes and guidance are realized through managers in the federal government. These leaders are periodically surveyed so that their perspectives can be used to inform recommendations about how evaluation evidence can be optimally deployed. Using principles for good evaluation embraced across the evaluation field (i.e., transparency, independence, ethics, relevance and rigor), CEMI provides criteria to help advise Engagement Teams who answer questions directed by the US Congress about the extent to which evaluation evidence is adhering to those principles and meeting mandates set by legislative directives. Recently, CEMI partnered with the Strategic Issues (SI) mission team to gather insights from federal managers (i.e., GS13-GS15 and SES equivalents). The CEMI team revised the program evaluation section of the Federal Managers Survey that has been administered in the past to track how federal managers are using various types of evidence in order to comply with these statutes set forth in law to ensure government is working effectively, efficiently, ethically, equitably, and economically. CEMI staff revised the program evaluation section in the most recent administration of the survey to gauge managers’ use of evaluation evidence and to identify correlates with this use. With data collected between July 2020 and December 2020, we surveyed 3993 federal managers from a population of 150,447 civilian managers at 24 major federal agencies to ask for their views on the availability and use of performance information, practices that can promote the use of this information, and their agencies’ data-driven reviews. The results of that online survey include managers’ perspectives on how these various factors correlate with use. In this presentation, several CEMI members of the larger research team responsible for administration of that survey will discuss the construction of the questionnaire used in the program evaluation section of the questionnaire. We discuss the thinking and analysis that went into revising the program evaluation section of the Federal Managers Survey to reflect current thinking and direction of program evaluation use gleaned from US GAO's own work and the work from other government agencies, academics, think tanks, and practitioners in the field. The collective insights of these sources of information informed the construction of a survey section that operationalized several key factors: Evaluation Use, Familiarity with Evaluation, Evaluation Roles Served, Evaluation Quality, Human Capital Capacity, and Other Facilitating Factors. Panel members discuss the items included in this scale's various components and share the source of the content that undergird the questionnaire construction. We further l discuss more broadly, the use of testimonial evidence as an illustration of stories shared by informed consumers of evaluation information.