The Evaluation Capacity of Czech Social Services Organizations

Abstract Information: The increasing call for the evaluation of nongovernmental organizations’ (NGO) programs is evident in the Czech republic where their nonprofit sector has substantially grown in the past 30 years since the Velvet Revolution. Such demand typically comes from the Czech Ministry of Labor and Social Affairs and the European Union Social Investment Fund. However, many Czech nonprofits are small and lack aspects of evaluation capacity, defined as “the human capital (skills, knowledge, experience, etc.) and financial/material resources” that are necessary for the practice of evaluation (Boyle, Lemaire, & Rist, 1999, p. 5). This is accompanied by the professionalization of the Czech sector along with the evolution of the evaluation profession there. Evaluation capacity building (ECB), defined as “the intentional work to continuously create and sustain overall organizational processes that make quality evaluation and its uses routine” (Stockdill, Baizerman, & Compton, 2002, p. 1) and similarly, “the design and implementation of teaching and learning strategies to help individuals, groups, and organizations, learn about what constitutes effective, useful, and professional evaluation practice” (Preskill and Boyle, 2008) therefore is necessary. ECB is context dependent and stakeholder driven, and therefore is multi-dimensional; dynamic; has economic, sociological, and psychological implications; and is different in size, scope, factors, and components from one organization to another. This mixed methods study examines the evaluation capacity of Czech social services nonprofit organizations, and discovers challenges and successes accompanied by recommended practice.

Relevance Statement: Evaluation capacity building (ECB) is dynamic, complex, driven by context and stakeholders, and its success is largely determined by organizational culture, organizational learning, leadership, systems and structures communication, and resources (Alaimo, 2008; Bamberger and Mabry, 2020; Bourgeois and Cousins, 2013; Boyle, Lemaire and Rist, 1999; Newcomer, Hatry, and Wholey, 2015; Preskill and Boyle, 2008; Volkov and King, 2007). As the body of knowledge on ECB grows and evolves, practice in it and in evaluation will expand and improve. This body of knowledge is now international as ECB is being studied and practiced around the world and evaluators are learning from other countries. This study of ECB in the Czech Republic will add to this body of knowledge and the evolution of recommended practice. External and internal contexts can be compared with other countries, so practitioners may be better equipped to practice ECB and strengthen the practice of evaluation. This study also helps tell the Czech story around evaluation which is not only interesting but helpful when understanding evaluation in a global cultural context. The lead author of this study developed an ECB framework drawing from existing frameworks (Bourgeois and Cousins, 2013; Bourgeois, Osseni, Hotte, and Simmons, 2016; Cousins, Goh, Elliott, and Bourgeois, 2014; and El Hassar, Poth, Gokiert , and Bulut, 2021). This framework can be applied to studies of ECB in other countries and service delivery contexts where research results can be combined with recommended practice so organizations can increase their evaluation capacity. The framework in turn was used to develop an online survey that was sent to all 1,032 Czech social service nonprofit organizations registered with the Czech government yielding a 41% response rate. The survey was coupled with semi-structured interviews with directors of these organizations to dig deeper into what is behind the survey results. These insights will be helpful to any evaluator engaging in ECB practice with their client organization. In these results 67% of the organizations were not familiar with the Czech Evaluation Society’s Evaluator’s Code of Ethics. Similarly, 71% were not familiar with the society’s evaluation standards. While results indicating evaluation results use and learning were generally positive, 65% stated their services (programs) did not have a theory of change and 74% indicated they did not have a logic model raising the question for what the basis was for how services were designed and delivered. These results and other insights paint a picture of a culture of compliance likely largely driven by the fact that they rely on the government for 65% of their revenue. This compliance culture typically is counter to ECB efforts as organizations engage in evaluation more so out of necessity and to please external stakeholders rather than for their own organizations and the clients they serve. There is clearly an opportunity in the Czech Republic to continue the professionalization of their nonprofit sector, and expand the role of government, the EU funds and the Czech Evaluation Society in helping organizations formalize and improve their approach to evaluation.