Use of Big Data to evaluate the impact of academic international mobility on funding

Abstract Information: Analyses of the effects of international mobility on academic careers already incorporate big data but are still based predominantly on databases and bibliometric methodologies. These analyses should consider individual, institutional and systemic factors to avoid generalist studies. Applying algorithms makes it possible to classify authors according to gender, except in some countries such as China. The information on authors and citations also allows the analysis of co-authorship networks. Some studies build proxies on career position by considering the number of years from Ph.D., first published article or position in the article authorship list. However, the proxies could be more accurate. Moreover, relevant variables regarding sociodemographic characteristics, such as race and nationality, and institutional aspects, are unavailable. In this sense, the paper discusses the limitations of bibliometric analyses and presents the possibilities of integrating other secondary databases to analyze the effects of mobility on obtaining research funding. The study uses data from the academic mobility programs of the São Paulo Foundation in Brazil. The country has bases that can be used, including administrative data from funding agencies, information on formal employment, a large curriculum vitae platform, and transparency portals. It is also possible to obtain data via the law on access to public information, which is part of the Brazilian Open Government Policies. The challenges lie in integrating the data, getting the funding figures, and attributing causality.

Relevance Statement: Currently, the use of big data is more concentrated in applications for implementation, coordination, and management of programs, and not necessarily in evaluations. However, given its potential in evaluation design, understanding program context, data collection and analysis, and dissemination of results, there are several initiatives underway to advance its use in this area (York & Bamberger, 2020). According to Silva (2020), there are at least five relevant challenges of using big data in evaluation. Firstly, data quality in terms of credibility, ease of access, consistency, and the possibility of integration. Secondly, challenges related to manipulation, as analysis techniques can skew data and impair decisions. Thirdly, the interpretability of data in terms of pointing out false causal relationships between data and lack of contextualization. Fourthly, there are concerns regarding ethics and privacy in terms of consent, data ownership, and digital exclusion. Finally, the use of big data demands specific skills from data scientists (York & Bamberger, 2020). The Brazilian advance in digital government opens up possibilities to obtain data for evaluation, including funding value from microdata in the Transparency Portals or requests via the law on access to public information. York & Bamberger (2020) point out some factors determining the applicability of big data and data analytics to program evaluation. Using this framework, the evaluation of the impacts of FAPESP's programs focused on mobility in obtaining funding for research can benefit from some characteristics. The programs use easily measurable and readily available data from most sources. The indicators for evaluation are well established with high construct validity, both concerning inputs and outcomes and impacts. These are programs with a relatively long duration and where time series data can be generated. These programs have many potential variables and no articulated theory of how outcomes are expected to be achieved because there is no consolidated theory on the subject. Finally, these programs have no political concerns about ownership, control, access or privacy. There are different situations in the specific case of data for evaluation of FAPESP programs. Some data are publicly available from federal funding agencies (CNPq, Sucupira and Geocapes from CAPES). In other cases, it is possible to access the data, but not the base itself (Lattes CV). Some are available through Application Programming Interface -API (Dimensions and Altmetrics) or by webcrawler (Scopus, Google Scholar and INPI). Finally, others require negotiation (CAPES, RAIS, FAPESP). By obtaining and integrating data from these sources, it is possible to use quasi-experimental designs for impact evaluation. Silva, V. M. (2021). ‪Futures studies and foresight for science, technology and innovation: trends of using big data and machine learning. (Doctoral Dissertation). University of Campinas. Retrieved March 17, 2023, from https://discovery.ebsco.com/c/7zh5jk/details/wn6x7ir54b?q=vinicius%20muraro York, P. (2020). Bamberger, M. Measuring results and impact in the age of big data: The nexus of evaluation, analytics, and digital technology. The Rockefeller Foundation - Measurement & Evaluation. Retrieved March 17, 2023, from https://shre.ink/kHyw