176 - Connecting disparate data sources to understand access, engagement and collaboration in a Community of Practice in STEM

Abstract Information: This paper offers a unique approach to connecting disparate data sources to understand the interactions and perceptions of over 1,000 STEM educators collaborating in an online Community of Practice (CoP). Connecting Our NASA Network of Educators for Collaborating Together in STEM (CONNECTS) Community of Practice (CoP) members express passion for the joint enterprise of STEM education and their desire to learn from and contribute to the community with their variety of experience and expertise (Lave & Wenger, 1991; Wenger & Snyder, 2000). The NASA CONNECTS is designed to facilitate collaboration amongst STEM educators. Metadata from the CONNECTS website offers viable sources of information to understand how users engage with online resources and collaborate. For example, these metadata can provide information on users' engagement activity like the average time an individual spends on CONNECTS, the total number of clicks an individual makes once they log in to CONNECTS, the number of unique individuals who have viewed any content, and the number of individuals who actively engage with content on CONNECTS. Cross-tabulated results on these engagement metrics by demographic can help provide insights into how educators interact with online resources and with other members of their community of practice. Additionally, these data analytics can help evaluators carefully identify and solicit feedback from CoP members who can provide meaningful qualitative data for focus groups and interviews. Results from focus groups and interviews offer rich descriptions of users’ experience with the CONNECTS platform. Moreover, these qualitative results can provide evidence of the validity of the metadata metrics on users’ online activity. In addition, Social Listening (Stewart & Arnold, 2017) provides a unique tool for evaluators to scan online comments to identify important topics that CoP members care about. Evaluators can leverage data analytics capabilities to conduct sentiment analysis, which includes scanning comments posted by individuals on the CONNECTS platform. Analysis of online conversations can provide valuable for CoP leaders and administrators to provide timely support on pressing topics in STEM. Powerful analytic tools from the environmental science discipline were also applied to understand the reach of the CONNECTS platform across the nation. Geographic Information System (GIS) combines geographic information and other available census data (e.g., poverty, age, education level, etc.) to provide insights about where CoP members are located and the characteristics of their location. The evaluation methods and approaches discussed in this paper may support evaluators and administrators in their efforts to understand online CoP members' access to STEM resources and propose alternative approaches to measuring online CoP engagement and collaboration. By connecting disparate data sources proposed, CoP leaders and administrators can leverage existing data to provide a more comprehensive story of how CoP members engage and collaborate and under what circumstances those engagements occur. Additionally, the proposed methods may support Community of Practice leaders and facilitators to share relevant and timely information to increase access to STEM resources and improve student outcomes.

Relevance Statement: This paper offers a unique approach to connecting disparate data sources to understand the interactions and perceptions of over 1,000 STEM educators collaborating in an online Community of Practice (CoP). The NASA CONNECTS is an online Community of Practice (CoP) designed to facilitate collaboration among educators through discussion, networking, and the opportunity to share best practices. Educators can access NASA mission updates, content, resources (i.e., knowledge articles), lesson plans, activities, and educator professional development opportunities. This presentation will share information gathered from a study that examined how and to what extent NASA CONNECTS: 1) increased educator access to STEM resources; 2) displayed efficiency of use; 3) incorporated features that enhanced collaboration; 4) enabled communication with NASA; and 5) encouraged continued engagement with fresh and current content. The evaluation study adopted a sequential mixed-method design where quantitative data are collected first. The initial data collection effort investigates users' engagement activity on CONNECTS by leveraging available data analytics on users' engagements with the social media platform. These descriptive statistics may include but are not limited to, the average time an individual spends on CONNECTS, the total number of clicks an individual makes once they log in to CONNECTS, the number of unique individuals who have viewed any content, and the number of individuals who actively engage with content on CONNECTS. The evaluation team will generate cross-tabulated results by demographic to understand better how educators interact with online resources and other members of their community of practice. Additionally, mailing zip codes provided by CONNECTS members during the online registration process will be used to gather geographic and demographic information using the Geographic Information System (GIS). The GIS tool allows evaluators to understand information such as poverty level, population age, education level and other available census data that can help provide insights about the access and reach of the CONNECTS platforms across the nation. While descriptive statistics offer valuable insights into individuals' activities on CONNECTS, these metrics alone do not provide deeper insights into why members engage with the platform. To help meet this need, the evaluation team will execute a follow-up qualitative data collection and analysis in two stages. The first stage will leverage quantitative data results to identify and strategically sample individuals with differential activity patterns on the CONNECTS platform to solicit qualitative data through focus groups and interviews. The focus of the interviews will revolve around their activities on the CONNECTS platform. The second stage will leverage data analytics capabilities to conduct sentiment analyses called Social Listening (Stewart & Arnold, 2017), which includes scanning comments posted by individuals on the CONNECTS platform within a specified period. This process involves coordinating with CONNECTS administrators and identifying keywords or phrases that could signal specific sentiments among users' engagement with CONNECTS content. By connecting disparate data sources proposed in this paper, CoP leaders and administrators can leverage existing data to provide a more comprehensive story of how CoP members engage and collaborate and under what circumstances those engagements occur.