158 - Choosing the Most Suitable Software for Analyzing Complex Educational Survey Datasets

Abstract Information: This presentation will provide information and suggestions on choosing the most appropriate software for analyzing large educational survey datasets that are drawn from complex samples, such as those from the National Center for Education Statistics (NCES). Information gained from this presentation will be helpful for evaluators and researchers who wish to use large public datasets for evaluation or research purposes. Advantages and disadvantages of different software packages, including IBM SPSS statistics, AM Software, and the R Software packages of EdSurvey and Survey are discussed. Additionally, considerations for evaluators and researchers when analyzing a large educational dataset are explained. Examples will be provided, and a recommendation for using the R Software Survey package is justified.

Relevance Statement: This proposal came about as a result of conducting analysis of the NCES’s Education Longitudinal Study of 2002 dataset. At this time, we knew that we would need a statistical software which could consider the complex survey aspects of our data during analysis; when searching for the most appropriate software to analyze our data, we ran into a few challenges. First, to use IBM SPSS statistics, we would need to purchase an expensive add-on module which would allow for analyzing complex survey datasets. Secondly, the American Institutes of Research’s recommended software (AM Software) is no longer supported. When considering using R, we needed to find a package which would allow for consideration of complex design aspects. We found that while Ed Survey would allow us to consider these elements, it would not allow for flexibility of using other packages for data munging and visualization, etc. Accordingly, we found that the R Survey package was the most appropriate package for analyzing complex survey datasets because of its versatility. From our experience, we first provide necessary considerations for evaluators and researchers when analyzing complex survey packages. As evaluators, we have access to many large educational survey datasets for evaluation or research purposes. To tell an engaging and informative story using complex survey data, we must first choose an appropriate software to analyze, describe, and visualize that data. Evaluators can use this information when designing evaluation methodologies which use large or complex survey datasets so that they may tell a story which is bias free. It is important to consider both the complex nature of our data as well as the flexibility we need for analysis and reporting results when we choose an appropriate dataset. For example, we must consider the unique aspects of the data we choose to use, such as unequal sample selection probabilities, non-responses, and coverage errors, before choosing the appropriate software. Additionally, researchers/evaluators must choose a software which provides flexibility for data visualization, so that they can tell a more accurate and capturing story. Advantages and disadvantages of different software packages are discussed, including those from IBM SPSS statistics, AM Software, and the R Software packages of Ed Survey and Survey. Examples will be provided, and a recommendation for using the R Software Survey package is justified. There is great power in using a story in our evaluations and using evaluations to tell a story. However, when using complex survey datasets, it is important that the story we tell is fueled by unbiased results which stem from proper consideration of our data and appropriate choice of software.