Investigating the Use of Bifactor Models in the Context of Item Wording Method Effects

Abstract Information: The use of unidimensional scales to measure latent constructs is common in evaluation practice. Data from unidimensional scales are often collected via self-report instruments. To prevent certain response styles (e.g., acquiescence) from occurring, these scales often contain both positively and negatively worded items. However, the inclusion of such items often leads to method effects, which can distort the instrument’s psychometric properties. One modeling technique that has gained attention in recent years related to item wording effects is the bifactor model, which specifies a general construct that explains the covariation among all items, and two method factors that explain additional covariation among items with similar wording. The purpose of this study was to investigate the performance of the bifactor model-based indices, explained common variance (ECV), omega hierarchical (omegaH), and the percent of uncontaminated correlations (PUC), under data conditions frequently encountered when item wording method effects are identified, including unbalanced method factors (unbalanced in size and magnitude). Results from this Monte Carlo simulation study indicated the unidimensional model can be quite robust when method factors are ignored under certain conditions, particularly if the method factors are found to contribute equally to additional item-level variance. In general, results from this study offer evaluators utilizing unidimensional scales that contain both positively and negatively worded items guidance with how to subsequently treat their data if method factors are identified. Ultimately, item wording method effects may not be as problematic as once feared.

Relevance Statement: The use of unidimensional scales to measure latent constructs is common in evaluation practice. For example, evaluators may administer the Rosenberg Self Esteem Scale (RSES) to a group of students participating in an afterschool mentoring program to assess the program’s impact on students’ self-esteem. Unidimensional scales, such as the RSES, are intended to measure one construct, with a composite score (i.e., a sum score) serving as an empirical indicator of the construct. Data from unidimensional scales are often collected via self-report instruments. To prevent certain response styles (e.g., acquiescence) from occurring, these scales often contain both positively and negatively worded items. However, the inclusion of such items often leads to method effects, which can distort an instrument’s psychometric properties. Dimensionality investigations of instruments that contain both positively and negatively worded items often lead to a rejection of the theorized unidimensional model in favor of multidimensional structures, leaving researchers at odds for how best to treat their data. One modeling technique that has gained attention in recent years related to item wording effects is the bifactor model, which specifies a general construct (i.e., the target construct) that explains the covariation among all items, and two method factors that explain additional covariation among items with similar wording. Recent applications of the bifactor model have called for utilizing select model-based indices to establish “essentially unidimensional” models (i.e., Reise, Scheines, et al., 2013; Rodriguez et al., 2016). The purpose of the current study was to investigate the performance of the select bifactor model-based indices, explained common variance (ECV), omega hierarchical (omegaH), and the percent of uncontaminated correlations (PUC), in establishing essentially unidimensional models. A Monte Carlo simulation study was conducted to examine the performance of these indices under conditions frequently encountered when item wording method effects are identified within unidimensional scales, including, (1) the number of items per method factor, and (2) item loading values and patterns (i.e., balanced vs. unbalanced) between the method factors and the general factor. Outcomes examined in this study included the relative bias in factor loadings estimated from a misspecified unidimensional model (i.e., ignoring the method factors) when data were simulated from a bifactor model. Results indicated the presence of unbalanced method factors (both in size and magnitude) required a substantially “stronger” general factor, as evidenced by higher ECV (>0.80) and omegaH (>0.85) values, to reach “essentially unidimensional” status. In general, findings from this study will aid evaluators in meeting and adhering to the AEA Evaluator Competency 2.0 – Methodology. The findings offer evaluators utilizing unidimensional scales that contain both positively and negatively worded items guidance with how to subsequently treat their data if method factors are identified. More specifically, this study highlights how the use of a bifactor model, and the model-based indices, can provide evaluators with a technique to assess the strength of the general factor in relation to the identified method factor(s), and the degree to which the data can be treated as unidimensional. Specific recommendations and guidelines will be discussed in the paper presentation.