Professor of Psychology and ASPIRE Evaluator California State University, Stanislaus Turlock, California, United States
Abstract Information: The chi-square test is a popular tool for determining whether participants prefer some response options more than others in forced-choice situations. For example, if college students are asked to identify the one issue that concerns them most from a list of 15 options, chi-square tests can determine if some issues are chosen significantly more often than others, or if the issues chosen most often by, say, students from historically underrepresented groups differ significantly from those chosen most often by other students. But what happens if students are asked to choose the top three issues, rather than just one? Chi-square testing produces highly inaccurate p-values when each participant makes several forced-choice responses. However, there’s no need to abandon the test or limit participants to choosing just one response; Monte Carlo techniques can be used to generate adjusted p-values that are appropriate for assessing statistical significance when participants make several forced-choice responses.
Relevance Statement: Evaluators often engage clients or program participants in forced-choice tasks that require selecting responses from a list of options. For example, clients may be asked to identify priorities for organizational change by identifying the most critical challenge(s) from a list of options, while participants in a successful program that must reduce its budget may be asked to identify the element(s) they find most beneficial. If each respondent makes only one selection, a chi-square test can be used to determine if some options are chosen significantly more often than others, or whether the options chosen by one group of respondents differ significantly from those chosen by other groups. However, standard chi-square testing is not appropriate when each respondent makes several choices, because the test’s independence assumption is violated. This is evident from noting that, each time respondents choose a response, they have fewer options from which to make their next choice. This forces uniformity on the response distribution; even if one response is favored more than others, this preference will be masked by the selection of other, less-favored responses. The result is grossly inflated p-values, which can make it difficult to identify the responses that are chosen more often than would be expected by chance, or that are chosen more often by one group of respondents than by others. One potential solution is to adjust the degrees of freedom associated with the number of response options. For example, instead of assuming the degrees of freedom is one less than the number of options (which is correct when each respondent chooses only one response), the degrees of freedom could instead be calculated by subtracting the number of selections made by each respondent from the total number of response options. However, with this adjustment the resulting p-value is smaller than it should be, because only it describes the degrees of freedom associated the very last choice made by each respondent. A mathematical solution to this problem is neither obvious nor necessary. This presentation describes a Monte Carlo approach to determining the appropriate p-values. The approach produces results identical to the normal chi-square test when respondents make only one forced-choice response. However, the Monte Carlo approach allows practitioners to extend the chi-square test to applications in which respondents make several forced-choice responses.
