Use of Crossed Random-Effects Models to Assess Multiple-Choice Items: An Experimental Study

Including general options on items, such as None-of-the-above (NOTA), could lead to worse psychometric properties. Further, personality traits are related to academic performance and could influence NOTA effects. Therefore, this study aims to test the effects of NOTA by manipulating its appearance and its use as the correct option or as a distractor, while considering the examinees' personality traits. The sample consisted of 449 psychology students who answered a statistics concept inventory. A crossed random-effects model was conducted to model the probability of answering an item correctly depending on items and students' covariates. The appearance of NOTA negatively affects the probability of answering correctly and changes between examinees. Conscientiousness, Extraversion, Emotional Stability, and previous knowledge of statistics are also significant predictors. The results point to increased items' difficulty when NOTA is included, although not all the examinees seem to show the same behavior when dealing with NOTA options.