A comparison of free-response and multiple-choice questions on the American Board of Pathology primary certification examinations

The American Board of Pathology is exploring alternate forms of assessment, and herein has undertaken a comparison between free-response, short-answer questions with paired, construct-equivalent standard multiple-choice, closed-response questions. Four short-answer, free-response questions were compared with four paired multiple-choice, closed-response questions that had equivalent stems and measured the same constructs. The percentage correct responses of the paired question types were compared using chi square analysis, with a level of significance set at P < 0.05. Candidates were surveyed regarding the perceived difficulty of the short-answer question type, the question type that they preferred, and why the preferred question type was favored. All four short-answer items had fewer correct responses compared with their paired multiple-choice questions. The percent-correct responses for the short-answer items compared with the paired multiple-choice items, respectively, were: item 1: 73.5% vs 91.6%, item 2: 79.0% vs 90.8%, item 3: 3.3% vs 28.6%, and item 4: 64.7% vs 90.0%. All differences were statistically significant at P < 0.01. The short-answer questions, even when liberally graded, were much more difficult compared with the corresponding multiple-choice questions, even though the questions targeted for this evaluation were simple recall type items. The short-answer item responses raised questions about the clarity of the stem for one item, whereas the response to another item clearly disclosed a lack of understanding concerning the construct tested. The candidate survey regarding these question types disclosed an overwhelming candidate preference for multiple-choice questions.