How much item formulations affect the probability of a correct answer? An experimental study

Different words, numeric values or semiotic registers, figures, graphs or tables, namely elements of item formulation, affect students’ probability of encountering an item correctly. An experimental study was carried out to compare different formulations and their effect for validating a comparative technique: students’ answers were equated within the framework of the Rasch analysis to make them directly comparable on the same scale and then analyzed and compared to each other. Four anchored math achievement tests were administered to a sample of 1647 students attending grade 8. In particular, we used different formulations of the same items to explore students’ misconception about the relationship between perimeter and area. Results confirmed that item formulation channels students’ solving strategy and thus modifies the probability of a correct answer more than item content. This quantitative technique combined in a mixed-method approach with qualitative analysis may provide useful didactical issues.