Following a unit of instruction, an instructor decided to give a quick review quiz to determine if the participants were ready to move on to additional content. The quiz had 10 multiple choice questions. Each question offered four possible options for the answer. After scoring the quizzes, the instructor noticed unusual trends in participants’ responses to three of the assessment questions. The trends are described below.
Question #2: The correct answer is A, but equal numbers of participants have chosen either A or D. Few chose B or C as the correct response.
Question #6: The correct answer is C, but less than 10% of the participants selected that response.
Question #9: 80% of the high-performing participants missed this question, while 75% of the lower-performing students got it correct.
Consider the following with regard to each of the three questions above:
- What are some possible reasons for the trends the instructor identified?
- What might the instructor do to improve test validity and reliability when this test is administered in the future?
“An effective test item discriminates well between students who generally understand the test’s subject and those who don’t” (Suskie, 2009, p. 267). With this in mind, this discussion posed these three scenarios for analysis:
Question 2: The correct answer is A, but equal numbers of participants have chosen either A or D. Few chose B or C as the correct answer.
Without more specific information, such as high versus low performing students, it is likely the issue lies within the distractors. This could possibly be a best answer question, and distractor D is too close to distinguish from A. Clearly distractor D is a big source of the problem. If the stem of the question is not inherently complete and the alternatives are overly long, this could create confusion and misinterpretation as to the expectations of the question (Popham, 2011). The other issue is that few students chose B or C. Those two distractors need to be reworked to be competitive with A and D (Suskie, 2009; Popham, 2011).
Question 6: The correct answer is C, but less than 10% of the participants selected that response.
Again, without more specific information as to the number of students who picked the other alternatives, high-performing, low-performing, etc., the most likely possibilities are (1) the material needs to be retaught because none of the students understood it well enough to make the discrimination; or, (2) the question is too difficult either because the distractors are too similar to one another or something about C is causing the students to purposefully not select it as the correct answer (Suskie, 2009; Popham, 2011). Lastly, this question in its current incarnation (and possibly in any incarnation) is not appropriately discriminating learning of any kind. Thus, it should probably be thrown out completely.
Question 9: 80% of the high-performing participants missed this question, while 75% of the lower-performing students got it correct.
This trend is clearly the result of negative discrimination (Oosterhof, Conrad, & Ely, 2008; Suskie, 2009; Popham, 2011). “Because they unfairly penalize the top students who read too much into them, they should generally thrown out and the tests rescored” (Suskie, 2009, p. 268). The best course of action is to throw out the question completely.
Oosterhof, A., Conrad, R. M., & Ely, D. P. (2008). Assessing learners online. Upper Saddle River, NJ: Pearson.
Popham, W. J. (2011). Classroom assessment: What teachers need to know (6th Ed.). Boston, MA: Pearson Education, Inc.
Suskie, L. (2009). Assessing student learning: A common sense guide (2nd Ed.). Retrieved from EBSCOhost