Multiple-choice Questions on Assessments

This is the first in a series of blogs I will write about multiple-choice questions on assessments. I'm using it to set the groundwork and define a few terms I will use in future posts.

Many companies I have worked with standardize on a four-option multiple-choice assessments. This format has a number of advantages:

• Delivery and grading are easy to automate.
• Grading is relatively quick even when done manually.
• Learners are comfortable with the format.

Multiple-choice assessments also have a few drawbacks:

• Questions often give away the answers to other questions.
• Results can depend on the interpretation of an ambiguous phrase.
• Learners can use clues in questions to guess correctly.
• Questions are limited to the cognitive domain—what learners know, rather than what they can do.

Multiple-choice questions consist of three parts:

• The stem
• At least one correct answer
• Distracters

The stem is the question itself. Stems can take several forms:

• Actual questions — “What is the maximum number of DIMMs that can be installed on the …”
• True/False statements — “True or False: The official language of Panamá is English.”
• Fill in the black statements — “Fill in the blank: You can have _______ servers in a 42U rack filled with PowerEdge™ M1000e enclosures.”

Every question should have at least one correct answer. Learner frustration and arguments can arise over which one is the correct answer. For this reason, I recommend some sort of an appeals process, at least until you have enough data to determine which questions have an acceptable performance curve.

Many Learning Management Systems (LMSs) allow for multiple correct answers. I my experience, LMSs seldom if ever give partial credit for partially correct answers (correctly choosing three out of the four correct answers). This all-or-none aspect makes questions with multiple correct answers more difficult that questions with only one correct answer. Use them with discretion or you may develop an assessment that very few can pass.

Distracters are incorrect options. They should all be plausible. That is, they should all seem reasonable to someone who does not know the correct answer. Or as Randall (2003) put it, a distracter “compellingly and confusingly attracts in the wrong direction.” Randall also distinguishes foils from distracters as something that sets off “another thing to advantage or disadvantage by contrasting with it.”

So why is it that so much dispute arises over multiple choice questions? I’ll focus on distracters soon.

References

• Randall, V. (2003, May 17). Recognizing distracters and foils. (University of Dayton School of Law) Retrieved May 15, 2008, from Passing the Bar.
• University of Minnesota, Office of Measurement Services. (2008, January 22). Writing multiple-choice items. Retrieved May 15, 2008, from Course Exams.
• University of Wisconsin Teaching Academy. (n.d.). Multiple Choice. Retrieved May 15, 2008, from Exam question types and student competencies: how to measure learning accurately.