Tuesday, April 8, 2008

Multiple-choice Questions on Assessments

This is the first in a series of blogs I will write about multiple-choice questions on assessments. I'm using it to set the groundwork and define a few terms I will use in future posts.

Many companies I have worked with standardize on a four-option multiple-choice assessments. This format has a number of advantages:

  • Delivery and grading are easy to automate.
  • Grading is relatively quick even when done manually.
  • Learners are comfortable with the format.

Multiple-choice assessments also have a few drawbacks:

  • Questions often give away the answers to other questions.
  • Results can depend on the interpretation of an ambiguous phrase.
  • Learners can use clues in questions to guess correctly.
  • Questions are limited to the cognitive domain—what learners know, rather than what they can do.

Multiple-choice questions consist of three parts:

  • The stem
  • At least one correct answer
  • Distracters

The stem is the question itself. Stems can take several forms:

  • Actual questions — “What is the maximum number of DIMMs that can be installed on the …”
  • True/False statements — “True or False: The official language of Panamá is English.”
  • Fill in the black statements — “Fill in the blank: You can have _______ servers in a 42U rack filled with PowerEdge™ M1000e enclosures.”

Every question should have at least one correct answer. Learner frustration and arguments can arise over which one is the correct answer. For this reason, I recommend some sort of an appeals process, at least until you have enough data to determine which questions have an acceptable performance curve.

Many Learning Management Systems (LMSs) allow for multiple correct answers. I my experience, LMSs seldom if ever give partial credit for partially correct answers (correctly choosing three out of the four correct answers). This all-or-none aspect makes questions with multiple correct answers more difficult that questions with only one correct answer. Use them with discretion or you may develop an assessment that very few can pass.

Distracters are incorrect options. They should all be plausible. That is, they should all seem reasonable to someone who does not know the correct answer. Or as Randall (2003) put it, a distracter “compellingly and confusingly attracts in the wrong direction.” Randall also distinguishes foils from distracters as something that sets off “another thing to advantage or disadvantage by contrasting with it.”

So why is it that so much dispute arises over multiple choice questions? I’ll focus on distracters soon.

References

Thursday, April 3, 2008

When Should Assessments Be Open-book?

One of my clients asked me today if there was a sound rationale in instructional design for open-book assessments. As with most instructional design questions, the answer one of my father’s coworkers gave to any question applies. “It all depends upon.” In this case, it depends on what you are trying to assess. Open-book and closed-book assessments both have their uses.

Even though we are using terms in common use (maybe because of that), a few definitions are in order for the sake of clarity:

  • An assessment measures learner performance against a set of learning objectives.
  • An open-book assessment allows learners to use materials such as textbooks, notes, or the Internet to answer the questions on the assessment.
  • Closed-book assessments allow learners no resources beyond their own memories.

Closed-book assessments are a great way to assess lower-level objectives such as knowledge or memorization of facts. Jenkins (1995) noted that learners can gain high marks on closed-book exams simply by memorizing lecture notes because of the focus on facts rather than application or synthesis. At the same time, learners with a better grasp of the principals and applications of the material may score lower on traditional assessments because they have not memorized the details.

Open-book assessments can eliminate the need to test for low-level knowledge (Jenkins, 1995). Learners do not need to memorize what they can look up. In fact, open-book assessments can be especially useful in a training environment where the requisite skill is finding facts and applying them to a given situation. One company for which I designed training gave learners three weeks to become familiar with their technical library. That library encompassed more than 10 million pages, and learner needed all of it. Obviously, three weeks training did not provide sufficient time for learners to read, much less assimilate, 10 million pages of information. They needed to learn how to find specific pieces of that information quickly, and what better way to determine if they could than with an open-book assessment? In fact, because the technical data and policies taught were subject to change, memorization was discouraged (Bruns, 2004). As one learner in a different study said, “Memorization is a thing of the past.… Even the best geographer in the world is no match for the CIA World Factbook online” (Johnson, 2006).

This example points out another benefit of open-book assessments. They can closely resemble the work environment in that learners use resources to solve problems, prepare reports, or communicate information to co-workers (Davis, 1993; Jenkins, 1995). In business and life, learners look for help wherever they find it. Ronald Jones, founder and former CEO of SongPro (a company that made music modules for game systems) called life an open-book test. (Thomas, 2003). Others have also made the same observation.

But if quick data retrieval is not one of the learning objectives, do learners perform better with open-book assessments? I was not able to find a study that compared the results of different testing methods in the business world, but several studies in education show mixed results.

  • Early research showed that learners performed about the same on closed- and open-book assessments (Verduin, 1960; Clift and Imrie, 1981; Crooks, 1988).
  • Kumar (1999) attributed better performance on an online assessment over a paper assessment in a computer science class to the availability of a compiler.
  • Eilertsen and Valdermo (2000) showed that open-book assessment strengthened understanding in both traditional and cooperative learning classrooms in Norway. One learner quoted in that study said, "I prepared for the test as I’ve always done. But because I knew books were available, I did not rush myself, and I noticed that I remembered better."
  • Agarwal et al (2007) found that learners given open-book assessments did better initially but fared no better than other learners on a closed-book assessment given two weeks later. Unfortunately, the questions provided in the study seemed designed to test factual recall rather than higher-level objectives.

But while neither open-book nor closed-book assessments appear to be better than the other in all situations, one variation of an open-book assessment may offer consistent advantages. Wachsman (2002) determined that learners who prepared cheat sheets for an assessment did better on assessments than those who did not, even when, just before the assessment, they were forbidden to use the cheat sheets when taking the assessment. Wachsman says this supports coding hypothesis—the idea that because of limited space, learners had to code the information to fit the space. That is, they had to prioritize the information by what they thought would be on the assessment or by their own perceived areas of weakness.

References

Tuesday, April 1, 2008

Why Punctuation Is Important*

Opinion

I have seen a disturbing tend in writing lately: ignoring or omitting punctuation. I think it started with newspapers dropping the comma before the conjunction in lists, but modernist writers have taken it to extreme. Some, like ee cummings, even do away with case. Lyrics in album librettos are seldom punctuated beyond line breaks. We seem to have forgotten how important those lovely little unspoken marks are in communication.

As I was waiting for a traffic light to turn, I noticed the truck beside me. Printed just beneath the store logo, the back panel proclaimed:

THIS TRUCK IS BEING DRIVEN UNSAFELY CALL (512)###-####

The truck was not being driven unsafely; it was motionless—waiting for the light to change, just as I was. But the first time I parsed the sign, I saw two distinct sentences, “This truck is being driven unsafely. Call…”

Since there was no punctuation to guide me, the lack of case affected how I interpreted the symbols, making me interpret the message in the emphatic. On reflection, I’m also sure that the furniture company meant to say, “If this truck is being driven unsafely, call…” Apparently, someone decided it was too expensive to pay for punctuation and that one little word.

So I want to call and let them know that their truck is telling everyone it is being driven unsafely and asking for help. I won’t, but I want to…

*Originally posted to my personal blog, I copied it here because it seemed relevant.