Thursday, April 3, 2008

When Should Assessments Be Open-book?

One of my clients asked me today if there was a sound rationale in instructional design for open-book assessments. As with most instructional design questions, the answer one of my father’s coworkers gave to any question applies. “It all depends upon.” In this case, it depends on what you are trying to assess. Open-book and closed-book assessments both have their uses.

Even though we are using terms in common use (maybe because of that), a few definitions are in order for the sake of clarity:

  • An assessment measures learner performance against a set of learning objectives.
  • An open-book assessment allows learners to use materials such as textbooks, notes, or the Internet to answer the questions on the assessment.
  • Closed-book assessments allow learners no resources beyond their own memories.

Closed-book assessments are a great way to assess lower-level objectives such as knowledge or memorization of facts. Jenkins (1995) noted that learners can gain high marks on closed-book exams simply by memorizing lecture notes because of the focus on facts rather than application or synthesis. At the same time, learners with a better grasp of the principals and applications of the material may score lower on traditional assessments because they have not memorized the details.

Open-book assessments can eliminate the need to test for low-level knowledge (Jenkins, 1995). Learners do not need to memorize what they can look up. In fact, open-book assessments can be especially useful in a training environment where the requisite skill is finding facts and applying them to a given situation. One company for which I designed training gave learners three weeks to become familiar with their technical library. That library encompassed more than 10 million pages, and learner needed all of it. Obviously, three weeks training did not provide sufficient time for learners to read, much less assimilate, 10 million pages of information. They needed to learn how to find specific pieces of that information quickly, and what better way to determine if they could than with an open-book assessment? In fact, because the technical data and policies taught were subject to change, memorization was discouraged (Bruns, 2004). As one learner in a different study said, “Memorization is a thing of the past.… Even the best geographer in the world is no match for the CIA World Factbook online” (Johnson, 2006).

This example points out another benefit of open-book assessments. They can closely resemble the work environment in that learners use resources to solve problems, prepare reports, or communicate information to co-workers (Davis, 1993; Jenkins, 1995). In business and life, learners look for help wherever they find it. Ronald Jones, founder and former CEO of SongPro (a company that made music modules for game systems) called life an open-book test. (Thomas, 2003). Others have also made the same observation.

But if quick data retrieval is not one of the learning objectives, do learners perform better with open-book assessments? I was not able to find a study that compared the results of different testing methods in the business world, but several studies in education show mixed results.

  • Early research showed that learners performed about the same on closed- and open-book assessments (Verduin, 1960; Clift and Imrie, 1981; Crooks, 1988).
  • Kumar (1999) attributed better performance on an online assessment over a paper assessment in a computer science class to the availability of a compiler.
  • Eilertsen and Valdermo (2000) showed that open-book assessment strengthened understanding in both traditional and cooperative learning classrooms in Norway. One learner quoted in that study said, "I prepared for the test as I’ve always done. But because I knew books were available, I did not rush myself, and I noticed that I remembered better."
  • Agarwal et al (2007) found that learners given open-book assessments did better initially but fared no better than other learners on a closed-book assessment given two weeks later. Unfortunately, the questions provided in the study seemed designed to test factual recall rather than higher-level objectives.

But while neither open-book nor closed-book assessments appear to be better than the other in all situations, one variation of an open-book assessment may offer consistent advantages. Wachsman (2002) determined that learners who prepared cheat sheets for an assessment did better on assessments than those who did not, even when, just before the assessment, they were forbidden to use the cheat sheets when taking the assessment. Wachsman says this supports coding hypothesis—the idea that because of limited space, learners had to code the information to fit the space. That is, they had to prioritize the information by what they thought would be on the assessment or by their own perceived areas of weakness.

References

Posted by Lee at 16:03
Labels: , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)