Think-Aloud
Home Up

A "think-aloud protocol" is a procedure for capturing part of the thinking that occurs when students take tests. In this sample, one question out of 50 was processed by a student who spoke aloud his reasoning for accepting or rejecting each of four answers as the "best." Here are the question and some notes about his reasoning.

A Sample Question

What kind of evidence would be most likely to make this article more convincing?

  1. Several more examples should be used to show how pervasive employee theft is.
  2. A chart should be included to show what percentages of employee thieves are affected by each of the causes listed for employee theft.
  3. An estimate of what percentages of employee thieves were prosecuted, only fired, or recycled should be included to show a need for more prosecution.
  4. More information about the California computer company scam and what happened to the staff who were caught should be included to show how dangerous employee theft is and to warn employees that they can be punished severely.

One Student’s Response

The Testing Situation—a Make-up Exercise: Five students missed the in-class test; their make-up exercise was to do a think-aloud protocol for 5 of the questions. The protocol would be recorded in my office. "John" selected item C because he thought that presenting such statistics in a chart would be a strong visual aid to the article. Since only B offered a chart, is John getting the question correct (the target answer is C) because he sees how forceful the statistics on consequences would be or because he talked himself into combining choices B and C to make an answer he perceived to be "best"?

John felt real concern about the issue of employee theft, hoped that more prosecution would result and not just firing, though he understood that businesses were in an awkward position if they have to prosecute too many employees. Having summarized the article section by section and then been asked to reflect on it, John realized that "companies aren’t doing enough to stop" employee theft, but he still feels shock from the examples and wishes more could "be done to stop it," or at least that prosecution might increase as a tactic to "even out a little more between firing and prosecution." Why would a teacher want to capture any of this moralizing and reflecting on the issue and how might it be done?

Revising a Test Item: The question above was shown by item-analysis to not be too easy, too hard, nor a weak discriminator. That is, the question wasn’t anomalous; but that doesn’t mean it shouldn’t be revised. On follow-up survey, 20% of the respondents picked this question as "particularly bad, dull, or unfair"—only 11th in the number of complaints out of 47 questions listed. But only 39% of the test takers who worked on that question got it right.

Basically, the question above asks students to choose among the impact of examples, a graph, statistics, and a case study. The question might be revised to ask instead which sort of information should be shown in a graph to make the article more persuasive—and offer answer choices that involve various kinds of number patterns.

 

Copyright3.gif (24311 bytes) 1999-2008+ by the Virginia Community College System. Prepared for the VCCS by Professor Eric Hibbison, 1998-2001 MRCTE Chair and Chief Chair of RCTE from  2000-2005. Permission is granted to use this content for professional development or other educational, nonprofit purposes.  Animations used on this site are either part of the Front Page theme or from a royalty free collection called "Web Clip Empire 250,000" ©1997, 1998 by Xoom, Inc., and its Licensors.  

Reminder for folks new to the Web: UNDERLINED WORDS (and some graphics images) ARE HOT LINKS. To preview them, hold your mouse on the hotlink (the arrow becomes a hand as you "mouseover" a link) and read the "URL" (Web address) in the "status line" (bottom) of your maximized Web browser. To visit, just click.