|
What are some of the ways that students process test items as they work through a
test?
Here are some observations made by collecting students' test and answer papers,
surveys about the test given at the end of the testing session, and during 5
"talk-aloud protocols."
Observed Ways That Students Process
Multiple-Choice Questions
about a Magazine Article
 The following observations were based on observations made from 5 think-aloud
protocols, 7 written protocols, markings on nearly 300 answer sheets for a test on a
magazine article called "Stealing from the Boss," and responses on
surveys about the two 50-item multiple-choice tests used in the study.* These observations
represent an attempt to figure out what do students really do and think during a
challenging multiple-choice test.
ACTIONS
Physical Actions
 | Students made physical markings on the test paper, e.g. a dash, circle, dot, checkmark,
or star to mark one or more choices as probable. |
| Crossing out letters of eliminated choices |
Procedures
| Stipulate the extent of a search for an answer in the "geography" of the
article, e.g. "The answer ought to be in this paragraph" or "up in this
corner" or "near the bottom of this column." (A previous study had
documented that readers create a sort of map or index of things they read as they go.) |
| Prioritize the importance of words in the question stem or in the answer choices |
Mental Processing
| Assemble parts of an expected answer in order to evaluate the given answer choices |
| Set traits for the expected answer based on the crucial words in the question |
| Hold a choice for further processing in order to understand it better before accepting
or rejecting it |
| Reconsider close alternatives or the importance of some word, especially after noticing
a word match between a word in an answer choice and a word in the article. (At least one
previous study established "word matching" as one of the most significant
test-taking strategies of students, particularly weaker students who used rote
memorization of isolated terms and definitions to study for the test.) |
| Eliminating answer alternatives because they are redundant (duplicate another answer in
some way) or because the answer choice is recognized as inconsistent with some information
in the article. |
Select the best answer.
AFFECTIVE REACTIONS
Positive
| Praising the question as good, challenging, or fair, or for asking an opinion, being
extensive or specific or calling for thought |
| Moralizing over the good or ill of the action described in the question |
Negative
| Complaining over confusion about dull, bad, unfair, or odd items, or about the
subjectivity, density, redundancy, ambiguity, or quantity of the questions |
| Expressing dislike due to "boredom," the topic, being asked to speculate,
having to scan the article for particular paragraphs or sections referenced in a question,
level of difficulty, or being asked opinions
|
COGNITIVE REACTIONS
With Questions
| Confirm givens in a question stem by word or idea match, by memory, or by looking back
at the article. |
With Answer Choices
| Interpret an answer choice by providing an example to clarify the choice or by giving a
word to clarify a vague word |
| Estimate the relevance of the answer choice, the overlap of the purpose between a quoted
speaker and the writer of the article, the overlap of choices, the writer’s motives,
the likelihood of a choice being the best or "target" answer by its centrality
(relationship to the main idea of the article), its position or the amount of space it
took up, or by clues in the wording of the answer choice or the way it was written |
| Speculate or interpolate missing data to assess an answer choice |
| Consider the validity of an answer separate from its truth value (that is, even a false
statement could be the desired answer if it makes a valid observation about life, theft,
how a business operates) |
| Contrast the answer choice with relevant cases from personal experience |
| Justify a selected answer by its magnitude (e.g., "The answer I chose is right
because it deals with an important issue in the article" or maybe even the
"test-wiseness" principle that the longest answer is the best answer) |
With the Article
| Scan the article to locate a passage |
| Connect a topic idea with data in the article that supports the topic |
| Assess the image of the intended reader or the meaning of a phrase in its context |
| Extrapolate or predict something based on the article’s data (e.g., "That
probably means the guy caught at my job taking stuff won’t get prosecuted but just
fired.") |
Counter-Productive Cognitive Reactions
| Assume something on the basis of a prior opinion that the estimated statistics in the
article are unconvincing |
| Reshape a distractor to fit the main criterion for the student’s expected answer |
| Advocate each answer choice instead of seeking to eliminate most of them |
______________________________________________
* Hibbison, Eric. "Prompting Inferences: Two Ways to Write Multiple-Choice
Comprehension Questions for a Complete Article." Dissertation. Indiana University of
Pennsylvania, 1990.
|
1999