Empirical Studies of Software Engineering Expertise

« Progress Report for Week 4 | Main | Week 4 Progress »

May 1, 2005

Thinking aloud: Reconciling theory and practice by Boren & Ramey (2000)

Boren, M. T., & Ramey, J. (2000, September). Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication, 43(3), 261-278.

Online version:
http://ieeexplore.ieee.org/iel5/47/18778/00867942.pdf

This paper begins with a discussion of think-aloud protocols. They first describe the original theoretical framework put forth by Ericsson & Simon (1984/1993), and then bring it into the context of research in usability testing. Field observations of usability testing showed that researchers do not appear to adhere to standards with regard to think-aloud protocols (the description of the variability is pretty interesting). Methods vary widely, and some are carried out without any explanation or basis in theory (i.e., no background given or sources cited in the literature). This paper is particularly useful because it summarizes some of the key points from Ericsson & Simon’s theory, and then offers an alternative theoretical framework (speech communication). One thing for us to consider is whether the concepts that this alternative framework is based upon are more pertinent to the realm of usability testing than they would be for us (i.e., should we consider departing from the original think-aloud framework).

According to Ericsson & Simon's (1984/1993) model, there are three types of verbalizations:

1. Level 1 verbalizations are those that need not be transformed before being verbalized during task performance, e.g., verbalizing a sequence of numbers while solving a math problem.

2. Level 2 verbalizations are those that must be transformed before being verbalized during task performance, e.g., images or abstract concepts, as long as this transformation into words is the only mediating cognitive process between short-term memory and verbalization.

3. Level 3 verbalizations require additional cognitive processing beyond what is required for task performance or verbalization; e.g., filtering (verbalizing only info that is related to a specific topic), making inferences about one’s own cognition, and retrieving information from long-term memory at the researcher’s request.

--Level 3 verbalizations are not considered reliable data under Ericsson & Simon's theory.
--Any outside influence during a task (e.g., a comment or prompt from a researcher) can turn verbalizations into Level 3 data, by altering the normal flow of information into short-term memory during the task.
--Verbalization has been critiqued by Nisbett & Wilson (1977), who claim that people cannot accurately report on their own mental processes. Ericsson & Simon, however, argued that this critique is directed at specific types of verbalization (Level 3), and that Level 1 and 2 verbalizations would constitute reliable data.
--If Level 1 or 2 verbalizations are collected properly, they can reveal what information the subject heeded and in what order.
--In their model, verbalizations are not to be valued for their subjective content (this aspect is addressed in another article, "Reconceptualizing think aloud methodology..." [Yang, 2003]).
--Additional verbalizations like feelings and value judgments would not be considered data.

Additional key points from Ericsson & Simon (as described in the current paper):

1. Collect and analyze only "hard" verbal data
--No inference, introspection, or opinion; only what would be present in short-term memory during a task, i.e., what the subject attends to and in what order

2. Give detailed initial instructions for thinking aloud
--Make the distinction between thinking aloud and explaining
--Encourage the participant to speak constantly as if he were alone, and warn him in advance that he will be given reminders if he falls silent
--Practice thinking aloud before the actual task!

3. Remind participants to think aloud
--After a predetermined period of time (e.g., 15-60 seconds)
--Short and nondirective (e.g., "keep talking")—do not make the subject more aware of the researcher’s presence or create a sense of personal contact

4. Otherwise, do not intervene

An alternative theoretical framework: Speech communication

1. Assumes that all verbal processes are meant for the purpose of communication, and the idea that the participant would suspend his awareness of the listener (so he may think aloud as if he were alone in the room) is flawed.

2. Speakers expect that listeners will react to what they say.

3. If this is true, and a communication framework is inevitable, then the goal becomes to create a highly asymmetrical speaker/listener relationship (where the speaker does almost all of the talking), rather than trying to have the researcher become essentially invisible.

--Roles: The participant should be cast as an important contributor (i.e., expert, and the researcher as the interested learner and listener). This should be defined at the start and maintained throughout.
--Make observational technology as unobtrusive as possible (decrease awareness of being observed).
--Use "acknowledgment tokens" continuously (e.g., "mm hmm" or "uh huh," which are both conversation "continuers," but remain uncommitted and carry no connotation). These provide the response of an "engaged listener" but still allow the researcher to keep a low profile. Tokens should carry little or no content and should not redirect the subject’s attention (and should require little or no processing on the subject’s part).

Posted by Erin at May 1, 2005 8:55 PM

Comments

Post a comment