Think-aloud interviews are widely used in usability testing, education, and related fields to see how people approach tasks. In education research, they can be used to learn how students think about tasks, elicit common misconceptions, validate possible assessment questions, and assess whether students develop “expert” thinking skills.
Modesty forbids me from mentioning our paper on think-aloud interviews of introductory statistics students.
See also Student assessment, Cognitive task analysis.
Ericsson, K. A., & Simon, H. A. (1998). How to study thinking in everyday life: Contrasting think-aloud protocols with descriptions and explanations of thinking. Mind, Culture, and Activity, 5(3), 178–186. doi:10.1207/s15327884mca0503_3
Explains the cognitive theory of think-aloud interviews, defending the claim that “the course of the thought process can be inferred in considerable detail from thinking-aloud protocols.”
Bowen, C. W. (1994). Think-aloud methods in chemistry education: Understanding student thinking. Journal of Chemical Education, 71(3), 184. doi:10.1021/ed071p184
Considers a study which tests two different instructional methods, and interviews students to see how the instructional methods affected how they solve problems. Includes a lab-based item where students actually use chemistry lab equipment to solve a problem.
Suggests randomly assigning students to interviewers and making sure the interviewers are blind to student attributes (e.g. if SAT scores were collected as a baseline covariate, the interviewer should not know the scores). Then:
Greet the subject in a pleasant way so they are not intimidated and relax a bit.
Describe the purpose of the study and emphasize that correctness isn’t the point – “we are not as interested in the answer you come up with as we are with how you are thinking about the tasks”. But don’t give away specific research questions.
Give a simple practice task which is fairly easy so the student gets practice thinking aloud.
Give the student the question on paper, ask them to read it aloud, and proceed. Allow the student to skip questions they are stumped by, instead of lingering forever.
Occasionally prompt the students to keep thinking aloud, or ask clarifying questions.
Interviews are recorded and fully transcribed, then coded. (In this case, they were interested in which way the students mentally represented the problems, so the transcripts were coded for representations.) Suggests presenting example excerpts to summarize the research, or even making composite transcripts by pasting together several interviews to make a “stereotype” transcript illustrating a common behavior or problem.
Adams, W. K., & Wieman, C. E. (2011). Development and validation of instruments to measure learning of expert-like thinking. International Journal of Science Education, 33(9), 1289–1312. doi:10.1080/09500693.2010.512369
Considers two kinds of interviews in a project to assess how student thinking differs from expert thinking:
Less-structured interviews to “understand student thinking about these topics and where and how it deviates from expert thinking”. These can start with observations of course help sessions, followed by selected think-aloud interviews; some probing of students during the interview is okay, to help find specific misconceptions and narrow down problems. Avoid questions that direct the students, but questions like “what does this term mean to you?” are still useful.
Validation interviews for proposed assessment questions, to ensure students understand them as intended, that they arrive at the correct answer for the right reasons, and that distracter answers catch real misconceptions. These interviews have less probing, to avoid making the students think in ways they wouldn’t ordinarily think while doing problems on their own; at the end of the entire test, the interviewer can go back to ask questions, being aware that the answers “do not represent the student thinking while taking the test.”
Some tips for both kinds of interviews:
Interviews should be recorded.
Interviewers should take half an hour after interviews to summarize the interview, making notes about the student’s responses and misconceptions.
Short icebreaker questions and practice exercises ensure that the student will be comfortable thinking aloud on the actual questions.
Interview as wide a range of students as possible.
Someren, M. W. van, Yvonne F Barnard, & Sandberg, J. A. C. (1994). The think aloud method: A practical guide to modelling cognitive processes. London: Academic Press. http://hdl.handle.net/11245/1.103289
An entire book on think-alouds (PDF available at link above). Chapter 4 focuses on “Practical procedures in obtaining think aloud protocols”. Some practical tips:
Emphasize the confidentiality of the data and make the student at ease.
Clearly state “Try to say everything that goes through your mind” – not just “tell me what you think”, which sounds like a solicitation for an opinion, not a stream of consciousness.
Give “a few minutes to a quarter of an hour” of warmup, with “a task which is not too different from the target task” but is perhaps easier. Don’t start with the real questions until the subject is good at thinking aloud; end the interview early if they prove not to get the idea.
Do not prompt the student with anything but “Keep on talking”. Do not correct the student.
Record the session. Make a complete transcription, including interruptions, pauses, and off-topic remarks.
Review the transcript with the student as soon as it’s available, to get more information about their intent and ideas (being aware these thoughts are retrospective).
Further discussion in later chapters focuses on how to analyze the transcripts effectively.