Coding Cognitive Interviews: An Approach to Enhancing the Value of Cognitive Testing for Survey Question Evaluation

Cognitive testing has become routine for well-designed surveys. However, the protocols for cognitive testing vary widely, and observers have been concerned that analysis of the results is not systematic, that results are not replicable, and that the bases for conclusions are not transparent. To address some of those concerns, in this study the cognitive interviews were structured so that interviewers gathered information specifically about whether there was a problem with any of the four standard issues in question answering: comprehension, retrieval, estimation, and providing an answer. From the tape-recorded interviews, the percentage of respondents who had “no problem” with each element of question answering was tabulated for each question. In addition to being systematic and transparent, a strength of the approach is that it documents the positive features of questions, potentially providing evidence for the validity of answers as well as identifying possible question problems.