Cognitive testing
The purpose of cognitive testing is to explore, understand and explain the range and diversity of ways in which people go about answering survey questions. This allows the researcher to ascertain whether or not a question is working as intended, and whether the information that respondents need to enable them to answer accurately is obtainable. Cognitive testing can also aid the development of questionnaires by suggesting improved and unambiguous question wording, layout and routing.
Cognitive testing generally takes place as a one-to-one interview, although it is possible to use a focus group situation. The interviewer probes the respondent to elicit what they understand a question to mean, and how they came to the answer they gave. Techniques used include concurrent and retrospective probing. Probing is informed by a cognitive model of the mental processes involved in responding, which comes in five parts (Tourangeau (1984) and Eisenhower et al. (1991)):
- encoding in memory (respondents have to have some knowledge or memory of what is being asked about)
- comprehension (respondents understand the question and relevant concepts)
- retrieval (respondents retrieve the information from memory)
- judgement (respondents assess the completeness and relevance of what they remember)
- communication/response (respondents decide whether their answer fits the answer categories provided and also decide whether they actually want to provide an answer or provide one that might be socially acceptable)
Cognitive question testing can be combined with in-depth interviewing.
Data Collection Methodology projects which have used cognitive interviews:
The reports/papers resulting from the following projects are available to download from this page.
Cognitive testing was used in the development of a module of questions on Public Confidence in Official Statistics, subsequently asked on the National Statistics Omnibus Survey in July 2004 and March 2005.
A study was conducted on behalf of the Socio-Economic Inequalities Branch (SEIB) of the Office for National Statistics (ONS) in 2004 as part of a wider research project investigating the provision of social capital statistics at the local area level. It involved the design and development of a questionnaire to enable the social capital harmonised question set to be converted from interviewer-administered to self-administered format, changing the mode of data collection from face-to-face to postal. Cognitive interviews were conducted to pre-test the prototype self-administered questionnaire.
Cognitive testing techniques were used alongside in-depth interviewing to explore responses to key Labour Force Survey questions on economic inactivity. The range of perceptions and attitudes about this subject were looked at in order to help distinguish different states of inactivity and propensities to start or return to paid work. The resulting paper won the Social Research Association's Mark Abrams Prize in 2004.
Cognitive testing was used to compare two approaches to collecting information on personal assets - savings and investments - from pensioners. The first approach was that used by the Family Resources Survey. The second approach was that used on a Survey of Entitled Non-Recipients (ENR) of the Minimum Income Guarantee and of possible future pension credit recipients. A key aim of this study was identifying ways in which the collection of such information could be improved. This was a joint project with the National Centre for Social Research, conducted for the Department for Work and Pensions (DWP).
References:
Eisenhower D, Mathiowetz N A and Morganstein D (1991) 'Recall Error: Sources and Bias Reduction Techniques', in Biemer P et al (ed) Measurement Error in Surveys, Wiley and Sons: New York.
Tourangueau R (1984) 'Cognitive Science and Survey Methods' in Jabine T, Straf M, Tanur J, and Tourangeau R (eds) Cognitive Aspects of Survey Methodology: Building a Bridge between Disciplines, National Academy Press: Washington DC.
