An international investigation into guidelines for observers in qualitative research

• This working paper focuses on the use of observers specifically in qualitative interviews and focus groups. 

• There are four main types of observers identified: subject area experts, notetakers and timekeepers, shadowers, and other survey stakeholders.

• The guidelines all work towards the highest possible quality in the research combined with respect for the research participant.

• Attention to detail is most important for notetakers.

The Office for National Statistics (ONS) qualitative leadership group identified a need to produce guidelines for the use of observers in qualitative research, as no guidance was available on this topic within the network. The authors of this working paper used their professional knowledge and experience in qualitative research to produce a draft set of guidelines and then contacted other national statistical institutes to explore international perspectives in this area.

This working paper is an equal collaboration between the Office for National Statistics (ONS, UK), Australian Bureau of Statistics (ABS), United States Census Bureau (USCB), Statistics Canada, and Statistics New Zealand (Stats NZ) to produce these joint guidelines.

This working paper has been produced as a guide for: 

• when and how observers should be used in the context of qualitative research, specifically relating to the observation of interviews and focus groups

• advantages of having an observer who acts as a notetaker

The Office for National Statistics (ONS) authors of this paper proposed a first draft of observer guidelines based on their expert knowledge of the subject matter and firsthand experience with the methods. International colleagues, who are qualitative research and question design experts, were then contacted via email, shown the draft, and invited to share their knowledge and any existing observer guidelines for use in this working paper. Discussions took place over one year, with drafts being shared, commented on and re-circulated until agreement for a final paper was agreed.

These guidelines refer specifically to two methods used in qualitative research where observers may be present.

Interviews

Interviews include:

• cognitive testing - an evidence-based method specifically designed to investigate whether survey questions, or other data collection tools, fulfil their intended purpose; for instance, investigating whether a social survey diary makes sense, can be answered, and is understood consistently across participants

• cogability testing - this has the same aim as cognitive testing with the addition of also testing the usability of the respondent-facing materials; for instance, testing whether an online survey instrument is easy to navigate and enter data into 

• in-depth interviews - these offer the opportunity to capture rich, descriptive data about how people think and behave; for instance, conducting research into how respondents conceptualise their national identity

Focus groups 

Focus groups include a small group of carefully selected participants who contribute to open discussions for research. For instance, a group of 10 people who identify as bisexual, plus a group moderator who facilitates discussion about how bisexual people conceptualise their identity. Or a group of pensions experts discussing new regulation requirements and whether they are willing and able to answer questions about them.

Observers

There are four main types of observers: 

• subject area experts (active observers)

• notetakers and timekeepers (active observers)

• shadowers (inactive observers)

• other survey stakeholders (inactive observers)

Subject area experts

During qualitative interviews and focus groups, an expert from the subject area may observe. For Business Survey research, this could be an expert with an in-depth knowledge of National Accounts. For Social Survey research, this could be someone from a special interest group or topic area expert, such as a survey manager, participating in the testing of questions within the survey they look after.

The aim of the observer here is to enable a more agile research experience for important stakeholder(s). As an observer, the subject area expert can observe the results as they arise and has a full understanding of the research experience. This allows experts to help with respondent questions requiring deep subject matter knowledge when asked, or when this knowledge would aid the interview.

For instance, when redeveloping the Office for National Statistics (ONS) Financial Services Surveys, which use many technical terms, it is helpful to include an observer who can interpret complex financial language, and apply it to the respondent's situation if needed. Any questions the subject area expert has for the respondent can also be answered immediately at the end of the interview. Where subject area experts are more removed from the process, they may not be as readily available to help with technical queries.

Notetakers and timekeepers

During qualitative interviews, an observer can be present to take notes as the interview progresses. This can help with speed of analysis, if quick high-level findings are needed. Also, if someone has refused to be recorded, or recording is not appropriate, notetaking ensures there will still be a summary of the interview.

Even when interviews are recorded, the notetaker plays a crucial role in contributing to the accuracy of transcripts and conclusions, as they are much better able to observe and interpret body language than cameras. With focus groups, this individual can also have a timekeeping role, and can summarise the discussion at the end of the meeting, where feasible. This aids the main moderator, helps the group to run smoothly, and introduces an extra level of quality assurance to the process, to check that the respondent's views have been interpreted correctly.

The person who fulfils the role of notetaker and timekeeper can also act as someone who enhances the safety of the main interviewer or moderator(s) if they do not feel comfortable attending an interview or focus group alone, or without support from colleagues. 

Shadowing

"Shadowing" describes an observer being present at an interview, purely as a learning experience to improve their research skills, and remaining silent throughout the interview or focus group. For instance, a person new to the research team who is observing a qualitative interview.

This person can also act as someone who enhances the safety of the main interviewer if they do not feel comfortable attending an interview alone.

Other survey stakeholders

Other survey stakeholders, who may or may not be subject area experts, observe passively and do not contribute to the interview. Watching the research unfold live is a valuable way to increase stakeholder engagement with, and support for, the test findings, especially if a question or element of a survey is performing poorly.

Mode of interview

Carrying out qualitative interviewing online has increased in frequency since the coronavirus (COVID-19) pandemic. While these interviews proceed in a similar way to face-to-face interviews, the change in context has some impact for the observer.

For instance, they could be either on or off camera. It is also important that observers are introduced overtly and that they state their role, as their presence may not be obvious online.

This section sets out seven important guidelines for using active and inactive observers and the main reasons for their inclusion.

The term interviewer can be used interchangeably with moderator, to also be relevant for focus groups. 

1. Consent should be sought from the participant(s) to allow an observer to be present. The timing of this depends on context.

This shows respect for the participant, allows them to have more control over the interviewing environment and mitigates the risk of someone cancelling an interview because they feel uncomfortable with too many people present. 

2. Interviewers and observers present at a qualitative interview or focus group should not exceed a maximum of three people. If it is a face-to-face group; the same maximum number should be allowed, but the interview space should be checked to ensure that there is enough room. 

This reduces the risk of the participant feeling self-conscious or uncomfortable, especially if the interview concerns a personal or emotive topic, such as finance or crime. 

3. The person(s) observing should be introduced at the start of the interview or focus group, their role explained, and a verbal agreement sought that this is still acceptable.

This helps the participant(s) feel informed, in control, and at ease. This will aid the flow of the interview, increase rapport between the interviewer and participant, and improve the quality of the data received by the interviewer. 

4. Subject area experts should only intervene when prompted by the lead interviewer.

This ensures the participant is not influenced by the presence of the expert and that the trained professional qualitative researcher is in control of the process. This protects the quality of the interview.

5. Notetakers should not intervene in the process unless asked to summarise at the end.

This ensures the participant is not influenced by the presence of the notetaker and the trained professional qualitative researcher is in control of the process. This protects the quality of the interview.

6. The observer must adhere to the same criteria as the main interviewers concerning confidentiality and data protection.   

There are strict confidentiality and data protection laws that exist to protect the participant from any harm coming to them by participating in the research. It is very important that observers also adhere to these laws to ensure the interviewing experience is safe and properly conducted.  

7. If it is not feasible or advisable for a stakeholder or subject area expert to observe directly, the interview can be recorded for later viewing by them. This is subject to following the correct ethical and data protection approval processes, including on data sovereignty. It is also subject to approval being attained to allow the recordings to be shared. [note 1] 

Sometimes it is difficult to schedule an interview for the time that all participants in the process can make. Recordings can be very useful in these circumstances, providing that:

• the participant has agreed

• the correct confidentiality and data protection laws are followed

• the recording device does not alter the respondent's answers

• the recording device does not make the respondent feel uncomfortable

These could all adversely affect the quality of the data from the interview.  

Footnotes for Section 6, Guideline 7: Data sovereignty

1. Part of Stats NZ's Privacy Impact Assessment process is considering data sovereignty concerns related to indigenous data and data about Māori. Stats NZ's preference would be for information/ data about Māori groups and individuals to be stored onshore. This is so that it is not subject to overseas jurisdiction that may be contrary to the interests of Māori. Where possible, they aim to use notetaking methods that enable secure storage of information in Aotearoa/ New Zealand. Any recordings made have planned storage and access processes to minimise the risk of data security and privacy breaches and are deleted as soon as they are not required (such as if sufficient notes have been taken from them).

Advantages of having an observer who acts as a notetaker  

An observer can note participant behaviours that will be useful in informing the analysis and conclusions drawn from the qualitative interview or focus group. For instance, any hesitation or uncertain facial expression from the participant when presented with a particular topic or survey question. They can also begin the process of identifying emerging themes pertinent to the topic being explored.  

The notetaker can observe in detail the way a survey or interview performs and take notes of every problem that is encountered. Behaviours can inform conclusions about a respondent's understanding of an electronic or paper business surveys form, or a social or census survey, or one particular question, as if it were in the field and conducted by field interviewers.   

Guidelines for notetaking when conducting qualitative interviews for questionnaire design (self-completion) 

This section further emphasises the advantages of having a notetaker present by highlighting fourteen areas that can be focused on when an active observer is present. This information may be missed if only an audio recording is used and the main interviewer is busy ensuring the interview is proceeding smoothly.  

Considerations for notetakers  

• Does the respondent look at the entire form contents first, or do they start at the beginning and work through to the end?  

• How much of the notes and instructions does the respondent appear to read? Are they only reading the notes for clarification when they have a problem understanding the question?  

• Does the respondent re-read or refer back to earlier instructions, questions or notes? This may mean the person has doubts about the answer to a previous question or that the present question is misunderstood.  

• Does the respondent make any corrections or changes to their answers?  

• Which parts of the form does the respondent spend more time on, and which do they skim over?    

• How long does it take the respondent to answer a particular question or section?  

• What sequence does the respondent complete the questions in? Do they miss any questions or instructions?  

• Does the respondent misread any questions or instructions? Make a note of anything that is read aloud differently to what is written down and note any resulting behaviour.  

• Does the respondent ask any questions about content? For example, "I'm wondering what this means".  Make a note if the person makes a comment and come back to it later.   

• Does the respondent appear to scan around the page without answering? Try to take note of where they are looking. This usually indicates that the person is uncertain and is looking for clues to meaning.  

• Does the respondent consult any records? At what stage do they do this? 

• Note any emotional reactions - do they seem uncomfortable, annoyed, or particularly happy to see certain questions? 

Considerations for notetakers observing sessions including a posted-out survey form 

• Observe the way the respondent physically handles the package (envelope, covering letter and form).   

• Does the respondent refer to any of the accompanying material in the package? At what stage do they do this?   

Good practice for interviews and focus groups

These guidelines set out good practice standards for the use of observers in interviews and focus groups. This section provides more detail on the practicalities of carrying out interviews and focus groups for different types of activity, when observers are present.

This section uses the following abbreviations:

• cognitive interviewing (Cog)

• face-to-face interviews (F2F)

• focus groups (FG)

• online interviews (OL)

For interviews and focus groups, observers should:

• use a private location where others without appropriate clearance cannot see or hear the discussion (OL) 

• be introduced (on camera if OL) and the reason for their presence should be explained (Cog, F2F, FG, OL) 

• have proper identification (Cog, F2F, FG, OL) 

• be present if it enhances the safety of the interviewer (Cog, F2F, FG)  

• take note of respondents' behaviours (Cog, F2F, OL) 

• remain present throughout the interview to the end (Cog, F2F, FG, OL)  

• respect confidentiality and comply with all data protection regulations (Cog, F2F, FG, OL) 

• be ready to join the online meeting 5 to 10 minutes before the scheduled start (OL) 

• keep their microphones muted and cameras deactivated (except for initial introductions) (OL)    

• use the chat function, during the interview, to ask the moderator any questions for the participant, making sure that they are chatting directly with the moderator by using a separate online chat window, rather than the main meeting chat window (OL)

• be aware that not every minute of the interview will be meaningful; some discussions are transitions from one topic to another (Cog, F2F, FG, OL)  

• be aware that participants will stray from the topic of conversation from time to time (Cog, F2F, FG, OL) 

• listen for more than a confirmation or validation of their own viewpoint; they should listen to what participants are saying (and not saying) (Cog, F2F, FG, OL) 

• listen to misinformation and find in it an insight into the thinking of respondents (Cog, F2F, FG, OL)   

• observe what is, and what is not, being said (Cog, F2F, FG, OL)

• be alert to the nuances of meaning and the language participants use to present their perceptions, opinions, beliefs and attitudes (Cog, F2F, FG, OL)

• ask observers to use unconditional positive regard (UPR) [note1]

• remain as neutral as possible if the respondent directs comments to them, and direct any queries back to the interviewer (Cog, F2F, FG, OL)

For interviews and focus groups, organisers should:

• consider including subject matter experts for complex surveys (Cog, F2F, FG, OL)  

• train subject matter experts to be neutral and not leading in any responses to questions directed at them (Cog, F2F, FG, OL) 

• co-ordinate with others to ensure there are no more than two observers per interview (Cog, F2F, FG, OL) 

For interviews and focus groups, observers should not:

• be included if it is not necessary, for example, if the interview is being recorded and there is adequate time for analysis (Cog, F2F, FG, OL)  

• interrupt the interview process, unless asked to do so on a matter of clarity or explanation of a term, for example, if they are a subject matter expert (Cog, F2F, FG, OL)  

• divert attention away from the interviewer (Cog, F2F, FG, OL)

• identify individual participants or business names in their notes; instead, they should use identifiers like "Participant 1" or "Interview 1" and treat all information, notes and observations as strictly private (Cog, F2F, FG, OL)

• arrive late or leave before the end of the interview (Cog, F2F, FG, OL)  

• send a chat message tagging everyone in the group (OL)  

• observe a meeting for which they have not signed up (Cog, F2F, FG, OL) 

• expect each participant to be asked every probing question (Cog, F2F, FG, OL) 

• judge participants based on their looks, clothing, age, education, income, level of articulateness, sexual preference, and so on; remember that each participant represents an actual respondent to the survey, and everyone has something valuable to say (Cog, F2F, FG, OL)

• intrude on the observation and notetaking process of others by chatting, laughing at (not with) participants (Cog, F2F, FG, OL)

• demeaning the comments of participants because of speech, demeanour, dress, lifestyle, income, race, age or degree of education (Cog, F2F, FG, OL)  

• defend their organisation; this may stop the respondent answering further and with candour (Cog, F2F, FG, OL)

Footnotes for Section 6, Good practice for interviews and focus groups

1. This section includes links to third-party sources for illustrative purposes only. Linking to third-party sources is not an endorsement of their content.