Impact of COVID-19 on ONS social survey data collection

Introduction to operational social survey changes

The Office for National Statistics is responsible for many of the largest surveys of households and individuals in the UK. Social surveys are now more important than ever, providing timely information on the coronavirus (COVID-19) pandemic's impact on society. The coronavirus pandemic has also meant social surveys have required widespread and rapidly-implemented operational changes.

This article discusses operational changes caused by the coronavirus pandemic, and the impact they have had on our surveys. We look at the changes in response rates and the distribution of characteristics among those responding to our surveys. The social surveys and waves included are:

• Labour Force Survey (LFS) (wave one only)

• Survey on Living Conditions (SLC) (wave one only)

• Living Costs and Food Survey (LCF)

• Wealth and Assets Survey (WAS) (wave one only)

• National Survey for Wales (NSW)

Initial changes to our survey operations

Prior to the coronavirus pandemic, interviewers conducted in-person, in-home interviews for the surveys and waves covered. Survey interviewing was paused from when national lockdown was declared on 17 March 2020 until mid-April 2020. A strategy to move data collection from face-to-face to telephone interviewing was therefore rapidly implemented to ensure survey data collection could continue, while protecting respondent and field interviewer safety.

 The immediate priorities were:

• to adapt the questionnaire to telephone interviewing, in consultation with stakeholders

• to communicate changes in data collection to respondents and users

• to equip interviewers to work from home

The changes implemented in the questionnaires included:

• shortening questionnaires for several surveys (SLC, LCF, WAS and NSW) to reduce respondent burden and impact of change on response quality

• adapting questions for telephone interviewing; for example, we reworded questions using showcards and added standardised prompts to questions that used to require showcards

• introducing new COVID-19 related questions

• redesigning the NSW questionnaire based on Welsh government requirements

We also changed how we contacted people sampled for our surveys by:

• including compliments slips with business-as-usual materials to inform respondents of changes to data collection processes

• extending use of telematching

• asking residents in sampled households in their advance letter to enter their telephone number onto an online portal, and informing them that ONS surveys would also collect information related to COVID-19

Telephone interviewing started in mid-April 2020 for LFS, SLC, LCF, and WAS and in May 2020 for NSW.

We introduced the online portal during a second phase of changes, because only a small proportion of addresses (between 10% to 15%) was successfully matched with telephone numbers in the telematching process. The online portal was implemented in May 2020 for LFS, SLC, LCF and WAS.

NSW was the only survey for which previous respondents from 2018, 2019 and 2020 were recontacted between May and December 2020. This was similar to the Crime Survey for Crime Survey for England and Wales (CSEW), which also recontacted previous sampled respondents to take part in the survey over the phone. NSW used a new sample of respondents that could enter their contact details into the online portal from January 2021 onward.

Further changes to our survey operations

Relying on people selected for survey participation to proactively provide the ONS with their telephone numbers caused a decline in survey response rates. We also saw a change in the characteristics of survey respondents.

We developed several strategies to counteract falling response rates and changes in the characteristics of respondents, including:

• increasing sample sizes

• reviewing incentive values

• changing how we contact respondents

Increasing sample sizes

Sample sizes were increased for some surveys (LFS, SLC, LCF, WAS and NSW). This oversampling strategy was also recommended by Eurostat (for example, to collect data for the European Surveys on Income and Living Conditions).

We boosted sample sizes for LCF, SLC and WAS by adjusting the selected quota sizes. The LCF, SLC and WAS sample sizes were initially doubled between June and September 2020. They were then revised from October 2020 to account for:

• achieved sample sizes between June and September 2020, compared with previous years and target sample for the year

• impact of the introduction of knock-to-nudge (ktN) intervention on response rates

• costs of KtN and operating with a bigger sample sizes in relation to materials, incentives, interviewer time and subsistence

Surveys that underwent a sample boost included:

LFS (wave one)

The survey had an original sample size of 16,640 addresses in June 2020. This was boosted by 100% in July 2020 to March 2021, and by 60% in April to May 2021.

SLC (wave one)

The survey had an original sample size of 795 addresses in May 2020. This was boosted by 100% in June to September 2020, by 80% in October to December 2020, by 67% in January to February 2021, and by 73% in March 2021.

LCF

The survey had an original sample size of 1,180 addresses in May 2020. This was boosted by 100% in June to September 2020, by 70% in October to December 2020, and by 40% in January to March 2021.

WAS (wave one)

The survey had an original sample size of 832 and 806 addresses in April and May 2020, respectively. This was boosted by 100% in June to September 2020, by 85% in October 2020 to January 2021, and by 54% in February and March 2021.

NSW

The survey had an original sample size of approximately 2,050 addresses a month in 2019 to 2020. During the recontact sampling period, this increased by 107% in May 2020 to account for interviewing being temporarily paused during April, while optimising the questionnaire for telephone during the coronavirus pandemic. It later decreased by 5% in June to October 2020, decreased by 11% in November 2020, and increased by 35% in December 2020. During the fresh sampling period, the sample size increased by 47% in January to February 2021, and by 60% in March 2021.

Reviewing incentive values

In 2018, ONS conducted an internal incentive strategy review.

A literature review found that monetary incentives can increase survey responses (PDF, 99.6 KB) for both face-to-face and telephone surveys. While incentives often increase response rates, they do not necessarily have to be of high monetary value. Small incentive amounts can be more cost effective and still successfully increase response rates.

However, we should also consider respondent burden when deciding incentive strategies. For example, for LCF, filling out the expenditure diary is a big respondent burden; to compensate for this, a higher conditional incentive has proven effective.

Following the onset of the coronavirus pandemic, we:

• increased the value of the unconditional incentive for the LFS to £10; this was in response to falling response rates and the results obtained at pre-coronavirus pandemic randomised controlled trials

• increased the value of unconditional incentives for the SLC, LCF and WAS from £5 to £10 in June 2020; however, there was no measurable impact on response rates, so the incentives decreased to £5 again in October 2020

• raised the conditional incentive for NSW from £10 to £15 in May 2020

The list of conditional and unconditional incentives for LFS, SLC, LCF, WAS and NSW is summarised in Annex 1.

Changing how we contact respondents

Despite extending the use of telematching as an alternative method for obtaining respondent contact details, only a small percentage of cases had successfully matched contact numbers. As a result, we introduced the KtN intervention. This involves interviewers visiting households to "nudge" those selected to participate in ONS surveys (following stringent health and safety protocols).

From June 2020, government measures were relaxed, and a small-scale KtN intervention trial was put in place in July. This involved several interviewers who volunteered to visit households to "nudge" those selected to participate in ONS surveys. Larger KtN intervention trials were put in place for August and September 2020, in regions with low COVID-19 incident rates.

Survey participants had already received an advance letter from ONS to provide their telephone numbers via the online portal, the ONS survey enquiry line, or directly in response to interviewer letters. We targeted households based on dynamic risk assessments. These were carried out by each interviewer in relation to health and safety while working during the coronavirus pandemic.

KtN offered an opportunity for survey participants to schedule a telephone appointment with the interviewer at the doorstep, and for the interviewer to build a rapport with the respondent. For non-contacts, a call today card, indicating that the interviewer has visited the address, was posted through the door.

 The aims of the trials were:

• to estimate how successful KtN was at obtaining survey participants' telephone numbers

• to understand how many KtN visits led to successful interviews

• to estimate the cost and lessons learnt on carrying out KtN within stringent health and safety guidelines

Once we obtained the outcomes of the small-scale trials, we scaled up the KtN intervention. As part of this, we implemented KtN on:

• SLC and LCF in October 2020

• WAS and NSW in January 2021

• LFS in April 2021

Sampled addresses were selected for the KtN intervention if they:

• had not entered their contact details via the online portal

• did not have telephone details captured via telematching

• had not contacted ONS or an interviewer directly

The proportion of households selected for KtN intervention, compared with the total number of households selected for telephone interviewing, varied by surveys and by month. This depended on:

• each survey's total sample size

• the number of interviewers volunteering for KtN

• the coronavirus situation and rapidly changing government guidelines

• the prioritisation and allocation of interviewer work across surveys

Datasets used for analysis

We will compare survey response rates and characteristics of responding households and individuals for all surveys across three different time periods and modes of data collection.

The first group of unweighted data covers April 2019 to February 2020, when respondents were interviewed face-to-face in their homes for all surveys studied. These datasets do not include the month of March 2020 because data collection was suspended on 17 March 2020.

The second group of unweighted data covers a time period when respondents were interviewed via telephone, without knock-to-nudge (KtN) intervention. This time period varies between surveys because KtN was implemented at different times for different surveys.

Telephone numbers of respondents to the Labour Force Survey (LFS), the Survey on Living Conditions (SLC), the Living Costs and Food Survey (LCF), and the Wealth and Assets Survey (WAS) were obtained via the online portal, a telematching provider, or in response to interviewer letters. However, telephone numbers of respondents to the National Survey for Wales (NSW) were already available, because the sample was re-sampled from the previous financial year.

The dataset for NSW for the telephone period includes May to July 2020; this is so we can compare sample characteristics with the previous financial year. After July 2020, NSW sampled respondents that took part in NSW in 2017 to 2018; therefore, we could not explore any direct influence of the mode from one financial year to the other.

The third group of unweighted data includes cases where respondents were contacted by interviewers via KtN and subsequently interviewed by telephone. Most of these respondents would have had a doorstep interaction with an interviewer, with very few having only received a "called today" card. For the purpose of this paper, no dataset from June 2021 onwards was included; this controlled for the impact that the gradual lifting of most coronavirus (COVID-19) restrictions may have had on the characteristics of respondents.

The third group of data only includes respondents whose telephone numbers were not obtained via the online portal or by telematching, which introduces a self-selection bias. Also, not all cases without a phone number were allocated to KtN.

However, looking at the distribution of the characteristics of respondents who received KtN intervention is useful. It shows us whether KtN helps obtain respondents with characteristics that were under-represented in our survey datasets prior to the KtN intervention (when telephone numbers were obtained via an online portal or by telematching only).

Dataset geography and time periods

LFS (wave one)

The geography covered in LFS datasets was Great Britain (GB). Dataset 1 (face-to-face mode) took place from April 2019 to February 2020. Dataset 2 (telephone mode) took place from April 2020 to March 2021. Dataset 3 (telephone mode filtered for cases that received KtN) took place from April to May 2021.

SLC (wave one)

The geography covered in SLC datasets was GB. Dataset 1 (face-to-face mode) took place from April 2019 to February 2020. Dataset 2 (telephone mode) took place from April to September 2020. Dataset 3 (telephone mode filtered for cases that received KtN) took place from October 2020 to March 2021.

LCF

The geography covered in LCF datasets was GB. Dataset 1 (face-to-face mode) took place from April 2019 to February 2020. Dataset 2 (telephone mode) took place from April to September 2020. Dataset 3 (telephone mode filtered for cases that received KtN) took place from October 2020 to March 2021.

WAS (wave one)

The geography covered in WAS datasets was GB. Dataset 1 (face-to-face mode) took place from April 2019 to February 2020. Dataset 2 (telephone mode) took place from April to December 2020. Dataset 3 (telephone mode filtered for cases that received KtN) took place from January to March 2021.

NSW

The geography covered in NSW datasets was Wales. Dataset 1 (face-to-face mode) took place from April 2019 to February 2020. Dataset 2 (telephone mode) took place from May to July 2020. Dataset 3 (telephone mode filtered for cases that received KtN) took place from January to March 2021.

Northern Ireland data were not included in the analysis for LFS, SLC, LCF and WAS. Data in Northern Ireland are collected by the Northern Ireland Statistics and Research Agency (NISRA), which has not implemented KtN.

Variables used for analysis

We wanted to compare characteristics of respondents and households across surveys and different modes of collection. For the analysis in this paper, we therefore included person-level variables (age, ethnicity, marital status, and National Statistics Socio-economic classification (NS-SEC)) and household-level variables (tenure, household size, and indices of multiple deprivation in quintiles).

NSW explored variables at household reference person-level only, whereas other analysis for surveys accounted for all household members when exploring person-level variables.

More details about the variables explored can be found in Annex 2.

Reference estimates of the GB population are used to assess the biases introduced by the change of data collection mode for some of variables. However, the main ONS social surveys do not target the whole GB population but exclude certain small subgroups. For example:

• LFS includes residents in private households, residents in NHS accommodation, and young people living away from their parental home during term time (about 98.5% of the total UK population), but excludes people not in households (such as people in care homes and prisoners)

• SLC includes slightly fewer people than LFS because residents in NHS accommodation are also excluded

• LCF and WAS only include residents in private households (about 97% of the population)

Because of the differences in target populations, the proportions for the various categories (such as age categories) of the variables of interest (such as age) in the GB population may therefore vary slightly between surveys.

Statistical tests

The Pearson's chi square of association statistical test was used to see whether there was any association among the selected person-level or household-level variables and modes of data collection or ways of gathering contact details.

A chi-square test compares the observed frequencies with those you would expect to get by chance if there was no association. This test is used on categorical or ordinal data.

A p-value of less than 0.05 is reported as significant. This means that the statistical test is showing a statistically significant result and an association between a variable and the mode of collection, for example, which is greater than would be expected by chance. It should be noted that a chi-square test is highly sensitive to sample sizes. For example, for large sample sizes, a weak association between two variables could become significant.

For a fully valid statistical test, the sampling design should be taken into account. Surveys are clustered and the dependence between observations that is introduced by clustering can lead to biased results. In particular, p-values may become too small and confidence intervals too narrow. However, design-adjusted tests require the full weighted microdata with cluster and stratum information, which are not yet available for all surveys. The standard statistical software packages offer tests, which deal with weighted proportions (a more common scenario). None of the tests offered by standard statistical packages are appropriate for the KtN scenario, where the null hypothesis involves the comparison of raw unweighted proportions.

As it was not feasible to run design-adjusted tests, standard chi-square tests were used to provide some information on whether the observed results could simply occur by chance and sampling fluctuations. Small p-values imply that a design-adjusted test would also produce a highly significant result. Similarly, large p-values imply that the design-adjusted p-values would also be large. No clear conclusion can be drawn in cases where p-values are neither very small nor large.

There is some ambiguity in this approach, because it is unclear when exactly a p-value should be considered neither very small nor large. However, this does not affect the key messages in this paper, which are largely based on descriptive statistics and exploring the characteristics of different datasets.

Changes to response rates

Average response rates decreased when moving from face-to-face to telephone mode of data collection for:

• the Labour Force Survey (LFS) wave one

• the Survey on Living Conditions (SLC) wave one

• the Living Costs and Food Survey (LCF)

• the Wealth and Assets Survey (WAS) wave one

This was during the coronavirus (COVID-19) pandemic and prior to any knock-to-nudge (KtN) intervention.

For example, the LFS average response rate decreased by more than 20 percentage points, from 55.2% (April 2019 to February 2020) to 28.7% (April 2020 to March 2021).

The variation in monthly response rates also increased for these surveys. For example, LCF monthly response rates ranged between 39.6% and 43.7% during face-to-face data collection (April 2019 to February 2020), compared with between 22.0% and 30.5% during telephone data collection (April to September 2020).

The lower response rates achieved during the period of telephone interviewing in comparison with face-to-face interviewing for ONS social surveys (except NSW) also support the existing literature on response rate differences by mode. Examples include:

• An experimental comparison of national telephone and personal interview surveys

• Alternative approaches to survey data collection for the national election studies

• Personal versus telephone surveys for collecting household health data at the local level

An increase in non-response during the coronavirus pandemic was also reported for other surveys across the world. For example, the US Census bureau reported a significant increase in survey non-response for their Current Population Survey (CPS) (PDF, 1,380 KB). Eurostat described the decrease in response rates during the coronavirus pandemic as a challenge for many European labour force surveys (PDF, 643 KB). The Scottish Health Survey (PDF, 172 KB), which also moved from face-to-face to telephone mode, similarly saw a decrease in response rates The average response rates for NSW were higher during telephone data collection (72.6% in May to July 2020), compared with face-to-face data collection (58.5% in April 2019 to February 2020).

This is different from other surveys because NSW respondents were re-sampled from the previous financial year at that time, whereas other surveys selected new respondents for their first waves (see Annex 4 for NSW response rates).

Changes in the distribution of household characteristics

This section reports on distribution of responding household characteristics when moving from face-to-face to telephone interviewing during the coronavirus pandemic. The variables used for this analysis were tenure status, indices of multiple deprivation (IMD) and household size.

Tenure status

An increase in the proportion of households who owned their property, and a decrease in the proportion of those who rented their property, was observed across all surveys studied when the mode of collection changed.

For example, Figure 1 outlines that 47% of SLC households interviewed by telephone during the coronavirus pandemic owned their property (April 2020 to September 2020), compared with 37% of those interviewed face-to-face prior to the coronavirus pandemic (April 2019 to February 2020). In comparison, 24% of SLC households interviewed by telephone rented or shared their accommodation, compared with 35% of those who were interviewed face-to-face.

Index of Multiple Deprivation (IMD) quintile

Each quintile from the Index of Multiple Deprivation (IMD) (the most deprived to the least deprived) should contain about 20% of households in the Great Britain (GB) population. This shows that those in the least and the most deprived quintiles should each represent around 20% of the total sample of respondents.

A decrease in the proportion of households in the most deprived quintile was observed across all surveys studied when the mode of collection changed (Table 3). When respondent details were obtained via an online portal or a provider, and when interviewing was conducted via telephone during the coronavirus pandemic, a noticeable under-representation of respondents in the most deprived quintile (compared with the reference estimate) was observed across all surveys.

An increase in the proportion of households in the least deprived quintile was observed for the LFS, SLC and LCF when the mode of collection changed (Table 3). For WAS and NSW, the proportion of households in the least deprived quintile remained similar when the mode of data collection changed. A noticeable over-representation of respondents in the least deprived quintile was observed for WAS for both face-to-face interviewing (24% in April 2019 to February 2020) and telephone interviewing during the coronavirus pandemic (24% in April to December 2020).

Household size

An increase in the proportion of households with two people was observed across all surveys studied when the mode of collection changed (Table 4).

A decrease in the proportion of households with three people or more was observed for LFS, SLC, LCF and WAS when the mode of collection changed (Table 4).

Changes in the distribution of person characteristics

This section reports on the distribution of responding respondent characteristics when moving from face-to-face to telephone interviewing during the coronavirus pandemic. The variables used for this analysis were age, marital status, National Statistics Socio-economic classification (NS-SEC) and ethnicity.

Age

Mid-year UK population estimates suggest that 43% of the private household population were aged 46 years and over, 38% were aged 16 to 45 years, and 19% were aged 0 to 15 years in 2019.

There was an increase in the proportion of respondents aged 46 years and over, and a decrease in the proportion of those aged 0 to 15 years and aged 16 to 45 years across all surveys studied, when the data collection changed (Table 5). The only exception was the LCF, where the proportion of those aged 16 to 45 years remained constant (34%) when the data collection mode was changed.

Marital status

An increase in the proportion of respondents classified as married, in a civil partnership, or separated, and a decrease in the proportion of respondents classified as living on their own, or cohabiting with a partner or others, was observed for the majority of surveys when the data collection mode changed (with the exception of the LCF).

Even though NSW re-sampled previous respondents, 51% of those interviewed via telephone during the coronavirus pandemic reported being married, in a civil partnership or separated (May to July 2020), compared with 47% of those interviewed face-to-face prior to the coronavirus pandemic (April 2019 to February 2020).

National Statistics Socio-economic classification (NS-SEC)

An increase in the proportion of respondents in higher managerial, administrative and professional occupations (NS-SEC categories 1 and 2) and a decrease in the proportion of those in routine occupations (NS-SEC categories 6 and 7) was observed for all surveys when the data collection changed. NSW was not included in the analysis as NS-SEC is not collected for this survey.

For example, Figure 3 shows that 49% of WAS respondents contacted and interviewed by telephone during the coronavirus pandemic (April to December 2020) were in higher managerial, administrative and professional occupations, compared with 44% of those interviewed face-to-face prior to the coronavirus pandemic (April 2019 to February 2020).

The proportion of WAS respondents in routine occupations decreased from 31% during face-to-face interviewing to 26% during telephone interviewing.

Ethnicity

An increase in the proportion of respondents belonging to the White ethnic group and a decrease in the proportion of respondents belonging to a Black, Asian, Arab, Mixed or Other ethnic group was observed for LFS and WAS when the data collection changed.

Figure 4 shows that 91% of LFS respondents contacted and interviewed by telephone during the coronavirus pandemic reported being White (April 2020 to March 2021) compared with 87% of those interviewed face-to-face prior to the coronavirus pandemic (April 2019 to February 2020).

The proportion of LFS respondents belonging the Black, Asian, Arab, Mixed or Other ethnic group decreased from 13% of the respondents during face-to-face interviewing to 9% during telephone interviewing.

In contrast, SLC saw an increase in the proportion of respondents who belonged to the Black, Asian, Arab, Mixed or Other ethnic group (three percentage points) and a decrease in the proportion of the White ethnic group (two percentage points) when the data collection changed. However, the results need to be interpreted with caution, because the question on ethnicity in the SLC questionnaire is currently being reviewed to improve quality and accuracy.

Changes in response rates when introducing knock-to-nudge (KtN)

An increase in response rates was observed across all surveys when the KtN intervention was introduced (Table 6). It was not possible to compare monthly average response rates for telephone mode prior to and after KtN for NSW, owing to a different sampling strategy implemented for the switch from face-to-face to telephone mode (see Annex 4 for NSW response rates).

For example, the monthly average response rate for the SLC was 27.5% during the April to September 2020 period. When the KtN intervention was introduced for the SLC, the monthly average response rate increased by 6.8 percentage points (October 2020 to March 2021).

The variation in response rates decreased when the KtN intervention was implemented, with more stable response rates across the months when KtN was used, compared with the months when telephone numbers were collected without KtN intervention.

For example, for WAS wave 1, the response rates ranged between 19% and 29% during telephone mode prior to KtN intervention, and between 30% and 32% when KtN was introduced.

This can be taken to imply the effectiveness of KtN intervention. It suggests that relying either on telematching or on respondents contacting the ONS themselves (in response to letters or using the online portal) could lead to significant drops in survey response rates in some months. The monthly response rate figures presented here indicate that this can be prevented by interviewers knocking and nudging survey participants to provide their telephone numbers, and possibly arranging interviews on the doorstep.

Before the coronavirus pandemic, nudging was mainly considered in relation to additional mailings to increase survey take-up. It was previously considered elsewhere in relation to push-to-web survey designs to increase response. However, this article suggests that an in-person nudge is also an effective measure to increase response rates for telephone surveys.

Exploratory internal research further supports this by suggesting that an in-person nudge is effective for a push-to-web survey using both online and telephone modes for survey completion. In this instance, respondents were offered an online mode to complete their surveys, and non-respondents were followed up with a KtN intervention, where they were prompted to complete the survey using the telephone mode (ONS, internal report, 2021).

Impact of introducing KtN on the distribution of household characteristics

This section reports on the distribution of responding household characteristics when respondents' contact details were obtained either without a KtN intervention, or with a KtN intervention. The variables used for this analysis were tenure status, indices of multiple deprivation and household size (Annex 2).

Among those who did not enter their contact details onto the online portal, only a certain proportion had an interviewer knocking on their door to nudge them. Therefore, the impact of KtN on the overall distribution of respondents by various characteristics varies for each survey, depending on the proportion of respondents who received the KtN intervention.

Tenure status

A higher proportion of respondents reported sharing or renting their accommodation when they were interviewed by telephone and contacted via KtN, compared with when they were interviewed by telephone prior to KtN intervention for all surveys. In comparison, the proportion of respondents owning their property was lower among those interviewed by telephone and contacted via KtN than among those interviewed by telephone prior to KtN intervention for all surveys.

However, those sharing or renting their accommodation might be more likely to respond to an interviewer knocking on their door. Rented properties are also more likely to be within blocks of flats, where advance letters might not reach respondents easily.

It is important to note that KtN was only applied to households who did not enter their contact details in the online portal. Those who owned their property were more likely to have entered their contact details in the online portal for all surveys. This explanation may not apply for NSW, as respondents were re-sampled from previous waves and therefore did not enter contact details in an online portal.

For example, as seen in Figure 5, just over a third (36%) of SLC respondents reported sharing or renting their accommodation when they were contacted via KtN (October 2020 to March 2021). This compares with just under a quarter (24%) of SLC respondents prior to KtN intervention (April 2020 to September 2020).

Index of Multiple Deprivation (IMD) quintile

A higher proportion of respondents were classified in the most deprived Index of Multiple Deprivation (IMD) quintile when they were interviewed by telephone and contacted via KtN, compared with when they were interviewed by telephone prior to KtN intervention for all surveys. This was more marked for WAS and NSW than for LFS, SLC and LCF (Table 7).

In comparison, the proportion of respondents in the least deprived quintile tended to be lower among those interviewed by telephone and contacted via KtN than among those interviewed by telephone prior to KtN for all surveys. This was more marked for WAS and NSW than for LFS, SLC and LCF.

It could be possible that those in the most deprived IMD quintile were more likely to respond to an interviewer knocking on their door. However, it is important to note that KtN was only applied to households who did not enter their contact details in the online portal and those in the least deprived IMD quintile were more likely to have entered their contact details in the online portal for the SLC and the LCF.

Household size

A higher proportion of respondents reported being part of a larger household of three or more people when they were interviewed by telephone and contacted via KtN, compared with when they were interviewed by telephone prior to KtN intervention for all surveys. This was more marked for LFS, SLC and LCF than for WAS and NSW (Table 8).

In comparison, the proportion of respondents in households of two people was lower among those interviewed by telephone and contacted via KtN than among those interviewed by telephone prior to KtN for all surveys.

It could be possible that those living in households of three or more people were more likely to respond to an interviewer knocking on their door. It is important to note that KtN was only applied to households who did not enter their contact details in the online portal, and those in households of two people were more likely to have entered their contact details in the online portal for most surveys.

Changes in the distribution of person characteristics

This section reports on the distribution of respondents’ characteristics when contact details were obtained either with or without a KtN intervention. The variables used for this analysis were age, marital status, National Statistics Socio-economic classification (NS-SEC) and ethnicity.

Age

The proportions of respondents aged 0 to 15 years and 16 to 45 years were higher among those interviewed by telephone and contacted via KtN than among those interviewed by telephone prior to KtN intervention for all surveys (Table 9). In comparison, the proportion of respondents aged 46 years and over was lower among those interviewed by telephone and contacted via KtN than among those interviewed by telephone prior to KtN for all surveys.

It could be possible that those aged under 46 years were more likely to respond to an interviewer knocking on their door. It is important to note that KtN was only applied to households who did not enter their contact details in the online portal and those aged 46 years and over were more likely to have entered their contact details in the online portal.

Marital status

The proportions of respondents classified as living alone or cohabiting with a partner or others were higher among those interviewed by telephone and contacted via KtN than among those interviewed by telephone prior to KtN intervention for all surveys (this was not significant for LCF).

In comparison, the proportion of respondents classified as married, in a civil partnership or separated was lower among those interviewed by telephone and contacted via KtN than among those interviewed by telephone prior to KtN for all surveys.

It could be possible that those classified as living alone or cohabiting with a partner or others were more likely to respond to an interviewer knocking on their door. It is important to note that KtN was only applied to households who did not enter their contact details in the online portal and those married, in a civil partnership or separated were more likely to have entered their contact details in the online portal for all surveys.

Figure 6 shows that just over a third (35%) of NSW respondents who were contacted via KtN (January to March 2021) were classified as living alone or cohabiting with a partner or others, compared with 25% during telephone interviewing prior to KtN (May to July 2020), when survey participants were re-sampled from previous waves.

National Statistics Socio-economic classification (NS-SEC)

Figure 7 indicates that a third (33%) of WAS respondents who were contacted via KtN (January to March 2021) were in routine occupations, known as the National Statistics Socio-economic classification (NS-SEC), categories 6 and 7. This is compared with 26% during telephone interviewing prior to KtN (April 2020 to December 2020).

An increase of 4 percentage points in the representation of respondents in routine occupations was observed for LFS for the samples with telephone mode and KtN intervention, compared with the samples with telephone mode prior to KtN intervention.

In comparison, the proportion of respondents classified as in higher managerial and professional occupations (NS-SEC categories 1 and 2) decreased for WAS (by 11 percentage points) and LFS (by 4 percentage points). This was for the samples with telephone mode and KtN intervention, compared with the samples with telephone mode prior to KtN intervention.

There was not any significant association between NS-SEC and method of survey contact for SLC and LCF.

Ethnicity

The proportion of respondents in the White ethnic group was lower among those interviewed by telephone and contacted via KtN than among those interviewed by telephone prior to KtN intervention for all surveys. In comparison, the proportion of respondents belonging to a Black, Asian, Arab, Mixed or Other ethnic group was higher among those interviewed by telephone and contacted via KtN than among those interviewed by telephone prior to KtN for all surveys.

For example, Figure 8 shows that 91% of LFS respondents reported belonging to the White ethnic group when interviewed over the telephone prior to KtN intervention (April 2020 to March 2021), compared with 87% among those interviewed over the telephone when contact details were gathered via KtN (April 2021 to May 2021).

There was an increase in representation of LFS respondents belonging to the Black, Asian, Arab, Mixed or Other ethnic group, from 9% when being interviewed over the telephone without KtN intervention (April 2020 to March 2021) to 13% when being interviewed over the telephone with KtN intervention (April 2021 to May 2021).

Several factors have the potential to impact on response rates and the distribution of characteristics among respondents, including:

• the change in mode of interviewing

• the ways of contacting survey participants

• people's behaviours during the coronavirus (COVID-19) pandemic

In response to the coronavirus pandemic, the mode of data collection for our social surveys (Labour Force Survey (LFS), the Survey on Living Conditions (SLC), the Living Costs and Food Survey (LCF), the Wealth and Assets Survey (WAS) ) and the National Survey for Wales (NSW)) changed from face-to-face to telephone interviewing from mid-April 2020 onwards.

Following this, the distribution of the characteristics among those responding to our surveys changed. The proportion of respondents who were aged 46 years and over, were married or in a civil partnership, owned their property, lived in the least deprived areas, and worked in higher managerial and professional occupations increased. The proportion of respondents who were younger, single, renting or sharing their accommodation, living in the most deprived areas, and working in routine occupations decreased.

The changes were less marked for the LCF, perhaps because of the larger incentive provided by this survey compared with other surveys. In particular, LCF respondents aged 16 to 45 years were as likely to respond when interviewed over the telephone as they were face-to-face.

It is difficult to distinguish the mode effect from the coronavirus pandemic effect. We must consider whether switching to telephone mode affects social surveys' sample composition. For example, there might be different response behaviours from respondents with specific characteristics (such as having more trust during face-to-face interviewing or telephone interviewing).

The survey topic might have an influence too. For example, a separate analysis was carried out for the Crime Survey of England and Wales (CSEW). This indicated that differences in the characteristics of respondent between face-to-face and telephone mode were small for tenure but significant for age.

As the data collection moved from face-to-face to telephone from mid-April 2020 onwards, we relied on the respondents who received their advance letters to proactively contact us to provide their contact details. This could have introduced a self-selection bias, meaning respondents with certain attributes (higher civic engagement or more available time) were more likely to provide us with their contact details. Certain groups of people might have also been more likely to contact us because of the ONS's increased profile as a result of the COVID-19 Infection Survey and Census 2021.

The coronavirus pandemic also affected many things, which could have altered people's behaviours, including:

• their social activities

• whether they lived alone or with other people

• their adherence to coronavirus guidelines

• their health

• whether they were working from home, outside the home or were furloughed

For example, a Coronavirus and homeworking report indicates that people aged 25 to 45 years were the most likely to be working from home (54.3%), followed by those aged 65 years and over (43%) and those aged 45 to 64 years (42%). In comparison, those aged 16 to 24 years were the least likely to be working from home (30.2%).

Around 67% of managers, directors, and senior officials and 70% of those in professional occupations were likely to be working from home, compared with 15% in the caring and leisure industry, 16% in sales and customer services and 5% of those working as process plant and machine operatives. It is plausible that those working from home at the start of the coronavirus pandemic were more likely to proactively contact us to provide their contact details than those who were not working from home.

The introduction of Knock-to-nudge (KtN) saw a change in the characteristics of respondents. We captured a larger proportion of younger respondents who were not married, lived in larger households of three or more (possibly with their families), were renting or sharing their accommodation, and were living in the most deprived areas. Previous research has suggested that families with young children are more likely to be at home and therefore easy to contact. The larger the household, the more likely that someone will be at home to answer a knock on the door.

A sample must reflect the population and be representative with respect to all variables measured in a survey. Ideally, the distribution of characteristics among respondents are assessed by comparing survey characteristics of respondents with reference data that represent the distribution of the characteristics of respondents in the population. Access to such information was possible for some variables (such as age), but challenging for other variables (such as tenure). The improved access to administrative data containing information on a range of characteristics should improve the assessment of respondent characteristics in the future.

To derive population estimates from survey statistics, weighting is an important technique used to correct for the lack of representation in a survey, with calibration bringing the estimated sample in line with population estimates. The weighting methodologies applied to the surveys studied when the mode of collection was face-to-face are described elsewhere.

To adjust for any potential impact of the mode change during the coronavirus pandemic, the LFS introduced tenure in its calibration. Similarly, tenure has been introduced in SLC and LCF weighting methodology from April 2020. In addition, LFS added a further change to their weighting methodology by adjusting the population changes for UK and non-UK born in line with the changes observed in the HM Revenue and Customs (HMRC) pay as you earn (PAYE) real time information. With WAS collection and processing of data being over a two year round, weights that cover the coronavirus pandemic have yet to produced; similarly to other surveys, consideration will be given to the introduction of new variables for calibration.

Following the change from face-to-face to telephone interviewing, we observed a drop in responses rates for our surveys (LFS, SLC, LCF and WAS). After introducing KtN, we then saw an increase in response rates across all surveys and a decrease in the variation in response rates.

Lower response rates often indicate a decrease in survey quality because it increases the chance of non-response bias (although higher response rates are not necessarily associated with a lower response bias). Non-response biases are a function of the differences between respondents and non-respondents, occurring when there is an interaction between the likelihood of responding to the survey and particular socio-demographic characteristics. This can introduce error into the survey results, if the characteristics of those who respond (for example, females and the elderly) are different to the characteristics of those who do not respond (for example, males and students). This can result in certain groups of people being under-represented in the results; however, this analysis is outside the scope of this paper.

The ONS Census Non-Response Link Study (CNRLS) run by the Methodology and Quality Division (MQD) in a highly secured environment will inform on any potential non-response bias in social surveys. Linking social survey households to information captured within census returns facilitates analysis of non-response determinants to social surveys.

This linkage allows us to connect non-responding social survey households, who we otherwise have no information on, to the information contained on that household within the census. Having auxiliary variables for both responding and non-responding social survey households will allow us to assess the efficacy of KtN as an additional way of contact.

Following the analysis presented in this paper, it will be important to understand the lasting impact of COVID-19 on our approach to data collection, and the respective roles of telephone interviewing, in-home interviewing and the KtN technique to contact respondents.

We are currently planning a face-to-face trial for the SLC, where a subsample of SLC survey participants is asked by an interviewer during the KtN intervention whether they are comfortable to proceed with a face-to-face interview. We need to develop a more complete understanding of how different possible approaches (face-to-face first or telephone first, with or without KtN intervention) affect response rates, the characteristics of respondents and data quality. This is particularly challenging in a constantly evolving COVID-19 context. The outcome of the SLC trial will be used to inform the future strategy for LCF and WAS.

There are currently no plans to implement face-to-face interviewing on the LFS. This may be considered once evidence from initial trials for other surveys is available. However, any potential face-to-face implementation would need to be considered alongside the further development of the online-first multi-mode transformed Labour Force Survey (sometimes previously referred to as the Labour Market Survey), which is currently being developed.

Our results are also informative in the context of the Office for National Statistics' (ONS) strategic business plan, which states the importance for data produced by the ONS to represent the country. The report says we should make our overarching data collection approach as inclusive as possible, using a range of data sources (including administrative and social survey data). To be inclusive and understand if our social surveys are representing the country as best as possible, we must understand how representative of the population our surveys are. It is also important to understand any non-response biases that may be introduced because of people being unwilling or unable to respond to our social surveys, owing to certain characteristics that distinguish them from those who do respond.

Beyond the techniques of nudging, the importance of ethics and the obtention of a social contract between data producers and participants cannot be understated. The Inclusive Data Taskforce 2021 recommendations report highlights the importance of enhancing trust and trustworthiness in the provision and use of data to improve inclusivity in our social surveys. Each intervention suggested by the Inclusive Data Taskforce, and any further interventions or operational changes we implement, could potentially have an impact on response and non-response. Therefore, it is important for all aspects of data quality to be continually monitored.

The newly launched ONS survey strategy clarifies our vision for all surveys to be fully inclusive by design, ensuring that our data and workforce fully represent the UK population. This means we will take a respondent-centric approach, putting respondents' needs at the heart of our surveys. For example, we will:

• segment samples

• use more inclusive sample frames

• produce easily accessible materials

• offer multiple collection modes

• engage businesses and communities to understand their needs and build support for our surveys

It also means that we will carry out our data collection operations in a responsive way that includes representation, and we will introduce the ability to prioritise our field resources around sub-groups of the population and different units of collection. Finally, we will also build a diverse and expert survey community, creating a centre of excellence for surveys which will provide through leadership, stewardship and support development.

This article has only included some household and respondent characteristics, based on their possible impact on key population estimates and their importance for weighting. However, the Inclusive Data Taskforce 2021 recommendations report identifies a range of other characteristics that represent critical data gaps. Obtaining more granular data for certain groups would require us to look beyond representativeness. The recent report is an important milestone in paving the road for the future of data collection.

Waves

Traditional to longitudinal surveys that re-sample households over a period of time to understand changes in society. This article only focuses on the first wave of longitudinal surveys.

Showcards

Used in a face-to-face survey so that respondents can choose an answer on the card rather than reading and answering with an answer option. The answers listed can be in the form of numbers, scales, words, pictures of other graphical representations.

Telematching

A process whereby a contractor provides an online facility to match telephone numbers against sampled addresses, with landline and mobile numbers provided from sources such as the electoral register and British telecommunications. Prior to the coronavirus (COVID-19) pandemic, this was already a routine process on the Labour Force Survey (LFS) for sampled addressed located north of the Caledonian canal.

Cross-sectional survey

Respondents are asked to take part in a survey at one specific point in time.

Longitudinal survey

Respondents are asked to take part in a survey over a period of time.

Survey and respondent burden

Reflects the time and effort in answering survey questions and can be influenced by questionnaire design, survey length, topic and mode.

Randomised controlled trials

A form of research trial where individuals are assigned randomly to different experimental conditions. There are usually one or more experimental conditions, where the impact of the experimental conditions is compared with a control condition that has to intervention applied.

Noncontacts

Sampled addresses where the interviewer has not been able to establish any contact. That is, the respondent has neither conducted the interview nor refused the survey request. This differs from ineligible addresses, which are unoccupied or not suitable for the survey request.

Incentives

A form of compensation for a respondent’s time and effort to fill in a survey. Incentives can be monetary and non-monetary and offered either ahead of survey participation (unconditionally) or after the respondent had taken part in the survey (conditionally). Incentives are also effective in increasing responses to the survey.

Online portal

Was set up at the beginning of the coronavirus pandemic, urging respondents to provide ONS with there telephone number to facilitate social survey interviewing over the phone. The online portal asked for the respondents’ Unique Access Code provided in the advance letter and subsequently their telephone number. This information was iteratively fed through to survey interviewers for telephone interviewing.

Quota

The number of addresses assigned to individual interviewers.

Annex 1

Annex 2

Annex 2a

Household variables explored.

Four household-level variables are included in the social surveys analysed in this article. These are:

• tenure

• household size

• Indices of Multiple Deprivation (IMD).

The tenure variable has three categorisations. These are:

• own (the household is owned outright)

• mortgage (the household is owned through a mortgage)

• share or renting; this is a collapsed category including sharing, renting, living rent free, and squatting (squatting was only provided by the WAS Survey)

NSW had differing categories for tenure. These were mapped onto the three main categories to produce a harmonised variable.

The household size variable has three categorisations. These are:

• 1 (a single occupant household)

• 2 (a two person household)

• 3 plus (three or more people living in the household)

Indices of Multiple Deprivation (IMD) has five categorisations. These are:

• most deprived 20%

• most deprived 40%

• deprived 50%

• least deprived 40%

• least deprived 20%

This variable is the official measure of relative deprivation for small areas (or neighbourhoods) in Great Britain (GB). It is common to describe how deprived an area is by saying whether it falls among the most deprived 10%, 20%, and so on.

In this report, IMD quantiles were used rather than deciles. The IMD scores are based on seven different domains of deprivation:

• income

• education

• skills and training

• employment

• health and disability, crime

• barriers to housing and services

• living environment

IMD was also used for analysis on NSW for consistent analysis across surveys, although usually the Welsh Index of Multiple Deprivation (WIMD) is preferably used for NSW. For this analysis, the November 2019 csv file was used.

Annex 2b

Person-level variables explored.

Four person-level variables are included in the social surveys analysed in this article. These are:

• age

• ethnicity

• marital status

• The National Statistics Socio-economic classification (NS-SEC)

The age variable has three classifications, which are:

• aged 0 to 15 years

• aged 16 to 45 years

• aged 46 years and over

NSW was the only survey that did not collect any information for respondents aged under 16 years.

The ethnicity variable has three categorisations. These are:

• White

• Black, Asian, Arab, Mixed and Other

• Missing

The ethnicity group categories Black, Asian, Arab, Mixed and Other were grouped together owing to small numbers within the sub-categories. The category of Missing was also included for some surveys when it was a substantial category and was not missing at random.

The collection of the ethnicity variable did vary across the surveys. For example:

• WAS did not include the Arab ethnicity group as an answer option or allow for ethnicity to be captured by proxy

• NSW did not ask this question in the telephone mode when operating without knock-to-nudge (KtN)

• the ethnicity question was temporarily excluded from the SLC (April to May 2020) and LCF (April to July 2020) questionnaires to optimise the surveys for the telephone mode

The marital status variable has four categorisations. These are:

• single

• married, civil partnership or separated

• divorced or dissolved civil partnership

• widowed or surviving civil partner

This variable was filtered for respondents that were aged 16 years and over. Similar answer categories in relation to legal status categories were grouped together. For example, married, separated and in a civil partnership falls within the same category legally, so were grouped for the purposes of this research.

The NS-SEC variable has three categorisations. These are:

• higher managerial, administrative, and professional occupations

• intermediate occupations

• routine and manual occupations

The NS-SEC variable was filtered for respondents that were aged 16 years and over. It was also filtered to include employed respondents only.

Annex 3

We got estimates for:

• age using mid-year estimates from 2019, including Northern Ireland

• ethnicity using weighted LFS data from the financial year 2019 to 2020, including Northern Ireland

• marital status using 2019 GB population estimate data from England and Wales, and Scotland (PDF, 381 KB)

• household size using weighted LFS data from the financial year 2019 to 2020, including Northern Ireland

• NS-SEC using weighted LFS data from the financial year 2019 to 2020, including Northern Ireland

• tenure using weighted LFS data from 2019, including England, Scotland and Wales

• IMD using statistical random sampling, which assumes each group should contain around 20% of households, owing to the construction of the category

Annex 4