Skip to content

Social Survey Non-response Update This product is designated as National Statistics

Released: 25 February 2011 Download PDF

Abstract

The UK Statistics Authority assessment of the Labour Force Survey (LFS) for National Statistic status, reported on in March 2010, identified areas where ONS could strengthen its compliance with the Code and listed 7 requirements essential to the designation as National Statistics. The first of these was to communicate clearly the falling response rates on the LFS and the implications this would have for data quality. The purpose of this brief interim report is to provide an update on the non-response surveys conducted by ONS to address this requirement.

Background

The response rate for the UK Labour Force Survey is currently at its lowest level since the survey started in 1970. Low survey response rates not only affect the confidence limits at the Local Area Level, but also potentially induce non-response bias in the key LFS estimates. One of the practices listed under Principle 4 of the UK Statistics Authority Code of Practice states that users should be "informed about the quality of statistical outputs, including estimates of the main sources of bias and other errors”.

A non-response program was commenced in 2007 to analyse falling response rates in social surveys. Research carried out by Hopper et al. [1] in 2008 examined non-response patterns for six different surveys including the LFS and found that households residing within certain OAC super-groups are usually under- represented because they are more difficult to contact. Hopper also investigated calling patterns and recommended a variable pattern be implemented. Based on Hopper’s results, the ONS conducted a number of studies specific to the LFS between 2008 and 2010.

A study by Beavan-Seymour in 2008 [2] took into account demographic characteristics relating to tenure, household composition, ethnicity and nationality, the Output Area Code (OAC) coding frames, as well as sampling frame variables e.g. Government Office Region (GOR). The study found that there was a correlation between the geo-demographic characteristics and the response rate. The study also concluded that refusal rates were consistent across OAC codes but not contact rates. Following this work more consistent contact patterns and different calling strategies were implemented on social surveys. Also, ONS continued to provide Avoiding Refusal Training (ART) and basic non-response training. It was also recommended that the three studied outlined below were undertaken. This recommendation was endorsed by the UK Statistics Authority.

Using paradata to model non-response

Building on Beavan-Seymour’s fingings, this project (James et al. in 2010 [3]), collected further paradata for both responders and non-responders. ‘Paradata’ refers to information that relates to the process by which survey data are collected. It can also refer to basic characteristics of a sampled case such as the time of day that an interview was conducted or the physical characteristics of a sampled address in a household survey. Information is collected for both responders and non-responders there is potential to use these data for methodological modelling in order to adjust for bias which arises as a result of non-response.

For this study paradata was collated from the LFS and the Living Costs and Food Survey (LCF). These data included dwelling type, data collection mode (face-to-face, telephone), GOR (Government Office Region), day of the week when the contact was attempted, OAC (using only the 7 super group). Some descriptive analysis was carried out on these variables to provide an initial picture of the pattern of response for each variable and to provide an indication of the quality of the data. Following technical improvements made to data collection programmes a replacement LFS dataset (second quarter of 2010), was obtained to replace quarter 1 2010. Logistic regressions were conducted on both the LFS and LCF datasets to determine the influence each variable had on non-response rates.

The outcome of the logistic regression suggests that these paradata variables can be used to predict non-response. Increasing floor levels in flats was also found to consistently increase
non-response. The analysis showed a clear relationship between the paradata variables and response. Whilst this study provides strong evidence that paradata can be used to predict non-response, and highlights a number of variables that exhibit influence on the response rate, more data are needed to identify whether there are controlling factors such as socio-demographic variables nested within these paradata variables.

The findings suggest that OAC and GOR are robust paradata variables that influence non-response across at least two ONS surveys. Further research is needed to determine the amount of variance explained in non-response by paradata variables. It will also necessary to identify whether there are variables nested within parade that are responsible for the interactions observed in this research. Age of participant appears to influence response rates, however, further research is required following the 75+ adjustments to the LFS to establish if this age remains significant.

LFS Refusal Follow-up

One of the recommendations of the Quality Task Force Review of the LFS states that information on the characteristics of non-respondents should be regularly collected in order to assess and adjust for non-response bias and to improve fieldwork strategies. Following a bidding process, the ONS has been funded by Eurostat to conduct a refusal follow-up study on the LFS. The aim of the research is to provide an insight into how survey design can be improved to increase the levels of participation amongst certain sections of the population where response rates are declining as well as providing a framework for the creation of more accurate calibration total, thereby controlling the level of bias within the LFS.

The project will collect information on the characteristics of individuals who are classified as non-respondents on the LFS. Specifically, the LFS Refusal Follow-up (LFS RFU) will focus on outright refusals on wave 1 of the LFS in order to understand whether people with particular characteristics are more likely to refuse to participate. This project will also pilot a process to regularly collect the characteristics of non-responders.

Data will be collected from individuals who refuses to take part in the LFS. The individual will be asked a few short questions related to their socio-demographic characteristics. The following key attributes will be collected: age, sex, ethnic group, nationality, household composition, household size, economic status. Information will be collected face-to-face, by telephone and by self-complete postal questionnaire.

Fieldwork on the RFU will be carried out in the April to June 2011 quarter of the main LFS. Planned analysis of the data will aim to investigate the traits of these non-responders. An in-depth picture of non-responders will help to identify which groups of individuals are likely to be underrepresented in the LFS. Additional analysis will be conducted on sub-sets of the data and comparative work will be carried out between the main LFS and the subsets in order to assess whether there is potential bias on the LFS.

Next steps

The three ongoing studies, Paradata, CNRL and the RFU outlined above, investigate the two different categories of ‘non-responder households’:

  • Non-contact: when a contact cannot be made with any responsible household member 

  • Refusal: when a contact has been made but the person does not wish to be interviewed.

We will continue to improve the quality of the paradata recorded for responders and non-responders on the LFS. Further analysis will be carried out to examine the relationships between paradata and non-response.

The LFS refusal follow-up uses the ‘Re-approaching’ strategy to gather information from an individual who refuses to take part in the LFS. The data collected will help ONS to profile specific non-responders and determine whether some particular groups in society are under or over-represented. As a result, any bias that is present in the key LFS estimates due to non-response can be quantified and estimated. The final report to Eurostat is timetabled for September 2011.

The CNRLS will obtain matched records from social surveys. Linking Census data to survey data provides a rich source of auxiliary variables which can be used to assess non-response bias. It provides the opportunity to assess how non-response is changing over time, allows non-response weights to be re-calculated and a reassessment of whether non-response weights should be applied to the LFS. In addition, it provides the opportunity to measure the extent to which determinants of non-response also correlate with the response given (for respondents). Analysis is not expected to start until late 2012 as the timetable is driven by the availability of the Census data.

References

[1] N. N. Hopper – Calling pattern analysis by Output Area Classification - ONS Internal Technical Report 2008.

[2] C. Beavan-Seymour – Non-response investigation – ONS Internal Technical Report 2008.

[3] R. James – An investigation into whether it is appropriate to use paradata to model non-response in the Labour Force Survey – ONS Internal Technical Report 2010.

Background notes

  1. Details of the policy governing the release of new data are available by visiting www.statisticsauthority.gov.uk/assessment/code-of-practice/index.html or from the Media Relations Office email: media.relations@ons.gsi.gov.uk

    These National Statistics are produced to high professional standards and released according to the arrangements approved by the UK Statistics Authority.

Get all the tables for this publication in the data section of this publication .
Content from the Office for National Statistics.
© Crown Copyright applies unless otherwise stated.