The UK Statistics Authority assessment of the Labour Force Survey (LFS) for National Statistic status, reported on in March 2010, identified areas where ONS could strengthen its compliance with the Code and listed 7 requirements essential to the designation as National Statistics. The first of these was to communicate clearly the falling response rates on the LFS and the implications this would have for data quality. The purpose of this brief interim report is to provide an update on the non-response surveys conducted by ONS to address this requirement.
The response rate for the UK Labour Force Survey is currently at its lowest level since the survey started in 1970. Low survey response rates not only affect the confidence limits at the Local Area Level, but also potentially induce non-response bias in the key LFS estimates. One of the practices listed under Principle 4 of the UK Statistics Authority Code of Practice states that users should be "informed about the quality of statistical outputs, including estimates of the main sources of bias and other errors”.
A non-response program was commenced in 2007 to analyse falling response rates in social surveys. Research carried out by Hopper et al.  in 2008 examined non-response patterns for six different surveys including the LFS and found that households residing within certain OAC super-groups are usually under- represented because they are more difficult to contact. Hopper also investigated calling patterns and recommended a variable pattern be implemented. Based on Hopper’s results, the ONS conducted a number of studies specific to the LFS between 2008 and 2010.
A study by Beavan-Seymour in 2008  took into account demographic characteristics relating to tenure, household composition, ethnicity and nationality, the Output Area Code (OAC) coding frames, as well as sampling frame variables e.g. Government Office Region (GOR). The study found that there was a correlation between the geo-demographic characteristics and the response rate. The study also concluded that refusal rates were consistent across OAC codes but not contact rates. Following this work more consistent contact patterns and different calling strategies were implemented on social surveys. Also, ONS continued to provide Avoiding Refusal Training (ART) and basic non-response training. It was also recommended that the three studied outlined below were undertaken. This recommendation was endorsed by the UK Statistics Authority.
Building on Beavan-Seymour’s fingings, this project (James et al. in 2010 ), collected further paradata for both responders and non-responders. ‘Paradata’ refers to information that relates to the process by which survey data are collected. It can also refer to basic characteristics of a sampled case such as the time of day that an interview was conducted or the physical characteristics of a sampled address in a household survey. Information is collected for both responders and non-responders there is potential to use these data for methodological modelling in order to adjust for bias which arises as a result of non-response.
For this study paradata was collated from the LFS and the Living Costs and Food Survey (LCF). These data included dwelling type, data collection mode (face-to-face, telephone), GOR (Government Office Region), day of the week when the contact was attempted, OAC (using only the 7 super group). Some descriptive analysis was carried out on these variables to provide an initial picture of the pattern of response for each variable and to provide an indication of the quality of the data. Following technical improvements made to data collection programmes a replacement LFS dataset (second quarter of 2010), was obtained to replace quarter 1 2010. Logistic regressions were conducted on both the LFS and LCF datasets to determine the influence each variable had on non-response rates.
The outcome of the logistic regression suggests that these paradata variables can be used to predict non-response. Increasing floor levels in flats was also found to consistently increase
non-response. The analysis showed a clear relationship between the paradata variables and response. Whilst this study provides strong evidence that paradata can be used to predict non-response, and highlights a number of variables that exhibit influence on the response rate, more data are needed to identify whether there are controlling factors such as socio-demographic variables nested within these paradata variables.
The findings suggest that OAC and GOR are robust paradata variables that influence non-response across at least two ONS surveys. Further research is needed to determine the amount of variance explained in non-response by paradata variables. It will also necessary to identify whether there are variables nested within parade that are responsible for the interactions observed in this research. Age of participant appears to influence response rates, however, further research is required following the 75+ adjustments to the LFS to establish if this age remains significant.
One of the recommendations of the Quality Task Force Review of the LFS states that information on the characteristics of non-respondents should be regularly collected in order to assess and adjust for non-response bias and to improve fieldwork strategies. Following a bidding process, the ONS has been funded by Eurostat to conduct a refusal follow-up study on the LFS. The aim of the research is to provide an insight into how survey design can be improved to increase the levels of participation amongst certain sections of the population where response rates are declining as well as providing a framework for the creation of more accurate calibration total, thereby controlling the level of bias within the LFS.
The project will collect information on the characteristics of individuals who are classified as non-respondents on the LFS. Specifically, the LFS Refusal Follow-up (LFS RFU) will focus on outright refusals on wave 1 of the LFS in order to understand whether people with particular characteristics are more likely to refuse to participate. This project will also pilot a process to regularly collect the characteristics of non-responders.
Data will be collected from individuals who refuses to take part in the LFS. The individual will be asked a few short questions related to their socio-demographic characteristics. The following key attributes will be collected: age, sex, ethnic group, nationality, household composition, household size, economic status. Information will be collected face-to-face, by telephone and by self-complete postal questionnaire.
Fieldwork on the RFU will be carried out in the April to June 2011 quarter of the main LFS. Planned analysis of the data will aim to investigate the traits of these non-responders. An in-depth picture of non-responders will help to identify which groups of individuals are likely to be underrepresented in the LFS. Additional analysis will be conducted on sub-sets of the data and comparative work will be carried out between the main LFS and the subsets in order to assess whether there is potential bias on the LFS.
The Census Non-response Linked study (CNRL) is another alternative for obtaining matched records in the absence of survey response. The method aims to link all households that have been sampled in a social survey within a few months of the Census date to their Census record. This allows us to explore determinants of response propensity by comparing characteristics of responders with those of non-contacts and refusals. This approach may lead to the development of optimal data collection strategies to target households more likely to be associated with non-response.
The primary purpose of the CNRL is to compare the characteristics of non-respondents and respondents, estimate non-response bias and evaluate methods of non-response adjustment. The 1991 and 2001 linked datasets have been used to develop non-response weights which are used to help compensate for non-response in social surveys.
The participating surveys in the 2011 study include:
Labour Force Survey
Annual Population Survey
General Lifestyle Survey
Living Costs and Food Survey
The study will investigate the characteristics of non-respondents, assess non-response bias and investigate the effect of re-weighting methods that might be used to correct for non-response bias. For each call to a sampled address interviewers will record a number of key observations including type of accommodation, existence of barriers (such as locked gate, warden control, and so on), visible security (such as burglar alarm, bars/grills on windows and so on), their personal safety in area, outcome of call, information left at address, how first contact was made with the household, person (male/female) first contact made with, and approximate age of the main person contacted. Fieldwork will take place from March 2011 through to June 2011.
The planned analysis for the 2011 CNRLS will be similar to that offered for the 2001 including analysis of the characteristics of non-responding households and persons (such as household composition, housing tenure and characteristics of the accommodation, characteristics of the Household Reference Person, characteristics of all adults in non-responding households). Multivariate analysis of non-responding households will be conducted. The analysis will use chi-squared analysis and logistic regression.
The three ongoing studies, Paradata, CNRL and the RFU outlined above, investigate the two different categories of ‘non-responder households’:
Non-contact: when a contact cannot be made with any responsible household member
Refusal: when a contact has been made but the person does not wish to be interviewed.
We will continue to improve the quality of the paradata recorded for responders and non-responders on the LFS. Further analysis will be carried out to examine the relationships between paradata and non-response.
The LFS refusal follow-up uses the ‘Re-approaching’ strategy to gather information from an individual who refuses to take part in the LFS. The data collected will help ONS to profile specific non-responders and determine whether some particular groups in society are under or over-represented. As a result, any bias that is present in the key LFS estimates due to non-response can be quantified and estimated. The final report to Eurostat is timetabled for September 2011.
The CNRLS will obtain matched records from social surveys. Linking Census data to survey data provides a rich source of auxiliary variables which can be used to assess non-response bias. It provides the opportunity to assess how non-response is changing over time, allows non-response weights to be re-calculated and a reassessment of whether non-response weights should be applied to the LFS. In addition, it provides the opportunity to measure the extent to which determinants of non-response also correlate with the response given (for respondents). Analysis is not expected to start until late 2012 as the timetable is driven by the availability of the Census data.
 N. N. Hopper – Calling pattern analysis by Output Area Classification - ONS Internal Technical Report 2008.
 C. Beavan-Seymour – Non-response investigation – ONS Internal Technical Report 2008.
 R. James – An investigation into whether it is appropriate to use paradata to model non-response in the Labour Force Survey – ONS Internal Technical Report 2010.
Details of the policy governing the release of new data are available by visiting www.statisticsauthority.gov.uk/assessment/code-of-practice/index.html or from the Media Relations Office email: firstname.lastname@example.org
These National Statistics are produced to high professional standards and released according to the arrangements approved by the UK Statistics Authority.