1. Background

The UK Labour Force Survey (LFS) National Statistics Quality Review (NSQR) was published in February 2014. The review focused largely on implications for accuracy arising from the LFS design, fieldwork and estimation procedures. In addition, it outlined that increasing requests for single month estimates have called into question the timeliness of results. The main conclusion from the review was that the LFS enables the production of good quality estimates from the survey outputs. However, 27 specific recommendations were put forward to improve the LFS design and procedures.

Back to table of contents

2. Progress

An Office for National Statistics (ONS) response to the National Statistics Quality Review (NSQR) was published in March 2016, outlining how the high level recommendations would be addressed. Of the 27 recommendations, eight are complete or closed, four are on hold due to dependencies with other work and 15 are in progress. These cover a range of topics, a number of which are relating to continuous improvement of the questionnaire and Labour Force Survey (LFS) quality. Four recommendations are being taken forward using resource from the ONS Social Surveys Transformation programme, including investigation of a rolling data collection period, moving reference weeks and electronic data collection. Other projects are being taken forward within the LFS research team or via projects led by other experts at ONS. A number of these are due for completion in 2019, such as a project to redesign the stint areas to improve efficiency, and the evaluation of data collection through an adaptive fieldwork design in the telephone unit.

As a result of work to address NSQR recommendations, a number of methodological papers have been published:

Progress against each of the recommendations is provided in Section 4.

Back to table of contents

3. Governance

Many of the recommendations of the review have been addressed or are incorporated into other projects and therefore further developments and will be reported as part of those projects. The Labour Force Survey (LFS) Management Board, which meets quarterly and has coordinated response to the National Statistics Quality Review (NSQR) to date, continues to provide high-level oversight of the actions to address the recommendations.

Back to table of contents

4. Labour Force Survey National Statistics Quality Review response

The following table lists each of the 27 recommendations from the 2014 NSQR and their progress. Details of each recommendation including the status in 2016 are included in the following sections.

Recommendation 1: Allocate dedicated resource to continuous quality improvement on the Labour Force Survey

ONS response to recommendation, March 2016
Examples of resource allocated to continuous quality improvement are as follows.

From a response rate perspective: a Local Area Response Project Team has been set up to research fieldwork patterns and develop action plans, in interview areas where response rates have fallen most since 2011.

From a questionnaire perspective: Office for National Statistics (ONS) has implemented workshops aiming to identify questions for removal, such that questionnaire focus can be kept relevant whilst not increasing its length. Further, ONS has worked with the Department for Business, Innovation and Skills (BIS) to initiate a review of Labour Force Survey (LFS) education questions.

From the perspective of providing data to meet user needs: ONS will conduct an LFS reweighting process each year, to use the most up-to-date population estimates and projections. Further, ONS has put in place regular production of longitudinal datasets based on Annual Population Survey (APS) data, to supplement LFS longitudinal data. Data from Eurostat ad hoc modules is now available to government users on the main APS files.

From the perspective of support for data users: in 2015, in addition to providing updated LFS User guides 2, 3, 4 and 6, resource was allocated to publishing a new User guide 10 ‘Analysis of data collected by the Labour Force Survey’.

Update on progress, February 2019
This work is ongoing. Further examples of continuous quality improvement work between 2016 and 2019 are given in this section.

Response rates: a restinting project is under way and due to be completed in 2019 (see recommendation 5). The Behavioural Insights Team has carried out some investigatory work to identify some behavioural barriers to response and a number of experiments are being taken forward, such as telephone scripting and reminder texts. The concept of the Messenger Effect on advance materials is also being tested from November 2018.

Questionnaire: dedicated resource has been added to the Questionnaire Development Team. A review of the education section of the survey has been instigated to improve quality and reduce respondent burden, and a piece of work is under way to look at the collection of nationality data with the Northern Ireland Statistics and Research Agency (NISRA).

Providing data to meet user needs: the LFS reweighting process has been reviewed to use the latest mid-year population estimates and subnational population projects, producing reweighted datasets every other year at the minimum. Three-year pooled APS datasets have been produced for users from August 2017, to be released on an annual basis.

Support for data users: User guide 9 ‘Eurostat’ was updated in 2018. Separate User guide 9a created to provide information relating to EU-LFS ad hoc modules. This volume is updated annually to coincide with the publication of ad hoc module data on the APS. An update of User guide 5 ‘Classifications’ will be published in spring 2019.

Status, February 2019
Ongoing

Recommendation 2: Ensure LFS representation and liaison with ONS projects whose outcomes have major implications for design of LFS

ONS response to recommendation, March 2016
ONS is represented on Eurostat task forces, for example, those considering the proposed introduction of household income questions and the 2018 ad hoc module. For the former task force, research was conducted by ONS recommending that household income not be included in LFS; this recommendation was implemented by Eurostat.

ONS also participates in Eurostat’s biannual Labour Market Statistics Workshops and annual European LFS Methodology Conference, and will host the latter in 2016. Of particular note is Eurostat’s draft regulation IESS (Integrated European Social Statistics), targeted for introduction in 2019: ONS is actively planning and communicating necessary activities to comply with IESS. It will participate in IESS Implementation Task Forces in 2016 to 2017.

Substantial liaison has taken place with ONS’s Electronic Data Collection Programme, and going forward with the Social Statistics Transformation Programme, focusing on implementing online data collection and utilising administrative and big data.

Update on progress, February 2019
ONS has had continued involvement in Eurostat task forces and working groups and will continue to engage on an international level.

ONS Social Statistics Transformation, which covers projects to develop the question content and processes, are represented on the LFS Steering Group and Management Boards, and Social Surveys representatives also sit on the equivalent Transformational Boards, both at an operational and senior level.

Status, February 2019
Ongoing

Recommendation 3: Retain the Postcode Address File for now, but move to AddressBase in future

ONS response to recommendation, March 2016
The relevance and benefit of the Postcode Address File and AddressBase is being considered by the ONS Address Register project, which has a cross-government focus. LFS, in line with other ONS surveys, will utilise address register findings to define future sample frame methodology.

Update on progress, February 2019
The Social Statistics Transformation Programme, which will deliver a mixed-mode, administrative data-first approach to labour market statistics, will use AddressBase as its primary sampling frame. The move to this frame for the LFS and other social surveys is still being considered.

Status, February 2019
In progress

Recommendation 4: Discontinue sampling of NHS accommodation

ONS response to recommendation, March 2016
A project will address this work package. It has not been given highest priority to date given the small number of NHS addresses in the LFS sample.

Update on progress, February 2019
Not yet progressed due to priorities of other work.

Status, February 2019
On hold

Recommendation 5: Explore the efficiency of interviewer stint areas, with a view to redesigning

ONS response to recommendation, March 2016
Analysis has suggested that the current interviewer areas are inefficient, as numerous fieldwork demands have changed since the areas were defined. As a result, ONS is conducting a project to implement a new restint design, in 2016 or 2017. It is anticipated that future restinting exercises will be required, driven by developments in online interviewing and use of administrative data.

Update on progress, February 2019
The stint areas are being redesigned as part of a project, which is due to be implemented in 2019.

Status, February 2019
In progress – completion mid-2019

Recommendation 6: Explore potential for adaptive fieldwork to focus resources more efficiently

ONS response to recommendation, March 2016
A modelling approach has been developed (drawing on advice from ONS Methodology and from Southampton University) relating to cases entering week two of data collection in Telephone Operations. Domains of cases are identified and a propensity model applied to prioritise cases more efficiently. A test of the approach will be implemented in 2016.

Update on progress, February 2019
An adaptive fieldwork design trial has been run in the Telephone Unit from August 2018. Results will be evaluated in spring 2019.

Status, February 2019
In progress – completion mid-2019

Recommendation 7: Further reduce the number of wave 1 telephone interviews

ONS response to recommendation, March 2016
The proportion of wave 1 cases handled by Telephone Operations was halved in 2014, to approximately 10 to 15%. It is estimated that this helped improve wave 1 response rates by 1 to 2%.

Update on progress, February 2019
The proportion of telephone interviews in wave 1 has reduced further from January 2018 in order to further improve response. Only wave 1 cases north of the Caledonian Canal are now collected by telephone, around 1 to 2% of the Great Britain sample. Impact analysis is being carried out to determine if this had a significant effect.

Status, February 2019
Complete

Recommendation 8: Pursue international agreement on a more standardised approach to reporting response and for greater clarity in Eurostat publications about differences in how response is measured and reported

ONS response to recommendation, March 2016
A Eurostat Task Force presented recommendations that were proposed for testing by some national statistical institutes (NSIs) in 2015. The results of the testing are awaited. ONS will review outcomes and consider the best way forward, taking into account harmonisation with other surveys.

Update on progress, February 2019
Not progressed due to priorities of other work.

Status, February 2019
On hold

Recommendation 9: Explore ways of exploiting the option to have a longer data collection period

ONS response to recommendation, March 2016
This is being considered as part of ONS’s Social Statistics Transformation Programme, in particular online interviewing development.

Update on progress, February 2019
This work is being considered as part of ONS's Social Statistics Transformation Programme. A longer, rolling data collection period is being used as part of a mixed-mode statistical test being carried out from October 2018.

Status, February 2019
In progress

Recommendation 10: Explore the option of using a moving reference week, in the context of the Electronic Data Collection Project

ONS response to recommendation, March 2016
This will be considered as part of ONS’s Social Statistics Transformation Programme, in particular online interviewing development.

Update on progress, February 2019
See recommendation 9.

Status, February 2019
In progress

Recommendation 11: Work with Electronic Data Collection on other issues relating to reference week and field period as well as effects on response rate and non-response bias

ONS response to recommendation, March 2016
This will be considered as part of ONS’s Social Statistics Transformation Programme, in particular online interviewing development.

Update on progress, February 2019
See recommendation 9.

Status, February 2019
In progress

Recommendation 12: The potential costs and benefits of adding a sixth wave of interviews into the rotation design be explored

ONS response to recommendation, March 2016
Analysis conducted by ONS Methodology identified that, in combination with the Annual Population Survey targets requirement, this approach would provide very limited cost benefit, while producing some concerns relating to quality of time series data. As such, it is not being progressed. However, the approach may be revisited as an aid to meeting Eurostat precision targets at local level specified in Integrated European Social Statistics regulation (IESS).

Update on progress, February 2019
Not being actively progressed.

Status, February 2019
Closed

Recommendation 13: Ensure that annual reviews are held aimed at reducing or containing interview length

ONS response to recommendation, March 2016
A workshop has been conducted every six months over the past two years, with members of the LFS Steering Group. These workshops focus on use of existing questions, challenging the need for asking them at all, asking them on each type of dataset and at the current frequency of collection. The interview length has not increased since the workshops were introduced. The process has been extended to include a review of education questions on LFS, as notified to wider users via StatsUserNet.

Update on progress, February 2019
Interview length is managed via the LFS Steering Group, with a current strategy to only introduce new questions if others with equal burden are removed. However, there is a requirement to include additional topics to meet the government's needs, and as part of EU-Integrated Social Statistics (IESS) regulation in 2021.

The education section is being reviewed in collaboration with the Department for Education (DfE) to aim to improve quality and reduce question length.

Status, February 2019
Ongoing

Recommendation 14: Assess the costs and benefits associated with non-response and attrition weighting and the desirability of integrative weighting

ONS response to recommendation, March 2016
Analysis has been conducted focusing on the impact of attrition on estimates, with an associated paper planned for publication in the ONS Survey Methodology Bulletin. This analysis is to be reviewed and decisions to be taken regarding implementation of recommendations.

Update on progress, February 2019
A paper on LFS attrition was published in the spring 2016 Survey Methodology Bulletin (PDF, 1.5MB), which did not recommend implementing a regression-based attrition adjustment in the LFS production system.

A report on Exploring the use of external data to assess for observed bias in Labour Force Survey estimates has been published in September 2018, recommending that a non-response bias adjustment should not be made to the LFS at this point in time. Further work is recommended using additional administrative data.

Status, February 2019
Ongoing

Recommendation 15: Assess the feasibility of adding APS boost samples to main LFS estimates

ONS response to recommendation, March 2016
Initial analysis conducted in 2015 will be considered further with particular reference to meeting Integrated European Social Statistics (IESS) precision requirements.

Update on progress, February 2019
This is being considered as part of the solution to meet Eurostat precision requirements under IESS. The sample size of the survey will also be considered as part of the Social Statistics Transformation Programme.

Status, February 2019
In progress

Recommendation 16: Assess more fully the process of attrition, its consequences for LFS estimates and promote sample retention

ONS response to recommendation, March 2016
Consequences of attrition are addressed against recommendation 14. Attrition will be considered in more detail as part of Social Statistics Transformation.

Update on progress, February 2019
See recommendation 14.

Status, February 2019
Ongoing

Recommendation 17: Develop a new monthly weighting methodology based on that used for quarterly estimates

ONS response to recommendation, March 2016
This work was carried out successfully as part of the project to develop the state space modelling approach (recommendation 24).

Update on progress, February 2019
Complete

Status, February 2019
Complete

Recommendation 18: Investigate opportunities for re-weighting the longitudinal flow estimates

ONS response to recommendation, March 2016
A paper was published with findings, in the Survey Methodology Bulletin, autumn 2015, which concluded that the current methodology is appropriate and will be retained.

Following this review, more detailed flows statistics have begun to be published on a quarterly basis. These include estimates of job to job flows and flows between different occupational groups.

Update on progress, February 2019
The first recommendation from the autumn 2015 report was to consider the replacement of the current economic activity calibration variable with that of a lower breakdown level. This recommendation is currently being reviewed.

Status, February 2019
In progress

Recommendation 19: Continue to produce rolling quarterly estimates as the preferred monthly headline indicator

ONS response to recommendation, March 2016
In place, unchanged.

Update on progress, February 2019
Complete

Status, February 2019
Complete

Recommendation 20: Apply minor modifications to the seasonal adjustment of single month estimates

ONS response to recommendation, March 2016
Completed – series have been reviewed and seasonal adjustment has been improved. Wave data were previously using rolled forward seasonal factors but are now fully seasonally adjusted every month.

Update on progress, February 2019
Complete

Status, February 2019
Complete

Recommendation 21: Continue with current single month estimates benchmarking method

ONS response to recommendation, March 2016
In place – continuing as usual until the state space modelling approach is introduced.

Update on progress, February 2019
Complete

Status, February 2019
Complete

Recommendation 22: Commence regular production of estimates of sampling variability at earliest opportunity

ONS response to recommendation, March 2016
Not to be developed given progress against recommendation 24.

Update on progress, February 2019
Not being actively progressed.

Status, February 2019
Closed

Recommendation 23: Improve metadata for single month estimates

ONS response to recommendation, March 2016
From January 2016, wave analysis has been published alongside single month estimates with clearer series descriptions.

Update on progress, February 2019
Complete

Status, February 2019
Complete

Recommendation 24: Continue the existing project to develop state space models, but with higher priority

ONS response to recommendation, March 2016
A prototype state space model has been developed that produces single month estimates of unemployment, employment and inactivity from quarterly LFS microdata. In short, the estimates represent time series with the identifiable survey design effects removed.

The methodology has undergone initial peer review and is sufficiently developed to enable the monthly publication of an experimental time series. The plan is to publish these Experimental Statistics alongside the existing LFS single month estimates from mid-2016 onwards.

In the longer-term it is anticipated that the modelled estimates will replace the existing estimates and will be provided for use in the Eurostat monthly unemployment release. The quarterly LFS aggregates will remain as the headline National Statistics.

Update on progress, February 2019
ONS continues working on the prototype state space model. Additional work is required before the estimates can be published alongside the current single month estimates.

Status, February 2019
In progress

Recommendation 25: Developments of the communal establishment sampling frame from Beyond 2011 are considered for their potential for sampling purposes in the LFS

ONS response to recommendation, March 2016
This is a longer-term recommendation of the National Statistics Quality Review. In the interim the estimated impact of not surveying communal establishments in the LFS was reviewed as part of the project that reconciles LFS estimates of jobs with those produced by ONS business surveys. The review took account of information from the 2011 Census of Population and a methodological article (PDF, 134KB) was published.

Update on progress, February 2019
This is not being actively progressed at this point in time. The recommendation may be taken forward using AddressBase as a sampling frame.

Status, February 2019
On hold

Recommendation 26: When available, the potential of improving the weighting using administrative data be examined

ONS response to recommendation, March 2016
This will be considered as part of Social Statistics Transformation.

Update on progress, February 2019
This work has not yet started as ONS does not yet have access to suitable administrative data.

Status, February 2019
On hold

Recommendation 27: Introduce web data collection when systems and design issues are resolved

ONS response to recommendation, March 2016
This will be considered as part of Social Statistics Transformation.

Update on progress, February 2019
Development of an online data collection system is continuing as part of the Social Statistics Transformation work. Initial testing in 2017 established that a good initial response rate (about 25 to 30%) can be achieved to an online first survey invite.

A statistical test is being carried out from October 2018 to establish what mixed mode response rate can be achieved with an online first approach, and how this compares with the current LFS. At the same time, this test will compare estimates from the new survey with comparable estimates in the current LFS. In addition, there will be early tests of new data collection systems.

Status, February 2019
In progress

Back to table of contents

Contact details for this Methodology

Debbie Curtis
socialsurveys@ons.gov.uk
Telephone: +44 (0)1633 45 5678