Comparison of labour market data sources

This page is designed to be a working reference document. We intend to update this page as we develop a better understanding of our sources and the differences between them.

Labour market statistics measure many different aspects of work and jobs and provide an insight into the economy.

A framework for labour market statistics is used to describe the major concepts that exist within the labour market and their relationship to each other. The framework is based on the concepts of labour supply and demand, which together provide a comprehensive picture from the perspective of both households and businesses. This approach is long-established and has wide international acceptance, including by the International Labour Organization (ILO).

Labour supply consists of people who are employed as well as those people defined as unemployed or economically inactive. It is measured using surveys of households. Labour demand is represented by employers, who have a need for work to be done and who offer compensation for this work (earnings) to the employees who undertake it. Labour demand is mainly measured using surveys of businesses. There is also some administrative data used as part of labour market statistics, and information about earnings is collected from both households and businesses.

As part of wider transformation plans, we have been seeking to identify and introduce new data sources, such as those from government departments and the private sector, to give a quicker and/or more complete picture of our economy.

The coronavirus (COVID-19) pandemic and the government response have had a significant effect on the UK labour market. Many businesses have ceased operating or have had to change their working practices, while recent government interventions have allowed for workers to be paid while away from work because of coronavirus-related reasons. The introduction of social distancing has also changed the way individuals work and their ability to look for and find employment.

The combined effect of the impacts has not only changed the UK labour market but also our ability to measure it (for more information, see Coronavirus and the effects on UK labour market statistics). Most of our information relies on data collected from surveys of households and businesses. The temporary closure of businesses, increase in remote working and introduction of social distancing are having an impact on collecting survey data from households and businesses.

Ensuring we provide the information necessary as the situation unfolds, through multiple data sources alongside changing how our surveys operate and reviewing methodology, brings about its own challenges when giving a comprehensive picture of the labour market.

This page compares the strengths and weaknesses of the main data sources used to produce the labour market figures, including the advantages of new administrative data sources and limitations of some of our published figures. We intend to update this page as we develop a better understanding of the new sources.

The concept of employment, which is about people and whether or not they have work, differs from the concept of jobs, since a person can have more than one job and some jobs may be shared by more than one person.

The Labour Force Survey employment versus HMRC’s Pay as You Earn Real Time Information

There are two main sources of information about employment:

• Labour Force Survey (LFS) employment

• Experimental Statistics from Real Time Information (RTI) payrolled employees (Experimental Statistics)

The LFS and HM Revenue and Customs’ (HMRC’s) Pay As You Earn (PAYE) RTI measure different things. The LFS follows the International Labour Organization (ILO) definition of employment, which consists of people aged 16 years and over who did one hour or more of paid work in the previous week and those who had a job they were temporarily away from (for example, because they were on holiday, off sick or, now, on furlough).

RTI measures the number of people who are being paid through the PAYE system via company payrolls. It could include self-employed people if someone has both an employee job and is self-employed (regardless of which is their main job) and does not include those whose jobs are not part of a company payroll or those whose company does not pay them through a PAYE scheme.

The RTI data used for publication come from a dataset processed for use in statistics, rather than the raw operational data, a separate article details the methodologies used.

Main differences

There are a number of differences between these sources, and more detail of these differences can be found in Annex 1.

Coverage is different since RTI only includes individuals who are being paid through the PAYE system via company payroll, whereas the LFS employment includes all forms of employment. LFS employment is estimated by a sample from UK residential addresses that excludes people living in some communal establishments (NHS accommodation is included, while residential care homes, prisons and defence establishments are excluded). The LFS will cover those earning below the lower earnings limit as well as the self-employed, those on government training schemes (some of which will be included in RTI), unpaid family workers and those who were employed, but not paid, during a particular reference period.

The LFS is collected via an interviewer, giving opportunity to clarify answers and grouping respondents into the ILO definition of employment. Responses are therefore subjective based on whether a responder feels attached to the labour market, for example, some of those temporarily away from work that they expect to return to will consider themselves as employed, while others might not. Additionally, a respondent self-defines their employment status between employee or self-employed; the definition of self-employment is those who in their main employment work on their own account, whether or not they have employees. Some of those who are actually paid through payroll might classify themselves as self-employed as they own the business. RTI data are collected through employers’ legal obligation to provide PAYE information to HMRC; they are based on individuals receiving pay through payroll and do not fully align with international definitions of employment. RTI classifies any person with an employee job as being an employed person, whereas LFS only classifies a person as being an employee if their main job is an employee job.

There are differences in reference periods: RTI is an average of the estimated number of employees in paid employment on each respective day of the month, while the LFS series is based on interviews regarding a respondent’s labour market situation within a reference week. The LFS estimates are then calculated based on interviews from reference weeks across a three-month period.

Since the LFS is a survey, bias from non-response and sampling variability is another explanation for some differences seen, although the Office for National Statistics (ONS) invests significant effort and resources into obtaining high response rates. The LFS is weighted to a version of already published population projections, rather than being used to determine the size of the population. Consequently, it can only ever apportion to an existing population projection, with estimates always summing back to that original population total. The LFS is not designed to measure short-term changes in the population. In October 2020, the LFS was reweighted to account for the impact of the coronavirus (COVID-19) pandemic on survey interviewing methods. There may also be further refinements to the methodology, that lead to further improvements and revisions in the future. We will continue investigating these options and implement them if they are of sufficiently significant benefit.

RTI payments are imputed to forecast tax submissions expected (but not yet received) for each employment. These are then adjusted using calculated grossing factors reflecting the expected probability those imputed payments will eventually be received. However, this process predominantly affects the more recent history and, consequently, affects the most recent periods much more than earlier months.

Comparison of latest data

The two sources are related and would often have similar properties and move in similar ways. However, because they measure different things, there are times when you would not expect them to follow similar paths. Here, we look at the differences between employment and employees from the LFS and payrolled employees from RTI.

Figure 1 shows a comparison of a three-month moving average of the headline RTI employees series and the headline LFS employment and employee series. As shown, the LFS employment series is consistently higher as this includes those self-employed, unpaid family workers and those on government training schemes. Between 2014 and 2019, the three series follow a similar trend. Since March 2020, while RTI has seen a sharper fall, LFS employment has also fallen. However, we are not seeing the same falls in the number of LFS employees.

Figure 2 shows the growth rates of employment and employees in LFS and RTI payrolled employees compared with the same period of the previous year. Over the past five years as a whole, the two series show a broadly similar rate of employment growth. However, larger differences are seen since April 2020.

The self-employed group goes some way to explaining the difference between the LFS employment and LFS employees. Since April to June 2020, we have seen record falls in self-employment, coinciding with large flows of people moving from self-employment into being an employee (as can be seen in Figure 3), although the number who had changed jobs had not increased from normal levels.

Consequently, some of the fall in self-employment comes from an increase in the number of people who have changed to classifying themselves as an employee, even though they have not changed jobs. As mentioned earlier, this categorisation is self-defined in the LFS and it is possible that recent coronavirus (COVID-19) government support schemes, such as the Coronavirus Job Retention Scheme (CJRS), have influenced respondents’ views on whether they are in fact self-employed or an employee. More work is required to understand the self-employed group, and those that are reclassifying themselves as employees.

Government legislation around off payroll working was also due to change in April 2020, bringing more alignment between different employment statuses. While implementation of this legislation has been delayed until 2021, it is possible that some firms may have already made preparations to be compliant by April 2020, which could also possibly have an impact on the numbers of employees.

Since February 2020, the number of RTI payroll employees has fallen by around 767,000; the larger falls were seen at the start of the coronavirus pandemic. In comparison, the number of people in employment according to the LFS has fallen by around 566,000 between December to February 2020 and July to September 2020 (over the same period LFS employees fell by 9,000). There were also around 210,000 people away from work because of the pandemic and receiving no pay in September 2020.

Those who are not receiving pay through payroll should not be included in the RTI figures; the RTI definition states that a person must not have missed more than one period of pay (that is, for someone paid weekly, not to miss more than one week’s pay, for someone paid monthly not to miss more than one month) to be included in the count of payrolled employees. However this group would remain within the LFS while they still reported having jobs, which goes some way to explaining the greater fall in the RTI figures.

Figure 4 shows the difference in LFS employees since February 2020 and those away from jobs and not being paid from the weekly LFS data, alongside the fall in RTI payrolled employees. The falls to LFS employees have been relatively small, in comparison with the fall in RTI payrolled employees, even when adding the number of people away from jobs and not being paid, this difference between the sources persists.

Similarly, Figure 5 shows the fall in LFS employment and those away from jobs and not being paid from the weekly LFS data, alongside the fall in RTI payrolled employees. While the weekly LFS employment data are volatile, they show a similar story since February 2020 to that of RTI.

Summary

We go some way to explaining some of the differences between LFS and RTI in this article but there is more work to be done. While a full reconciliation will not be possible, the ONS will work closely with HMRC to further understand the differences and quantify where possible.

Both the LFS and RTI provide a measure of people who are working, and each have their own uses. RTI provides a timelier estimate of payroll employees for both UK and regionally but only includes those who are paid through the payroll. Also, it does not have the detail that is available in LFS; however, HMRC are continuing to develop their outputs and may include more breakdowns in future.

LFS employment includes employees and the self-employed as well as people on government training schemes and people who work unpaid for family businesses; therefore, it gives a more comprehensive picture of the labour market. The LFS employment data are available by many different breakdowns including sex, age, disability, ethnicity, occupation and industry.

The LFS-based employment levels and rates are also available back to 1971 and can be used when considering changes over longer periods of time.

There are two measures that are commonly used to understand spare labour supply. Unemployment, as measured through the Labour Force Survey (LFS) and the Claimant Count, based on administrative data from benefits systems. Both important measures of spare labour capacity in the UK economy however they each record different aspects of the non-utilisation of labour.

LFS unemployment versus Claimant Count

The official estimates of unemployment for the UK are provided by the Office for National Statistics (ONS). These are produced from the LFS in line with the International Labour Organization (ILO) definition of unemployment. This states that a person is unemployed if they are either:

• without a job, want a job, have actively sought work in the last four weeks, and are able to start work within the next two weeks

• out of work, have found a job and are waiting to start it in the next two weeks

Respondents' status is determined by their responses to a number of questions about their labour market activity. Only those people whose answers indicate that they satisfy these conditions are included in the UK unemployment estimates, which are published monthly in the Labour market overview statistical bulletin. Since the LFS is a survey, bias from non-response and sampling variability is another explanation for some differences seen.

The Claimant Count seeks to measure the number of people claiming benefit principally for the reason of being unemployed.

The Claimant Count has generally been a count of the appropriate benefits within the UK's current benefit regime that best meet that criteria, and therefore sensitive to any changes in the benefit regime. Currently, this is a combination of claimants of Jobseeker's Allowance (JSA) and claimants of Universal Credit (UC) who fall within the UC "searching for work" conditionality.

Those claiming unemployment-related benefits (either JSA or in the UC searching for work conditionality group) may be:

• wholly unemployed and searching for work
• employed but with low income and/or low hours, that make them eligible for unemployment-related benefit support
• under certain circumstances, not required to seek work and therefore economically inactive

UC, which has been rolled out to replace the previous income-based element of JSA, along with a range of other income-based benefits, is designed to be a broader benefit, covering more people with underlying eligibility than those who claimed JSA. As a household benefit, the holistic situation of all members of the household are considered, and their obligations to look for work considered. Therefore, the roll-out itself has caused notable changes to the level of the Claimant Count, significantly closing the previously existing gap between the count and unemployment levels. Consequently, the measure is currently an Experimental Statistic, as movements are not wholly caused by changes in the labour market.

The Department for Work and Pensions (DWP) also publishes an alternative Claimant Count; this Official Statistic seeks to model a data time series of the number of people that would be claiming unemployment-related benefits if UC had been fully rolled out in 2013. As such the series should better reflect changes in the labour market, by eliminating the impact of the rollout.

As with unemployment, the gap between the Claimant Count and the Alternative Claimant Count has narrowed as the UC rollout has progressed. In theory, when the rollout and migration to UC has been completed, the Claimant Count and Alternative Claimant Count should merge to the same statistic. However, there currently are some definitional differences preventing this, because of information becoming available later than the timetable for producing the Claimant Count allows. Hence the Alternative Claimant Count is lagged by an additional month

Main differences

The main differences between the two definitions can be summarised as:

• people who are unemployed are not necessarily eligible for unemployment benefits or may choose not to claim even when they are eligible; further details can be found in Annex 2

• people who are eligible for unemployment-related benefits may not fit the definition of unemployment (for example, it is possible to be employed or inactive while claiming benefits included in the Claimant Count)

• Claimant Count statistics are based on a particular "count date" falling on the second Thursday of the month, whereas the LFS unemployment figures are based on interviews across the whole of a three-month period

• the headline Claimant Count statistics do not include 16- to 17-year-olds because of historical changes in coverage of the benefits involved; also, people above State Pension Age are far less likely to fall within this part of the benefit regime

• LFS unemployment statistics are based on all people meeting the unemployment definition aged 16 years and over (under certain circumstances, 16- to 17-year-olds may fall within the Claimant Count and are included in some breakdowns but not in the headline Claimant Count)

Comparison of latest data

The ONS publishes a regular comparison of the difference between LFS unemployment and the Claimant Count as part of the monthly Labour market overview statistical bulletin (see Table X05). This comparison allows for the differences in age and time coverage of the two series, and for the ineligibility of full-time students, but still reflects other definitional differences.

Figure 6 shows that over time, the two measures have both converged and diverged. It is noticeable that in 2009 and 2010, even though both measures increased, the gap did not narrow. The levels of LFS unemployment remained consistently higher than the Claimant Count from early 1997 to mid-2019, although they moved closer together over the course of the rollout of UC. During the coronavirus (COVID-19) pandemic, Claimant Count figures have seen record increases while LFS unemployment levels have increased less and far more gradually.

The claimant count statistics are an accurate count of the number of people claiming unemployment- related benefits (and remain so under COVID-19).

A number of enhancements and easements have been made to Universal Credit, which impact the statistics:

• temporary suspension of the claimant commitment and work search requirements
• for the self-employed, relaxation of the Minimum Income Floor requirements
• issuing of COVID isolation notes to replace fit notes

In addition, claimants are accessing UC as a “top-up” to government support packages (such as the Coronavirus Job Retention Scheme and Self-Employment Income Support Scheme) to legitimately claim unemployment benefits whilst “furloughed”. A proportion of unemployed claimants will be “employed” – furloughed, or with low earnings or hours of paid work.

During the pandemic, we have seen a significant increase in people identifying themselves as temporarily away from paid work that they expect to return to. There are also a number of people who have said that they are temporarily away from work for COVID-19-related reasons and are not receiving any pay. Given that these people are not getting paid, it is likely they would be eligible for benefits, but they have stayed within the LFS definition of employed. During April to June, there were over 470,000 of these people, who would have remained in the LFS definition of employment, many of whom would have started claiming benefits. In July to September there were still an estimated 215,000.

During the early months of the pandemic, there was an unprecedented shift from LFS unemployment to economic inactivity. Many of these started returning to unemployment in later months, having not met the ILO definition of searching for and being available for work during the national lockdowns. While this move into economic inactivity would have suppressed the increase in unemployment during this time, it is likely that they would have continued to claim benefits and stayed within the Claimant Count.

We have also looked in a little more detail at people on the fringes of unemployment for July to September 2020, those that under ILO definitions are not LFS unemployed but who may be included in the Claimant Count. These include the inactive group who may begin to seek work and those temporarily away from work for coronavirus-related reasons, without earnings. This group decreased from 2.18 million to 1.72 million between April to June 2020 and July to September 2020 (see Figure 7).

For July to September 2020, an estimated 1.62 million people were unemployed, up by 318,000 on the year. The annual increase in unemployment is driven by those unemployed for up to six months, up by 224,000 on the year to 1.04 million. However, the number of those unemployed for over 12 months has also increased by 30,000 on the year. To estimate duration of unemployment, LFS respondents are asked how long they have been looking for work. Respondents are unlikely to discount short periods where they were not looking from this. Consequently, those who briefly stopped looking for work in the earlier stages of the pandemic, and were therefore classified as economically inactive, are likely to return to unemployment duration estimates in longer-term categories. It is unlikely that these people who became inactive for a while would have flowed out of the Claimant Count.

Summary

LFS unemployment, Alternative Claimant Count and Claimant Count measure different things. However, unemployment and the measures of the Claimant Count will normally perform similarly under similar labour market conditions. This means that while they do not measure unemployment, the Claimant Count measures can be useful indicators for how unemployment moves over time, or compares between different geographic areas.

The LFS-based headline unemployment estimates are best used to describe national and regional levels and rates of unemployment at any one point in time and when considering changes over longer periods of time.

Because of not being subject to sampling variability, Alternative Claimant Count data are generally best used to better understand short-term changes in the labour market but aren’t published as frequently or timely as the Claimant Count. The Claimant Count is also not subject to sampling variability but can be used to help understand the relative unemployment position within and between local areas. However, changes in the Claimant Count measures seen at the start of the pandemic do not wholly reflect changes in unemployment, because some people turning to the benefit system as support would not necessarily have lost a job at that time.

As stated earlier, the number of people with jobs is not the same as the number of jobs.

This is because a person can have more than one job.

Estimates of jobs can be compiled from a number of sources, including Short-Term Employment Surveys (STES), the Quarterly Public Sector Employment Survey (QPSES), the Business Register and Employment Survey (BRES) and the Labour Force Survey (LFS).

STES is a group of surveys that collect employment information from private sector businesses that are used to compile Workforce Jobs (WFJ). In December of each year, Workforce jobs benchmarks the private sector employee component to BRES private sector employee estimates. The private sector employee component of WFJ is not publicly available, therefore it is not possible to reconcile the WFJ and BRES totals that are published. Workforce jobs is a quarterly measure of jobs in the UK and is the preferred measure of short-term employment change by industry. A variety of outputs are produced, including industry and region.

BRES is an annual survey and the official source of employee and employment estimates by detailed geography and industry.

WFJ versus PAYE RTI

Main differences

Similar to differences between the LFS and Real Time Information (RTI), most differences between the datasets can be attributed to coverage and content.

WFJ can be broken down into employee and self-employed jobs; the latter is sourced from the LFS.

WFJ employees are the number of employees, aged 16 years and over, that an organisation directly pays from its payroll, whereas RTI counts anyone who has a paid employee job during the reference period; it is a monthly average of daily counts of the number of individuals in payrolled employment and not the number of jobs. WFJ employees are collected through a survey with a specific count date, whereas RTI is a monthly average of daily estimates.

Imputation and weighting applied to the WFJ employees and sampling variability as a result of being collected from surveys is another explanation for some differences seen. RTI payments are imputed to forecast tax submissions expected (but not yet received) for each employment. These are then adjusted using calculated grossing factors. However, this process predominantly affects the more recent history and, consequently, affects the most recent periods much more than earlier months.

Comparison of latest data

Figure 8 shows that the series generally follow a similar trend, with RTI showing stronger growth at the start of the series and then levelling off to be more in line with WFJ growth.

Since March 2020, RTI has seen a sharper fall.

In more recent months, estimates of employees from the Monthly Wages and Salaries Survey (MWSS) have also been analysed to provide additional insight into the labour market. MWSS works on similar principles to WFJ but has very similar concepts to RTI. The analysis has shown that MWSS employees have similar outcomes to RTI (that is, similar falls since the start of the year).

WFJ versus LFS

The concept of employment (measured by the LFS as the number of people in work) differs from the concept of jobs, since a person can have more than one job and some jobs may be shared by more than one person. WFJ and LFS series are reconciled annually. The LFS, which collects information mainly from residents of private households, is the preferred source of statistics on employment.

The LFS can also be used to produce estimates of the total number of jobs in the UK by adding together the headline employment figures (which are equivalent to main jobs) and those for workers with a second job. The WFJ series, which is compiled mainly from surveys of businesses, is the preferred source of statistics on jobs by industry since it provides a more reliable industry breakdown than the LFS.

Main differences

The 2006 National Statistics Quality Review of Employment and Jobs Statistics (PDF, 4.35MB) identified a number of reasons why the LFS and WFJ estimates of jobs can differ from each other. They include:

• the LFS excludes those living in communal establishments, whereas these would be included in WFJ
• the LFS collects information on main and second job, but no information is collected on people who have three or more jobs; these would be counted within WFJ

Some of these factors can be quantified approximately using information from the LFS and other sources, while others are more difficult to measure. The measurable factors causing differences between the LFS jobs and WFJ estimates are available in Dataset X03 is published alongside these data.

Comparison of latest data

Figure 9 shows headline LFS and WFJ data and two further series adjusted to take into account the measurable factors causing differences between LFS jobs and WFJ (as published in March 2020). Over the latest comparable three-month periods, the published LFS series shows a quarterly increase of 216,000 jobs (0.6%) and the published WFJ series shows an increase of 67,000 jobs (0.2%). On an annual basis, the LFS series shows an increase of 301,000 jobs (0.9%) and the WFJ series shows an increase of 541,000 jobs (1.5%). Once accounting for the measurable factors, the adjusted LFS estimate of total UK jobs is lower than the adjusted WFJ estimate by 1.046 million (3.0% of the LFS total).

Business Register and Employment Survey (BRES)

BRES is an annual survey of around 85,000 businesses. It provides employee and employment estimates at detailed geographic and industrial levels. BRES is the primary source for employee estimates at a detailed regional and industrial level. Timelier, less detailed, employment estimates are available in WFJ. WFJ benchmarks the private sector employee component to the BRES private sector employee estimates on an annual basis. The private sector employee component of WFJ is not publicly available, therefore it is not possible to reconcile the WFJ and BRES totals that are published.

Summary

RTI counts people in employment and is not a measure of jobs.

WFJ is the preferred source of jobs broken down by region of workplace and industry when considering all jobs in a timely manner. BRES should be used where more detailed analysis of region and/or industry is required or where less timely data for employees are of interest.

Both WFJ and BRES industry data are recommended in preference to industry data from household surveys, such as the Annual Population Survey (APS) and LFS, where the industry information would be obtained from the survey respondent rather than the business. However, LFS and APS are useful when looking at more detailed breakdowns not available from the WFJ and BRES business surveys, such as analysis of hours worked or redundancies by industry.

There are four measures of public sector employment:

• Public Sector Employment (PSE), mainly collected from the Quarterly Public Sector Employment Surveys (QPSES)
• public sector employee job estimates, collected from the Business Register and Employment Survey (BRES)
• residence-based public sector employment estimates, collected from the Annual Population Survey (APS)
• residence-based public sector employment estimates, collected from the Labour Force Survey (LFS)

The Office for National Statistics (ONS) PSE statistical bulletin provides the UK's headline official estimates of public sector employment. As well as estimates of total public sector employment, breakdowns are published by government sector, broad industry and region of workplace, together with Civil Service employment by government department.

PSE estimates are based on a complete census of local government and civil service and cover all public bodies with 20 or more employees. The primary source is the QPSES. PSE estimates count every individual with an employment contract who is being paid in the public sector, including those where public sector employment would be their second job.

BRES provides workplace-based public sector employee job estimates down to local authority level. BRES collects comprehensive information from businesses representing the majority of the economy. BRES produces estimates of employees, rather than workforce jobs. Self-employed jobs, HM Forces and government-supported trainees are therefore excluded. Individuals are classified to the public sector dependent on the legal status of the organisation for which they work, as classified on the Inter-Departmental Business Register (IDBR), and follows UK National Accounts concepts and definitions. Since BRES is a business survey, jobs are classified on a workplace basis (that is, a public sector job is counted within the local authority where the organisation is located).

The APS and LFS provide residence-based PSE. The APS enables PSE estimates to be broken down to local area region of residence. Individual respondents to the APS and LFS classify themselves as working within the public or private sector. Estimates of the number of people working in the public sector from the APS and LFS are generally much higher than the ONS' official quarterly estimates of PSE. This is partially because many people who work within public sector premises, while being employed by private sector organisations, will classify themselves as working in the public sector, for example, cleaners or security guards employed by a contractor to work at public sector premises.

As can be seen from the figures in Table 1, PSE and the BRES estimates are reasonably comparable, whereas APS and LFS estimates show a much higher proportion of people employed in the public sector; this is mainly because of the self-classification of sector on the APS and LFS.

Summary

PSE, mainly collected from the QPSES, is the official source of public sector employment and the best source to use for both headline estimates and comparisons over time. As PSE is not available below regional level, local area region of workplace can be derived from BRES.

The APS and LFS generally over-estimate the number of people employed in the public sector employment but can be used when required to be broken down by personal characteristics and region of residence.

There are several data sources on earnings and income; each has its uses, strengths and limitations. The majority of the Office for National Statistics (ONS) sources are detailed in data sources of earnings and income

Average weekly earnings (AWE) is the ONS' lead indicator of short-term changes in earnings. It is designed to capture monthly changes in the AWE of employees in Great Britain. AWE is based on the Monthly Wages and Salaries Survey (MWSS), which covers employees working in businesses with 20 or more employees in all industrial sectors in Great Britain (an adjustment is made for smaller businesses, using factors derived from the Annual Survey of Hours and Earnings (ASHE)). Aggregate data are collected from businesses on total amount paid and number of people being paid. This gives timely information on short-term trends in earnings but lacks the detail available from ASHE.

Alongside AWE, ASHE is another important ONS source of earnings statistics. ASHE is our main measure of earnings in terms of structural statistics; it provides information about the levels, distribution and make-up of earnings and hours paid for employees in the majority of industries and occupations across the UK. It is also the lead source of estimates for the gender pay gap and the number of jobs paid below the National Minimum Wage or National Living Wage. The level of detail available (such as full-time versus part-time, weekly pay, hourly, annual, by occupation, industry, low-level geography, sex and age) reflects a large sample size and resource-intensive data collection, which means it is published only on an annual basis. ASHE uses a reference date in April.

HM Revenue and Customs' (HMRC's) data include information on people paid through the Pay As You Earn (PAYE) Real Time Information (RTI) system. Statistics are based on people who are employed in at least one job, and monthly estimates of aggregate pay reflect the average of such people for each day of the calendar month. As is the case for ASHE, published figures focus on median monthly pay; however, mean pay is also available and so are breakdowns by region and pay distribution.

AWE versus PAYE RTI

Main differences

Because of the different methodologies of AWE and RTI, the headline statistics for average pay levels are not directly comparable. In a distribution such as earnings, where the higher end is skewed because of a small percentage of very high earners, the mean (which is the focus of AWE estimates) will be higher than the median (which is the focus of RTI estimates). Both RTI and AWE will include anyone who has been added to a payroll for just a month, for example, to receive a bonus or similar; however, these are treated differently between the two sources. If people joined halfway through the month, RTI calendarises them (they are counted as part of an employee), whereas AWE counts them as a whole employee (which contributes to a smaller average wage).

Both RTI and AWE estimates include the earnings of those employees whose earnings were reduced for any reason. Additionally, the RTI statistics cover Northern Ireland, HM Armed Forces and government-supported trainees paid via PAYE, pay-rolled redundancy payments and signing-on fees, all of which are excluded from AWE.

Another important difference is that RTI estimates are calculated on a person basis, while AWE estimates are calculated on a job basis. As people can have more than one job each, the pay from their multiple jobs would be summed together in RTI; this difference will also cause the RTI estimates to be higher than the AWE estimates.

As with the Labour Force Survey (LFS), AWE will be subject to sampling variability, which will also lead to short-term differences between the RTI and AWE estimates.

Comparison of latest data

Figure 10 shows both sources have a similar trend over time, when based on mean earnings. The RTI series is consistently higher than AWE, with the factors described earlier contributing to this pattern. In recent months to September 2020, growth recorded in RTI is stronger than for AWE. This reflects, in part, lower percentage growth in RTI in 2019. In addition, if people being made redundant receive substantial payments, RTI estimates may increase.

The composition of the full set of jobs affects pay growth. AWE has identified that changes in the proportion employed within individual industries has had very little impact on pay growth, whereas RTI has identified that a fall in the number of job inflows has had a notable upward impact. Neither source is able to estimate the impact of changes in job by occupation within industry, and reference to LFS estimates may provide insight on this aspect.

Median RTI is a less comparable measure here, since AWE figures include bonus payments that generally create data extremes, which are not reflected in median as an estimator of average.

ASHE versus PAYE RTI

Main differences

ASHE is based on a 1% sample of employees, sampled from HMRC PAYE data in January of the same calendar year. The January sample is updated with HMRC data supplied to the ONS in April to take account of new entrants to the labour market and those who have changed jobs since January. While the sample is employees, ASHE is completed by the employer in relation to the selected employees. RTI includes anyone who has a paid employee job during the reference period (that is, it is not based on a sample).

As ASHE uses a reference period within April each year and RTI is a monthly average of daily estimates, the most comparable published figures are ASHE’s median gross weekly pay of all employee jobs, for employees on adult rates of pay and whose pay was unaffected by absence in the reference period and RTI’s non-seasonally adjusted median weekly pay for April of each year. Note that the ASHE definition was revised for 2020 only, to include any furloughed employees whose pay was affected by absence, reflecting the unusual labour market situation and existence of the Coronavirus Job Retention Scheme (CJRS).

Comparison of latest data

Figure 11 shows median weekly pay from ASHE and RTI. The series trend well over time, but ASHE is consistently higher than RTI. This is caused in part by the exclusion of workers with absences and those not on adult pay rates in the published ASHE estimates; analysis of ASHE including these employee jobs increases the ASHE median by close to £20 (of an observed difference between RTI median and ASHE median of approximately £65) in 2020. Another important driver of difference - as yet not quantified - is that RTI figures measure pay per person, while the ASHE data measure pay per job. As a person can have several jobs, the pay from each job is summed for a person in the RTI data and pay per person will be higher than pay per job. The ONS will explore the impact of this factor.

ASHE versus AWE

A common misconception of the headline ASHE and AWE figures is that they are measures of pay rises. However, they are both designed to estimate earnings of all employees in the economy at a single point in time and therefore measure the change in average pay, rather than the average of changes in pay. This is the reason why average pay can increase without anyone having had a pay rise.

Main differences

Although ASHE and AWE measure similar aspects of the labour market, they are two different measures of employee earnings that can exhibit notably distinct differences when compared against one another. ASHE benefits from a large sample size that, coupled with the array of individual and geographic characteristics that are also gathered and a longitudinal aspect, allows for more detailed analysis than other sources of earnings data. However, as it is an annual survey, it is not as timely as other measures of earnings and there are known coverage issues of bonus and incentive payments resulting from the reference period (the highest bonuses are paid in March, rather than April).

AWE does not include information on businesses with fewer than 20 employees or detailed characteristics that ASHE contains, which (although compensated for by the inclusion of estimates calculated using factors extracted from ASHE) means some changes in earnings resulting from compositional effects could be missed. However, it has better coverage of bonus and incentive payments and provides a much timelier estimate of trends in earnings.

A detailed comparison of these sources, their strengths and their weaknesses was published in An overview of and comparison between ASHE and AWE: 2017.

Comparison of latest data

Figure 12 shows that the two series track each other closely up to 2010 but there is a notable divergence between the series for a few years before returning to general stability and narrowing from 2016. Some potential reasons for divergence in the two series are changes between occupational and industrial classifications, methodological change for AWE in the move from the Average Earnings Index (AEI) to AWE in 2010 and reclassification of banks, further education and large companies between public and private sectors.

Summary

More work is required to align the sources where possible to better understand the differences and where possible quantify.

The LFS also collects information on the earnings of employees. While the earnings data are known to be underestimated in the LFS they are a useful source of breakdowns not available in our other sources such as ethnicity and education. Further work will be carried out to compare LFS earnings with RTI.

ASHE is the preferred series for measuring levels of pay and detailed breakdowns because of its granularity, whereas AWE is more suitable for measuring rates of change as well as impact of bonus payments as it is a more frequent series. The frequency of AWE also allows for comparison with measures of inflation to calculate changes in real earnings.

RTI is the best source for timely earnings data by region and distribution.

ONS vacancy survey versus Adzuna

Vacancies are defined as positions for which employers are actively seeking recruits from outside their business or organisation. Vacancies over time are indicators of the demand for labour.

Official vacancy estimates are based on the Office for National Statistics (ONS) Vacancy Survey; this is a survey of businesses designed to provide estimates of the stock of vacancies across the economy, excluding agriculture, forestry and fishing (a small sector for which the collection of estimates would not be practical) and private households.

In response to the coronavirus (COVID-19) pandemic and the consequent demand for more timely data, the ONS has started publishing experimental figures for job adverts provided by Adzuna, an online job search engine that collates information from thousands of different sources in the UK. These range from direct employers' websites to recruitment software providers to traditional job boards, thus providing a comprehensive view of current online job adverts. More information about the methodology is available.

Main differences

The Vacancy Survey covers businesses in Great Britain only, although estimates for the UK are derived by grossing up the data for Great Britain, along with information about employment in Northern Ireland businesses. Businesses in Northern Ireland are not surveyed because of the risk of overlap with responses to other surveys conducted by Northern Ireland departments.

Adzuna has coverage of all job adverts in the UK but because of the fact that this source is limited to online vacancies, there will be some adverts missed such as casual work advertised through word-of mouth and internal vacancies, which are filled using other headhunting methods. While we are not able to estimate how many adverts are missed in the online jobs adverts, analysis has shown that larger businesses are more consistent between the two sources.

The ONS Vacancy Survey is based on a reference date – the first Friday of the month unless it is the first day of the month, in which case it is the second Friday of the month – and is published within six weeks of the reference date. Industrial category and organisation size breakdowns are available at UK level; however, no geographic data are available.

Adzuna weekly online job adverts publish vacancies with a lag of six days. Any vacancies that are not advertised online would not be included in the online job advert estimates. Data are available by Adzuna industrial category and by UK country and Nomenclature of Territorial Units for Statistics (NUTS1) region. The Adzuna industrial categories used do not correspond to Standard Industrial Classification (SIC) categories, so data broken down by industry are not directly comparable with the ONS Vacancy Survey.

Comparison of latest data

The online Adzuna job advert estimates show a decline of around 60% in total vacancies from early March to early May generally followed by an increase in vacancies through to September (Figure 13). There was a small decrease in the online job advert estimates at the start of September, which may be because of a seasonal component of the data.

The online job advert estimates for the first week in October show a smaller recovery compared with the first week in March than ONS Vacancy Survey. The ONS Vacancy Survey estimates show that the recovery for smaller businesses has been stronger than for larger businesses. Any vacancies that are not advertised online would not be included in the online job advert estimates; if small businesses are advertising vacancies through alternative methods, this could explain the difference between ONS Vacancy Survey and the online job advert estimates.

Summary

While the ONS Vacancy Survey is the best source to use over time and for headline estimates by industry and business size, Adzuna has additional breakdowns such as regional data. The combined use of these sources are being considered for future development, using the Adzuna regional information to produce estimates of more detailed breakdown of the Vacancy Survey data.

LFS redundancies versus Insolvency Service HR1 data

There are two potential sources on the number of redundancies. One is the Labour Force Survey (LFS) and the other is an administrative data source, HR1 forms submitted to the Insolvency Service by employers when intending to make collective redundancies.

Main differences

The LFS asks people whether they had left a job in the three months prior to interview and if so, why; answer categories include “made redundant” and “took voluntary redundancy”. The LFS estimates are based on interviews that take place across a three-month period, each asking about the previous three months (therefore, a recorded redundancy in the LFS would have happened somewhere between three and six months ago). Therefore, there is a lag between when redundancies occur and when they are reflected in the survey estimates.

The HR1 forms are collated through the Insolvency Service. HR1s are forms businesses must fill out if they plan to make more than 20 people redundant at one establishment. The data therefore reflect proposed redundancies that may not result in actual redundancies. Companies are also required to submit a HR1 form when making changes to employees' terms and conditions, and these are included in the data.

Comparison of latest data

As the LFS data are presented on a quarterly basis, the HR1 data have been aggregated to rolling quarters for comparability, as shown in Figure 14. Over time, the two sources have followed a similar trend, with the LFS series slightly lagged. However, the HR1 series show some peaks in proposed redundancies that have not followed through into the LFS redundancies series. Since the start of the pandemic in 2020, both series are showing increases.

Summary

LFS redundancy data are currently published monthly and are the most appropriate source for looking at actual redundancies. Data are available by age band, sex, region of residence and industry.

HR1 data are not currently regularly published but more recently have been made available through Freedom of Information (FOI) requests. They are a useful source in understanding potential planned redundancies.

There are a number of sources available that estimate the number of furloughed jobs.

HM Revenue and Customs (HMRC) publish the number of furloughed jobs in their Coronavirus Job Retention Scheme (CJRS) statistical bulletin. This information is sourced from administrative data and where possible linked to further sources to provide more detailed information.

The Labour Force Survey (LFS) collects information on people who are temporarily away from a job that they expect to return to. This would include people who are on annual leave, maternity or paternity pay and more recently those furloughed. In response to the coronavirus (COVID-19) pandemic, a number of questions were added to the LFS to better understand the impact of COVID-19. A follow-up question was asked to those temporarily away from work or working less hours because of COVID-19 on the amount of pay they were receiving.

The Business Impact of Coronavirus (COVID-19) Survey (BICS) was launched in April 2020 in response to the coronavirus pandemic. It included a question asking businesses to estimate the proportion of jobs furloughed. These proportions can be applied to existing jobs data to estimate the number of furloughed jobs. These estimates are publish bi-weekly in the BICS results.

Figure 15 shows a comparison of the sources that collect information on furloughing.

The sources differ greatly in terms of levels, but all show the same trend throughout, including a downward trend into September 2020.

Summary

The administrative data as published by HMRC are the best source of furlough data, and they are available by region, sector, age, gender and employer size. Alternative sources can offer useful insight but should be used with caution.

This page summarises the work done so far on understanding how different data sources can be used to help understand the labour market. We will continue to work with data providers and stakeholders to develop a better understanding of the sources and, where possible, work to quantify differences. We intend to update this page when greater understanding of the data sources is achieved and if new sources are introduced into our reporting of the labour market.

Recently, regional breakdowns of payroll employees and their earnings have been included in the Real Time Information (RTI) bulletin. From December 2020, the RTI bulletin will also include industrial breakdowns of payroll employees and their earnings. The next steps will be to further explore the regional and industrial differences between RTI and existing data sources and provide an update on this page.