Methods to produce provisional long-term international migration estimates

This methodology article summarises the methods used to produce experimental and provisional estimates of long-term international migration flows, published in our Long-term international migration, provisional: year ending December 2022 bulletin. It also summarises previous methods implemented since April 2021. We explain how our methods have changed and why future revisions of the estimates will be an important part of producing migration estimates going forward.

The methods detailed in this paper are part of an ongoing programme of work to transform population and migration statistics. This work was accelerated in response to the coronavirus (COVID-19) pandemic, and we continue to build on these developments. For more details, please see our How we are improving population and migration statistics article.

The International Passenger Survey (IPS), which underpinned our estimates of migration until 2020, has been stretched beyond its original purpose. We are considering all available sources and methods to estimate international migration flows (referred to as migration).

In March 2020, the IPS was suspended because of the coronavirus (COVID-19) pandemic. In response, we accelerated our approach for transforming migration statistics using new methods and administrative data, supported by statistical modelling. The IPS resumed in January 2021. However, because of its limitations in measuring migration, we continue to focus on using administrative data supported by statistical modelling.

Going forward, the revision of long-term international migration statistics will be an important part of the production of these estimates. Provisional estimates are released with the expectation they may be revised as more complete data become available. In addition, our methods are still evolving, and we will therefore revise the estimates as our methods mature. Further information on our approach to revisions can be found in our Population and International Migration Statistics Revisions Policy.

A common misconception is that it is easy to measure international migration by counting people in and out as they cross the border. There are many reasons why it is difficult to count migrants by monitoring cross-border travel data using passport scans at airports. For example:

• some people hold two passports and use different passports for incoming and outgoing journeys

• the UK and Ireland belong to a free travel zone called the Common Travel Area; people can travel freely between the two countries and movements across the land border between Northern Ireland and the Republic of Ireland are not tracked

Countries across the world have encountered similar measurement challenges and we work closely with many of them to learn and share best practice.

The Office for National Statistics (ONS) is committed to exploring all sources of information to produce migration estimates. This includes Advance Passenger Information, which provides passenger information for a large proportion of inbound and outbound air passengers.

Our experimental estimates for year ending (YE) December 2022 are produced using our latest method, which has been developed to make greater use of administrative data and rely less on the International Passenger Survey (IPS) data and statistical modelling. We moved from intentions-based IPS estimates to estimates based on actual observed activity in administrative data, as explained in our International migration: developing our approach for producing admin-based migration estimates article.

Our latest estimates use different data sources and methods for each nationality grouping. We currently publish estimates on immigration, emigration and net migration for non-EU nationals, EU nationals and British nationals.

We continue to use the United Nations' (UN) definition of a long-term migrant: a person who moves to a country other than that of their usual residence for at least a year.

Non-EU nationals

Non-EU migration refers to the migration of people who do not hold British or EU nationality. We use Home Office Borders and Immigration data, which combines visa and travel information, to link an individual's travel movements into and out of the country. For more information, please see the Home Office statistics on exit checks: user guide.

Our first step is to identify which travellers meet the definition of a long-term migrant, filtering out those on long-term visit visas. We use arrival and last departure dates within a visa period as an approximation for length of stay in the UK. Short trips abroad over the course of an extended period of residence are excluded. If either the first arrival or last departure information is missing, then visa start or end dates are used as a proxy.

Visa periods are constructed by linking together any consecutive or concurrent visas held. If there is a gap of more than seven days between visas, then a new visa period is started. We look at any previous visa period to determine if this is a new long-term immigrant or one who has previously been in the country. If no presence is identified in the country during the 12 months preceding first arrival on a given visa, or the previous visa period had a length of stay of less than 12 months, then this person will be considered a new long-term immigrant.

To measure emigration, we identify previous long-term immigrants with a last departure from the UK during the reference period. We record them as a long-term emigrant if they do not return to the UK within 12 months, or if they only return for a short-term stay.

An adjustment is applied to account for the over-estimation of both immigration and emigration in the most recent time periods where we do not have data available showing evidence of 12 months in or out of the country. We provide further details in our International migration research, progress update: November 2022 article.

EU nationals

Our latest methodology to estimate the migration of EU nationals is based on our Methods for measuring international migration using Registration and Population Interaction Database (RAPID) administrative data. RAPID currently provides the best insight into the migration of EU nationals.

RAPID is created by the Department for Work and Pensions (DWP) to provide a single coherent view of citizens' interactions across the breadth of systems in the DWP, HM Revenue and Customs (HMRC), and local authorities via Housing Benefit. RAPID covers everyone with a National Insurance number (NINo) and for each person, the number of weeks of "activity" within these systems is summarised in each tax year. Records are then categorised as either long-term or short-term by looking for patterns of interactions with the tax and benefits system.

RAPID is made available to the Office for National Statistics (ONS) on an annual basis in Quarter 3 (July to Sept) for the previous tax year. Currently we publish bi-annual international migration estimates, which requires forecasting RAPID for three or nine months.

Temporal disaggregation is used to perform both distribution and forecasting of the RAPID EU estimates. Distribution is the disaggregation of already known totals in the annual source to the monthly totals, which does not go beyond the time of RAPID data. Forecasting is the process of generating figures beyond the timeframe of RAPID data, based on the signals and trends in the higher frequency time series. The IPS is used as input for the higher frequency time series. We have acknowledged the long-standing issues with measuring the levels of migration, however, the IPS helps to incorporate seasonality and trend of migration flows.

These EU migration estimates are limited by the population coverage of RAPID. Anyone arriving in the UK needs to apply for a NINo to work, claim benefits, or apply for a student loan. The coverage is extensive for most migrants because of the wide range of data sources included. However, there are some populations who have less or no interaction with the source datasets, and we adjust our EU estimates to account for this under-coverage as outlined below.

Inflow and outflow adjustment

International migrants that have recently arrived or departed the UK will not necessarily meet the UN definition of a long-term international migrant (LTIM). We adjust the estimates from the more recent tax years to take account of this. This is done by calculating the proportion of migrants that have historically become LTIM, both immigrants and emigrants, and applying this to the number of arrivals and departures in the most recent tax years.

Students who may be excluded if they are not working or claiming benefits

To identify students immigrating into the UK long-term, we use the Higher Education Statistics Agency (HESA) dataset and HMRC Pay as You Earn Real Time Information (PAYE RTI) data as the best available data sources. Our latest method links these two sources to better understand how many international students are in employment alongside their studies. We are then able to adjust our estimates to include the proportion of students that do not show employment activity and are therefore not included in our initial estimates.

Children aged under 16 years (U16s)

The adjustment for children aged under 16 years (U16s) uses an adult-to-child ratio derived from the IPS. Where IPS data are not available (2020), a three-year average ratio (2018, 2019 and 2021) is applied. This ratio is then used to calculate the number of U16s to add to the RAPID estimate. This ratio is calculated separately for both immigration and emigration.

During the coronavirus (COVID-19) pandemic, some EU workers may also have been furloughed under the Coronavirus Job Retention Scheme. It is likely that some of these workers will have left the country for more than 12 months (and therefore should be measured as a long-term emigrant), but they will be missed in these estimates as they still appear active on RAPID.

Further research is required and planned for determining migration estimates using the RAPID migration dataset.

British nationals

Our research into British nationals is ongoing, but the complexity associated with identifying these migrants in administrative data means we cannot use such data at this time. For our latest estimates of migrants with British nationality, the IPS data are still our main source of information.

The IPS was reinstated in January 2021, and we use these data as our estimates for January 2021 to December 2022. To cover the period when the IPS was suspended (March to December 2020), we use the state space model (SSM) time series analysis. This takes the available IPS and administrative data and uses the relationship between them to estimate the missing IPS data. The full technical details of the SSM can be found in our Using statistical modelling to estimate UK international migration methodology.

The estimates undergo quality assurance to ensure they are plausible and meet the standards of experimental statistics set by the UK Statistics Authority. This includes comparing the trends with other comparable data and statistics, and consulting with the Government Statistical Service Migration Steering Group and the Migration Statistics Expert Group.

For our most recent Long-term international migration, provisional year ending December 2022 bulletin, the use of administrative data to produce our estimates means that we are currently exploring the best approach to quantify uncertainty. Our most recent estimates do not yet have uncertainty intervals associated with the main dataset. An overview of the ongoing work is available in our International migration research, progress update: May 2023.

The way we measure international migration is evolving, taking account of new data sources, improved methods, and the changing needs of our users.

We summarise the different methods we have used to calculate international migration, developed since April 2021.

Version one

The first version of the model was published in April 2021 and provided migration estimates for March to June 2020.

The state space model (SSM) projected forward the trends and seasonality of the previous IPS data and then adjusted it by the structural shift seen in the Home Office Borders and Immigration data for non-EU citizens. This version includes assumptions about EU nationals having different travel options during lockdowns. When airports were closed, EU nationals were still able to travel by ferries and Euro Tunnel.

Version two

This version was published November 2021 and produced estimates up to December 2020.

The model in version two remained similar to version one, however, we updated one of our assumptions in the model by creating an EU proxy series for estimating migration of EU nationals. This new EU trend used in the SSM was based on historical movements of EU nationals, rather than an assumption that EU nationals behave the same as non-EU nationals (as in version one).

Version three

This version was published in May 2022. This version relied more on administrative data and benchmarked the EU estimate to the Department of Work and Pensions (DWP) Registration and Population Interactions Database (RAPID) for the first time.

In version three, record-level Home Office Borders and Immigration data were not available for the final eight months of the time series (November 2020 to June 2021). This missing time period was estimated using an aggregated version of the data. We applied the pattern of change observed in the aggregate dataset with the Denton-Cholette method to predict the record level dataset for the missing period. This provided us with a rate of change for arrivals on a month-to-month basis. For more information on the Denton-Cholette method, please see the European Commission’s Temporal disaggregation, benchmarking and reconciliation guidelines (PDF, 2.3MB).

At the time, we did not have an equivalent method for measuring non-EU emigrants. Therefore, we calculated a ratio between emigration (numerator) and immigration (denominator) on a monthly basis from the aggregate dataset, which was applied to the calculated non-EU immigration estimates to estimate emigration. This assumed the trends in the aggregate dataset for the immigration series and emigration series are similar to the trends in the record-level data, as both are derived from the same source. However, subsequent estimates of emigration using record-level data (introduced in version four) entailed average revisions of around 200,000 emigrants compared with the version two approach. These revisions are presented in Section 8 of our Long-term international migration, provisional: year ending June 2022 bulletin.

Version four

This version was published in November 2022 and produced quarterly estimates for year ending June 2020 to June 2022.

In version four, for EU nationals we introduced a more comprehensive student adjustment based on the linking of Higher Education Statistics Agency (HESA) and HM Revenue and Customs (HMRC) Pay as You Earn Real Time Information (PAYE RTI) datasets to understand how many international students were in employment during their studies. We also introduced an adjustment for those aged under 16 years, which until this iteration had been excluded from our estimates. This adjustment was based on an adult-to-child ratio derived from the International Passenger Survey (IPS).

The non-EU estimates in version four used the complete time series for record-level Home Office Borders and Immigration data, which had been unavailable in version three. The methodology of emigration for non-EU nationals was made comparable with the methodology of the immigration of non-EU nationals, and an adjustment for recent arrivals and departures was implemented to avoid overestimation.

Further detail on these changes can be found in our International migration research progress update: November 2022 article.

Version five

This version, published in May 2023, produced quarterly estimates for year ending December 2018 to December 2022.

Changes to EU estimation

The EU estimates of international migration continue to be based on DWP’s RAPID. In our original research article into this administrative data source, we used four categories based on the amount of activity for the identification of long-term international migrants. For further information, please see our Methods for measuring international migration using RAPID administrative data.

The C1 and C2 categories most closely aligned with the UN definition of a long-term migrant. When we initially developed our method, we expanded on this definition of long-term activity to reflect the complexity of people’s lives. This created two further categories: C3 and C4.

As part of the post-census reconciliation of the population estimates, we have reassessed the international migration component based on our new methodology using administrative data. Through this process, we have acknowledged that those categorised using the C3 and C4 classifications are likely to have pushed the definition of long-term migration too far, resulting in the overestimation of migration. Therefore, we have redefined our long-term international migration (LTIM) estimates to only include those people who meet the C1 and C2 classification criteria.

In addition to this, we have also brought our student emigration adjustment in line with the methodology introduced in version four for the student immigration adjustment.

Changes to non-EU estimation

We have continued to refine our methods using Home Office Borders and Immigration data, and we have updated several components, which are summarised below.

We now calculate the recent arrival adjustment based on reason for migration rather than at the headline figure (all immigration). This is to reflect that different types of migrants have different behaviours.

We have also implemented an adjustment for the Ukraine schemes and British National (Overseas) visa route arrivals where we do not have enough information to suggest how many people do not have a long-term stay of 12 months or more. Internal Home Office analysis shows that many people left the UK before reaching the 12-month period. We have used these data to implement an adjustment for these groups and removed these arrivals from our long-term immigration estimates where they have been outside the UK for eight weeks or more.

Inclusion of asylum seeker applicants and returns, and resettlement scheme arrivals

The estimates now include asylum seekers and resettled refugees, which were not included in versions one to four. An asylum applicant (also referred to as an “asylum seeker”’) is someone who makes a claim to be recognised as a refugee under the Refugee Convention.

We added the total number of asylum applicants and resettlement scheme arrivals to the international immigration estimate. We used internal Home Office analysis to apply an adjustment to account for people who applied for asylum after entering the country on a long-term visa (for example, on a student visa), by removing them from the count of asylum applications.

We added the total number of asylum returns to the international emigration estimate.

To include asylum seekers and resettled refugees in the long-term international migration (LTIM) estimation, we have made the assumption that:

• all asylum seekers are LTIM based on processing times in recent years, assumed deportation lag, and declared intent to stay in-country
• returns are people who have been in-country for more than 12 months
• all resettled persons are LTIM based on declared intent to stay in-country
• all asylum seekers arrived in the UK no more than 12 months before applying

All assumptions will be reviewed going forward.

Changes to British nationals estimation

Following the inclusion of British National (Overseas) visa route arrivals in the non-EU estimates based on Home Office Borders and Immigration data, we have made an adjustment to the IPS based estimates to avoid double counting. We did this using a combination of country of birth and country of last residency, and removing these from the IPS estimates.

Strengths

The current approach, as published in May 2023:

• uses a wide range of data sources to observe the behaviour of migrants, greatly helped by the data-sharing powers of the Digital Economy Act 2017
• provides timely, provisional, estimates that are subject to revision as data are finalised and methods refined
• includes estimates increasingly derived by administrative data sources that capture actual, rather than intended, migration patterns (as was measured by the IPS)
• includes improved methods, taking account of new data sources, better processing methods and the changing needs of our users; it will join up with other Office for National Statistics (ONS) transformation work on population and migration statistics
• had greater involvement and collaboration of experts and other government departments in the process of producing these migration statistics; this leads to enhanced trust in, and understanding of, these statistics, as well as more consistent methods across all government international migration data

Limitations

The current approach, as published in May 2023:

• can only produce headline figures for migration by direction of flow and broad nationality groups (British, EU, and non-EU), which does not meet all our users' needs
• includes estimates that are provisional, which means that there is a degree of uncertainty around them as we refine our methods and data sources
• does not include the methodology for calculating uncertainty intervals using administrative data as this is still in development; we are therefore unable to provide estimates with uncertainty intervals at this time
• requires users to adapt to the new practice of regular revisions of migration estimates because of the transition to estimates based on activity rather than intentions-based

The Office for Statistics Regulation's (OSR's) review on migration statistics encouraged us to improve and broaden our user engagement as well as ensuring that we have coherent plans across our transformation work.

We recognise the need to continuously improve our methods with our users. If you would like to find out more or have any feedback, please email us at pop.info@ons.gov.uk.

We continue to explore other data sources as they become available and existing sources as they are updated. In the coming year, we plan to:

• explore insights available from Advance Passenger Information, including those related to measuring the migration of British nationals
• continue to investigate new visa requirements for EU nationals, set out in GOV.UK's New immigration system guidance, with the intention to incorporate associated data into our method
• explore how our methods can be improved to provide more timely and frequent measures of migration, and with breakdowns, including by reason for travel, geography, sex and age
• continue developing a simulation-based approach to measuring uncertainty in international migration estimates, of which the first working paper is due to be published on 1 June 2023
• work with other government departments to ensure published migration statistics are coherent
• implement a reproducible analytics pipeline (RAP) strategy to improve the quality of our analysis, increase trust in our analysis, and create a more efficient process