This article provides an overview of our transformation of population and migration statistics to put administrative data at the core of what we do. This includes how our plans have evolved because of the coronavirus (COVID-19) pandemic and our approach for delivering new measures of migration based on administrative data from November 2020 onwards, following the suspension of the International Passenger Survey (IPS) in March 2020.
This article also provides information on our latest progress towards an activity-based method for our admin-based population estimates as well as an overview of our "administrative data first" approach to transforming population characteristics.Back to table of contents
It is our mission to provide the best insights on population and migration, working with other government departments and using a range of new and existing data sources to meet the needs of our users. This is increasingly important in a rapidly changing policy and societal context, where we know our users need better evidence to support decision-making at both national and local levels.
As set out in our previous work on the Administrative Data Census project, the current population system is heavily reliant on the decennial census. While this provides granular data at the lowest levels of geography every 10 years, it delivers less detail throughout the interim years. Additionally, the quality of our population estimates declines as we move further away from the census year.
We have also long acknowledged that the International Passenger Survey (IPS), which underpins our existing international migration estimates, has been stretched beyond its original purpose and that we need to consider all available sources to understand international migration.
Enabled by data-sharing powers in the Digital Economy Act 2017, guided by our data security principles, and building on the ongoing collaboration and data-sharing under way with government departments, we are seizing the opportunity to make use of more data to give us a much richer understanding of how our population is changing.Back to table of contents
Population and migration statistics underpin a wide variety of other statistics (such as unemployment rates) to support a vast range of decisions and inform public debate. For example, the ability to forecast pensions, make decisions about local services (such as the number of school places or the provision of health services for an ageing population) and make decisions about where to site new businesses.
We know that our users are highly interested in how migration patterns are changing and what this means for society and the economy. For example, users are interested in the contribution to and impact on the UK labour market and public services, such as education and healthcare, that migration has. This includes both the national picture and what is happening at a more detailed regional and local level.
It is therefore essential that our population and migration statistics are robust and timely and meet this broad range of user needs.Back to table of contents
To help users understand the wider impact of the coronavirus pandemic on international migration and mobility, ONS has published an article bringing together available evidence from the Office for National Statistics (ONS), Civil Aviation Authority (CAA), Department for Transport (DfT), Home Office and Department for Work and Pensions (DWP). Using our data-sharing powers through the Digital Economy Act 2017, we have been progressing research into how we can bring a range of government data sources together to build an integrated system for measuring population and migration.
Our progress so far
In January 2019, we published a research engagement report that updated users, and we have since progressed a series of updates on our progress towards our ambition. However, there is still more to be done to embed administrative data in our population and migration statistics.
Current progress on our population statistics transformation
Our population statistics research has centred around creating population estimates from administrative data via our methodology of anonymously linking person records on administrative data sources. These admin-based population estimates (ABPEs) provide a measure of the usually resident population at both national and local level and will be used as a basis for producing admin-based household estimates and estimates of internal migration.
Previous iterations of our ABPEs (formerly known as Statistical Population Dataset) have used presence on multiple data sources as a main inclusion rule. Ongoing research consists of moving away from this approach to an activity-based approach using registration activity and other interactions with administrative sources as "signs of life" in our inclusion rules. We published our latest ABPEs in June 2019. Our design aim was to reduce the over-coverage observed in our previous versions when compared with the official estimates. This aim is driven by our understanding that to produce ABPEs of acceptable quality for our users, we need to supplement them using a Population Coverage Survey (PCS) and a tried and tested estimation method. An important condition for the latter is that our ABPEs have zero to minimal over-coverage.
We have been assessing the quality of our ABPE Research Outputs so far and evaluating whether we have achieved our design aim. We can conclude that moving to an activity-based approach has removed some of the over-coverage we observed in our previous ABPEs; however, over-coverage remains. This will need to be reduced further to enable the production of robust estimates using existing coverage adjustment methods.
A priority next step for us is developing our quality frameworks further and using the knowledge gained here to make the best use of the data. This is important for understanding how interactions with administrative sources change over time in both our existing and future data sources.
Given the current context of the coronavirus (COVID-19), we need these frameworks in place so we can understand any changes in the interactions with administrative sources, for example, students moving away from their university accommodation or deferring their courses. Now more than ever we see the need for more timely information about the size and structure of the population and population change.
Current progress on our migration statistics transformation
We are moving away from the International Passenger Survey (IPS) towards the use of administrative data to deliver new measures of migration.
We know that there are challenges we need to overcome to deliver new measures of migration based solely on administrative data, including establishing a clear framework for the use of administrative data sources to construct a new estimate of migration. Through our previous research, we have already identified a range of data sources held across government that can help us to better measure migration – including immigration, income, benefits and education data – and will build on this to design our new measures. So far, the main focus of this work has been on two sources of data, which have shown greatest potential for the measurement of long-term migration:
Department for Work and Pensions (DWP) Registration and Population Interaction Database (RAPID)
Home Office Visas and Border data
RAPID provides a single coherent view of citizens' interactions across the breadth of systems in both DWP and HM Revenue and Customs (HMRC) including benefits, employment, self-employment, pensions and in-work benefit. We are exploring the use of these data to determine signs of "activity" among the migrant population in the UK, and how we can use that information to infer migrant flows based on when that activity commenced (indicating arrival in the UK) and when it ends (indicating departure).
As RAPID covers everyone with a National Insurance number (NINo) it covers both migrants from EU countries and those from countries outside of the EU, although there are issues in relation to the identification of current nationality that still need to be addressed. While the coverage is extensive and most migrants will appear in RAPID there are some groups less well covered in the data. For example, migrant children under 16 years of age are not separately identifiable in RAPID and visiting students who do not hold a NINo will not be included in the dataset. Students who do hold a NINo are included in RAPID but may not be identified as resident in the UK if they do not undertake any activity that verifies residency, for example, some form of long-term work alongside their studies. We are working on how we can address these coverage gaps by using RAPID alongside other sources, including data from the Higher Education Statistics Agency (HESA) and NHS Digital.
Home Office Exit Checks data provide a more direct measure of movement and can be used to identify when non-European Economic Area (EEA) nationals have entered or exited the UK. Previous analysis published in February used Exit Checks data to explore patterns and definitions of long-term international immigration. Looking ahead, our work with these data will focus on refining our method of measuring long-term immigration by further exploring patterns of movement of individuals over time. Following this, we plan to develop similar methods to measure long-term emigration.
Throughout this period of development we are continuing to work closely with statisticians in other government departments in order to build our understanding of complex administrative data sources so that we can draw on the strengths of these together. In particular, we are collaborating with the Home Office to better understand how the new immigration system will inform our understanding of migration. From January 2021, EEA nationals will be included in Home Office travel and visa data, and together we are rapidly progressing work to understand the opportunities offered by these data.
Our regular statistical reports will look different as we adapt our methods and go through a period of transitioning to administrative data-based estimates. They will continue to be badged as Experimental Statistics throughout the transition, to support users in understanding the changes. We will also highlight where we are unable to provide certain statistics or breakdowns of data while we build our new migration measures and provide guidance on the quality and coverage to ensure the statistics can be interpreted appropriately. Our plans for forthcoming migration statistics publications are in the accompanying summary of recent updates. We plan to release a more detailed report on progress in developing new ABMEs in early 2021.
We are also collaborating closely with National Records of Scotland (NRS) and the Northern Ireland Statistics and Research Agency (NISRA) to ensure that we can produce comprehensive UK migration statistics. This includes incorporating the Northern Ireland migration estimates produced separately by NISRA and ensuring alignment with the migration statistics for local areas in Scotland and Northern Ireland, produced by the NRS and NISRA respectively.
We will continue to regularly engage with users as we go through this transformation journey and gather feedback to inform our development work.
Reporting on international migration since the coronavirus pandemic
To help users understand the wider impact of the coronavirus (COVID-19) pandemic on international migration and mobility, the Office for National Statistics (ONS) has published an article providing information on patterns of international mobility since the coronavirus pandemic, pulling together available evidence from the ONS, Civil Aviation Authority (CAA), Home Office and Department for Work and Pensions (DWP). In addition, the Home Office has also published a wide range of additional information in their Immigration statistics looking at changes in visas and other operational activity over the course of the pandemic.
The coronavirus pandemic has had a significant impact on administrative data used to produce migration statistics. For example, Home Office visa application centres were closed by 31 March 2020 because of the pandemic and at the same time the NINo allocation process for some applications was paused as a consequence of pandemic restrictions. In addition to this, travel restrictions imposed because of the pandemic have had a significant impact on mobility, meaning historical data trends, such as those based on the IPS and admin data, cannot currently be used to help understand international migration since the start of the coronavirus pandemic.
To respond to these challenges and to provide the best possible evidence on migration, the ONS are exploring options that can help us understand and estimate international migration since the start of the coronavirus pandemic. Our work in this area includes reviewing a range of data sources, which can provide up-to-date information during the coronavirus pandemic, as well as working with other national statistical institutes, academics and experts across the Government Statistical Service (GSS) . The ONS expects to share further updates on this work and the feasibility of methods in early 2021.
Population estimates for mid-2020
The population estimates for mid-2020 are planned for release in June 2021. The coronavirus pandemic has disrupted the collection of several data sources (such as the IPS) that underpin the production of population estimates. Further, the pandemic may have had impacts on how different groups, for example, students, are captured on different data sources used in the population estimates. Over the coming months, we will be working to further understand the impacts of these issues and will share our findings and consult with stakeholders as we progress.
Our plan for population and migration statistics from 2020 onwards
We will continue to ensure that the work to transform UK migration statistics in 2020 aligns with our ongoing research to produce population statistics using administrative data. Migration statistics are an important component of estimating population change, and so we will ensure our research is integrated as we move towards a transformed population and migration statistics system in 2023.
A summary of the revised milestones on this transformation journey, by target date and what we aim to deliver, follows.
published the August Migration Statistics Quarterly Report (MSQR); the last set of migration statistics based on International Passenger Survey (IPS) data
updated users on our approach for delivering new measures of migration using administrative data from 2020 (administrative data-based migration estimates (ABMEs)), including our data sources and methods
continued research into our approach for delivering population estimates based on more fully integrated administrative data sources in the future (known as administrative data-based population estimates (ABPEs))
improved population statistics (England and Wales) by embedding adjusted international migration figures into our mid-year population estimates (reference year: 2020)
continued research into our approach for delivering ABPEs
continued refinements to the ABMEs with reports to keep users informed of progress
2021 Census statistics published
population and migration statistics based on administrative data (reference year: 2021), including an understanding of how these are different when compared with census outputs and mid-year population estimates for England and Wales
- deliver transformed population and migration statistics system – informing the recommendation from the National Statistician on the future of the census
This transformation journey – alongside the work to transform surveys across the ONS – also supports our goal to produce census-type statistics in new ways using government-held administrative data and other data much more often and, importantly, that the outputs we publish are coherent and meet the needs of our users. This is essential for providing evidence to inform the final recommendation to the UK government in 2023 about the future of population and housing censuses in England and Wales. The ONS is also working with the devolved administrations on their respective programmes of work, including the NRS and NISRA, which are responsible for the census and producing population statistics in Scotland and Northern Ireland respectively. Ongoing collaboration will take place between the ONS, NRS and NISRA to consider harmonisation and ensure population statistics across the UK remain comparable.
It was announced in July that Scotland's census will be moved back by a year until March 2022. The census in Scotland is the responsibility of the NRS and Scottish Government ministers, and their decision is as a result of the coronavirus pandemic. Many important activities have been delayed or prevented because of the coronavirus restrictions, including planning and testing the effectiveness and security of systems and processes as well as engagement with certain organisations. This announcement does not affect the England and Wales census, which we continue to work towards delivering as planned in March 2021. The three census offices of the UK (the NRS, NISRA and, in England and Wales, the ONS) will work closely to minimise the impact on UK data coherence and to ensure that high-quality data are available to inform policy, investment decisions and the planning of services.
Decisions on the census after 2021 in Northern Ireland and after 2022 in Scotland will be made by the Northern Ireland Executive and Scottish Government ministers respectively.
We have taken the decision not to benchmark the administrative data census outputs with the outputs from Census 2021. Instead, we will ensure that we use the best available sources to produce the best possible outputs from the census – using our understanding of the differences between administrative data and the census and the strengths of all data sources. We will iteratively develop our transformed population and migration statistics system, taking on board feedback from users and making the best use of new data and methods as they become available. We will rigorously quality assure new methods and share the impact of any changes made, demonstrating the benefits and improvements before implementing them.
Alongside supporting the decision on the future of the census, our transformation journey also plays an important role in supporting our progress in measuring the global Sustainable Development Goal (SDG) indicators in the UK. Migratory status is one of the standard disaggregations required by the UN to measure and report on the global SDGs, and our transformation work will help to improve the evidence base by providing more granular data in the future.Back to table of contents
Our users sit at the heart of our framework for transforming these statistics (Figure 1), which describes the important questions we need to answer to meet their needs.
To meet our users' needs, we need to provide coherent statistics on the size (or stock) of the population and how it changes over time (flows, both nationally and locally). We also need to tell a clear story about what is contributing to this change and show how different groups in the population impact on society and the economy, including on our workforce, communities and public services such as the NHS and schools. This needs to be recognised as the story that is being experienced by our users.
Our users have also told us that they want us to deliver these statistics frequently and in a timely manner to be able to make evidence-based decisions. Our statistics also need to be relevant in a rapidly changing society, and we need to be able to report on their quality.
Feedback gathered shows support for how we are reviewing current data sources and looking to maximise the value of administrative data in the future, while challenging us to provide more evidence to ensure we instil confidence in these statistics.
To make sure that our transformation journey continues to be as open and transparent as possible, we will:
regularly publish research and methods as we develop them
continue to present analysis showing the coherence between different sources of information
engage with our users and stakeholders, seeking regular feedback
use our research findings, the best available data and methods, and the feedback from users to make decisions about which improvements to make each year
implement changes to our statistics when and where appropriate, clearly communicating these changes to users in advance of making them
How to contact us
We welcome your feedback on this update and on our transformation journey. If you would like to contact us, please email us at email@example.com.
We have recently presented our research at the 2019 Royal Statistical Society Conference, the British Society for Population Studies Conference, the Migration Statistics User Forum, the International Conference on Administrative Data Research and the 2020 International Forum on Migration Statistics in Cairo.
We are also working with local authorities, experts from academia and other government departments through invited engagement groups to help shape and quality assure our work. Additionally, we are engaging with other National Statistical Institutes (NSIs) across the UK and internationally to share experiences and seek feedback on our plans.Back to table of contents
For a comprehensive list of descriptions for the names and terms used in our outputs, see our glossary.
Research Outputs provide more information on using administrative data to produce population statistics.
Previous research using administrative data to produce estimates on the size of the population (previous admin-based population estimates (ABPEs)) is available.
Our August 2017 report highlights our progress towards developing a better understanding of student migration to and from the UK, alongside the Home Office exit checks report also published in August 2017.
In May 2018, we published an update on the migration statistics transformation programme. Our report on international migration data sources sets out how we are working with the Home Office in using their administrative data to further our understanding of international migration.
Our January 2019 report, Update on our population and migration statistics transformation journey: a research engagement report, updates users on our progress and seeks feedback on our future plans.
In May 2019, we published our research into international migration and the education sector, exploring what current data sources tell us about school places, pupil attainment and the number of teaching staff in schools.
In June 2019, we published our analysis using activity-based rules and records from single and linked data sources to develop our approach for producing ABPEs. Initial rules have been combined to produce the first ABPEs using this approach. Alongside this, we also published a short note outlining our discussions on the measurement of illegal migration.
On 15 August 2019, we published our analysis plans on the contribution and impact of international migration on the health sector. Alongside this, we published a report on international migration and the healthcare workforce and our analysis of migrant labour force within the tourism industry. On 21 August 2019, we published our latest research into understanding different migration data sources, setting out how we can draw on the combined strengths of survey and administrative data sources to provide our best assessment yet of international migration. This research was then reflected in the Migration Statistics Quarterly Report (MSQR) for the first time in August.
On 14 February 2020, we published an overview summarising how we are defining and measuring international migration, including developments of our current methods. Alongside this, we published our latest research investigating international student migrants' travel patterns using linked administrative data and alternative definitions for international migration using Home Office exit checks data. We also published a report on applying a statistical quality framework to longitudinally linked administrative data, aiming to identify potential errors within datasets.Back to table of contents
For national and subnational mid-year population estimates for the UK and its constituent countries, see the Population estimates for the UK, England and Wales, Scotland and Northern Ireland statistical bulletins.
All information and publications on international migration produced by the Office for National Statistics (ONS) Centre for International Migration are available on our International migration page. You can find out more about the ONS Centres and their priorities on our landing page.
All information and publications on the size of the population produced by the ONS are available on our Population estimates page.
You can find more information on the main concepts and definitions used for migration, country of birth and citizenship across the Government Statistical Service (GSS) in the relevant GSS harmonisation guidance.Back to table of contents
Contact details for this Article
Telephone: +44 (0)1329 444661