1. Managing our end to end data journey


Figure 1 illustrates our data use and management policies that manage our end to end data journey.

Back to table of contents

2. Data acquisition

Office for National Statistics (ONS) follows a three stage model to acquire data under the Statistics and Registration Service Act 2007. The detailed processes will vary by provider and data source, but will cover the following basic steps:

  1. the initial stage is to develop a clear specification of need through collaboration between ONS teams and the data provider; this will include the preparation of a detailed business case, which provides a justification of the intended and potential use of each requested variable

  2. initial data supply to enable feasibility exploration; this includes the agreement of delivery mechanisms and security approach, including a review of all relevant legal obligations

  3. ongoing supply of data; this will include regular reviews to ensure all practices remain legally compliant and secure, and data are only acquired that ONS has an ongoing need for

Back to table of contents

3. Data collection

At ONS, we collect data from both survey and non-survey sources. We are transparent and open with data providers and value their time and contribution, collecting data only when necessary. We collect information from individuals, households, public bodies and businesses through a range of surveys we administer. We also conduct the census in England and Wales every 10 years.

To explain what respondents taking part in our surveys can expect, we publish a Respondent Charter for Surveys of Households and Individuals and a separate survey. Similarly, for each census we explain publicly how we will conduct the census and what will happen to the data collected, including our commitment that census records will be kept confidential for 100 years.

We design our surveys to minimise the burden placed on respondents, only collecting the data necessary to fulfil our functions, meet users’ needs and serve the public good. We publish details of all the surveys we run.

We further reduce the need for surveys by using non-survey data sources in compiling some of our statistics. This includes administrative data obtained by public or private sector organisations in the course of undertaking their normal operations, rather than our collecting data specifically for statistical purposes.

Wherever possible we follow the principle of “collect once, use many times” to produce statistics from existing data sources and avoid duplicate requests being made to those providing data.

When collecting data, regardless of whether from survey or non-survey sources, we comply with the relevant legislation.

Summary of main data collection principles:

  • we will only collect data for statistical and research purposes and where access is lawful

  • we will collect data transparently and treat data providers with respect

  • we will minimise the burden placed on respondents to surveys

  • we will aim to fully exploit non-survey data for statistical purposes

  • we will keep individuals’ and businesses’ information secure and confidential, regardless of whether it has been collected from survey or non-survey sources

  • we will collect data via web-scraping for the production of statistics and research that serve the public good, following practices and procedures set out in the ONS web-scraping policy

Back to table of contents

4. Quality Assurance of Administrative Data

ONS complies with the UK Statistics Authority’s Quality Assuring Administrative Data and implements practices outlined in the Quality Assurance of Administrative Data Toolkit.

Back to table of contents

5. Web-scraping

ONS collects data via web-scraping to use in the production of statistics and research that serve the public good. ONS’s web-scraping policy defines the following principles that must be followed:

  • seek to minimise burden on website owners

  • honour requests made by website owners to refrain from scraping their website

  • protect all personal data in all statistics and research outputs and seek ethical advice when scraping data that may identify individuals

  • apply scientific principles in the production of statistics and research based on web-scraped data and consider other sources of data

  • abide by all applicable legislation and monitor the evolving legal situation

These principles are applicable to all ONS staff activities involving web-scraping or which use web-scraped data. Additionally, when obtaining or procuring web-scraping services from a third party, ONS will ensure that these main principles have been followed. The policy does not cover the use of application programming interfaces (APIs) and is not applicable to government departments outside of ONS.

Back to table of contents

6. Social media for research

The Social Media for Research Policy sets out the practices and procedures that ONS staff will follow when collecting or using data obtained from social media platforms to produce statistics and conduct statistical research, including exploratory research, which serves the public good.

The Social Media for Research Policy outlines the main ethical considerations of using social media data and provides practical guidance to ensure that we use social media data ethically and consistently, in line with the National Statistician’s Data Ethics Advisory Committee (NSDEC) principles.

The following principles will help ensure social media data are used fairly, ethically and lawfully:

  • data are used for producing statistics, or statistical research that has clear public benefit that outweighs any associated risks and ethical implications of research on the data subjects

  • ONS uses the most appropriate data to realise the potential benefits of statistical research

  • data are used lawfully

  • data are used ethically and fairly

  • statistics, analysis and advice based on social media data are produced using scientific principles; following professional best practice and guidance

  • statistics, analysis and advice based on social media data are disseminated transparently with appropriate disclosure control measures

Back to table of contents

7. Data use

We use data from surveys, the census and administrative sources for statistical and research purposes only. In our work, we adopt statistical methods that are professional, ethical and transparent.

At ONS we use data to produce official statistics related to the economy, population and society at national, regional and local levels. We produce official statistics for the public good – our statistics are for the benefit of society and the economy generally.

We are always looking to engage with users of our statistics to ensure that we are using data, including data used for research purposes, in ways that are helpful and which inform decision-making by government, public services, business, researchers and the public.

In our use of data, we are committed to meeting consistently high standards of professionalism and ethics. We follow the principles and protocols for the production of official statistics set out in the Code of Practice and supplement this with independent advice from leading experts.

On ethical considerations concerning data, we seek advice from the National Statistician’s Data Ethics Advisory Committee.

We publish details of the statistical methods we adopt and why we adopt them on the methodology pages of our website.

We follow the principle of “collect once, use many times” to make full use of data and to secure the greatest possible public benefit from the data we hold. By linking datasets together we can understand better the relationship between different attributes, or corroborate analysis using alternative sources. We adopt linking processes that deliver efficient and accurate linkage while protecting the identities of individuals and businesses.

We use the data we collect from surveys and administrative sources for statistical and research purposes only. Respect for confidential personal information is embedded in the Code of Practice and is a fundamental tenet of the UK Statistics Authority and ONS, and the Statistics and Registration Service Act 2007.

Summary of main data use principles:

  • the purpose of our work is to inform decision-making by government, public services, business, researchers and the public

  • we comply with the Code of Practice for Statistics as fully as possible in the production of our statistics

  • we only use data from surveys, the census and administrative sources for statistical and research purposes

  • we seek expert advice to ensure the statistical methods we develop and adopt are fit-for-purpose, ethical and in line with international standards

  • we document online the methods we adopt and why we adopt them

  • as the executive office of the UK Statistics Authority, we share the statutory objective of “promoting and safeguarding the production and publication of official statistics that serve the public good”

Back to table of contents

8. Geography and geospatial

The spatial dimension of data plays a critical role in the production of statistics and use of data for research purposes.

Accurate geographic information is essential to:

  • locate and reference events

  • provide the geographical framework for sampling, collecting, linking and processing survey, administrative and other data

  • analyse and present statistics and research outputs – unlocking new information about places and relationships

  • select and access information for a range of geographic areas to meet policy needs

ONS is a leading player in maintaining the UK’s geographic frame. Spatial referencing and methods are embedded at all stages of our process – from data collection though to analysis.

The following are main principles of our Geospatial strategy:

  • the geospatial element is treated as a fundamental part of all of our data – with geospatial thinking, standards and methods embedded into all stages of the data journey

  • in time, data in the ONS data platform will be geo-referenced and stored to agreed and published architecture and standards – enabling the integration of data across domains and the use of common geospatial reference data

  • we will adopt the most appropriate spatial infrastructure and tools – continually reviewing our use of geospatial technology to respond to emerging and future data sources and technologies

  • geographic information system and geospatial analysis tools will be made available, promoted and supported for everyone involved in analysis of ONS data for statistical and research purposes

ONS maintains and complies with the Government Statistical Service Geography Policy.

Back to table of contents

9. Data classifications

A classification follows prescribed rules and guidelines, which are generally recommended and accepted to ensure that data are classified consistently. Classifications are a central part of statistical infrastructure, and are critical for producing coherent statistics and drawing meaningful comparisons.

We ensure that we use and maintain statistical frameworks in line with the best practice outlined by the United Nations Statistical Commission.

Back to table of contents

10. Data retention and destruction

The Data Retention, Data Archiving and Data Disposal Policy sets out clear principles for the management of data throughout their lifecycle. The policy ensures that data owners understand their responsibilities for regular review of data according to prescribed timeframes from stakeholders, or to ensure compliance with data protection legislation.

The data owner will need to consider any other potential uses of the data at the point they are no longer required to ensure maximum benefits are identified, taking into account any supplier contract conditions or terms of use.

Summary of main data retention, data archiving and data disposal principles:

  • datasets acquired from third parties are retained for the time period defined by the memorandum of understanding or contract

  • datasets generated by ONS are retained

  • data minimisation is applied for personal, or special category data

  • regular data and metadata reviews are performed dependent on the sensitivity of the data

  • additional uses of the data are considered before destruction to maximise the benefits of our data holdings

  • archiving of data is considered at the point the data are no longer of use by ONS to ensure their continued availability and functionality

Back to table of contents

11. Linking and matching

Linking datasets has vast potential to deliver public benefit by enabling deeper insight into the complex questions that face society. The value of linking datasets must be balanced with ensuring that the privacy of individuals remains protected. We achieve this by applying the principles of data minimisation and ensuring that, where possible, data are de-identified. Specifically, we minimise the number of users who work with identifiable data and the work of these users is primarily to link and de-identify data to protect the privacy of individuals. We continually invest in developing our linking and matching methods to ensure that the value of linking and matching datasets is valued.

ONS continually invest in developing its linking and matching methods to ensure that the value of linking and matching datasets is valued.

You can find more information on ONS and wider Government Statistical Service (GSS) linking and matching on the Government Statistical Service website.

This policy is under review to reflect the UK Statistics Authority’s systemic review

Back to table of contents

12. Data organisation

The management of data in ONS must comply with the ONS data principles and data standards. The data lifecycle extends across the following stages:

  • data acquisition

  • data storage

  • data manipulation and processing

  • data access

  • data sharing and dissemination

  • data archiving

  • data destruction

Data should be validated and cleansed on receipt, and then modified (as necessary) to conform to the data standards and to ensure that data are harmonised to facilitate re-use. Standardised statistical datasets should be linked to reference data (as appropriate) and links to geospatial co-ordinates should be established to enable geospatial analysis of data. Reference datasets include statistical classification codes and also statistical indexes, such as the business index and the address index.

Back to table of contents

13. Metadata

Metadata provide information about other data, including a description of the data. This includes information that provides context to the data, for example, how they were collected, or the coverage of the data. Metadata typically include publication date, description and search keywords. In order to understand the data we hold, ONS will hold and manage a comprehensive set of unambiguous metadata relating to the data we manage.

The metadata will be categorised and managed according to the ONS metadata management framework, which includes data principles and data standards relating to metadata. The metadata standard defines the “minimum set of metadata” for ONS datasets. All data acquired by ONS must be accompanied by metadata.

Metadata items will be completed by data owners, who will be responsible for periodically reviewing the metadata is correct and of high quality. The data owner will initiate corrective action to address any issues found with accuracy and quality of the metadata.

Back to table of contents