Figure 1 illustrates our data use and management policies that manage our end to end data journey.
Back to table of contents
Office for National Statistics (ONS) follows a three stage model to acquire data under the Statistics and Registration Service Act 2007. The detailed processes will vary by provider and data source, but will cover the following basic steps:
the initial stage is to develop a clear specification of need through collaboration between ONS teams and the data provider; this will include the preparation of a detailed business case, which provides a justification of the intended and potential use of each requested variable
initial data supply to enable feasibility exploration; this includes the agreement of delivery mechanisms and security approach, including a review of all relevant legal obligations
ongoing supply of data; this will include regular reviews to ensure all practices remain legally compliant and secure, and data are only acquired that ONS has an ongoing need for
At ONS, we collect data from both survey and non-survey sources. We are transparent and open with data providers and value their time and contribution, collecting data only when necessary. We collect information from individuals, households, public bodies and businesses through a range of surveys we administer. We also conduct the census in England and Wales every 10 years.
To explain what respondents taking part in our surveys can expect, we publish a Respondent Charter for Surveys of Households and Individuals and a separate survey. Similarly, for each census we explain publicly how we will conduct the census and what will happen to the data collected, including our commitment that census records will be kept confidential for 100 years.
We design our surveys to minimise the burden placed on respondents, only collecting the data necessary to fulfil our functions, meet users’ needs and serve the public good. We publish details of all the surveys we run.
We further reduce the need for surveys by using non-survey data sources in compiling some of our statistics. This includes administrative data obtained by public or private sector organisations in the course of undertaking their normal operations, rather than our collecting data specifically for statistical purposes.
Wherever possible we follow the principle of “collect once, use many times” to produce statistics from existing data sources and avoid duplicate requests being made to those providing data.
When collecting data, regardless of whether from survey or non-survey sources, we comply with the relevant legislation.
Summary of main data collection principles:
we will only collect data for statistical and research purposes and where access is lawful
we will collect data transparently and treat data providers with respect
we will minimise the burden placed on respondents to surveys
we will aim to fully exploit non-survey data for statistical purposes
we will keep individuals’ and businesses’ information secure and confidential, regardless of whether it has been collected from survey or non-survey sources
we will collect data via web-scraping for the production of statistics and research that serve the public good, following practices and procedures set out in the ONS web-scraping policy
ONS collects data via web-scraping to use in the production of statistics and research that serve the public good. ONS’s web-scraping policy defines the following principles that must be followed:
seek to minimise burden on website owners
honour requests made by website owners to refrain from scraping their website
protect all personal data in all statistics and research outputs and seek ethical advice when scraping data that may identify individuals
apply scientific principles in the production of statistics and research based on web-scraped data and consider other sources of data
abide by all applicable legislation and monitor the evolving legal situation
These principles are applicable to all ONS staff activities involving web-scraping or which use web-scraped data. Additionally, when obtaining or procuring web-scraping services from a third party, ONS will ensure that these main principles have been followed. The policy does not cover the use of application programming interfaces (APIs) and is not applicable to government departments outside of ONS.Back to table of contents
We use data from surveys, the census and administrative sources for statistical and research purposes only. In our work, we adopt statistical methods that are professional, ethical and transparent.
At ONS we use data to produce official statistics related to the economy, population and society at national, regional and local levels. We produce official statistics for the public good – our statistics are for the benefit of society and the economy generally.
We are always looking to engage with users of our statistics to ensure that we are using data, including data used for research purposes, in ways that are helpful and which inform decision-making by government, public services, business, researchers and the public.
In our use of data, we are committed to meeting consistently high standards of professionalism and ethics. We follow the principles and protocols for the production of official statistics set out in the Code of Practice and supplement this with independent advice from leading experts.
On ethical considerations concerning data, we seek advice from the National Statistician’s Data Ethics Advisory Committee.
We publish details of the statistical methods we adopt and why we adopt them on the methodology pages of our website.
We follow the principle of “collect once, use many times” to make full use of data and to secure the greatest possible public benefit from the data we hold. By linking datasets together we can understand better the relationship between different attributes, or corroborate analysis using alternative sources. We adopt linking processes that deliver efficient and accurate linkage while protecting the identities of individuals and businesses.
We use the data we collect from surveys and administrative sources for statistical and research purposes only. Respect for confidential personal information is embedded in the Code of Practice and is a fundamental tenet of the UK Statistics Authority and ONS, and the Statistics and Registration Service Act 2007.
Summary of main data use principles:
the purpose of our work is to inform decision-making by government, public services, business, researchers and the public
we comply with the Code of Practice for Statistics as fully as possible in the production of our statistics
we only use data from surveys, the census and administrative sources for statistical and research purposes
we seek expert advice to ensure the statistical methods we develop and adopt are fit-for-purpose, ethical and in line with international standards
we document online the methods we adopt and why we adopt them
as the executive office of the UK Statistics Authority, we share the statutory objective of “promoting and safeguarding the production and publication of official statistics that serve the public good”
The spatial dimension of data plays a critical role in the production of statistics and use of data for research purposes.
Accurate geographic information is essential to:
locate and reference events
provide the geographical framework for sampling, collecting, linking and processing survey, administrative and other data
analyse and present statistics and research outputs – unlocking new information about places and relationships
select and access information for a range of geographic areas to meet policy needs
ONS is a leading player in maintaining the UK’s geographic frame. Spatial referencing and methods are embedded at all stages of our process – from data collection though to analysis.
The following are main principles of our Geospatial strategy:
the geospatial element is treated as a fundamental part of all of our data – with geospatial thinking, standards and methods embedded into all stages of the data journey
in time, data in the ONS data platform will be geo-referenced and stored to agreed and published architecture and standards – enabling the integration of data across domains and the use of common geospatial reference data
we will adopt the most appropriate spatial infrastructure and tools – continually reviewing our use of geospatial technology to respond to emerging and future data sources and technologies
geographic information system and geospatial analysis tools will be made available, promoted and supported for everyone involved in analysis of ONS data for statistical and research purposes
ONS maintains and complies with the Government Statistical Service Geography Policy.Back to table of contents
A classification follows prescribed rules and guidelines, which are generally recommended and accepted to ensure that data are classified consistently. Classifications are a central part of statistical infrastructure, and are critical for producing coherent statistics and drawing meaningful comparisons.
We ensure that we use and maintain statistical frameworks in line with the best practice outlined by the United Nations Statistical Commission.Back to table of contents
The Data Retention, Data Archiving and Data Disposal Policy sets out clear principles for the management of data throughout their lifecycle. The policy ensures that data owners understand their responsibilities for regular review of data according to prescribed timeframes from stakeholders, or to ensure compliance with data protection legislation.
Summary of main data retention, data archiving and data disposal principles:
datasets acquired from third parties are retained for the time period defined by the memorandum of understanding or contract
datasets generated by ONS are retained
data minimisation is applied for personal, or special category data
regular data and metadata reviews are performed dependent on the sensitivity of the data
additional uses of the data are considered before destruction to maximise the benefits of our data holdings
archiving of data is considered at the point the data are no longer of use by ONS to ensure their continued availability and functionality
The management of data in ONS must comply with the ONS data principles and data standards. The data lifecycle extends across the following stages:
data manipulation and processing
data sharing and dissemination
Data should be validated and cleansed on receipt, and then modified (as necessary) to conform to the data standards and to ensure that data are harmonised to facilitate re-use. Standardised statistical datasets should be linked to reference data (as appropriate) and links to geospatial co-ordinates should be established to enable geospatial analysis of data. Reference datasets include statistical classification codes and also statistical indexes, such as the business index and the address index.Back to table of contents
Metadata provide information about other data, including a description of the data. This includes information that provides context to the data, for example, how they were collected, or the coverage of the data. Metadata typically include publication date, description and search keywords. In order to understand the data we hold, ONS will hold and manage a comprehensive set of unambiguous metadata relating to the data we manage.
The metadata will be categorised and managed according to the ONS metadata management framework, which includes data principles and data standards relating to metadata. The metadata standard defines the “minimum set of metadata” for ONS datasets. All data acquired by ONS must be accompanied by metadata.
Metadata items will be completed by data owners, who will be responsible for periodically reviewing the metadata is correct and of high quality. The data owner will initiate corrective action to address any issues found with accuracy and quality of the metadata.Back to table of contents