Summary
Sample Design and Estimation (SD&E) is the name of one of the centres in the Office for National Statistics’s (ONS’s) Methodology Group; its staff work across our sites on a variety of research and support projects. This work helps maintain and develop the sampling and weighting methods used to derive the Office’s statistical outputs.
SD&E can be contacted via the Methodology email address.
What are sample design and estimation?
Sample design
Sample design covers all aspects of how the samples in our surveys are specified and selected.
The design of samples is a particularly important aspect of survey methodology, as it provides the basis for sound measurement of economic and social phenomena from surveys of businesses and households in the UK.
Topics include:
sample frames, which list the businesses and household addresses we might select for a survey
types of sampling (for example, stratified and clustered designs), which specify how we group businesses and households when sampling to provide more efficient and effective designs
sample size specification, with implications for cost, respondent burden, and output quality
sample selection mechanisms, which are used for choosing, with a random element, the particular units (households, people, businesses, schools) that will make up any given survey sample
Estimation
Estimation is also known as weighting.
It describes the method by which we produce estimates about the characteristics of the population as a whole from the responses given by those people and businesses selected in the sample survey. Those responses from the sample are weighted to ensure they represent the entire population without bias and produce good quality outputs.
Alongside the estimates themselves, we also calculate variances. These are measures of precision, and give an indication of how much the estimates might vary if different samples had been drawn, rather than the particular sample realised on this occasion. Related measures often presented in publications are standard errors, coefficients of variation (CVs) and confidence intervals.
Aims of the Sample Design and Estimation Centre
to provide support for the production of official statistics at ONS and elsewhere in the Government Statistical Service (GSS), through the specification and implementation of appropriate sample design and estimation methodology
to improve the overall quality of statistical outputs under given constraints through research into, and application of, effective sampling and estimation methods
to promote integration, harmonisation and other improvements in the consistency of ONS outputs through the use of, for example, common methods and processes
to conduct research in areas that include:
- use of alternative sampling frames from different or multiple sources
- design of appropriate sampling schemes for a range of survey objectives
- consideration of the burden placed upon survey on respondents, ensuring this is minimised as far as possible and distributed in a reasonable and justifiable way
- weighting and estimation methods, including for weighting adjustments to account for non-response
- use of auxiliary information to improve the efficiency of sampling and estimation, and which provides estimates that are calibrated and consistent with other ONS outputs
- detection and treatment of outliers in surveys
- estimation of variances and other quality measures
- making better use of administrative data in various ways in estimation and sampling
Within the GSS, the SD&E Centre also provides consultancy and training. These are usually delivered via the Methodology Advisory Service and GSS Learning Academy.
Structure of the SD&E Centre
The SD&E Centre is split into teams that specialise in different applications:
social surveys
business surveys
population statistics, which covers the design and estimation of the census
These teams carry out similar work in principle, but on surveys that have different topic areas and in which the application of the principles may take different approaches.
Social surveys
SD&E (Social Surveys) branch provides support services for both the regular surveys that ONS runs, such as the Labour Force Survey and for other, ad-hoc surveys carried out by ONS, sometimes for other government departments or those won by tender.
The branch designs sampling procedures for these surveys and works closely with the Sampling Implementation Unit in ONS, which is responsible for drawing the samples. The designs employed are typically stratified, multi-stage samples. Some surveys are purely cross-sectional, whereas others have a longitudinal element to the design and follow the same people or households from one wave of the survey to the next.
Samples of the private residential population are usually drawn from Royal Mail’s Postcode Address File (PAF). Addresses are selected from the frame using a systematic sampling procedure, which ensures a good geographic spread in the sample and includes provision for the more complex sample designs.
The vast majority of ONS social surveys are concerned with estimating characteristics of people and households. The branch develops the weighting and estimation strategies, which take account of the complex sample design and, for one-off surveys, often implements the estimation for survey output areas by providing weights for the responding sample. The final weights provided take account of non-response and are calibrated so that the sum of the weights in a number of sub-groups corresponds to the mid-year population estimates published by ONS.
The branch also derives methods that enable the production of standard errors that take account of the complex survey designs and go beyond the limits of generic statistical survey estimation software.
SD&E (Social Surveys) work has included:
research into designs for integrating social surveys and use of modular survey designs; much of the theory was developed for ONS’s Integrated Household Survey, but will prove useful for possible design used in connection with electronic data collection
development of methodology for weighting longitudinal surveys, which enables ONS to develop new estimates and outputs; examples of this sort of work include:
designing the sample and weighting schemes for the Child Dental Health Survey
re-designing aspects of the International Passenger Survey sample, helping to improve estimates of migration
developing efficient statistical computing algorithms to estimate standard errors when using complex survey designs; such software had previously been unavailable, or was impractical with large data sets
developing various initiatives to understand better response patterns in social surveys and to update non-response weighting procedures using information from the census; the aim of the latter work is to reduce the potential for non-response bias
Business surveys
SD&E (Business Surveys) branch has a role of maintaining and improving the sampling and estimation methods used in ONS business surveys, the responses from which form the main input to the UK’s principal economic indicators, such as gross domestic product, retail sales and measures of inflation.
Most business surveys’ samples are drawn from the Inter-Departmental Business Register (IDBR). The sample designs are usually stratified by economic activity and size of business, with fixed sample sizes that are allocated across the different strata (groups of businesses) in a way that results in best precision overall for the survey outputs.
The IDBR itself is updated continuously from administrative sources and annually, from a specially designed, updating survey. Part of the branch’s role is to support ONS’s Register Unit in maintaining the IDBR.
The branch also supports survey output areas during results processing (estimation) and works closely with them when developing new methods, and quality assures the implementation.
SD&E (Business Surveys) work has included:
re-designing many business survey samples to accommodate revised, internationally consistent methods and classifications for measuring the economy
researching re-sampling methods to estimate variances in the case of complex estimators
investigating potential coverage issues on the IDBR
improving sample selections in a way that more fairly distributes response burden
use of administration data in the short-term statistics as a possible replacement for survey data
analysing the impact of electronic data collection initiatives
Population statistics
SD&E (Population) branch develops the sampling and estimation methodologies for estimating the size of the population from the 10-yearly census for England and Wales. This includes the sample design for the Census Coverage Survey, the use of administrative data to improve the process and the production of confidence intervals for the resulting population estimates.
Work is underway for the 2021 Census alongside considering the potential for producing population estimates using administrative data alone.