Skip to content

National Statistics Methodology Advisory Committee 12th meeting 11 May 2007

Twelfth meeting, 11 May 2007

The following work-in-progress papers were considered:

Paper 1: Population Statistics: Methods and Quality Measurement

This paper looked at problems associated with evaluating the relative benefits of new methods for population estimates. Assessment has been limited owing to the difficulty in obtaining objective quality measures when the methods used to derive the estimates draw upon multiple data sources and when there is a lack of independent data sources with which to corroborate. The paper presented a project to better understand the sources of uncertainty associated with population estimates and to investigate the use of simulation based methods to measure the propagation of error. It is proposed that the benefits of Bayesian approaches are investigated as a longer-term objective.

Committee conclusions:

  • This is a complex problem and the principles in understanding uncertainty need to be defined

  • Models are required and this is challenging for administrative data. Whatever method is used would need to be simple but fairly robust. We would need to view it with caution owing to uncertainties and there is no obvious method that would provide confidence intervals

  • Methods that are used must be anchored with what is known, such as calibrate to the census.
    Peter Kenny 1984 - 86 published in Economic Trends on simulations that may be useful

  • Bayesian models take into account much uncertainty but are underpinned with subjective judgement. The results can only be as good as the model and for an effective model we need good priors

  • Look at the work of J. M. Ahlo, who had produced a modelling programme for simulation

  • Understanding the error processes in administrative data was important and getting at them was critical in the use of these data sources. Comparisons between sources are important; they can be used for more intensive checking and validation of data

Paper 2: Combining Data: Developing a Centre in Methodology Directorate to meet the challenges

This paper discusses data sharing and matching. These have the potential to improve the quality and timeliness of current statistical outputs and to enable new ones to be developed. As a result there are a growing number of projects within the Office for National Statistics (ONS) that are concerned with linking administrative data to other administrative or survey sources at the individual level. Methodology Directorate need to develop knowledge and expertise in record linkage techniques and methods for using combined data sources to provide support and leadership within ONS and the Government Statistical Service (GSS).

Committee conclusions:

  • It is important to develop quality standards and performance criteria for results. Performance measures should be dependent on aims

  • An advantage with using combining approaches is that they may help identify potential weaknesses in the sources

  • More advanced methods could be explored including global matching, rather than simply one-to-one matching

  • Public acceptability needs to be considered. There are clear legal issues when exploring the use of data for purposes other than those for which they were collected

  • Meta-analysis is important, in particular summaries of statistical matching. Issues around what to do when there have been changes in the definitions used in administrative sources need to be addressed.
    Expectations around what can be achieved with combining data needs to be carefully managed. Quality issues around the matched sources will still remain. An emphasis should be made on understanding quality in terms of the limits of the different kinds of inference possible

  • An Economic and Social Research Council (ESRC) invitation to tender has been raised into secure data linkage

  • Ownership of linked data is a possible problem in future as both original owners of the data may claim ownership

Paper 3: In Search of a Question on Sexual Identity

There is an increasing requirement for information on sexual orientation. A programme of work was established to determine the most suitable way to meet this need. This paper discusses the background to this project, the data collection issues surrounding implementation, and the proposed research methodology employed by ONS. It is aimed to provide advice on best practice with regard to data collection in this field, and also examine the feasibility of providing benchmark data. The primary outputs from this project will be a question, or suite of questions, asking people to self-identify to a particular sexual orientation, along with advice on administration. Alongside the question(s), a user guide will be produced discussing the conceptual issues as well as the methodological issues, such as context and mode effects.

Committee conclusions:

There is a need to maximise the benefits of each stage of the research, so it is important to be focused on a well-defined objective, particularly at the focus group and qualitative testing stages.

Paper 4: Business Register and Employment Survey: Progress on Design

This paper updates the committee on progress in designing the new Business Register and Employment Survey (BRES) and highlights some issues that have arisen during the development. A detailed survey methodology has been developed following investigations into:

  • sample designs and strata boundaries

  • the effectiveness of the ratio estimator using local unit responses

  • the current validation and automatic editing rules in the Business Register Survey and an assessment of their appropriateness for use in BRES

  • the most accurate imputation method for item non-response

  • the performance of a range of outlier treatments

Committee conclusions:

  • Plans should be made for dealing with difficulties in detecting and correcting for trend breaks owing to the implementation of a the new survey

  • With respect to the sample design, stratification was by size and complexity; stratification by type of activity should be considered

  • On the issue of editing and validation, it was suggested that it may be possible to exclude certain validation such as total checks given that they could be calculated directly from the data

  • Testing of the imputation methods may only be valid for the simulation of the data, if possible it would be better to use real data

  • A solution to the problem of gender splits would be to use ratio models from other surveys, possibly using the Annual Survey of Hours and Earnings (ASHE)

  • Look at the possibility of using; small area estimation, synthetic estimators and as many automatic checks as possible at the data collection stage

  • Regarding register updating, the necessary for this was questioned as out-of-date information does not necessarily imply bias, although it may reduce the efficiency of estimators

Paper 5: Savings to Surveys in 2007/08

ONS resources are limited and it has to make savings on its existing business and household surveys to meet new statistical pressures. This paper discusses some of the issues. It looks at what the short-term approach should be. It describes ONS's prioritisation tool 'OPTIONS' that should help ONS prioritise different surveys/activities/outputs to take into account changing customer needs, new technology and budgetary pressures. It also discusses what the longer-term strategic approach should be, specifically on the editing of business surveys and ONS's development of SNOWDON.

Committee conclusions:

  • Business plans include information about the likely effects of reductions in budgets on plans. There is a need to balance the benefits against the costs. However, costs are easier to measure than benefits particularly in a framework where users do not pay for products, as there is no good measure of benefits and different claims need balancing

  • Within 'OPTIONS' it would be preferable if the continuous variable was not reduced to a categorical one

  • SNOWDON was a useful means of looking at impacts of changes to validation gates

  • A useful tool would be a matrix which gives costs of surveys/outputs and options for cuts. Customers consultation on how they would choose to cut £X would give ONS information on the relative importance of different surveys or outputs. This would provide better information on how to allocate funds.
    Beware of making up front savings, as sometimes this can increase costs later, should have a holistic view of savings

  • Low response rates should not be tolerated as this increases bias

Content from the Office for National Statistics.
© Crown Copyright applies unless otherwise stated.