• cymraeg

• UK Snapshot
• Neighbourhood
• Economy
• Census
• About ONS
• Jobs

5.1 Identification of risks related to reliability

Statistical outputs and services will never, by their nature, be of perfect quality. Nonetheless they must be of adequate accuracy to fit their main purposes, they must measure the appropriate concepts and they must be timely and coherent.

The Code of Practice for National Statistics will highlight the principles of achieving high quality and trusted National Statistics. It will provide protocols to assist best practice and provide a framework for identifying and managing any particular areas of concern.

Components of risk relating to reliability are identified by considering the various elements of the statistical process from user consultation and development through to publication and archiving of the derived information. For each statistical process, the associated risks are identified. Table 2 sets out the various stages in the statistical process and the risks that are considered at each stage. The Office for National Statistics (ONS) takes part in international discussions, conferences and research to ensure reliability in statistical data through various stages. Methods used across International Statistical Agencies are compared and further developed. Benchmarking and peer review across agencies is undertaken to ensure that risks to reliability are identified.

An important means of identifying risk to the reliability of statistics is through well trained and motivated staff. These people are then able to identify risk as it arises in day-to-day work, and to develop effective management strategies. ONS places high priority on recruitment of people with the right skill base, training and development activities. It is accredited under Investors in People.

A number of business areas use the Business Excellence framework for this, while others use less formal approaches. Risks relating to business plans are prepared as part of annual planning, and a risk register is included in the annual business plan. All new collection activities, or significant changes to activities (including redevelopments, or the development of new statistical infrastructure) are managed as projects by the organisation, generally using procedures modelled on the PRINCE2 project management methodology, with risks identified and managed as part of this methodology.

A significant aspect of the reliability of statistics for users is trust that the figures have been objectively obtained using best scientific method. To achieve this, ONS provides openness in all aspects of its statistical work, and invites comments on its methods. It approaches both users and other statisticians, and through the delivery of papers to conferences and to relevant professional bodies. In addition ONS publishes descriptions of the methods used for all statistical outputs, and measures of quality achieved, on its website. As well as providing openness of information to guarantee objectivity, this approach encourages users and other professionals to help ONS identify areas of potential risk. ONS also has a methods area responsible for assuring that best possible methods are in place, and for development of a strategy to identify and manage risks in the methodological environment. This area uses expert consultants to assist in reviewing specific methods and identifying any further risks on a regular basis.

For each of its key economic and social statistics, ONS undertakes a continuing round of review work as part of the regular work of each area. This review work is aimed at identifying weak elements in our processes, methods and systems. For example, following each release of the quarterly National Accounts and Balance of Payments there is a review undertaken of the quarter which identifies particular areas of concern that might be addressed to reduce risk to accuracy or timeliness for subsequent quarters. A similar exercise takes place in RPI 'quality days'. Methodological reviews are held regularly to ensure that the National Accounts and Balance of Payments statistics are compiled according to the specifications of the ESA95 regulation and BPM5.

A similar quality assessment process is applied following the preparation of each monthly LMS First Release - if problems arise, an assessment is made of how to avoid them in the future.

A review of Census planning in respect of the 1991 Census was undertaken as input to the planning for 2001. Reviews of all aspects of the 1991 census have provided input to risks and issues associated with the implementation of the 2001 Census.

For each component of its outputs, and as part of the ongoing work, ONS tests its statistics against other information available. At the unit level, data from large businesses in one survey are confronted with data from related surveys for consistency. For medium and smaller businesses, the same is done at aggregated levels. In the context of labour market statistics, a programme of data reconciliation work has been put in place to investigate differences between the data for employment and earnings obtained from business and household survey sources. Failure of statistical outputs to provide a coherent picture points to potential quality risks, and the characteristics of any discrepancy can help identify the particular concern.

In addition to ongoing ONS review activity, the National Statistics Framework requires that all significant statistical outputs be reviewed at least once every five years, with the review including the use of independent expertise. In addition, the Statistics Commission has a remit to undertake particular reviews of National Statistics, to gain a view of the quality and independence of the statistics. This review activity provides an additional mechanism to identify risks to the quality of statistics.

Table 2: Statistical Processes and Associated Risks to Reliability

5.2 Processes for judging the likelihood and impact of identified risks related to reliability

The likelihood and impact of identified risks to reliability of statistics is assessed in the same way as the risk is identified. Assessment of the likelihood is made by the relevant statistical experts, based on past knowledge and experience, as well as international experience. In some areas the likelihood might be informed by specific measures, for example response rates are monitored during data collection to enable a timely assessment of whether an adequate level of response is likely to be achieved. If not, strategies can be set in place to increase response measures for the particular collection.

A significant element of the development work for implementing new products or methods and of the ongoing review work described in the previous section, is to not only identify risks to quality, but also to analyse, and where possible obtain a measure for the likelihood of the risk and the magnitude of its impact. In this work, simulation studies, including sensitivity analyses, are often used to better understand the likely impact of a particular risk for a set of statistics. In addition to specific evaluation work undertaken as part of the review activity, the impact of risk is assessed on the basis of past experience, the size of the problem, the nature of the statistical output and its use, and by discussions with users on the likely impact on their use.

5.3 Processes for evaluating risk management strategies related to reliability

The review processes used to identify new risks for statistics, also provide a means of evaluating past risk management strategies. Where a review picks up a problem with an aspect of an output, the problem is corrected, and an effort made to understand why the error occurred in the first place.

Where a problem arises, and after it has been solved, the business area is responsible for reviewing why the problem arose. There is not currently a formal approach adopted by each area, but there would generally be an analysis that would involve reviewing the risk register to ensure the risk had been listed and see if response actions could have been more appropriate. If the risk was not included on the register, the responsible area would be expected to consider why the risk had been missed and review both the risk register and risk identification procedures to see if further risks of a similar type may also have been omitted.