Jennifer Davies, Tony Crook and Matthew Greenaway
This article is part of the series of articles launched on 14 September (Guled, 2012), which intend to keep users informed of statistical continuous improvements being carried out across the Office for National Statistics (ONS). This is the third article in the series and reports on the evaluation of the sample design and estimation methodology used for the Construction Output Survey. Adjustments to the sampling methodology are recommended and areas for further investigation are identified. Progress on the recommendations and further investigations into the Construction survey will be reported every six months.
The investigations detailed in this article suggest that the current sample design is performing quite well, but it is possible to make some small gains in precision to estimates of Total Output without extra costs. We recommend re-allocating the sample, minor adjustments to the turnover element of the stratification, and suggest potential improvements to the sampling fractions and stratification of businesses with few employees.
As part of the ONS commitment to being the place where people come first for trusted statistics, ONS continuously reviews and improves data sources, methods and systems. This also assists users in their understanding of the quality of the statistics, their fitness for purpose and their relevance. This article covers continuous improvement work being undertaken in relation to the sample design and estimation for the Construction Output Survey.
Responsibility for the Construction Output Survey passed from the Department for Business, Innovation and Skills (BIS) to ONS in January 2010. Based on BIS quarterly data a number of changes to the survey were made, including changing the periodicity of the survey from quarterly to monthly (Office for National Statistics, 2010). As several years of monthly data are now available, a full evaluation of the sample design and estimation methods used on the survey is possible.
This article is solely concerned with sampling error – that is, random error caused by taking a sample rather than a census. We assume the current sample size of 8,000 businesses will remain unchanged and therefore we focus on identifying cost-neutral reductions in sampling error. A wider post-implementation review of the Construction Output Survey will be investigating non-sampling error and other methodological issues (such as seasonal adjustment): for more details see Jones and Walker (2012).
The Construction Output Survey is a monthly survey designed to estimate output in the Construction industry in Great Britain. The survey selects 8,000 businesses per month from 220,000 Construction businesses on the Inter-Departmental Business Register (IDBR). Construction estimates are released monthly – in Output in the Construction Industry – and contribute 6.8% to the output measure of Gross Domestic Product (GDP).
This report evaluates the sample design and estimation methods used on the survey.
Sample design refers to the method used to (randomly) select a sample from a sampling frame. In relation to the Construction Output Survey these are the methods used to select 8,000 businesses per month from the 220,000 Construction businesses on the IDBR. The focus of the sample design evaluation is on the sampling fractions, which determine the probability that a particular business will be included in the sample.
Estimation refers to the method used to produce estimates for all units in the population of interest using the collected survey sample data. In relation to the Construction Output Survey these are the methods employed to use the Construction Output data returned by the 8,000 selected businesses to produce estimates of Construction Output for the 220,000 Construction businesses on the IDBR.
Estimates produced from data collected from a sample of 8,000 businesses will differ from those that would be produced if data were collected from all registered Construction businesses on the IDBR (the ‘population’). If all businesses were selected (i.e. a census), population totals would be (in theory) 100% accurate. The difference between these ‘true’ population totals and the population estimates derived from a sample is known as sampling error. If the estimation method is unbiased, the sampling error will be random – i.e. the population estimate is equally likely to be greater or smaller than the true population value.
The average sampling error – otherwise known as the standard error – is used to calculate confidence intervals, which give an indication of the plausible range of a survey estimate. For example:
in August 2012, estimated total construction output was £9,647 million with a standard error of £111 million and a 95% confidence interval of £9,647 million ±£217 million. If a large number of samples were taken then the confidence intervals from 95% of these samples would cover the true value and therefore the confidence interval gives an indication of the interval in which the true value could plausibly fall.
In addition to calculating standard errors, to assess statistical efficiency, this report also uses the coefficient of variation, which is the standard error expressed as a percentage of the estimate to which it refers.
Standard errors, confidence intervals and coefficients of variation are published in Table 11 of the Output in the Construction Industry release. The coefficients of variation reported here may differ from those published as more up-to-date survey data has been used.
3. Selecting the sample
International best practice on surveys of businesses is to utilise stratification. This entails splitting the population up into non-overlapping sub-populations called strata and sampling independently from each stratum. If the strata are defined such that businesses within a stratum are similar with respect to the variable of interest then stratification will result in smaller standard errors (Cochran, 1977); and stratification based on the domains for which estimates are published can help ensure all published estimates are based on sufficient sample sizes.
Sample selection for the Construction Output Survey involves two processes: (1) defining how the construction business population will be grouped into strata; and (2) selecting the sample in each strata. The following sections provide a summary of investigations into improvements of these processes.
4. Defining Construction Output Survey Strata
The Construction Output Survey is currently stratified according to industrial classification based on SIC 2007 (Office for National Statistics, 2009) and several ‘employment bands’. The industrial classifications used are based on the domains for which estimates are produced, which offers substantial advantages as it allows us to control the variability of domain-level estimates, and therefore we have not investigated altering this part of the stratification. Instead, investigations on statistical efficiency have focussed on adjusting the employment bands. The existing employment bands have different sampling scheme approaches using either random samples or including all businesses in the band. The current employment bands are shown in table 1 and show that construction businesses allocated to bands 4 and 5 are all selected for inclusion in the sample (full enumeration).
Table 1: employment bands currently used
|Employment band||Description||Sampling scheme|
|1||0-4 employment||randomly sampled|
|2||5-9 employment and 10-19 employment with turnover<£60m||randomly sampled|
|3||20-99 employment with turnover<£60m||randomly sampled|
|4||100+ employment||full enumeration|
|5||10-99 employment with turnover £60m||full enumeration|
As part of the evaluation alternative definitions of the employment bands were considered within each industry group – two such alternatives are given in table 2. The band for businesses with 10-99 employment and turnover of £60 million or greater (band 5) was maintained throughout. One set of alternative stratum boundaries fixes the 100+ threshold for fully enumerated businesses, while another allow the employment threshold at which businesses are fully enumerated to vary. The position of the employment boundaries were determined using standard methods for setting stratum boundaries implemented using the R package ‘stratification’ (Baillargeon & Rivest, 2011; R core development team, 2010).
When fixing the 100+ threshold the resultant employment bands do not change largely from the current bands. When the 100+ full enumeration threshold is not fixed, the theoretical optimum boundaries for full enumeration varied between 24 and 70. Businesses with employment exceeding these thresholds would all be fully enumerated and therefore the census part of the population would increase meaning the sample size for the sampled bands (bands 1-3) would decrease.
Results from a sample re-allocation – i.e. deciding on the number of businesses to sample within each stratum – using the current and the alternative stratum boundaries showed no systematic improvement in the coefficients of variation.
Table 2: alternative employment bands considered for each industry group
|Industry group||Employment band keeping 100+ threshold fixed||Employment band allowing 100+ threshold to move|
|Construction of residential and non-residential buildings||0-3, 4-17, 18-99, 100+||0-2, 3-7, 8-24, 25+|
|Construction of roads and railways and Construction of utility projects||0-2, 3-11, 12-99, 100+||0-5, 6-21, 22-63, 64+|
|Construction of other civil engineering projects||0-2, 3-8, 9-99, 100+||0-3, 4-10, 11-35, 36+|
|Demolition and site preparation||0-4, 5-19, 20-99, 100+||0-4, 5-17, 18-58, 59+|
|Electrical installation||0-3, 4-15, 16-99, 100+||0-3, 4-12, 13-56, 57+|
|Plumbing, heat and air-conditioning installation||0-3, 4-15, 16-99, 100+||0-3, 4-13, 14-69, 70+|
|Other construction installation||0-3, 4-16, 17-99, 100+||0-2, 3-10, 11-39, 40+|
|Plastering||0-2, 3-9, 10-99, 100+||0-2, 3-8, 9-33, 34+|
|Joinery installation||0-3, 4-15, 16-99, 100+||0-3, 4-11, 12-49, 50+|
|Floor and wall covering||0-2, 3-9, 10-99, 100+||0-2, 3-8, 9-41, 42+|
|Painting and glazing||0-3, 4-13, 14-99, 100+||0-3, 4-11, 12-57, 58+|
|Other building completion and finishing||0-2, 3-8, 9-99, 100+||0-2, 3-6, 7-23, 24+|
|Roofing activities||0-3, 4-14, 15-99, 100+||0-3, 4-10, 11-40, 41+|
|Other specialised construction activities||0-4, 5-18, 19-99, 100+||0-3, 4-9, 10-27, 28+|
With regards to band 5, investigations did suggest that lowering the turnover threshold to around £30 million would reduce standard errors; while lowering the threshold to less than £30 million means the benefit from taking a census of these businesses is outweighed by the loss in precision seen from taking a reduced sample in bands 1-3. It is worth noting that lowering the threshold would increase the number of businesses which are fully enumerated every month, so placing more burden on these businesses which may already be involved with other ONS surveys. We recommend that the turnover threshold for band 5 be decreased to £30 million, which will result in approximately 80 extra businesses being fully enumerated.
5. Sample allocation
5.1 Overall level
The proportion of the population included in the sample in each stratum is called that stratum’s ‘sampling fraction’ – so, for example, a sampling fraction of 0.5 means half the sampling frame in that stratum is included in the sample. The optimal sampling fraction in each stratum can be determined using a Neyman allocation, which minimises the standard error of estimates of overall totals or means subject to a fixed sample size (Cochran, 1977). The optimal sampling fraction given by a Neyman allocation will depend on the observed variability within a stratum; the more variable the responses in a stratum the larger the sampling fraction.
The sampling fractions currently used were determined using quarterly data from BIS. As the ONS has been running the Construction Output Survey on a monthly basis for nearly three years the sample was re-allocated using these newer data, without changing the definition of the strata.
Although the allocation is performed at the strata level, a summary of the Neyman allocation and current allocation is presented by employment band. It is assumed that a census is still carried out of business with 100+ employment or employment between 10 and 99 with turnover of £60 million or greater (bands 4 and 5).
Table 3: the current sample size and the sample sizes under the Neyman allocation in employment bands 1 to 3
|Employment band||Population count January 2011||January 2011 sample size||Neyman allocation||January 2011 sampling fraction||Neyman allocation sampling fraction|
The sampling fractions for the three sampled employment bands are not substantially changed between the current and alternative allocation, but the strata-level sampling fractions do vary.
The coefficients of variation for the estimates for Total Output in each month of 2011 are provided under the current and Neyman allocations. In general the coefficients are improved by reallocating the sample, particularly towards the end of the year.
Table 4: coefficients of variation of the monthly estimates for 2011 produced under the current allocation and the Neyman allocation
|Month||CV, Current allocation||CV, Neyman allocation|
5.2 Sector level
Estimates for Construction Output by sector are produced in addition to estimates for total Construction Output, and the accuracy of the sector estimates is also important. An alternative (multivariate) allocation was investigated which based on the coefficients of variation of output in each sector as well as Total New Work, Total Repair and Maintenance and Total Output (Preston, 2004). This allocation balances minimising the variance of Total Output with reducing the variability of some of the more variable sector estimates.
The multivariate allocation was performed at the strata level, and a summary is provided by employment band in table 5. The coefficients of variation under the current allocation and the multivariate allocation are provided in table 6. The multivariate allocation reduces the coefficients of variation towards the end of the year compared to the current allocation, but again the gains are not large, and the multivariate allocation does somewhat worse than the Neyman allocation.
Table 5: the current sample size and the sample sizes under the multivariate allocation in employment bands 1 to 3
|Employment band||Population count January 2011||January 2011 sample size||Multivariate allocation||January 2011 sampling fraction||Multivariate allocation sampling fraction|
Table 6: coefficients of variation of the monthly estimates for 2011 produced under different allocations
|Month||CV, Current allocation||CV, Multivariate allocation||CV, Neyman allocation|
The multivariate allocation was produced to meet a set of constraints on the coefficients of variation of the sector level estimates. In the example presented above, the constraints have focused on reducing the larger sector-level coefficients of variation under the current allocation. The coefficients of variation under the current allocation and the multivariate allocation one such sector level estimate – Infrastructure New Work - is shown in figure 1. Under the current allocation, the coefficients of variation for Infrastructure New Work are larger than most other sector level estimates towards the end of the year. The multivariate allocation reduces these by around three percentage points for some of the later months of 2011.
Figure 1: coefficients of variation of the monthly estimates for Infrastructure New Work in 2011 produced under the current allocation and the multivariate allocation
Figure 1 download (27 Kb Excel sheet)
Although it is possible to use a multivariate allocation to limit the variability of the more variable sector level estimates, this does have a detrimental impact at the overall level - the average coefficient of variation for Total Output under the multivariate allocation is 1.4% compared to 1.3% for the Neyman allocation. In practice the accuracy of the estimate for Total Output is more important as it is this estimate which contributes to the Output measure of GDP.
5.3 Small businesses
To help manage the burden which surveys may place on small businesses, the ONS implements the Osmotherly Guarantee (Osmotherly, 1996). This limits the number of periods for which a business with fewer than 10 employees may be included in a monthly survey to 15 months, while larger sampled businesses are currently included in many ONS monthly surveys for 27 months. The current 5-19 employment band overlaps this 10+ employment boundary, meaning that all businesses in this band are rotated out every 15 months. Splitting this band into two bands, 5-9 and 10-19 would allow the 10-19 businesses to remain in the survey for the full 27 months. A longer rotation period allows for greater overlap of businesses being in the sample at two different points in time meaning that period on period change estimates are improved. Longer rotation periods also have substantial practical advantages, as they reduce the number of new businesses entering the survey each month and therefore alleviate some of the practical difficulties associated with businesses being included in the survey for the first time.
Splitting the 5-19 band into two new bands, 5-9 and 10-19, and then drawing a Neyman allocation based on these new bands could improve the accuracy of the estimates of Total Output, producing coefficients of variation for each month of 2011 of around 0.1 percentage points lower than the Neyman allocation on the current five band system, and this change would aid the implementation of the Osmotherly Guarantee. We therefore recommend that the practicality of adding an extra employment band is investigated.
Smaller businesses may find surveys more burdensome than larger businesses and also the response rates of small businesses can be lower. Although deviating from the optimal sample allocation will increase standard errors, it is of interest to consider the extent of the impact on standard errors of sampling fewer small businesses and compensating by sampling more medium and large businesses.
A Neyman allocation allows costs to be specified for particular employment bands, and several different ‘costs’ were tested for small businesses. The results were surprising – sampling fewer small and more medium/large business did not substantially increase the coefficients of variation for Total Output. This may be because the variance is a relatively flat function and therefore deviating slightly from the optimal allocation does not have a large impact on the variance. If burden and/or response rates are an issue for small businesses in the Construction Output Survey, the option of defining cost functions in the Neyman allocation should be explored further. This approach could be considered by ONS in general and not just specifically for Construction Output.
Estimation refers to the process by which an estimate for the population is obtained from sample data.
The Construction Output Survey currently uses ratio estimation, which reduces the standard errors of estimates by utilising extra information from an ‘auxiliary’ variable in addition to sample data. Ratio estimation corrects for some of the randomness of sampling by using the difference between the auxiliary variable in the sample and the auxiliary variable in the population. The auxiliary variable needs to be known for each sampled business and for the whole population, and ratio estimation will only reduce standard errors if the auxiliary variable is well-correlated with the variable of interest.
The Construction Output Survey currently uses turnover on the Inter-Departmental Business Register (IDBR) as an auxiliary variable, and IDBR employment was investigated as an alternative. The resulting average coefficients of variation across 2011 for Total New Work, Total Repair and Maintenance and Total Output for 2011 are presented in Figure 2.
Figure 2: average coefficients of variation of the monthly estimates for 2011 produced using different auxiliary variables
Figure 2 download (17 Kb Excel sheet)
Using turnover as the auxiliary variable consistently produces lower coefficients of variation for Total New Work and Total Output. However, for Total Repair and Maintenance, employment performs somewhat better. This is also true of the sector estimates – turnover tends to perform better for the New Work variables, while employment tends to perform better for the Repair and Maintenance variables.
Table 7: average correlation of turnover and employment with Total New Work, Total Repair and Maintenance and Total Output
|Total New Work||0.62||0.46|
|Total Repair and Maintenance||0.30||0.42|
The difference in performance can be explained by the correlations shown in table 7 - turnover is more highly correlated with New Work variables and employment with Repair and Maintenance variables. As New Work constitutes a larger part of Total Output, and turnover correlations are in general higher than employment, turnover performs better for total output, and therefore turnover is the better choice of auxiliary variable. We therefore do not recommend any change to the auxiliary variable used in ratio estimation method.
This article has evaluated the sample design and estimation methods used on the Construction Output Survey, utilising the monthly survey data that have been collected since January 2010. Only issues relating to sampling error have been considered – i.e. variability related to the fact estimates are based on a sample of 8,000 businesses rather than a census of all 220,000 businesses.
The Construction Output Survey utilises stratified sampling and the sample was originally designed using quarterly survey data from BIS. We have evaluated the impact of re-allocating the sample using an optimal Neyman allocation in accordance with international best practice, and also of altering the stratum boundaries using statistical theory on sample design. In general, the gains in precision from altering the sample design are small, indicating that the current sample design is performing quite well given the current sample size of 8,000 businesses. It would be possible to reduce sampling error by more by increasing the sample size.
As it is possible to make some gains in precision to estimates of Total Output under the current sample size, we recommend re-allocating the sample using a Neyman allocation. We have also investigated allocation accounting for sector-level estimates, but as the coefficients of variation of these estimates are not prohibitively large, we recommend using the Neyman allocation as this optimises the estimate at the overall level. We have recommended minor adjustments to the turnover element of the stratification, and suggested potential improvements to the sampling fractions and stratification of businesses with few employees.
Estimation on the Construction Output Survey currently uses a ratio estimator with turnover as the auxiliary variable. This is confirmed as the approach producing the smallest standard errors, and therefore no change is recommended to the estimation methodology.
Baillargeon, S. and Rivest, L.-P. (2011) Stratification: Univariate Stratification of Survey Populations, R package version 2.0-3. accessed on 23/11/12.
Cochran, W. (1977) Sampling Techniques, Wiley.
Guled, G. (2012) Reviewing and Improving ONS Statistics: Measurement of UK Private Non-Financial Corporations’ Overseas Deposits and Loans, accessed on 10/12/12.
Jones, J. and Walker, G. (2012) Continuous Improvement of Gross Domestic Product: Sources, Methods and Communication, ONS Statistical Continuous Improvement, accessed on 23/11/12
Office for National Statistics (2009) UK Standard Industrial Classification of Economic Activities 2007 (SIC 2007): Structure and explanatory notes, accessed on 23/11/12
Office for National Statistics (2010) New Construction Output Series: Analysis of improvements and their impact (114.6 Kb Pdf) , accessed on 23/11/12
Osmotherly, E., Graham, T. And Pepper, M. (1996) Statistical Surveys: Easing the Burden on Business, accessed on 10/12/12
Preston, J. (2004). Optimal Sample Allocation In Multivariate Surveys: An Integer Solution, Unpublished document, Australian Bureau of Statistics.
R Development Core Team (2010). R: A language and environment for statistical computing, R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, accessed on 23/11/12
UK Statistics Authority (2009) Code of Practice for Official Statistics.