Service Lives of R&D Assets: Questionnaire Approach

Abstract

This publication presents information gathered from a new question on R&D service lives included in the 2011 Business Enterprise Research and Development survey. The origins and development of the question are overviewed and results are presented, along with assessment of statistical reliability. Findings indicate that the average service life of UK R&D lies between 6.0 and 10.5 years and that R&D lives vary across different industrial sections.

Acknowledgements

The author would like to thank Christopher Steer, Helen Meaker, and Walter Mkandawire of the Office for National Statistics for their valued support and contributions.

Introduction

The forthcoming capitalisation of Research and Development in the National Accounts of EU Member States and other countries creates a requirement for new information on the useful ‘service life’ of the resulting R&D assets.  Two alternative methods have typically been used to estimate R&D asset lives; collecting information through survey questions, and using information on patent renewals extracted from national Intellectual Property Protection systems.

This publication is from a suite of three which provide a comprehensive overview of Office for National Statistics (ONS) research into R&D service lives.  This paper describes the design and implementation, and presents results of a new service lives question on the UK Business R&D questionnaire.  ‘Service Lives of R&D Assets: Patent Approach’ (Ker, 2013c) derives estimates from patent renewals data, while ‘Service Lives of R&D Assets: Background and Comparison of Approaches’ (Ker, 2013a) provides context for the research, compares the methods and results for the UK, and draws conclusions.  It is suggested that these are read together.

Survey Approaches to Estimating R&D Service Lives

Statistical agencies usually carry out two surveys relevant for estimating asset lives; discards of assets by producers, and purchase dates and expected remaining useful lives of assets.  However, these concepts are not readily applicable to R&D; most R&D is produced on own account and is therefore not purchased or sold.  Furthermore, the concept of ‘discarding’ knowledge does not have obvious meaning; knowledge can be ‘kept’ indefinitely but at some point it will probably be rendered obsolete and cease to be an asset in terms of storing value derived from its usefulness in production.

This approach uses survey questions to collect information about how long respondents benefit from R&D by asking them directly.  Questions may be general; asking about the how long the respondent would expect to benefit from a typical investment in R&D, or specific; asking about the number of years over which a product embodying R&D was produced and sold.

The 2006 joint meeting of the ‘Canberra II’ expert group on the measurement of non-financial assets, and the National Experts on Science and Technology Indicators (NESTI) favoured survey approaches, ‘that could target major R&D performers in various industries to test whether they were able to provide expectations of the service lives of R&D’ (OECD, 2010, p. 62).  Following this resolution, exploratory surveys were undertaken in Israel, Germany, and the UK.  This added to experience from Japan and South Korea and together led the OECD to conclude that ‘it appears that obtaining service lives by surveying respondents is viable, but an assessment of full blown surveys by several countries are required to confirm this is so’ (OECD, 2010, pp. 62-66).

As a response to this, in 2012 ONS decided to build upon previous research into R&D service lives by designing a question for implementation in ONS R&D surveys.

Previous UK Surveys

In 2008, the ONS and the National Endowment for Science Technology and the Arts (NESTA) conducted an interviewer-led face-to face survey employing ‘cognitive’ techniques to detect differences in understanding between the surveyor and respondent (Whittard, Franklin, Stam, & Clayton, 2009).  Responses came from technical personnel, who were shown to be best able to provide answers, and covered 120 R&D projects across 40 firms.  The concept of a three-stage service life of ‘development’, ‘transition’ (to use), and ‘use’ stages was applied.

The survey found an average ‘use’ period of 8.6 years for ‘technical R&D’ (which has greater scientific and technological uncertainty) and service lives were shown to be longer in firms classified as ‘high-tech’, lasting 13.1 years in use on average compared to 8.4 years in ‘low-tech’ firms.  For non-technical R&D (focused on knowledge commercialisation and business process improvements) an average use of 5.0 years was found across ‘high’ and ‘low-tech’ businesses.  It also appeared that lives were shorter in services than manufacturing – perhaps because Intellectual Property Protection (IPP) systems cater more to physical products.

The overall conclusion reiterated that firms can answer questions of this type, though respondents from larger firms often struggled with the breadth of the questionnaire as their activities comprised a great many different types of R&D.  A closer focus on key questions of interest was recommended for the future.  Some firms also struggled to differentiate between “in-house” and “bought in” R&D, especially in larger projects.

In 2009 and 2011 the ONS conducted two rounds of the ‘Investment in Intangible Assets’ Survey (IIA) which collected information on intangibles such as training, software, branding, design, business process, and R&D.  These voluntary surveys gained good response rates of around 44 percent from 2-2.5 thousand private sector businesses drawn from the Inter-Departmental Business Register (Awano, Franklin, Haskel, & Kastrinaki, 2010), (Field & Franklin, 2012).  The survey asked “on average, how long does the business expect to benefit from a typical investment in Research and Development?” 

Once adjusted for outliers and ‘zero’ responses, which are theoretically challenging as rational businesses should not invest in an activity that they do not expect to benefit from, both surveys suggest an unweighted average of around 4.5 years.

However, the IIA estimates much lower total UK business R&D expenditure than the Business Enterprise R&D (BERD) survey which makes more comprehensive coverage of key R&D performers (£5.7bn compared to £16bn in 2011).  Furthermore, only around 90 of the R&D-performing respondents answered the question on service lives in each round and there was also no data from key R&D performing sectors such as pharmaceuticals and the Scientific R&D industry.

These results do not provide sufficient coverage for use in the National Accounts.  However, the IIA showed that firms could respond to questions of this type and so this was taken as the starting point in developing a new question for inclusion on the 2011 R&D surveys.

New Question

Starting with the IIA survey wording, two rounds of cognitive testing were undertaken to develop the question and ensure that firms can correctly interpret and answer it.  Responses were requested for ‘basic research’, ‘applied research’, and ‘experimental development’ separately to provide additional insight and offer a way for respondents to disaggregate their answers.  These different ‘types’ of R&D are defined in the Frascati Manual (OECD, 2002) and are used in other questions on the UK Business R&D survey.

At first some respondents were considering the useful lives of end products embodying R&D rather than the R&D itself. It was necessary to clarify that the period over which R&D is used to make (and sell) products is relevant.  Also, some respondents confused National Accounts R&D capitalisation with business accounting practice.  Modified supporting notes clarified that no alteration to International Financial Reporting Standards (IFRS) is implied by this change and advised respondents (who are often employed in firms’ accounting departments) to seek input from researchers and technical experts when answering this question.

Round two showed that the additional notes and guidance had addressed these problems and that respondents understood a response for a typical R&D investment or project was required. 

Implementation in ONS Surveys

The new question was implemented in the Business, Government, and Private Non-Profit R&D surveys.  The final wording is presented in Annex 1.

The Business Enterprise R&D (BERD) survey is an annual collection of expenditures, employment, and funding relating to R&D from ‘business enterprises’ as defined in the Frascati Manual (OECD, 2002).  It therefore excludes government organisations, higher education establishments, and charities.  BERD was designed to provide data on Science and Technology (of which R&D is a major part) for Government monitoring, planning, and policy uses, as well as being used extensively in academic research.  Businesses selected are legally required to respond.  Ker and Greenaway (2012) provide a detailed overview and evaluation of the BERD.

A census is made of approximately 400 ‘key responders’ that made the greatest R&D expenditure (more than approximately £3.3 million) in either of the previous two surveys.  These businesses made around 80 per cent of UK business R&D expenditures, which itself constituted 51 per cent of total UK R&D spending in 2011.  This group, which is relatively stable between years, receives a full BERD questionnaire requesting detailed R&D expenditure and employment information. In 2012 this form carried the additional questions on R&D service lives. 

The BERD also takes a sample survey of businesses spending smaller amounts on R&D.  They receive a shorter questionnaire which did not include service lives questions.  No imputations were made for these respondents or for un-sampled firms as the methods used for other variables are not appropriate for service lives.

Additionally, questions were carried on all forms in Northern Ireland where the BERD is administered by the Northern Ireland Statistics and Research Agency.  Northern Irish business R&D is around two per cent of the UK total (Office for National Statistics, 2012) but constituted 47 per cent of service lives responses.  However, any bias this introduces is likely to be small (especially when estimates are weighted by expenditure shares), while the benefit from this significant number of additional responses which also represent lower-spending businesses may be considerable.

In total the responding firms made £10.9bn of current expenditure on R&D in 2011.  This implies that these results are directly representative of around 66 per cent of all UK business R&D.

The Government R&D survey (‘GovERD’), which takes a census of all UK central Government departments also carried the question, as did the Private Non-Profit R&D Survey. These results will be presented in a later publication.

Question Response

Although the BERD is a mandatory survey, response to individual questions varies.  The service lives question achieved an 86 per cent response rate (349 responses) from the 403 key responders in the 2011 survey.  A further 310 responses were received from firms in Northern Ireland; a considerably lower response rate of around 28 per cent.  While the businesses which spend the most on R&D are often able to estimate service lives, those spending smaller amounts appear to have more difficulty.

Of the 659 responses provided, 10 were ‘zero years’ (or <6 months which had been rounded down to zero in processing).  These responses are puzzling as rational businesses would not perform R&D if they do not expect to gain some benefit.  Responses could relate to R&D that had been written off as unsuccessful (though the question specifies successful research) or may have been produced for sale so that the purchaser would be the beneficiary of the R&D (whereas the producer would only gain an instantaneous financial benefit from the sale).  However, there was insufficient information to draw any conclusions and these cases were filtered leaving 649 cases. This gives a ‘usable response rate’ of 43 per cent.

49 per cent of businesses answered for one type of R&D only, while 23 per cent answered for two different types of R&D and 28 per cent answered for all three types of R&D.  From these 649 businesses ONS received 1,167 individual answers (each relating to one type of R&D) as presented in Table 1:

Table 1, Responses by Number and R&D Type

Table source: Office for National Statistics

Composite Service Lives

Responses relate to three separate categories of R&D identified in the Frascati Manual; basic research, applied research, and experimental development.  However, it is desirable to find some service life representative of the firm’s R&D as a whole.

This can be achieved by weighting together each business’ individual responses based upon the relative share of each R&D type in their total R&D expenditure to produce a ‘composite’ estimate.  However, one fifth of firms had provided service lives for one or more type of R&D that they had not performed in 2011.  This is anticipated as the survey question asks about a typical or average investment rather than R&D performed in 2011 specifically.  Weighting using 2011 R&D spending only would ignore this information and therefore average expenditure shares for the years 2002-2011 were used. 

Despite this, 53 cases (8.2 per cent) remained where service lives had been provided for at least one R&D type which the business had not recorded expenditure against within the last 10 years.  In lieu of other information these estimates were ignored in the weighting process.  This led to 13 businesses which had provided service lives for only one type or R&D and had not spent on this type of R&D over the previous 10 years exiting the sample to give 636 composite service lives.

Outliers

Boxplots were inspected for outliers.  One very extreme outlier was adjusted to the next most extreme value of 35 years as the error in the expected life over such a timescale is likely to be substantial and sensitivity analysis showed a tangible impact on the mean.  Though there were other outliers these were much less extreme and were kept.

Results

Table 2 presents service lives for each type of R&D spending, along with the composite estimates.  With positively skewed data such as this the median is preferred as it will not be biased upwards by extreme values in the right tail of the distribution.  The median life for each type of R&D is 5.0 years, although the composite estimate is longer at 6.0 years reflecting the relative importance of longer lived R&D within firms’ overall R&D spending.

The mean lives are considerably longer and show more variation with the shortest, 7.7 years, in ‘Experimental Development’ and the longest, 8.1 years in ‘Applied Research’.  Again the composite life is longer, but only slightly in the case of the mean, at 8.2 years.

Table 2, Average expected service lives of R&D

Table source: Office for National Statistics

Table notes:

1. Average of [(BRspend * BRlife) +(ARspend * ARlife) + (EDspend * EDlife)]/∑R&D spend for each firm
2. Business' spend on R&D type as a proportion of all businesses' spending on that R&D type; (eg. basic research/∑basic research)

As R&D is valued by the sum of input costs in the National Accounts, weighting by firms’ R&D expenditure gives a reasonable proxy for each firm’s share in total R&D output.  This ensures that the responses of firms that produce the most R&D receive the most weight.  Weighting by expenditure shares yields longer service lives, suggesting that businesses which spend more on R&D generally expect to benefit for longer periods.  The weighted median composite life is 10 years, as are the medians for ‘Basic-’ and ‘Applied Research’, while the median for ‘Experimental Development’ is considerably shorter at 7 years.  However, overall the median life estimates appear rather low, generally lying below the 10-20 year range suggested by the OECD (2010, p. 62).

‘Basic Research’ now shows the longest mean service life of 12.4 years with ‘Experimental Development’ remaining the shortest, and the composite mean toward the middle at 10.5 years. 

Standard errors are relatively small for all means but Median Absolute Deviations (MADs) are large when expenditure weighting is applied.  However, MADs are known to be relatively inefficient and problematic with skewed data such as this (Rousseeuw & Croux, 1993).

These service lives are considerably longer than those found in the Intangible Assets Survey but all weighted and un-weighted estimates are within the 5.0 to 13.1 year range from the 2009 UK pilot study (Whittard, Franklin, Stam, & Clayton, 2009).  All unweighted estimates lie below the 10-20 year range suggested by the OECD (2010, p. 62), and all weighted estimates lie in the lower half of that range.  Only the expenditure-weighted medians for ‘Applied Research’, ‘Experimental Development’, and the expenditure-weighted composite are equal to the EU default of 10 years (see Ker (2013a)).

The breakdown in Table 3 shows the variation in composite R&D service lives between industries.  The longest median life is in the R&D sector (10.0 years), while the shortest life of 5.0 years is found in three industries.  Weighting by shares in total R&D spending extends the R&D sector life to 12.0 years, while a shorter life of 4.0 years is estimated for software sector firms’ R&D.  This suggests that while software is R&D intensive, software firms’ R&D is a relatively small part of total business R&D spending in the UK.

The longest mean life is in the R&D sector (10.9 years), while the shortest life is in software (5.1 years). Findings are similar when weighting by firms’ expenditure shares, though in this case the R&D sector life is now longer at 13.8 years and software shorter at 4.1 years.  The data were insufficient to provide individual results for other industrial sections.

Table 3, Composite R&D service lives by industrial section

Table source: Office for National Statistics

It is important to consider that this classification is based upon the industry of the R&D producer.  This may be different from the industry of the eventual owner or user of the R&D; R&D may be produced for sale or the producer may be the ‘R&D branch’ of a larger entity operating in a different industry (or even country).

Despite this, the ‘great majority’ of R&D is produced ‘wholly or partially’ on own-account (OECD, 2010, p. 9) and thus would not be transferred between industries.  New questions introduced to the BERD in 2012 showed that 72.3 per cent of UK businesses’ R&D would be owned by their UK business - the producer itself or the enterprise it is part of, which is typically classified in the same industry group as the producing unit (Steer & Ker, 2013).  Therefore such an industrial breakdown can provide useful insight into the typical service life of the majority of R&D conducted in any of these industries.

Figure 1 presents the expected survival profile of R&D assets.  It gives the proportion of R&D assets remaining in use at each age or ‘vintage’.  The curves decline at successive vintages as R&D progressively retires from the productive stock.  Weighting by expenditures implies that the stock of R&D declines more gradually because businesses that spend more on R&D typically expect benefits for longer.

Figure 1, R&D Survival Profiles UK BERD, 2011

Source: Office for National Statistics

Download chart

• XLS format
  (19.5 Kb)

Statistical Reliability

Figure 2 plots the expected life of R&D as a proportion of all responses, broken down by R&D type.  The tendency of estimates towards round ‘focal’ numbers is striking; there are notable peaks at five and ten years, and expected lives of 15 years and above almost all occur at focal numbers (15, 20, 25, 30, 35, 40).  This phenomenon has been noted in survey sources previously (see for example Omerod & Ritchie (2006) in the case of earnings data) and should be expected when using a question requesting a ‘representative estimate’ rather than a specific measurement.  Composite estimates are relatively more dispersed due to the averaging in their calculation but still show considerable peaking.  Weighting by expenditure exacerbates the effect (not shown).

The question is whether or not this introduces any systematic bias; are respondents more likely to round up to the nearest focal number than down for example?  It may be desirable to assume that these responses are distributed around the peaks in some way (to smooth the survival profile) but such an adjustment would be highly subjective.

Figure 2, R&D Service Lives; Proportional Frequency

Source: Office for National Statistics

Download chart

• XLS format
  (19 Kb)

Figure 2 also indicates that the data are positively skewed and this was confirmed by statistical evaluation.  The data are therefore not normally distributed and so violate the assumptions of the one-sample t test for statistical significance (and other parametric tests) which require data to be normally distributed and to have homogeneous variance across groups.  The optimal Box-Cox transformation (Box & Cox, 1964) was applied but the data remained significantly skewed and kurtosed.  Furthermore, histograms showed that the data – particularly composite estimates - are not unimodal due to clustering in responses, while a number of outliers also remain.

Kitchen (2009) explained that in the presence of only one arbitrarily large outlier the mean becomes arbitrarily large; by contrast, the median will not breakdown as long as only a minority of observations are arbitrarily large.  Therefore, in the presence of skewed data such as this, the median is likely to provide a more appropriate measure of central location.

Non-parametric tests are more robust to such violations.  The Wilcoxon Signed Rank test found that all estimates were significantly different from the European default of 10 years (see Ker (2013a)).  The Kruskal-Wallis H test for differences in the distributions of service lives across industries was also statistically significant and follow-up pairwise Mann-Witney U identified statistically significant differences between most industries.  However, the quality of these results relies on the data being fairly evenly distributed around the median and also on sufficiently similar distributions across groups.

Bootstrapping was used to estimate 95 per cent confidence intervals for the median and mean service lives.  These are shown in Table 4:

Table 4, Mean and median service lives with bootstrap 95 per cent confidence intervals

Table source: Office for National Statistics

Wider confidence intervals show that there is greater uncertainty around the estimated lives for the different industrial sections.  For unweighted means, the confidence intervals shown are relatively narrow, with the widest for basic research.  Intervals for medians are wider, though this is partially because the data are not strictly continuous (as most responses were given in round years rather than years and months) and because the bootstrapping method is better suited to means.  In particular, the confidence interval around the weighted median life in ‘Information and Communications (excluding software)’ is extremely broad.  However, this is driven by the small sample of only 45 cases in this section; bootstrapping performs better with larger samples, ideally of 100 or more cases should be used (Hesterberg et. al, 2003).  Only the manufacturing sector and overall total estimates have samples greater than 100, although ‘all other industries’ contains 96 cases and the ‘R&D industry’ contains 89 cases.

A full overview of the methods used is given in Annex 2.

Evaluation and Conclusions

This research shows that, with methodical design and testing of questions, businesses can interpret and respond to questions on expected service lives.  However, just because firms can respond does not mean their answers are accurate or consistent. Answers to the question used require a high degree of aggregation over R&D projects and different respondents may be more or less scientific in their approach, may have more or less data or experience to draw upon when estimating such an abstract answer, may or may not request help from technical colleagues as suggested etc.

Different data treatments produce different estimates and conclusions. From the outset, the decision to consider means or medians is not neutral; means are more susceptible to upward bias from outliers in the right hand tail of the distribution but median lives are considerably shorter and further away from the 10-20 year range suggested by the OECD and the EU default.

Producing composite estimates by weighting together lives of different R&D types based on individual firms’ expenditures appears to work well.  However, weighting overall estimates by firms’ relative R&D spending, while theoretically appealing, may in practice give ‘too much weight to too few firms’ so that results are largely determined by the responses of a small number of big spenders. This appears to be the cause of multimodality introduced into the bootstrap distributions of some industry sections by weighting.

This approach can provide information on different types of R&D and different industries. Non-parametric tests suggest that service lives are significantly different between R&D types, while bootstrap analysis gives the conflicting result that the confidence intervals for R&D types overlap considerably.   Non-parametric and bootstrap results both suggest that service lives are significantly shorter for software industry R&D and longer for R&D industry R&D.  They also suggest a difference between manufacturing (which performs the most R&D of any industry) and other industries.  The data are insufficient to permit more detailed industry analysis but it seems desirable to distinguish at least these groups in estimates.

There is also a question of broader generalisation. This is not a representative sample of UK R&D performers; in the main these results represent the biggest spenders on R&D and even within this sample there appears to be positive correlation between spending and expected life. This suggests that results may not be representative of businesses that do smaller amounts of R&D. However, this is a feature of the BERD survey design more generally and these results do represent around 80 per cent of total measured UK business R&D spending. The results therefore provide an important insight into the most sizable component of UK R&D.

References

Awano, G., Franklin, M., Haskel, J., & Kastrinaki, Z. (2010). Investing in Innovation; Findings from the UK Investment in Intangible Assets Survey. London: National Endowment for Science, Technology, and the Arts.

Box, G., & Cox, D. (1964). An Analysis of Transformations. Journal of the Royal Statistical Society , 211-252.

Field, S., & Franklin, M. (2012). Results from the second survey of Investment in Intangible Assets. Office for National Statistics.

Hawkins, D., & Wixley, R. (1986). A note on the transformation of chi-squared variables to normality. The American Statistician , 296-298.

Hesterberg, T., Monaghan, S., Moore, D. S.,  Clipson, A., & Epstein, R. (2003), Bootstrap Methods and Permutation Tests. W. H. Freeman.

Holm, S. (1979). A Simple Sequential Rejective Multiple Test Procedure. Scandinavian Journal of Statistics , 65 - 70.

Ker, D. (2013a). Service Lives of R&D Assets: Background and Comparison of Approaches. Newport: Office for National Statistics.

Ker, D. (2013c). Service Lives of R&D Assets: Patent Approach. Newport: Office for National Statistics.

Ker, D., & Greenaway, M. (2012). Coverage of the Business Enterprise Research and Development Survey. Office for National Statistics.

Kitchen, C. (2009). Nonparametric vs Parametric Tests of Location in Biomedical Research. Los Angeles: UCLA School of Public Health.

OECD. (2002). Frascati Manual. Paris: OECD Publications.

OECD. (2010). Handbook on Deriving Capital Measures of Intellectual Property Products. Paris: OECD Publishing.

Office for National Statistics. (2012). UK Business Enterprise Research and Development, 2011. Office for National Statistics.

Office for National Statistics. (2012). UK Gross Domestic Expenditure on Research and Development, 2010. Office for National Statistics.

Omerod, C., & Ritchie, F. (2006). Measuring Low Pay: Focus points and Rounding. Office for National Statistics.

Osborne, J. W. (2010). Improving your data transformations: applying the Box-Cox transformation. Practical Assessment, Research & Evaluation , 1-9.

Rousseeuw, P. J., & Croux, C. (1993). Alternatives to the Median Absolute Deviation. Journal of the American Statistical Association , 1273-1283.

Steer, C., & Ker, D. (2013). Ownership of R&D Assets. Newport: Office for National Statistics.

Whittard, D., Franklin, M., Stam, P., & Clayton, T. (2009). Testing an Extended R&D survey: Interviews with Firms on Innovation Investment and Depreciation. London: National Endowment for Science Technology and the Arts.

Zimmerman, D. (1998). Invalidation of Parametric and Nonparametric Statistical Tests by Concurrent Violation of Two Assumptions . Journal of Experimental Education , 55-68.

Annexes

Annex 1: wording of the R&D service lives question

From covering letter:

The National Accounts measure the economic activity of the UK; one of the key outputs is the UK GDP (gross domestic product) statistics. The National Accounts should not be confused with either UK GAAP (Generally Accepted Accounting Practice), which is the body of regulation establishing how company accounts must be prepared in the UK, or the IFRS (International Financial Reporting Standards) which set out international business accounting standards. The System of National Accounts is changing to better account for certain types of activity. One of the impacts is that research and development (R&D) will be re-classified as investment in intangible assets. This means that additional information is required about R&D performed in the UK to establish the value and ownership of R&D assets. The data you collect and compile for your accounts may not match these new requirements, but we encourage you to please use your knowledge of the R&D you perform to provide your best possible answers to these new questions.

Question: Life Lengths of R&D Assets

With the re-classification of R&D as investment in intangible assets, the length of time for which these assets will be useful needs to be determined. This information is required to create a benchmark for use in the National Accounts.

This question asks how long your business (or the owners of the R&D) would expect to benefit from a typical investment in each different type of research it performs.  If your business usually undertakes multiple projects, please provide an average estimate for each type of research.  We appreciate that you might have many varied projects of differing lengths, but please provide your best estimate of the average useful life of R&D assets produced in the UK.  This question is about your R&D in general and is not specifically concerned with the projects carried out in this particular reporting period.

The principle is similar to determining how long machinery or equipment would be used for in production.  However, as R&D is intangible, this is a more abstract concept.  This question is interested in how many years, on average, your business expects to make use of R&D.  It is the estimated length of life of your R&D which is relevant, not the life length of any products or tangible assets it contributes to.  It may be necessary to seek assistance in answering this question from your business’ technical experts.

On average, how long would the business expect to benefit from a typical investment in a successful project in the following research types?

Only answer for those types of research that are relevant to your business.

a) Basic research

Work undertaken primarily to acquire new knowledge without a specific application in mind ……………………………..………………………… XXX years

b) Applied research

Work undertaken to acquire new knowledge with a specific application in mind ………………………………………………………………………...... XXX years

c) Experimental research

Work using the results of basic and/or applied research for the purpose of creating new or improved products/processes ……….. XXX years

Annex 2: Analysis of Statistical Reliability

Figure 2 shows that the responses are not normally distributed. Highly positive skew was confirmed by assessing Q-Q normality plots along with the skewness, kurtosis, and Z scores as presented in Table 5 (21 Kb Excel sheet) . 

The Box-Cox suite of power transformations can be used to normalise data (including chi² distributed data (Hawkins & Wixley, 1986). The method outlined by Osborne (2010) was followed in which the data (y) were transformed such that:

Box-Cox Transformation

A number of different values of lambda are estimated and the optimal value selected to minimise skewness.  Optimal λ=0.10 to 0.14 across the different types of R&D.  The optimal lambda for the composite estimates of 0.12 was adopted and this improved skewness, kurtosis, and Z scores for unweighted estimates as also shown in Table 5.

Despite these improvements, and while Q-Q plots showed that the data were much closer to normal, distinct patterns remained.  Histograms showed that the data – particularly composite estimates are not unimodal (a requirement of parametric tests) due to clustering in responses and a number of outliers also remain.

Furthermore, when weighted by expenditure so that the responses of firms spending more on R&D are given more importance, even though unique skewness-minimising lambda values were taken for the different types of R&D there is still considerable kurtosis.

Non Parametric Tests

Kitchen (2009) explained that in the presence of only one arbitrarily large outlier the mean becomes arbitrarily large; by contrast, the median will not breakdown as long as only a minority of observations are ‘corrupted’.  Nonparametric tests use the median for location when evaluating the distribution as a whole and are more robust to violations of normality;  ‘in data where there exists skewness, extreme asymmetries, multimodality, or heavy tails, nonparametric tests such as Wilcoxon Rank Sum Test and Kruskall-Wallis offer a very satisfactory alternative to parametric tests’ (Kitchen, 2009).  Results of one-sample Wilcoxon Signed-Rank Tests (a non-parametric alternative to the one-sample t-test) of difference from 10 years, are presented in Table 6.

Table 6, Wilcoxon signed rank tests of difference from 10 years

Table source: Office for National Statistics

Table notes:

1. Based on positive ranks

In all cases the null hypothesis that the median difference of businesses’ service lives from the European standard of 10 years is zero can be rejected with a high degree of confidence.  This is also the case when expenditure weighting is applied, even though this yields median lives of 10.0 years for two types of R&D and the composite.

The Kruskall-Wallis H test is the non-parametric alternative to the one-way ANNOVA used to determine if there are significant differences between the distributions of different groups.  Applying this to the transformed composite R&D lives by industry section showed that significant differences do exist between one or more of the groups: Chi² (5) = 47.64, p < 0.005. This was even more true of the when weighted by expenditure: Chi² (5) = 1,636,334.34 p < 0.005.  

Table 7: presents the results of pairwise Mann-Witney U comparisons of the distributions of the industrial sections.  The main conclusion is that service lives of the R&D industry are significantly different from each of the groups shown except ‘all other industries’.  R&D in software is also significantly different from manufacturing, R&D, and ‘all other industries’.  Expenditure-weighted tests (not presented) are all highly significant (p<0.005) suggesting that lives vary across all these industries.

Table 7, Pairwise Mann-Witney U coefficients

Table source: Office for National Statistics

Table notes:

1. Itallics indicate significance at the 5% level (Holm's Sequential Bonferroni adjustment applied)

To reduce the chance of Type I error (i.e. incorrectly rejecting the null hypothesis that the distributions of each pair of industries are identical), ‘Holm’s sequential Bonferroni Adjustment’ (Holm, 1979) was applied. This computes more conservative 95 per cent thresholds based on the rank significance of each result.

Although the transformed data (and test statistics) were used, Zimmerman (1998) showed that simultaneous violation of only two assumptions increases the risk of Type I error – even for non-parametric tests. In particular, to provide a valid result the data must be symmetrical around the median; histograms show this is often not the case for this data, despite the transformation made.  This is exacerbated by expenditure weighting which is an intentional introduction of bias.  Therefore, these results should be treated with appropriate caution.

Bootstrap Analysis

As is often the case, the analysis presented here is based on only one sample from the population of interest – all UK R&D producers (or, more correctly, all UK R&D assets).  However, statistical testing is based upon the sampling distribution of the statistic of interest (mean, median) which can only be found by taking repeated samples from the population.  Parametric (and to a lesser extent non-parametric) tests make assumptions about the sampling distribution of the test statistic.  However, evaluation of this data has shown strong positive skew and kurtosis and, although the central limit theorem states that the sampling distribution of the sample mean should tend to normality as sample sizes increase, this is not always the case.

Bootstrapping provides an alternative tool that uses the distribution of the data itself rather than relying on a theoretical distribution which may not hold.  By taking repeated re-samples of the composite lives with replacement a bootstrap sampling distribution of 10,000 sample statistics (means, medians) was created for each R&D type and industry section.  Resample sizes equal the number of responses in the observed sample for that group.  Weighting prior to this process produces a bootstrap distribution of the weighted statistic of interest.

Provided the sample is sufficiently large ‘its shape and spread don’t depend heavily on the original sample and mimic the shape and spread of the sampling distribution’ (Hesterberg et. al, 2003). However, as usual, results will be more reliable with larger samples since ‘bootstrap distributions do not have the same centre as the sampling distribution; they mimic bias, not the actual centre’ (ibid) and the implicit assumption is that the distribution of the sample data is representative of the true population distribution.  Here the smallest sample was 45 in ‘J – Information and Communication (ex. software)’. With enough resamples, bootstrapping will introduce very little additional variation beyond the original sampling variation so ‘we can rely on a bootstrap distribution to inform us about the shape, bias, and spread of the sampling distribution’ (ibid). 

This bootstrap sampling distribution is used to draw conclusions about the distribution of the sample means. Histograms showed varying degrees of non-normality with combinations of skewness, kurtosis, and multimodality – especially in the bootstrap distributions of expenditure weighted means. Table 8 presents means, bootstrap 95 per cent confidence intervals, standard errors, and bias estimators for the different types or R&D at the top level and for composite lives by industry section.

Table 8, Bootstraped parameters for sample means

Table source: Office for National Statistics

Table notes:

1. Bootstrap standard error = standard deviation of the bootstrap distribution of resampled means.
2. Sample mean minus mean of bootstrap distribution; positive numbers indicate that the (observed) sample mean is positively biased.

Bootstrap 95 per cent confidence intervals are relatively narrow around the means for the different R&D types at 1.2 years in width on average. Confidence intervals for the different R&D types broadly overlap suggesting that their lives may not be statistically significantly different.  None of these confidence intervals include the 10 year European Default which can therefore be rejected at the total level with at least 95 per cent certainty.  Bootstrap standard errors are generally relatively small as are the bias estimators.

The picture is similar by industry though confidence intervals are around twice as broad on average. The confidence interval for the R&D industry does span the 10 year default and therefore there is greater than five per cent chance that the average life for R&D sector firms is 10 years.

Weighting by expenditure makes the confidence intervals much broader, especially in ‘J – Information and Communications’. Standard errors are larger accordingly.  There is also some evidence of bias, especially for basic research where the observed mean is 4.9 years larger than the mean of the bootstrap distribution.  All confidence intervals include the European default of 10 years except software which is expected to have much shorter R&D lives in general.  These results imply that a 10 year weighted mean is plausible for most industries and R&D types.  However, this is not the same as conducting a significance test for difference from 10 years and such a test might find that all estimates are significantly different from 10.  These tests require further bootstrapping and were not completed due to time constraints.

These confidence intervals employ the percentile method; more robust methods are unavailable in SPSS 12.  However, percentiles perform better than t-distribution based intervals when there is skewness and also improve with larger samples so the method should perform acceptably here (Hesterberg et. al, 2003).

Although median life estimates are preferred as a measure of central location due to being less susceptible to upward bias, bootstraps for medians rely upon the few data points around the centre of the sample and therefore bootstrap estimates for the median are less reliable than the mean (ibid). This is less problematic with sample sizes over 100 so the method should perform well at the total level and for Manufacturing but will suffer with the smaller sample sizes in other sectors. Median estimators are presented in Table 9 but this limitation must be considered when interpreting them.

Table 9, Bootstrapped parameters for sample medians

Table source: Office for National Statistics

Table notes:

1. Median average of the bootstrap distribution from the (observed) sample median.
2. Sample median minus median of bootstrap distribution; positive numbers indicate that the (observed) sample median is positively biased.

Unweighted medians for the R&D types generally lie toward the lower end of their confidence intervals.  When industrial section is considered the picture is more mixed.  Again, only the confidence interval for the R&D industry includes the 10 year default.  MADs are very small and there is no indication of bias.

Once again confidence intervals are much broader for the weighted results and only software excludes the EU default.  As with the expenditure weighted means, there is most evidence of bias for Basic Research which suggests that the observed median may overestimate the true life of basic research for this industry section.  This is clearly a product of the weighting, probably caused by the concentration of Basic Research in only a few companies with estimated lives below the median.

Background notes

1. Details of the policy governing the release of new data are available by visiting www.statisticsauthority.gov.uk/assessment/code-of-practice/index.html or from the Media Relations Office email: media.relations@ons.gsi.gov.uk