External Coherence, Measuring Coherence Between Official Estimates of Economic Activity and External Indicators

Abstract

This article analyses the extent of statistical coherence between Office for National Statistics (ONS) and Markit/Chartered Institute of Purchasing and Supply indicators (Markit/CIPS).

Summary

• The Office for National Statistics (ONS) and Markit/Chartered Institute of Purchasing and Supply (Markit/CIPS) both provide measures of economic activity in the manufacturing, services and construction sectors.

• ONS and Markit/CIPS indicators are not directly comparable due to differences in sampling, methodology and output. This article concludes that many of the perceived divergences between the two sets of indicators can be reconciled after these underlying differences are addressed.

• The only significant difference between the Index of Manufacturing (IoM) and the Markit/CIPS output index for manufacturing occurs during Jubilee bank holidays.

• The only significant difference between a coverage-adjusted Index of Services (IoS) and the Markit/CIPS services Purchasing Managers Index (PMI) occurs during the recent financial crisis.

• Construction and aggregate measures show no periods of statistically significant divergence, although there are periods where the trends appear to differ in magnitude. There are occasions where the two time series trend in different directions, but this does not lead to a statistically significant difference in the monthly series.

Introduction

In the first quarter of 2012 the UK economy recorded its second consecutive quarter of negative growth, signalling a return to recession. The announcement was controversial since private measures of economic activity suggested that economic performance was stronger.

Given their timeliness and high frequency, private indicators are viewed by many analysts as useful complements to official figures. Private indicators, however, have very different sampling and processing methods to official estimates (Meader and Tily, 2008), which mean that official estimates are preferable in terms of quality.

Private indicators increased in prominence through the recent financial crisis, as uncertainty and concern over revisions to official data (BBC, 2012) increased. Among the most reputable private indicators are the Purchasing Managers’ Indices® (PMI®s) produced by Markit Economics on behalf of the Chartered Institute of Purchasing Managers and Supply® (Markit/CIPS®).

It is important, therefore, to check the coherence between private indicators and official data. This paper updates and expands the previous analyses conducted by Chamberlin (2008) and Chamberlin et al (2011), by improving the methodology and expanding the analysis to include the construction sector.

The paper first considers the main differences between official estimates and Markit/CIPS data, informing the methodology used to compare the two series, which is detailed in section 3. The ONS and Markit/CIPS data are then compared on the performance of the manufacturing, services and construction sectors.

Analysis of the whole economy is conducted in two sections: one using a composite PMI created by weighting together the individual headline output indices, and one using the manufacturing PMI, which Markit/CIPS use as a leading indicator for the whole economy. The conclusion draws together the analysis and outlines areas for further work.

Differences between indicators

Direct comparisons of headline Office for National Statistics (ONS) and Chartered Institute of Purchasing and Supply (Markit/CIPS) figures are misleading. There are four main differences: in output, sample size, periodicity, and coverage.

I. Output

ONS data give an exact estimate of growth between one period and a reference period, whereas Markit/CIPS data reports a balance statistic – respondents are asked to judge if the measure has increased, decreased or remained unchanged. Responses are weighted so that a score of 50 reports no change; a score above 50 suggests an increase in activity and a score below 50 a reduction.

In sum, the ONS estimates aim to show both the direction and size of a change, while Markit/CIPS data mainly consider the direction.

II. Sample size

Markit/CIPS aims to publish in advance of ONS releases to provide a rapid response. Consequently, the sample size for the Markit/CIPS surveys that can be processed is considerably smaller than the size of corresponding ONS samples.

Sample size comparison

Table source: Office for National Statistics

III. Periodicity

The Markit/CIPS figures for services, manufacturing and construction are reported on a monthly basis. This allows a direct comparison to ONS headline figures for services and manufacturing, however construction and Gross Domestic Product (GDP) are only available from ONS (prior to 2010 in the case of construction) on a quarterly basis. Therefore a simple average of Markit/CIPS figures for three months is taken to produce a quarterly Markit/CIPS estimate, in order to allow comparison with ONS’s headline GDP and construction statistics. 

IV. Coverage

Both Markit/CIPS and ONS data are based on industries as classified by the 2007 Standard Industrial Classification (SIC 07). In broad terms, the manufacturing and construction surveys have the same coverage, although, due to the sample size, the PMI may not sample any particular industry as well as the ONS. There are a number of differences with respect to the coverage of the service sector. Just under one half of the service sector according to SIC weights is not covered by the services PMI, including:

• Wholesale and distribution (145 parts per 1000).

• Letting of dwellings (92).

• Public administration and defence (70).

• Education (77).

• Health and social services (96).

• Sewage and refuse disposal^2.

• Private households with employed persons (5).

A final note; Markit/CIPS releases are not usually subject to revisions, while ONS data are revised as more data are collected. While this does not affect the ability to compare the most recent headline figures, it does mean that ONS data for earlier periods are likely to be of higher quality, as more source data have been used in production.

Even after adjusting both series, some discrepancy is to be expected as the two different estimates provide slightly different information. The analysis that follows takes this into account.

Notes for Differences between indicators

1. Most of the monthly ONS data come from the Monthly Business Survey, which is a sample of ~30,000 firms drawn from a total of 1.24 million in the production and service industries.

2. The Index of Services publication does not explicitly include detailed industry figures for Sewage and refuse disposal, activities of membership organisations not elsewhere classified, or renting, but the PMI does not include these industries either.

Methodology

Although this paper follows the same underlying methodology as Chamberlin (2008), it has introduced improvements which have resulted in a more robust procedure. Specifically, the revised method addresses two main points:

• Accounting for the fact that both Chartered Institute of Purchasing and Supply (Markit/CIPS) and Office for National Statistics (ONS) data tend to be influenced by the data that immediately precedes it (this is called ‘autocorrelation’, and is a common problem in analysing time-series data).

• Ensuring datasets are of the same periodicity.

Addressing these issues has not changed the over-arching story of fundamentally similar series, but increases the validity of it. This section explains each step of the revised methodology.

The first step is a departure from Chamberlin (2008); instead of comparing the monthly Markit/CIPS series against the headline three-month ONS series for manufacturing and services, this paper compares the one-month ONS series against the Markit/CIPS data, to ensure ‘like for like’ comparison. After that, the data are standardised in exactly the same way as Chamberlin (2008), by subtracting the series mean from each data point and dividing by the series standard deviation.

From here on the methodology diverges; it can be broken down into two steps:

• Testing if and when shocks in both series are correlated.

• Finding periods of significant difference between the ONS and Markit/CIPS series.

For both steps, the raw standardised data are not used due to the presence of autocorrelation. The degree of autocorrelation¹ is first identified using statistical packages . This allows for the specification of an Autoregressive Integrated Moving Average (ARIMA) model. From these models, the differences between each individual observation, and what the model expects the observation to actually be (these differences are known as ‘residuals’) are extracted and these are then analysed.

The first step – testing correlation of shocks – requires the use of a cross-correlation function. This takes the residuals from the ONS and Markit/CIPS series, after correcting for autocorrelation through the ARIMA model, and considers whether those residuals are correlated across different lags. The output from the cross-correlation function is presented on a cross-correlogram, shown in Figure 1.

The important result is that there is a significant relationship between the residuals at the time of the shock taking place (lag zero), as evidenced by a result beyond the dotted line (95 per cent confidence interval). This shows that a shock causing a spike in residuals in one series also causes a spike in residuals in the other series, and both occur simultaneously. That other lags are significant is interesting, but for present purposes it is sufficient to identify significance at lag zero².

Figure 1: Cross-Correlogram of GDP* residuals against implied GDP from CIPS residuals

Source: Office for National Statistics

Notes:

1. GDP* is a weighted average of the Index of Manufacturing, coverage-adjusted Index of Services, and Output of the Construction Industry.

Download chart

• XLS format
  (17.5 Kb)

The second step focuses on the difference between the two series. From the two standardised series, the difference is calculated and tested for autocorrelation. This difference is then run through a regARIMA model, which tests whether each point is an outlier. If it is, then there is a significant divergence between the ONS and Markit/CIPS data (i.e. the difference is bigger than the model’s expectation)³ . Confidence intervals can also be constructed (determined by the number of observations tested), so the test can be presented visually. The end result, therefore, mirrors that of Chamberlin (2008).

It is important to note that the confidence intervals are set at 95 per cent - this means that 5 per cent of all data points are bound to fall outside the intervals, even if they are not truly significant. However, closer examination of any significant difference will take place to distinguish ‘true’ significance from ‘false’ significance.

To supplement the analysis, the series trends are compared. The trends are estimated by the use of X-12-ARIMA software, and provide a substantially smoother series than the monthly data⁴ . While visual inspection in this way is not rigorous, it highlights interesting stories that the volatility of monthly data obscures, and allows for an at-a-glance comparison of the underlying trends as opposed to the shocks, upon which the rest of the analysis is based. 

Notes for Methodology

1. The degree of autocorrelation refers to how many previous periods influence the current period’s value. In complex cases, where the degree of autocorrelation is not clear from visual inspection, the X-12-ARIMA program (US Census Bureau 2009) is run to generate an ARIMA model specification.

2. This process of extracting residuals is known as ‘pre-whitening’. For more information refer to Chatfield (2004).

3. For more information on the use of regARIMA models, please refer to Findley et al. (1998).

4. More detail about how X-12-ARIMA software can be used to extract series trends can be found in Findley et al. (1998) and Ladiray and Quenneville (2001).

Analysis

The analyses in the following sections are conducted in a standardised fashion. First, a visual inspection of the series trends for ONS and Markit/CIPS data is carried out, before a discussion of the results arising from the more rigorous statistical analysis detailed above is presented.

Manufacturing

The two data series that are compared for manufacturing output are the Index of Manufacturing (IoM) from the Office for National Statistics (ONS), and the Markit/Chartered Institute of Purchasing and Supply (CIPS) manufacturing output index.

It is worth noting that the Markit/CIPS manufacturing Purchasing Managers Index (PMI) is not used as it is a composite measure that includes a weighting of non-output variables (such as employment and new orders); however the manufacturing PMI will be used when considering whole economy comparisons.

Figure 2 plots the trends for the ONS and Markit/CIPS series against one another. In broad terms, the two series track each other fairly well, with both following the same direction over the business cycle. It is notable that there does seem to be a considerable difference in the magnitude of the trends over the cycle, with the Markit/CIPS data showing an exaggerated trend while the ONS data is relatively muted. This is most obvious with the recession, where the trends substantially differ in terms of the depth of the recession.

There could be a number of reasons for this, most noted in the ‘Differences between indicators’ section. Probably the most important factor is the way in which economic activity is measured – due to the Markit/CIPS survey only allowing for three answers, it is more likely to overestimate any given trend. If there is a recession, then most respondents will report a negative response in the Markit/CIPS survey, which will negatively impact the series, even if the fall in output is mild. That the recovery starts slightly later in the Markit/CIPS series also suggests an element of hysteresis – respondents’ answers are influenced by the answers of the previous month.

Figure 3 shows a plot of t-values resulting from the application of the regARIMA model process (outlined in the methodology section) to the difference series. As can be seen, most observations lie within the confidence bounds, suggesting that it is the general case that there is no statistically significant divergence between ONS and Markit/CIPS manufacturing indicators, once periodicity and output differences are accounted for. For the observations that lie outside the bounds it is useful to refer to the standardised month on month growth rates.

Figure 4 shows the ONS and Markit/CIPS standardised month on month growth rates. As can be seen, there are significant spikes in early 2002 and in mid 2012, which are the same dates of the outliers in Figure 3. These correspond to the Queen’s Golden and Diamond Jubilees respectively.

Both the Markit/CIPS and ONS data are sensitive to the shock, but Markit/CIPS data far less so. This is to be expected – the first shock, on the month of the Jubilee, will have a considerable impact on the magnitude of output relative to the previous month. However, since Markit/CIPS is a qualitative survey, there may be psychological factors preventing firms from reporting a negative response – one potential factor is that the ‘shock’ was well broadcasted in advance, so it may have already been ‘factored in’. Such an explanation would be in line with the story seen with the trends: with unexpected and prolonged shocks, the Markit/CIPS trend exaggerates the magnitude; but with expected/short ‘shocks’, Markit/CIPS under-estimates.

Figure 2: Trends for ONS and CIPS manufacturing data

Source: Office for National Statistics

Download chart

• XLS format
  (29 Kb)

Figure 3: Difference series for Manufacturing

Source: Office for National Statistics

Download chart

• XLS format
  (26 Kb)

Figure 4: Month on Month growth rate for standardized ONS and CIPS manufacturing data

Source: Office for National Statistics

Download chart

• XLS format
  (29.5 Kb)

Services

As noted in the ‘Differences between indicators’ section, comparison between Office for National Statistics (ONS) and Charted Institute of Purchasing and Supply (Markit/CIPS) service sector data is difficult and, even when standardised, misleading. This is because the coverage of the service sector is different in each survey, with the ONS including government and distribution sectors.

Although the following analysis provides a story congruent with that of the manufacturing sector (broad coherence with rare significant divergence), it is important to note that the two series exhibit far weaker cross-correlation; the cross-correlation coefficient at lag zero is the lowest of any of the sectors tested, and only just significant at the five per cent level.

Figure 5 shows the trends for the adjusted Index of Services (IoS) from the ONS and the Markit/CIPS services PMI. As can be seen, while the Markit/CIPS trend tracks the ONS trend on average, it has a tendency to exaggerate poor economic performance, and mildly overstate good performance. This was a behaviour seen in the manufacturing output index by Markit/CIPS as well, but to a more muted extent.

This relationship is seen most clearly during three periods; the economic slowdown in the late 90’s due to the Asian financial crisis, the slump in late 2001 due to the dot-com bubble, and the recent recession. In all these cases, the Markit/CIPS measure has shown a drastic drop in performance and strong recovery, while the adjusted IoS trend provides a more nuanced view of the same story. It is likely the reason driving this difference is the different ways in which the surveys record economic activity, as it was in manufacturing.

This reasoning is given some validation by analysis of the difference series, as shown in Figure 6. There is only one instance of significant divergence between the two datasets, occurring in 2009, although it is marginal. That is not to say that the two series are substitutes for one another – besides the different purposes of the surveys (Markit/CIPS focusing on being a more timely indicator), the relationship can be quite volatile, as shown by both the trends and number of difference series observations that come close to being significant.

Further, the nature of the shock that drove the divergence is different to that in manufacturing. In manufacturing, it was one that was known about well in advance, and could be accounted for easily. In services, the divergence occurred because of the recession – suggesting that the ONS and Markit/CIPS surveys are capturing slightly different responses from the service sector. The distinction between the ONS and Markit/CIPS surveys being quantitative and qualitative surveys respectively is perhaps an important factor in this, although further study will be needed to isolate the true reason.

The volatility of the difference series and different reactions to shocks notwithstanding, it is fairly clear that on a month-to-month basis, ONS and Markit/CIPS data are generally coherent after adjusting for coverage and autocorrelation. This is not to suggest that the headline figures for ONS and Markit/CIPS service sector surveys will share this coherence; while the adjusted IoS and unadjusted IoS are broadly similar, there are areas where they exhibit different behaviour due to the substantial magnitude of the omitted industries. Thus, the results found in this section are congruous with potential divergence between headline published ONS and Markit/CIPS service sector figures. 

Figure 5: Trends for adjusted IoS and CIPS Services PMI

Source: Office for National Statistics

Figure 6: Difference series for services

Source: Office for National Statistics

Download chart

• XLS format
  (26 Kb)

Construction

Due to data limitations, the analysis of the construction sector surveys is carried out on a quarterly basis, with the Markit/Chartered Institute of Purchasing and Supply (CIPS) growth rate being an average of the three months within the quarter.

The trends for the construction surveys appear, on visual inspection, to follow one another reasonably closely up until mid 2009, during the initial recovery from the recession (Figure 7). Before this point, there is no systemic under or over estimation by either survey, with both the Office for National Statistics (ONS) and Markit/CIPS data showing similar stories; the main departure from this is a slight fall in the ONS index just prior to the recession, where the Markit/CIPS series continued to rise.

Across the full length of time being considered however, the most notable divergence between the series trends occurs in early 2010, with ONS data showing a much stronger trend than Markit/CIPS data. One possible explanation is the difference in magnitude measurement; both ONS and Markit/CIPS show the same direction of change across the period, with turning points occurring simultaneously. It is also worth noting that the Markit/CIPS trend slightly exaggerates the depth of the recession by around one standardised unit.

When considering the difference series analysis (Figure 8), a relatively close relationship can be shown. There is no particular point where the t-values series even approaches the confidence bounds; the largest difference comes in 2010, around the time of the initial recovery and the trend dissonance, but is still well within the bounds.

One possible reason for the result (which contrasts with manufacturing and services to a degree), is the use of a quarterly series, as opposed to monthly. Most of the spikes in the manufacturing and services series were due to single-month shocks, as opposed to sustained divergence, so over a quarter that shock may be moderated – meaning that the difference no longer becomes statistically significant.

Given that the difference series is of a relatively low magnitude, it is unlikely that there are numerous instances of divergence on a month-to-month measure. The story appears to be similar to the other two sectors examined so far – once the data from the two series has been converted into a comparable format, there are no extended periods of divergence between them.

Figure 7: Trends for ONS and CIPS construction series

Source: Office for National Statistics

Download chart

• XLS format
  (21 Kb)

Figure 8: Difference series for construction

Source: Office for National Statistics

Download chart

• XLS format
  (20 Kb)

GDP* and MARKIT/CIPS implied GDP

This section compares the headline Gross Domestic Product (GDP) chained volume index and an implied GDP figure from Markit/CIPS, coverage differences having been accounted for to allow an accurate comparison. ONS and Markit/CIPS estimates for manufacturing and construction sectors are broadly similar in coverage, but the estimates differ in coverage for the service sector.

Markit/CIPS do not cover wholesale, retail, motor trades and government. Therefore these sectors have been excluded from the headline GDP figure to give an adjusted GDP figure which is referred to as GDP*. In order to aggregate the Markit/CIPS surveys into an implied GDP figure, Standard Industrial Classification weights have been used, these have also been adjusted to account for the exclusion of the categories that Markit/CIPS do not cover.

Figure 9 compares the trend for GDP* and the implied GDP figure from Markit/CIPS. The series track each other closely, especially during the pronounced fall in output during 2008 and recovery throughout 2009. That said, the relationship between the series trend has faltered in recent quarters; GDP* has been weaker than survey data has suggested.

Divergences of a similar magnitude have previously been seen during the latter part of 1998 and early 2004, but only for a period of two to three quarters before the series return to tracking each other reasonably closely. More quarters of data are required to see if the divergence of the series continues or only exists for a short period, as in 1998 and 2004.

Figure 10 shows a plot of t-values for the difference series after the regARIMA model process outlined in the methodology section. Despite Figure 9 showing a number of divergences between the two trends, Figure 10 shows that all differences between observations are within the confidence bounds.

Figure 9: Trends for GDP* and implied GDP from CIPS

Source: Office for National Statistics

Notes:

1. GDP* is a weighted average of the Index of Manufacturing, coverage-adjusted Index of Services, and Output of the Construction Industry.

Figure 10: Difference series for GDP* and implied GDP from CIPS data

Source: Office for National Statistics

Notes:

1. GDP* is a weighted average of the Index of Manufacturing, coverage-adjusted Index of Services, and Output of the construction industry.

Download chart

• XLS format
  (20 Kb)

GDP and Manufacturing PMI

The Markit /CIPS UK Purchasing Managers Index (PMI) is a composite index which is designed to provide an overall picture of the situation in the manufacturing sector and also acts as a lead indicator of the whole economy. The index is derived from five individual indexes including manufacturing new orders, output, employment, suppliers’ delivery times, and stock of items purchased.

As this index is seen as a lead indicator of economic activity for the whole economy, this next section analyses its relationship with the headline GDP chained volume index. Unlike the previous section, GDP has not been adjusted for coverage, as it is not a direct comparison between Markit/CIPS and ONS estimates of the same sector. It is a comparison of GDP and what is seen to be a lead indicator of it.

From visual inspection of Figure 11, the trend of GDP and Markit/CIPS manufacturing PMI track each other reasonably closely, albeit not as closely as GDP* and the implied GDP from Markit/CIPS. This suggests that the composite measure consisting Markit/CIPS estimates of the services, manufacturing and construction sector output are a better proxy for official statistics than the Markit/CIPS manufacturing PMI.

The relationship between the series falters since the end of 2009, and despite a close relationship during the fall in output in 2008, Markit/CIPS estimates show a much stronger recovery during 2010 compared to official statistics.  Notable divergences also exist in 1998 and 2004, the same periods that divergences exist between the trend of GDP* and the Markit/CIPS implied GDP.

Figure 12 shows a more detailed analysis of the relationship between GDP and Markit/CIPS manufacturing PMI. Despite being able to identify a number of periods visually where the series diverge, when statistically tested there is no divergence. All observations differences lie within the confidence bounds, suggesting there is no statistical difference between the series.

Figure 11: Trend for GDP and CIPS Manufacturing PMI

Source: Office for National Statistics

Download chart

• XLS format
  (21 Kb)

Figure 12: Difference series for GDP and CIPS Manufacturing PMI

Source: Office for National Statistics

Download chart

• XLS format
  (19 Kb)

Conclusion

The article has considered the theoretical and practical differences between ONS releases and external indicators, and found that direct comparisons are usually not possible for a number of methodological and coverage reasons. After standardising both ONS and PMI data and adjusting ONS data to fit the PMI coverage as closely as possible, a story of relative coherence is played out. In manufacturing, services, and on a whole economy level, the two datasets broadly agree with one another, with isolated and rare instances of significant divergence.

Where there are significant divergences, there is usually an exceptional economic event that accounts for it, as opposed to a systemic driver. The exceptions to this are the service sector surveys, which appear to react slightly differently to shocks, although usually not to a significant extent.

The coherence is much stronger when considering aggregate measures as opposed to single-sector measures, both when using the manufacturing PMI as a lead indicator, and when using a constructed ‘composite’ Markit/CIPS PMI. On a single-sector level, the Markit/CIPS data tends to exaggerate movements in the sector, but this is muted on an aggregate level.

Background notes

1. Details of the policy governing the release of new data are available by visiting www.statisticsauthority.gov.uk/assessment/code-of-practice/index.html or from the Media Relations Office email: media.relations@ons.gsi.gov.uk

References

1. BBC (2012) Viewpoint: Is UK GDP data fit for purpose?’ [online] [Accessed on 19th September 2012]

   Chamberlin G (2008) ‘Monitoring the coherence of ONS and Purchasing Managers Index data’, Economic & Labour Market Review, 2(5), pp 23–28.

   Chamberlin G, Fender V and Khan Z (2011) ‘Monitoring the Coherence between ONS and PMI Data – An Update’, Economic & Labour Market Review, 6(3), pp 66-74.

   Chatfield C (2004) The Analysis of Time Series: An Introduction, Sixth Edition, Chapman & Hall/CRC: Boca Raton.

   Meader R and Tily G (2008) ‘Monitoring the quality of the National Accounts’ Economic & Labour Market Review, 2(3), pp 24–33.

   Findley D F, Monsell B C, Bell W R, Otto M C and Chen B-C (1998) “New Capabilities and Methods of the X-12-ARIMA Seasonal-Adjustment Program,” Journal of Business and Economic Statistics, 16(2).

   Ladiray D and Quenneville B (2001) Seasonal Adjustment with the X-11 Method, New York: Springer-Verlag.

   U.S. Census Bureau (2009) X-12-ARIMA Reference Manual, Version 0.3, Washington, DC: U.S. Census Bureau.