1. Overview of online price changes data

During the coronavirus (COVID-19) pandemic, we have been providing timely indicators of the effect of the disease on the UK economy and society in our Coronavirus, the UK economy and society, faster indicators bulletin.

These faster indicators include experimental online price indices for several high-demand products (HDPs), covering items such as long-life food, health, and household and hygiene products. These indices are created using daily web-scraped data from several, large, online UK retailers and provide an indication of weekly price change for items in the HDP basket.

As experimental indices, these data are subject to revisions as we develop our methodology and systems. Since the first publication of these data on 2 April, we have made some changes. These are detailed in Section 4 - Timeline of changes to online price indices data.

Back to table of contents

2. How we measure online price changes data

Prices are scraped daily from a number of large online UK retailers (typically supermarkets and other prominent high-street chains with an online presence) starting at 16 March 2020, and continuing over the course of the coronavirus (COVID-19) pandemic for selected items chosen to form the HDP basket (Table 1).

An average weekly price is then calculated for each unique product.


Because of the nature of these data and the impact that daily changes in product availability can have on chain drift within a chained Jevons series, we have chosen to apply a movement splice GEKS-Jevons index to these data. The movement splice GEKS-Jevons method is essentially equivalent to the GEKS method but it uses a rolling window of 5 weeks to ensure the data remain relevant to the current time period.

For more information on the movement splice GEKS method and our wider research into suitable index methodology for alternative data sources such as web-scraped data, please see our latest project update The winning formula? A framework for choosing an appropriate index method for use on web-scraped and scanner data (PDF, 434KB).

Back to table of contents

3. Strengths and limitations of online price changes data

Comparability with other sources

It is not appropriate to compare the online price movements detailed in the Coronavirus, the UK economy and society, faster indicators release with the headline Consumer Prices Index including owner occupiers' housing costs (CPIH).

The CPIH is produced using different methods, data and quality thresholds, and incorporates a broader range of goods and services, such as housing. The online price movements data are experimental and are produced using experimental web-scraping techniques.

Strengths of the experimental online price indices

  • The data present an indicator of the price movements of high-demand products during the coronavirus (COVID-19) pandemic.

  • The data are available on a weekly basis, to allow week to week comparisons of price movements in these high demand products.

  • The data are available 4 days after the reference period. This makes them a timely indicator for users.

Limitations of the experimental online price indices

  • No comparable replacement or quality adjustments have been applied, which may lead to downward bias in the indices.

  • Because of the nature of web-scraped data, there are no expenditure weights available at the product level. Therefore, there is no accounting for consumer substitution within these items. As we follow the same unique product over time, any substitution towards cheaper or more expensive items will not have an impact on the index, unless these types of products follow a different price change trend.

  • For retailers that have a clear label indicating that a product is out of stock, we have removed those products from the analysis. However, other retailers may not have a clear indication of this, so there may be products currently out of stock that have still been included. If the price of these items do not change, this could cause the index to remain static.

  • Limited sample size and daily changes in product availability may result in volatility at the item level. In particular, the size of the sample means that sometimes single retailers can contribute to substantial movements at the item level.

Back to table of contents

4. Timeline of changes to online price indices data

The publication of online price indices data was included in the first faster indicators publication on 2 April. As experimental indices, these data are subject to revisions as we develop our methodology and systems.

On 20 April we noticed an issue with the way our automated web-scrapers collected prices from one particular high-street retailer, which meant that these experimental price indices were overstating online price changes for some categories of high-demand products. This led to the online price indices being suspended and not published in the faster indicator's publication on 23 April.

We resumed publication of these data on 30 April. The data were corrected for the initial data collection error. The impact of this error was large, in particular on categories such as pet food, cough and cold medication and rice, which all saw substantial changes to the overall index values.

We have also taken the opportunity to improve our retailer weights and classification methods. While the impact of these changes was much smaller than the correction of the data collection error, they still had a small impact on items such as spray cleaning products and antibacterial wipes.

An additional retailer was also introduced from week beginning 6 April (week 4), which began contributing to the index and weekly changes from week 5.

In the weekly publications on 7 May  and 14 May 2020, we made improvements to the way products are matched over time. On the 14 May, we also removed a single retailer from the series while we improved our methods for identifying unique products.

In the weekly publication on 21 May 2020, we reintroduced the retailer we removed in the previous week. We also improved the methods used to calculate the indices (the movement splice GEKS-Jevons method). The impact of these changes is small at the aggregate index level, although there were some larger changes for individual items, such as nappies and rice.

The updated historical time series for all individual HDP items are published in a dataset, updated weekly.

Back to table of contents

Contact details for this Methodology

Rob Kent-Smith
Telephone: +44 (0)1633 651 618