Web Scraping Drives Decision-Making In the UK & US Ecommerce

Over 82% of eCommerce organizations are now using web scraping to gather external data to help guide their decision-making. Web scraping overtakes internal data collection (81%), which is the number one method in nearly every other sector. This is one of the key findings from the new Oxylabs white paper, Alternative Data Defines Competition in the US & UK Ecommerce Sectors. It demonstrates how web scraping is shaping the industry.

In cooperation with Censuswide, Oxylabs surveyed 500 UK-based and 501 US-based senior data decision-makers from UK and US eCommerce companies. Oxylabs gathered insights on attitudes towards data collection in the sector.

 Web scraping takes the lead

Web scraping has entrenched itself as the primary vehicle for decision-making. This technology has become the most popular way to collect data in the eCommerce industry.

When asked what methods the respondents use to acquire data, 75% of eCommerce organizations selected web scraping. Also, 74% are using the runner-up, internal data collection. Other methods, such as the procurement of data from third-party aggregators, fall further behind at 65% for traditional, and 59% for alternative data. Such a trend isn’t surprising. Having control of the entire pipeline allows for greater flexibility and adaptability to business goals.

When asked which type of data respondents use to make business decisions, third-party aggregators came in third behind web scraping and internal data. Only 66% view it as the most crucial source of data-driven decision-making. Such a trend might be caused by eCommerce companies and retailers mainly dealing with regular consumers. Third-party aggregators are often oriented towards B2B data, making their offerings more useful to internal decision-making than direct revenue generation.

Gediminas Rickevičius, VP of Global Partnerships at Oxylabs, said: “Web scraping overtaking internal data collection in any industry is a milestone moment. External data collection is where businesses across many sectors will compete fiercely with one another in the future. eCommerce is an important trendsetter in this respect. As external data offers exciting, unique, and profitable insights, getting to them first should be a priority for every business, whether involved in eCommerce or not. The reliance on data is clearly important. Therefore it is no surprise that the budgets are also expected to increase over the coming year.”

Expanding data budgets

As the data access needs increased (cited by 85%, with 30% citing significant increases), so did the budgets for data departments (cited by 73%, with 27% citing substantial increases). Additionally, 70% of respondents expect to continue expanding their data department budgets in the coming 12 months.

Rickevičius continued: “Ecommerce isn’t the only industry heading towards significantly increasing the procurement of external data. These trends are visible throughout all industries. They show that external data, especially its advanced forms, has become an essential part of daily operations. Today’s business optimization is achieved through careful data-driven decision-making. It shows the immense importance of departments that support such processes.”

“In eCommerce, where margins are thin and the competition is fierce, companies will take absolutely any advantage they can get. Web scraping has now proven itself as the way to do so. Through external data, businesses can monitor the competition, adapt to changing environments. They can even predict consumer behavior, bringing it closer to optimal performance,” concluded Rickevičius.

To download your copy of the Oxylabs’ Alternative Data Defines Competition in the UK & US eCommerce sectors white paper, please visit HERE.