Big Data in 2022: What to expect?

Prominent trends in Big Data in 2022 will include data management professionals taking center stage in business, an increasing focus on ethics, and Big Tech under pressure from government worldwide about the acquisition and aggregation of personal data, according to experts at public data gathering solutions provider, Oxylabs.

Growing markets for external data

Tomas Montvilas, Chief Commercial Officer at Oxylabs, says that more industries will discover the benefits of using external data in the upcoming year. He lists a few:

“The market of SaaS products that use external data to provide insights for their clients will grow further in 2022. The successful IPOs of such companies as Semrush, Similarweb, and Zoominfo, among others, are driving further investments in the field and we are likely to see more stars emerging,” Tomas says.

Cybersecurity is another important area he sees for the web scraping industry’s growth. Cyber threats are becoming more advanced and require new measures of defense. This is where web monitoring and scraping technologies come in.

“Constant monitoring of both public web and dark web can help identify malicious sites and programs early. It can also help catch data leaks sooner by finding data sets when they go for sale on the dark web, and recognize the actions of hacker groups. Meanwhile, proxies can serve well in email protection, by providing the ability to scan emails from different IP addresses,” he explains.

Data management role in business further increasing

With the recent explosion in digitizing everything, data management and analytics became central roles in business. Data departments have experienced exponential growth the past few years, and the growth will continue in 2022.

Gediminas Rickevičius, Vice President of Global Partnerships at Oxylabs, notes that the increasing importance of data departments can be easily illustrated by budgeting trends. According to several recent surveys Oxylabs of the UK’s finance and ecommerce industries, 51% of ecommerce and 43% of financial services data departments expect to increase their budgets.

Another trend Gediminas predicts for data departments will be the increasing outsourcing of automated public web data gathering tools. There will be several reasons for this. First, as companies become dependent on external data, manual data gathering processes are simply not sufficient. Another important factor is the current job market.

“With “the great resignation” and lack of human resources being the dominant topics of 2021 it became even harder to find in-house professionals who could dedicate all their time for maintaining and adjusting web scraping infrastructure. Outsourcing this task optimizes resources and focus on data analysis, rather than acquisition, says Gediminas.

Pressure for Big Tech could affect web data industry

Pressure on Big Tech from governments around the world has grown in recent years. There will likely be a push for new regulations, especially around acquisition and aggregation of personal data in 2022.

The data gathering industry should not turn a blind eye on these processes, according to Denas Grybauskas, Head of Legal at Oxylabs. Some big tech companies might already be in the process of restricting access to public web data because of government pressure, which could affect many businesses.

“Some companies are preparing for the old life tactic – pointing fingers. That is what, at least in accordance with the leaked emails, Meta (Facebook) is planning to do in terms of personal data leaks and data scraping companies – to shift the attention from leaks by stating that personal data got out in the wild not due to Facebook’s mistakes, but those of scrapers,” Denas says.

Moving towards industry self-regulation

When it comes to strategic development of the data gathering industry, ethics and legal implications will remain the hot topic in 2022, pushing the industry to continue raising the standards. I believe, that ethical proxy acquisition and strong KYC practices will dominate the conversation.

As with most new technologies, web scraping is developing faster than the regulation that could safeguard it from potential misuse cases. Therefore, the industry itself has to take the lead in developing the self-regulation guidelines and standards for the proper use of technology.

The issue is set to become more mainstream in 2022. First of all, as the largest industry players are setting the tone, smaller players are likely to follow. Secondly, brands that use proxy services are putting more emphasis on the nature of proxies too, as potential misuse could damage their reputation as well.