Researchers at Activeloop AI Introduce ‘Deep Lake,’ an Open-Source Lakehouse for Deep Learning Applications

A data lake is a centralized repository where enterprises may store structured, unstructured, and semi-structured data. Data lakes improve data management, governance, and analysis. Furthermore, they enable breaking down data silos and discovering previously concealed insights in diverse data sources. Traditionally, first-generation data lakes gathered data into distributed storage systems such as HDFS or AWS S3. Unorganized data collections transformed data lakes into “data swamps,” giving birth to the second generation of data lakes led by Delta, Iceberg, and Hudi. They work only on top of standardized structured formats such as Parquet, ORC, and Avro and offer capabilities like as time travel, ACID transactions, and schema evolution.

To conduct analytical queries, data lakes easily interface with query engines like as Presto, Athena, Hive, and Photon. They also interface to frameworks like as Hadoop, Spark, and Airflow for ETL pipeline maintenance. In turn, the combination of data lakes and query engines with explicit compute and storage separation resulted in the introduction of systems such as Lakehouse that serve as alternatives to data warehouses such as Snowflake, BigQuery, Redshift, and Clickhouse. During the last decade, deep learning has surpassed standard machine learning algorithms for dealing with unstructured and complicated data such as text, photos, videos, and audio.

Deep learning systems outperformed not only previous approaches but also attained superhuman accuracy in applications, including cancer diagnosis from X-Ray pictures, anatomical reconstruction of human brain cells, playing games, driving automobiles, unfolding proteins, and producing graphics. Large language models with transformer-based topologies produced cutting-edge outcomes in translation, reasoning, summarization, and text completion tasks. Unstructured data is embedded into vectors in large multi-modal networks enabling cross-modal search. Furthermore, they are utilized to produce photorealistic visuals from the text.

Although the availability of large datasets such as CoCo (330K images), ImageNet (1.2M images), Oscar (multilingual text corpus), and LAION (400M and 5B images) has been one of the primary contributors to the success of deep learning models, it lacks a well-established data infrastructure blueprint similar to traditional analytical workloads to support such scale. On the other hand, because Modern Data Stack (MDS) lacks the functionality needed to build performant deep learning-based solutions, enterprises choose to design their own systems. Deep Lake, a lakehouse tailored for deep learning workloads, is introduced in this research.

Deep Lake preserves the benefits of a regular data lake with one noteworthy difference: it stores complicated data as tensors and instantly feeds the data to deep learning frameworks across the network without losing GPU utilization. It also supports a natural interface across deep learning frameworks like PyTorch, TensorFlow, and JAX. You can access all the resources of DeepLake on their GitHub page.

This Article is written as a research summary article by Marktechpost Staff based on the research paper 'Deep Lake: a Lakehouse for Deep Learning'. All Credit For This Research Goes To Researchers on This Project. Check out the paper and github.

Please Don't Forget To Join Our ML Subreddit