This AI Paper Proposes to Inject the 3D World into Large Language Models and Introduce a Whole New Family of 3D-LLMs

Over the last several years, they have seen a rise in large language models (LLMs) (like GPT4) that are excellent at various tasks, including communication and common sense reasoning. Recent research has looked at how to align pictures and videos with LLM for a new breed of multi-modal LLMs (like Flamingo and BLIP-2) that can comprehend and make sense of 2D visuals. However, despite the models’ effectiveness in communicating and making decisions, they are based on something other than the deeper notions found in the real 3D physical world, which includes things like spatial connections, affordances, physics, and interaction. As a result, such LLMs are insignificant compared to the robotic helpers shown in science fiction films, which can comprehend 3D situations and do reasoning and planning based on those understandings. To do this, they suggest incorporating the 3D world into large language models and introducing a brand-new class of 3D-LLMs that may process various 3D-related tasks using 3D representations (i.e., 3D point clouds with associated attributes) as input. 

LLMs benefit from two things when they use 3D representations of situations as input: (1) They can store long-term memories about the complete scene in the holistic 3D representations rather than episodic partial-view observations. (2) Reasoning from 3D representations may infer 3D features like affordances and spatial linkages, going much beyond the capabilities of language-based or 2D image-based LLMs. Data collecting is a significant barrier to training the proposed 3D-LLMs. The lack of 3D data makes it difficult to create foundation models based on 3D data, in contrast to the abundance of coupled 2D-images-and-text data on the Internet. Even more difficult to get are 3D data combined with verbal descriptions. 

They suggest a collection of distinctive data-generating processes that provide massive amounts of 3D data linked with language to solve this. They provide three effective prompting processes for communication between 3D data and language, specifically using ChatGPT. As illustrated in Figure 1, they can acquire 300k 3D-language data in this way, which includes information on various tasks such as 3D captioning, dense captioning, 3D question answering, 3D task decomposition, 3D grounding, 3D-assisted dialogue, navigation, and more. The next difficulty is finding useful 3D attributes that match language features for 3D-LLMs. One method is to train 3D encoders from scratch using a contrastive learning paradigm similar to CLIP, which aligns language and 2D pictures. This approach, however, uses a lot of data, time, and GPU resources. From a different angle, several recent efforts (such as idea fusion and 3D-CLR) construct 3D features from 2D multi-view photos. They also use a 3D feature extractor that creates 3D features from the 2D pretrained features of rendered multi-view pictures in response to this. 

Many visual-language models (such as BLIP-2 and Flamingo) have recently started using the 2D pretrained CLIP features to train their VLMs. They can easily employ 2D VLMs as their backbones and input the extracted 3D features to effectively train 3D-LLMs since they are mapped to the same feature space as 2D pretrained features. The fact that 3D LLMs are anticipated to have an underlying 3D spatial sense of information sets them apart from traditional LLMs and 2D VLMs in several important ways. As a result, researchers from UCLA, Shanghai Jiao Tong University, South China University of Technology, University of Illinois Urbana-Champaign, MIT, UMass Amherst and MIT-IBM Watson AI Lab create a 3D localization system that connects language to geographical places. They add 3D position embeddings to the retrieved 3D features to encode spatial information more effectively. Additionally, they add several location tokens to the 3D-LLMs. Localization may then be trained by producing location tokens based on linguistic descriptions of certain items in the sceneries. This would enable 3D-LLMs to record 3D spatial data more effectively. 

In conclusion, their paper makes the following contributions: 

•They present a new family of 3D-based Large Language models (3D-LLMs) that can process a range of 3D-related tasks using input from 3D points with features and language prompts. They concentrate on activities outside the purview of conventional or 2D-LLMs, such as those involving the knowledge of a whole scene, 3D spatial connections, affordances, and 3D planning. 

•They create innovative data-gathering pipelines that could produce much data in 3D language. Based on the pipelines, they gather a dataset with more than 300,000 3D-language data points spanning a wide range of 3D-related activities, such as 3D grounding, dense captioning, 3D question answering, task decomposition, 3D-assisted dialogue, navigation, etc. 

•They employ a 3D feature extractor, which takes rendered multi-view pictures and extracts useful 3D features. They build their training system using 2D pre-trained VLMs. To train the 3D-LLMs to collect 3D spatial information better, they added a 3D localization method. 

• ScanQA, a held-out assessment dataset, performs better in experiments than cutting-edge baselines. On ScanQA, 3D LLMs, in particular, perform much better than baselines (e.g., 9% for BLEU-1) than baselines. Their approach beats 2D VLMs in tests using held-in datasets for 3D captioning, task creation, and 3D-assisted discourse. Qualitative investigations show that their approach can handle a wide range of jobs in more detail.

•They want to make their 3D-LLMs, the 3D-language dataset, and the dataset’s language-aligned 3D features available for upcoming study.

Check out the Paper, Project, and Github. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 26k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.