What is Multimodal Learning? Some Applications

Multimodal Learning is a subfield of Machine Learning in which the system learns from multiple data modalities like text, speech, images, sound, video, etc. These data types are then processed using Computer Vision, Natural Language Processing (NLP), Speech Processing, and Data Mining to solve real-world problems.

Multimodal Learning allows the machine to understand the world better, as using various data inputs can give a holistic understanding of objects and events. This ability enables us to create better AI models and achieve much better results.

What are the benefits of Multimodal Learning?

Two main benefits of Multimodal Learning are:

• Multimodal Learning expands the capabilities of an AI model by making it more human-like. A multimodal AI system gets a wider understanding of the task by analyzing various data types. For example, an AI assistant can use images, pricing data, and purchasing history to offer better-personalized product suggestions.

• The accuracy of an AI model can be increased significantly using Multimodal Learning. For example, an apple can be identified not only by its image but also by the sound of being bitten or its smell.

Multimodal Deep Learning

Multimodal embeddings are valuable when we have multiple input data types that can inform a downstream task. One example is image captioning, where the photos are accompanied by text. Image captioning has many uses, such as helping the visually impaired or creating accurate and searchable descriptions of visual media available on the web. For this task, we can use not only the image itself but also the accompanying text, such as a news article or a reactionary social media post.

For this example, Images and the accompanying text are encoded by their respective embedding models. The resulting embeddings can then be fused and passed through a specially trained recurrent model to generate the caption or an alternative text. This model would show improved performance compared to its unimodal counterpart because of a richer understanding of the input data. 

Another example of Multimodal Deep Learning is visual question answering, i.e., given an image and a question, provide a textual answer. All such applications face challenges, but learning to create multimodal embeddings and develop their architecture is an important step forward. As deep learning techniques continue to permeate modern society, it becomes increasingly important that these models can handle multiple and often incomplete sources of information.

Traditional AI models are unimodal, i.e., they are provided with a single input and are required to perform a single task. For example, a face recognition system takes an image as input and analyzes and compares it with other pictures to find a match.

Using various data types expands the horizon of an AI system. We can better understand this by taking the example of doctors. They only make a full diagnosis once they’ve analyzed all available data, including, for example, medical reports, patient symptoms, patient history, and so on. Similarly, a unimodal system’s output with a single data input is limited.

The main benefits of Multimodal Learning over Unimodal are:

• Having multiple sensors to observe the same data helps the system to make robust inferences.
• The fusion of various sensors facilitates the capture of trends that individual sensors might not capture.

Despite the advantages of Multimodal Learning, most companies like IBM, Amazon, Google, and Microsoft continue to focus on unimodal systems predominantly. One of the reasons is that it is challenging to mitigate the conflicts in modalities and reconcile the difference in the quantitative influence of the modalities over the predictions.

Apart from image captioning and visual question answering, some of the applications of Multimodal AI are:

• By fusing the data from different modes of data, a multimodal AI model can predict the likelihood of a customer’s hospital admission during a visit to the emergency room or the length of a surgical procedure.

• A recent study by Google claims to have developed a system that can predict the video’s next dialogue. The model successfully predicted the next line in an assembling video. 

• Meta is working on a digital assistant project called CAIRaoke, which can interact like a human and even turn text into images and vice versa.

• Researchers have developed a prototype that can translate comic book texts requiring an understanding of the context. It can even identify the gender of the speaking character.

• By combining the data from various streams, multimodal AI systems can predict a company’s financial results and maintenance needs. If old equipment isn’t getting the required attention, the model can infer that it doesn’t need frequent servicing.

Don’t forget to join our Reddit page and discord channel, where we share the latest AI research news, cool AI projects, and more.

References:

• https://www.forbes.com/sites/forbestechcouncil/2022/03/25/six-ai-trends-to-watch-in-2022/?sh=366ac5f32be1
• https://research.aimultiple.com/multimodal-learning/
• https://www.clarifai.com/blog/multimodal-deep-learning-approaches
• https://research.aimultiple.com/multimodal-learning/
• https://www.abiresearch.com/blogs/2022/06/15/multimodal-learning-artificial-intelligence/