Amazon AI Researchers Propose A New Machine Learning Framework Called ‘GRAVL-BERT’: A BERT-Based Graphical Visual-Linguistic Representations For Multimodal Coreference Resolution

The use of multimodal data for AI training has gained popularity, particularly in recent years. The popularity of voice-activated screen devices like the Amazon Echo Show is rising due to their increased potential for multimodal interactions. Customers can refer to products on-screen using spoken language, which makes it easier for them to express their objectives. Multimodal coreference resolution (MCR) refers to this process of selecting the appropriate object on the screen utilizing natural language comprehension. In order to create the next generation of conversational bots involves resolving the references across many modalities, such as text and visuals.

On visual-linguistic tasks where they look for images that match a verbal description, multimodal models have in the past produced impressive results. However, coreference resolution is the most challenging, partly because there are many different methods to refer to an object on the screen. Most recent developments in this area use single-turn utterances and concentrate on straightforward coreference resolves. By merging Graph Neural Networks with VL-BERT, a research team from Amazon and UCLA developed GRAVL-BERT. This unified MCR framework incorporates visual relationships between objects, background scenery, dialogue, and metadata. The model uses scene photos to resolve coreferences in multi-turn conversations. 

This Amazon-UCLA model finished first in the multimodal coreference resolution task in the tenth Dialog State Tracking Challenge (DSTC10). The team’s research paper was also presented at the International Conference on Computational Linguistics (COLING). Visual-linguistic BERT (VL-BERT), a model trained on pairs of text and images, is the foundation of the GRAVL-BERT model. It extends the BERT model’s standard masked-language-model training, in which some parts of the input are hidden, and the model must learn to anticipate them. As a result, it gains the ability to predict images from text input and vice versa.

The team focused primarily on three significant adjustments to this strategy. Graph neural networks were used to generate the necessary graphical data from the relationships between the items in an image and display those relationships as graphs. Object metadata was included as additional knowledge sources to enable coreferencing based on nonvisual attributes like brand or pricing. The most recent change required active sampling from an object’s neighborhood to augment information about its surroundings and create captions that described the thing. The model generates a binary decision for each object in the current scene as to whether or not it is the object mentioned in the current exchange of dialogue.

GRAVL-BERT creates a graph using the relative positions of the scene’s objects, where the nodes are objects, and the edges are the connections between them. This graph is then sent to a network of graph convolutions, which creates an embedding for each node that contains data about that node’s immediate surroundings in the graph. The coreference resolution model then uses these embeddings as inputs.

Additionally, the object recognizer might not always be able to identify some components of a visual scene. However, consumers might still wish to utilize those components to describe objects. The researchers used knowledge of an object’s immediate surroundings to resolve such criteria. There were two ways to record this. The first technique involves creating eight boxes in eight different directions around the object. The visual input stream of the coreference resolution model was then supplemented with the visual features of the image regions contained within those boxes. In the second technique, the group trained an image captioning model to describe other items close to the object of interest. These can be things like shelves or racks that aided in the model’s ability to identify objects based on descriptions of their surroundings.

According to the team, their model was able to achieve first place in the DSTC10 challenge by combining these changes with the dialogue-turn distance measure. The F1 score, which considers both false positives and false negatives, was used to evaluate how well the challenger performed. By making it simpler for users of Alexa-enabled devices with screens to express their intents, Amazon expects that their work will benefit Alexa users.

Check out the paper, code, and reference article. All Credit For This Research Goes To Researchers on This Project. Also, don’t forget to join our Reddit page and discord channel, where we share the latest AI research news, cool AI projects, and more.