A New AI Research Proposes Compositional Exemplars for In-context Learning (CEIL): A Novel Approach That Uses Determinantal Point Process (DPP) for More Efficient In-context Learning

In the past few years, language models have become one of the quickest-growing fields in Artificial Intelligence. These models have been developed to process, produce and use natural language text to drive some creative and ground-breaking AI applications. Language models are revolutionizing and introducing us to a new era in AI expansion. The model developed by OpenAI called GPT-3, which recently gained popularity, possesses extraordinary capabilities and shows great performance. It uses a transformer architecture to process text, resulting in a model that can effortlessly generate content and answer questions as a human would. Not only this, the model is even capable of summarizing long texts, completing codes, and carrying out tasks with super good speed and accuracy.

Language models can operate flawlessly, thanks to the concept of in-context learning by which they generalize to unseen tasks. However, in-context learning (ICL) shows a slight limitation because of its sensitivity towards the selection of in-context examples and inability to take into account the inter-relationship between the in-context examples. The new approach, called Compositional Exemplars for In-context Learning or simply CEIL, formulates the process of choosing in-context examples as a subset selection problem. It is not based on simple heuristics like the previous methods but shows a great interaction between the input and the examples. 

In-context learning can be simply explained as learning in which the model learns something new and unique by looking at examples similar to the ones the model is trying to predict. This can be explained with the help of an example. While learning the addition of fractions in Mathematics, one learns so by first looking at examples involving the addition of fractions with the same denominator. The idea is to comprehend the patterns and rules to solve new and unseen problems. In terms of in-context learning, to make the model understand and classify positive and negative sentences, it is shown several examples and some context about the sentence, such as an app review or a tweet. 

Since traditional methods use basic estimations and show sub-optimal performance, CEIL is a better approach because it uses the Determinantal Point Processes (DPPs) concept. It does so to model the interaction between the given input and the in-context examples. DPP is a probabilistic model that selects various subsets of items from a bigger set. The determinants in DPP measure the volume of a subspace of a larger space spanned by a set of vectors. In CEIL, DPP has been used to choose diverse sets or subsets of examples for training a model. CEIL models all exemplar sets by learning its joint probability with a conditional DPP, followed by training it to align with the Language model score through a contrastive loss.

The team behind Compositional Exemplars for In-context Learning (CEIL) has validated the approach on 12 classification and generation datasets from 7 different Natural language Processing tasks. The data varied from sentiment analysis and paraphrase detection data to reasoning and open-domain question answering. The CEIL proved more efficient and effective than the standard methods because of its transferability and compositionality. Consequently, introducing Compositional Exemplars for In-context Learning (CEIL) seems like a game changer in Natural Language processing.  

Check out the Paper and Github. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 14k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.