Allen Institute for AI (AI2) Open-Sources ‘Macaw’, A Versatile, Generative Question-Answering (QA) System

OpenAI’s GPT-3 system is the best at many tasks, including question answering (QA), but it costs money and can only be used by approved users. While there are other pretrained QA systems out on the market, none has matched its few-shot performance so far.

As a possible solution to the above problem, a team of researchers from AI2 has just released Macaw. This versatile and generative question answering system exhibits strong zero-shot performance on a wide range of questions. The best part of Macaw is that it is publicly available for free.

According to a recent study, ‘Challenge300’ (300 challenge questions), Macaw outperformed GPT-3 by over 10%. This is despite the fact that it is an order of magnitude smaller (11 billion vs. 175 billion parameters). Macaw is an impressive (T5-based) language model with not quite as wide-ranging capabilities, but it’s still better than many other systems.

As an example in the research, the researchers gave Macau a riddle and found the answer below.

Macaw has three exciting features

1. Macaw can often produce high-quality answers to questions far outside the domain it was trained on, sometimes surprisingly so.
2. MACAW can be used in a variety of ways, from getting answers to giving them questions. This multi-angle QA capability allows MACAW’s versatility by allowing it to be applied recursively – even if you start with outputs as new inputs for your system.
3. MACAW creates explanations and answers, which is an unusual feature that the AI program can generate at all. Although they’re of lower quality than its answers, it still is an additional feature.

The research team has released the codes and paper. You can access both from the below links.

Paper: https://arxiv.org/pdf/2109.02593.pdf

Code: https://github.com/allenai/macaw

AI2 Blog: https://medium.com/ai2-blog/general-purpose-question-answering-with-macaw-84cd7e3af0f7

Suggested