Researchers from the University of Washington and AI2 Introduce TIFA: An Automatic Evaluation Metric that Measures the Faithfulness of An AI-Generated Image via VQA

Text-to-image generation models are one of the best examples of developments in Artificial Intelligence. With the constant progress and efforts made by the researchers, these models have come a long way. Though there have been significant advancements in text-to-image generation models, these systems usually fail to produce images that accurately match the provided written descriptions. Existing models usually need help in correctly combining several items inside an image, assigning characteristics to the appropriate objects, and producing visual text.

Researchers have been attempting to enhance the ability of generative models to handle these difficulties by introducing linguistic structures to direct the creation of visuals with many. Methods like CLIPScore, which employs CLIP embeddings to assess how similar the created image is to the text input, is an unreliable metric since it is constrained in its capacity to precisely count things and reason compositionally. Using image captions is an alternative strategy where an image is explained in text and then contrasted with the original input. This approach, however, falls short since labeling models could overlook crucial aspects of the image or concentrate on unrelated areas.

To address these issues, a team of researchers from the University of Washington and AI2 has introduced TIFA (Text-to-Image Faithfulness evaluation with Question Answering), an automated evaluation metric that makes use of visual question answering (VQA) to determine how closely an image-generated matches the associated text input. The team has used a language model to generate various question-answer pairs from a given text input. By examining whether well-known VQA models can correctly respond to these queries using the created image, it can be assessed how truthful the image is.

TIFA stands out as a reference-free metric that enables thorough and simple evaluations of the quality of output images. In comparison to other evaluation metrics, TIFA showed a stronger association with human judgments. Using this methodology as a foundation, the team has also presented TIFA v1.0, a benchmark that includes a wide range of 4K text inputs and a total of 25K questions divided into 12 different categories, such as objects and counting. Using TIFA v1.0, this benchmark has been used to evaluate existing text-to-image models holistically, highlighting their current shortcomings and difficulties.

Despite excelling in areas like color and material representation, the tests using TIFA v1.0 showed that modern text-to-image models still have issues accurately depicting quantities of spatial relationships and successfully composing images with multiple objects. The team has shared their aim of building a precise yardstick for evaluating developments in the field of text-to-image synthesis through the introduction of their benchmark. By providing priceless insights, they wish to direct all future research in the direction of overcoming the noted constraints and encouraging the further development of this technology.

In conclusion, TIFA is definitely a great approach to measure image-text alignment by firstly generating a list of questions by LLM and secondly by using Visual Question Answering on the image and computing the accuracy.

Check out the Paper, Project, and Github link. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 29k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..