Meet Animate-A-Story: A Storytelling Approach With Retrieval-Augmented Video Generation That Can Synthesize High-Quality, Structured, and Character-Driven Videos

Text-to-image models have recently gained a lot of attention. With the introduction of Generative Artificial Intelligence, models like GPT and DALL-E have been in the headlines ever since their release. Their rise in popularity is the reason because of why generating content like a human is no longer a dream today. Not only text-to-image models but also text-to-video (T2V) generation is now possible. Filming live-action or producing computer-generated animation is typically required to produce interesting storytelling videos, which is a difficult and time-consuming procedure. 

Though the latest advancements in text-to-video production have demonstrated promise in automatically creating videos from text-based descriptions, there are still certain limitations. Lack of control over the resulting video’s design and layout, which are essential for visualizing an engaging story and producing a cinematic experience, is a primary challenge. Close-ups, long views, and composition, among other filmmaking techniques, are crucial in allowing the audience to understand subliminal messages. Currently, existing text-to-video methods struggle to provide appropriate motions and layouts that adhere to the standards of cinema.

To address the limitations, a team of researchers has proposed a unique video generation approach, which is retrieval-augmented video generation, called Animate-A-Story. This method takes advantage of the abundance of existing video content by obtaining films from external databases based on text prompts and using them as a guide signal for the T2V creation process. Users can have greater control over the layout and composition of the generated videos when animating a story, using the input retrieved videos as a structure reference.

The framework consists of two modules: Motion Structure Retrieval and Structure-Guided Text-to-Video Synthesis. The Motion Structure Retrieval module supplies video candidates that match the requested scene or motion context as indicated by query texts. For this, video depths are extracted as motion structures using a commercial video retrieval system. The second module, Structure-Guided Text-to-Video Synthesis, uses the text prompts and motion structure as input to produce films that follow the storyline. A model has been created for customizable video production that enables flexible control over the plot and characters of the video. The created videos adhere to the intended storytelling elements by following the structural direction and visual guidelines.

This approach places a strong emphasis on preserving visual coherence between footage. The team has also developed a successful concept personalization strategy to ensure this. Through text prompts, this method enables viewers to select preferred character identities, preserving the uniformity of the characters’ appearances throughout the video. For evaluation, the team has compared the approach to existing baselines. The results demonstrated significant advantages of this approach, proving its capability to generate high-quality, coherent, and visually engaging storytelling videos. 

The team has summarized the contribution as follows: 

1. A retrieval-augmented paradigm for narrative video synthesis has been introduced, which, for the first time, allows the use of varied existing videos for storytelling.

2. The framework’s usefulness is supported by experimental findings, which establish it as a cutting-edge tool for creating videos that are remarkably user-friendly.

3. A flexible structure-guided text-to-video approach has been proposed that successfully reconciles the tension between character production and structure guiding.

4. The team has also introduced TimeInv, a new concept in the personalization approach that significantly exceeds its current rivals.

Check out the Paper, Github, and Project Page. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 26k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

🚀 Check Out 900+ AI Tools in AI Tools Club