Researchers from Meta GenAI Introduce Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis Artificial Intelligence Framework

Artificial intelligence has recently been used in all spheres of life. Likewise, it is being used for video generation and video editing. AI has opened up new possibilities for creativity, enabling seamless content generation and manipulation. However, video editing remains challenging due to the intricate nature of maintaining temporal coherence between individual frames. The Traditional approaches to video editing addressed this issue by tracking pixel movement via optical flow or reconstructing videos as layered representations. However, these techniques are prone to failure when confronted with videos featuring large motions or complex dynamics because pixel tracking remains an unresolved problem in computer vision.

Consequently, the researchers of Meta GenAI have introduced Fairy, a novel and efficient video-to-video synthesis framework designed specifically for instruction-guided video editing tasks. Fairy takes a video input with N frames and uses the natural language editing instruction to create a new video that follows the given instruction while maintaining the semantic context of the original video. Fairy uses an anchor-based cross-frame attention mechanism that transfers diffusion features among adjacent frames. By this technique, Fairy produces 120-frame 512 × 384 resolution videos in just 14 seconds, which marks a considerable improvement of at least 44x compared to earlier state-of-the-art systems.

Fairy can also preserve temporal consistency throughout the editing process. Researchers used a unique data augmentation strategy that imparts affine transformation equivalence onto the model. Consequently, the system can effectively manage alterations in both source and target images, further bolstering its performance, especially when dealing with videos characterized by expansive motion or intricate dynamics.

The developers devised a scheme where value attributes extracted from carefully selected anchor frames are propagated to candidate frames via cross-frame attention mechanisms. This subsequently enables the establishment of an attention map serving as a similarity measure, ultimately finetuning and harmonizing feature representations spanning various frames. This design substantially diminishes feature discrepancies, culminating in enhanced temporal uniformity in the final outputs.

The researchers evaluated the model by subjecting it to rigorous evaluations encompassing 1000 generated videos. The researchers found that Fairy demonstrated superior visual qualities to previous state-of-the-art systems. Moreover, it exhibited an impressive speed enhancement exceeding 44x, courtesy of eight GPU-enabled parallel processing capacities. But it also has some limitations. Despite identical text prompts and random initialization noises, it can have slight inconsistencies within input frames. These abnormalities can result from affine modifications performed to inputs or small changes occurring within video sequences. 

In conclusion, Meta’s Fairy is a transformative leap forward in video editing and artificial intelligence. With its outstanding temporal consistency and video synthesis, Fairy establishes itself as a benchmark for quality and efficiency in the industry. Users can generate high-resolution videos at exceptional speeds due to the innovative use of image-editing diffusion models, anchor-based cross-frame attention, and equivariant fine-tuning.

Check out the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 35k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..