Researchers from the National University of Singapore propose Show-1: A Hybrid Artificial Intelligence Model that Marries Pixel-Based and Latent-Based VDMs for Text-to-Video Generation

Researchers from the National University of Singapore introduced Show-1, a hybrid model for text-to-video generation that combines the strengths of pixel-based and latent-based video diffusion models (VDMs). While pixel VDMs are computationally expensive and latent VDMs struggle with precise text-video alignment, Show-1 offers a novel solution. It initially uses pixel VDMs to create low-resolution videos with strong text-video correlation and then employs latent VDMs to upsample these videos to high resolution. The result is high-quality, efficiently generated videos with precise alignment validated on standard video generation benchmarks.

Their research presents an innovative approach for generating photorealistic videos from text descriptions. It leverages pixel-based VDMs for initial video creation, ensuring precise alignment and motion portrayal, and then employs latent-based VDMs for efficient super-resolution. Show-1 achieves state-of-the-art performance on the MSR-VTT dataset, making it a promising solution.

Their approach introduces a method for generating highly realistic videos from text descriptions. It combines pixel-based VDMs for accurate initial video creation and latent-based VDMs for efficient super-resolution. The approach, Show-1, excels in achieving precise text-video alignment, motion portrayal, and cost-effectiveness. 

Their method leverages both pixel-based and latent-based VDMs for text-to-video generation. Pixel-based VDMs ensure accurate text-video alignment and motion portrayal, while latent-based VDMs efficiently perform super-resolution. The training involves keyframe models, interpolation models, initial super-resolution models, and a text-to-video (t2v) model. Using multiple GPUs, keyframe models require three days of training, while the interpolation and initial super-resolution models each take a day. The t2v model is trained with expert adaptation over three days using the WebVid-10M dataset.

Researchers evaluate the proposed approach on the UCF-101 and MSR-VTT datasets. For UCF-101, Show-1 exhibits strong zero-shot capabilities compared to other methods measured by the IS metric. The MSR-VTT dataset outperforms state-of-the-art models in terms of FID-vid, FVD, and CLIPSIM scores, indicating exceptional visual congruence and semantic coherence. These results affirm the capability of Show-1 to generate highly faithful and photorealistic videos, excelling in optical quality and content coherence.

Show-1, a model that fuses pixel-based and latent-based VDMs, excels in text-to-video generation. The approach ensures precise text-video alignment, motion portrayal, and efficient super-resolution, enhancing computational efficiency. Evaluations on UCF-101 and MSR-VTT datasets confirm their superior visual quality and semantic coherence, outperforming or matching other methods. 

Future research should delve deeper into combining pixel-based and latent-based VDMs for text-to-video generation, optimizing efficiency, and improving alignment. Alternative methods for enhanced alignment and motion portrayal should be explored, along with evaluating diverse datasets. Investigating transfer learning and adaptability is crucial. Enhancing temporal coherence and user studies for realistic output and quality assessment is essential, fostering text-to-video advancements.

Check out the Paper, Github, and Project. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 31k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..

We are also on WhatsApp. Join our AI Channel on Whatsapp..