Meet ProPainter: An Improved Video Inpainting (VI) AI Framework With Enhanced Propagation And An Efficient Transformer

The field of Artificial Intelligence is evolving like anything. One of its primary sub-fields, well-known Computer Vision, has gained a significant amount of attention in recent times. A particular technique in the domain of computer vision, called video inpainting (VI), fills in any blanks or missing areas in a video while preserving visual coherence and guaranteeing spatial and temporal coherence. The applications of this difficult task include video completeness, object removal, video restoration, watermark removal, and logo removal. The main objective is to seamlessly include the new footage into the video, giving the impression that the missing areas never existed.

VI is specifically challenging because it requires establishing accurate correspondence across different frames of the video for information aggregation. Many earlier VI methods performed propagation in the feature or picture domains separately. Isolating global picture propagation from the learning process can result in problems with spatial misalignment brought on by inaccurate optical flow estimation. The inpainted portions may not appear visually consistent as a result of this misalignment.

Another drawback is the memory and computational restrictions connected to the feature propagation and video transformer approaches. The time span during which these strategies can be used effectively is constrained by these limitations. Because of this, they are unable to investigate correspondence data from distant video frames, which is essential for ensuring flawless inpainting. To overcome the limitations, a team of researchers from S-Lab, Nanyang Technological University, has introduced an improved VI framework called ProPainter. 

ProPainter incorporates two main components: enhanced ProPagation and an efficient Transformer. With ProPainter, the team has introduced a concept called dual-domain propagation, which aims to combine the advantages of feature and picture-warping approaches. By doing this, it makes use of the benefits of international correspondences while ensuring accurate information dissemination. It fills the gap between image and feature-based propagation to produce inpainting results that are more precise and visually consistent.

ProPainter also has a mask-guided sparse video transformer in addition to dual-domain propagation. It maximizes efficiency in contrast to conventional spatiotemporal Transformers, which require substantial processing resources because of interactions between multiple video tokens. It accomplishes this by concentrating attention just on the pertinent areas discovered by inpainting masks. Since inpainting masks often only cover specific regions of the video and nearby frames frequently have repeated textures, this method eliminates pointless tokens, lowering the computational burden and memory needs. This allows the transformer to function well without compromising the quality of the inpainting.

ProPainter outperforms earlier VI approaches by a large margin of 1.46 dB in PSNR (Peak Signal-to-Noise Ratio), which is a standard statistic for evaluating the quality of images and videos. In conclusion, ProPainter is an important development in the field of video inpainting since it has improved performance while retaining a high level of efficiency. It addresses important problems with spatial misalignment and computational limitations, making it a useful tool for jobs like object removal, video completion, and video restoration.

Check out the Paper and Github. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 30k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..