Meet Inpaint Anything (IA): A Versatile AI Tool that Combines the Capabilities of Remove Anything, Fill Anything, and Replace Anything

Have you ever taken a photograph only to discover that something in the backdrop ruined the shot? Or you inadvertently destroyed a section of an image that you now urgently need to restore. Inpainting is a common technique in digital picture editing and restoration, and various software tools have been created to make the process quicker and more efficient. These programs analyze the surrounding areas of a damaged or missing picture section and then intelligently fill the gap to produce a coherent image. Inpainting fills in missing or damaged picture areas, resulting in smooth, natural-looking results.

Overall, inpainting is a powerful technique that can potentially restore and improve our images in previously imagined ways. It is now more accessible than ever because of the availability of advanced computing capabilities, and its applications are expanding. Researchers from the University of Science and Technology of China and the Eastern Institute for Advanced Study make the first effort at mask-free picture inpainting and suggest a new paradigm of “clicking and filling,” which they call Inpaint Anything (IA).

Why a new tool like Inpaint Anything was required? 

• State-of-the-art (SOTA) picture inpainting efforts, such as LaMa, Repaint, MAT, ZITS, and others, have made significant development. They can inpaint enormous areas, operate well with complicated repeating structures, and generalize well to high-resolution pictures. However, they often need detailed annotations for each mask, which are required for training and inference. 

• The Segment Anything Model (SAM) provides a robust segmentation basis by generating high-quality object masks from input prompts like points or boxes. It may be used to create detailed and precise masks for all items in a picture. Their mask segmentation predictions, however, have not been properly explored. 

• Furthermore, current inpainting algorithms can only replace the excised region with context. AIGC models provide new options for creativity, with the ability to satisfy great demand and aid people in creating new material. 

• As a result, by combining the benefits of SAM, SOTA image inpainters, and AI-generated content (AIGC) models, they create a robust and user-friendly pipeline for handling more general inpainting-related challenges such as object removal, new content filling, and backdrop replacement. 

What can Inpaint Anything do? 

• Inpainters SAM + SOTA for eliminating anything: Users of IA may quickly delete certain elements from the interface by clicking on them. Furthermore, IA allows users to fill the resulting “hole” with contextual data. They use the abilities of SAM and some SOTA Inpainters, like LaMa, to achieve this goal. After manually improved through corrosion and dilation, the mask predictions given by SAM serve as input for the inpainting models, offering unambiguous signs for the object portions to be erased and filled. 

• Filling or replacing anything using SAM + AIGC models: 

(1) Following the removal of items, IA gives users the option of filling the resultant “hole” with contextual data or “new content.” A robust AI-generated content (AIGC) model, such as Stable Diffusion, is used to produce new items via text prompts. For example, users may use the word “dog” or a statement like “a cute dog, sitting on the bench” to produce a new dog to fill the hole with. 

(2) users may also utilize IA to keep the clicked-on item and replace the remaining backdrop with the freshly produced scene. This IA scene replacement procedure allows you to prompt AIGC models in a variety of ways, for as, by displaying a different image as a visual cue or a short caption as a text prompt. Users can, for example, preserve the dog in a photograph while replacing the original indoor setting with an outdoor one.

The complete code source can be found on GitHub, along with instructions to use the tool.

Check out the Paper and Github. Don’t forget to join our 19k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more. If you have any questions regarding the above article or if we missed anything, feel free to email us at Asif@marktechpost.com

🚀 Check Out 100’s AI Tools in AI Tools Club