High Precision Semantic Image Editing with EditGAN

Generative Adversarial Networks or GANs have been enjoying new applications in the image editing industry. For the past few months, EditGAN is gaining popularity in the AI/ML industry because it’s a novel method for high-precision, and high-quality semantic image editing. 

We will be talking about the EditGAN model in detail, and let you know why it might prove to be a milestone in the semantic image editing industry.

So let’s start. But before we get to know what EditGAN is, it’s important for us to understand what is the importance of EditGAN, and why it is a significant step forward. 

Why EditGAN?

Although traditional GAN architectures have helped the AI-based image editing industry advance significantly, there are some major challenges with building a GAN architecture from scratch. 

1. During the training phase, a GAN architecture requires a high amount of labeled data with semantic segmentation annotations. 
2. They are capable of providing only high-level control. 
3. And often, they just interpolate back and forth between images. 

It can be observed that although traditional GAN architectures get the work done, they are not effective for wide scale deployment. Traditional GAN architecture’s sub-par efficiency is the reason why EditGAN was introduced by NVIDIA in 2022. 

EditGAN is proposed to be an effective method for high precision, and high quality semantic image editing with the capability of allowing its users to edit images by altering their highly detailed segmentation masks of an image. One of the reasons why EditGAN is a scalable method for image editing tasks is because of its architecture. 

The EditGAN model is built on a GAN framework that models images and their semantic segmentations jointly, and requires only a handful of labeled or annotated training data. The developers of EditGAN have attempted to embed an image into GAN’s latent space to effectively modify the image by performing conditional latent code optimization in accordance with the segmentation edit. Furthermore, to amortize optimization, the model attempts to find “editing vectors” in latent space that realizes the edits. 

The architecture of the EditGAN framework allows the model to learn an arbitrary number of editing vectors that can then be implemented or applied directly on other images with high speed, and efficiency. Furthermore, experimental results indicate that EditGAN can edit images with a never seen before level of detail while preserving the image quality to a maximum. 

To sum as to why we need EditGAN, it’s the first ever GAN-based image editing framework that offers

1. Very high-precision editing. 
2. Can work with a handful of labeled data. 
3. Can be deployed effectively in real-time scenarios. 
4. Allows compositionality for multiple edits simultaneously. 
5. Works on GAN-generated, real embedded, and even out of domain images. 

High-Precision Semantic Image Editing with EditGAN 

StyleGAN2, a state of the art GAN framework for image synthesis, is the primary image generation component of EditGAN. The StyleGAN2 framework maps latent codes that are drawn from a pool of multivariate normal distribution, and maps it into realistic images. 

StyleGAN2 is a deep generative model that has been trained to synthesize images of the highest quality possible along with acquiring a semantic understanding of the images modeled. 

Segmentation Training and Inference

The EditGAN model embeds an image into the GAN’s latent space using optimization, and an encoder to perform segmentation on a new image, and training the segmentation branch. The EditGAN framework continues to build on previous works, and trains an encoder to embed the images in the latent space. The primary objective here is to train the encoder consisting of standard pixel-wise L2 and LPIPS construction losses using samples from GAN, and real-life training data. Furthermore, the model also regularizes the encoder explicitly using the latent codes when working with the GAN samples. 

Resultantly, the model embeds the annotated images from the dataset labeled with semantic segmentation into the latent space, and uses cross entropy loss to train the segmentation branch of the generator. 

Using Segmentation Editing to Find Semantics in Latent Space

The primary purpose of EditGAN is to leverage the joint distribution of semantic segmentations and images for high precision image editing. Let’s say we have an image x that needs to be edited, so the model embeds the image into EditGAN’s latent space or uses the sample images from the model itself. The segmentation branch then generates y or the corresponding segmentation primarily because both RGB images & segmentations share the same latent codes w. Developers can then use any labeling or digital painting tools to modify the segmentation & edit them as per their requirements manually. 

Different Ways of Editing during Inference

The latent space editing vectors obtained using optimization can be described as semantically meaningful, and are often disentangled with different attributes. Therefore, to edit a new image, the model can directly embed the image into the latent space, and directly perform the same editing operations that the model learnt previously, without performing the optimization all over again from scratch. It would be safe to say that the editing vectors the model learns amortize the optimization that was essential to edit the image initially. 

It is worth noting that developers have still not perfected disentanglement, and edit vectors often do not return the best results when used to other images. However, the issue can be overcome by removing editing artifacts from other parts of the image by performing a few additional optimization steps during the test time. 

On the basis of our current learnings, the EditGAN framework can be used to edit images in three different modes.