Facebook AI Introduces A New Image Generation Model Called ‘IC-GAN’ That Creates High-Quality Images of Unfamiliar Objects And Scenes

Generative adversarial networks (GANs) have been used for few years to generate photorealistic images of objects or scenes that are very similar in style and content. However, until now, these models could only produce output related to datasets they were trained on – which had limitations because there was usually less diversity among those files than what you would find when generating new ideas. A conventional GAN trained on images of cars shows impressive results when asked to generate other images of cars or automobiles. But the trained GAN will likely fail if given a flower or any object outside its automotive data set.

Failure to show non-identical objects from the training dataset is a huge limitation, and it definitely needs to be resolved to meet the demand. Facebook is trying to solve the above problem by introducing Instance-Conditioned GAN (IC-GAN). The IC-GAN is a new image generation model that can produce high-quality images with some input, even if it doesn’t appear in the training set. The unique thing about the IC-GAN model is that it can generate realistic, unforeseen image combinations—for example, a camel in snow or zebras running through an urban cityscape.

It is no surprise that IC-GANs could be used to create visual examples for data sets with these new capabilities. This would allow artists and creators alike more expansive AI-generated content by creating art from photos or videos in the same way an artist might draw a picture using pencils and paintbrushes at their disposal.

In the proposed approach, the IC-GAN can be used with both labelled and unlabeled data sets. It applies a GAN framework to model a mixture of local clustering in one image and overlapping samples from different regions or neighborhoods across images similar to their nearest neighbors in the same dataset. They use “neighbors” for the input layer into Discriminator neural network, which helps the generator create realistic-looking sample pictures by mixing real pixels with predicted ones. Thus, the model could use data sets more efficiently because it avoided the problem of partitioning into small clusters.

Augmenting data with IC-GAN can help model builders include items or objects that are not commonly found in the training set. Furthermore, since it works across different domains and generates more diverse examples for object recognition models, this approach may be an effective way to improve your research endeavors. Traditional GAN models can only generate images of zebras in grasslands because their training data consists solely of those types of pictures. When you try and train a GAN using urban areas, such as New York City or Los Angeles, it will fail since there’s not enough examples from which the model can learn what features define these landscapes – like buildings or trees. The IC-GAN model can generate novel combinations of data by including objects that are not usually seen in standard datasets. For example, the research shows how it created an image of cows walking across the sand – this is just one way they choose from many possibilities.

Paper: https://arxiv.org/abs/2109.05070?

Code: https://github.com/facebookresearch/ic_gan?

Source: https://ai.facebook.com/blog/instance-conditioned-gans/

Suggested