Meet CityDreamer: A Compositional Generative Model for Unbounded 3D Cities

The creation of 3D natural settings has been the subject of a lot of research in recent years. Significant advancements have been made in the creation of several kinds of 3D objects, including 3D cities, 3D scenarios, and 3D avatars. Cities have many applications in areas like urban planning, environmental simulations, and game creation as a key type of 3D component. 

Models like GANCraft and SceneDreamer have produced images within 3D scenes using volumetric neural rendering algorithms, 3D coordinates, and semantic labels. These techniques have shown promise in creating 3D natural settings using SPADE’s fake ground-truth photos. Though the development of 3D natural settings has gained immense popularity, it comes with a number of limitations. The creation of 3D cities is fundamentally more complicated than the creation of 3D nature scenes. This is because, though buildings belong to the same class of objects, they still have a far greater range of appearances than the more uniform appearance of natural scene elements like trees.

A unique approach called CityDreamer has been introduced to overcome these challenges. It has opened the way for more accessible and realistic 3D city development. CityDreamer is basically a compositional generative model specifically created for unbounded 3D cities. Its unique strategy of isolating the creation of building instances from other background elements often present in cities, such as streets, parks, and water features, sets it different. Through several modules inside the model, this division has been achieved.

Two large datasets named OSM and GoogleEarth have also been created to improve the authenticity of the generated 3D cities in both their layouts and aesthetic appearances. There is a significant quantity of actual city imagery in these databases. By adding traits and differences from the real world, the addition of such data attempts to increase the realism of the 3D cities that are built. CityDreamer has proven to be superior to cutting-edge techniques in the field of 3D city development through thorough trial and review. It has demonstrated its capacity to create a variety of realistic 3D cities, overcoming the difficulties posed by the complexity of urban surroundings and the requirement for accurate, high-quality outcomes.

The key contributions of the CityDreamer project can be summarized as follows –

1. CityDreamer Model: The core contribution of this work is the introduction of the CityDreamer model, which is specifically tailored for generating unbounded 3D cities. It works on a unique approach of separating the generation of building instances from other background objects like roads, green areas, and water bodies, which has been achieved through distinct modules within the model, allowing for more precise control and enhanced realism in the generated urban landscapes.

2. Construction of Datasets: The first dataset, OSM, provides more realistic city layouts by sourcing data from OpenStreetMap. It includes semantic maps and height fields, offering valuable information about the locations of roads, buildings, green spaces, and water features. The second dataset, GoogleEarth, focuses on enhancing the visual appearance of cities as it comprises images captured using Google Earth Studio and includes multiview consistency, allowing for a more comprehensive and realistic representation of urban environments. 

3. Quantitative and Qualitative Evaluation: CityDreamer’s performance has been evaluated through both quantitative and qualitative assessments. It has been compared against existing state-of-the-art 3D generative models to showcase its capabilities in generating large-scale and diverse 3D cities. 

Check out the Paper and Project. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 30k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..