NYU and Intel Researchers Introduce Image Sculpting: A New Artificial Intelligence Framework for Editing 2D Images by Incorporating Tools from 3D Geometry and Graphics

The existing 2D image editing methods face a substantial amount of limitations as they heavily rely on textual instructions, leading to ambiguity and restricted control. This confined nature of these methods within 2D spaces hinders the direct manipulation of object geometry, resulting in imprecise outcomes. The lack of tools for spatial interaction also limits the creative possibilities and fine-tuned adjustments that can be made, leaving a gap in image editing capabilities.

The research includes exploration into generative models like GANs, which have broadened the scope of image editing to encompass style transfer, image-to-image translation, latent manipulation, and text-based manipulation. However, text-based editing has limitations in precisely controlling object shapes and positions. ControlNet is one of the models that address this by incorporating additional conditional inputs for controllable generation. Single-view 3D reconstruction, a longstanding problem in computer vision, has seen advancements in algorithmic approaches and training data utilization.

The Image Sculpting method, developed by researchers at New York University, addresses these limitations in 2D image editing by integrating 3D geometry and graphics tools. This approach allows direct interaction with the 3D aspects of 2D objects, enabling precise editing such as pose adjustments, rotation, translation, 3D composition, carving, and serial addition. 

Using a coarse-to-fine enhancement process, the framework re-renders edited objects into 2D and seamlessly merges them into the original image, achieving high-fidelity results. This innovation harmonizes the creative freedom of generative models with the precision of graphics pipelines, significantly closing the controllability gap in image generation and computer graphics.

While Image Sculpting presents promising capabilities, it faces limitations in controllability and precision through textual prompts. Requests regarding detailed object manipulation remain challenging for current generative models. The method relies on the evolving quality of single-view 3D reconstruction, and manual efforts may be required for mesh deformation. Output resolution falls short of industrial rendering standards, and addressing background lighting adjustments is crucial for realism. Despite its innovative approach, Image Sculpting represents an initial step, and further research is essential to overcome these limitations and enhance its overall capabilities.

To summarize, the key highlights of this research include:

• The proposed method of Image Sculpting integrates 3D geometry and graphics tools for 2D image editing.
• It directly interacts with 3D aspects, enabling precise edits like pose adjustments and rotations.
• Further re-renders edited objects into 2D, seamlessly merging for high-fidelity results.
• Attempts to balance creative freedom of generative models with graphics precision.
• Faces certain limitations in detailed object manipulation, resolution, and lighting adjustments, creating the need for further research and improvement.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..