CC3D: Layout-Conditioned Generation of Compositional 3D Scenes

September 28, 2023

Welcome back to the Talking Papers Podcast! In our latest episode, we had the privilege of hosting the brilliant Jeong Joon Park to delve into his groundbreaking paper, “CC3D: Layout-Conditioned Generation of Compositional 3D Scenes,” freshly published in ICCV 2023.

In this work, they introduce CC3D, a revolutionary conditional generative model that aims to redefine the very boundaries of 3D scene synthesis. Unlike conventional 3D GANs, which often limit themselves to single objects, CC3D takes a bold leap forward by focusing on crafting intricate scenes filled with multiple objects. The heart of this innovation lies in its unique utilization of 2D semantic scene layouts to guide the creation of 3D scenes. Additionally, CC3D introduces a novel 3D field representation enriched with geometric inductive bias, resulting in not only remarkable efficiency but also a significant enhancement in the quality of the generated scenes. It represents a giant stride toward more controlled and superior 3D scene generation, and we are eager to unveil the insights from this exceptional work in our upcoming episode.

Myjourney with JJ doesn’t begin here. I first encountered his remarkable contributions when I stumbled upon his influential SDF paper at CVPR 2019, a moment that left an indelible mark on my understanding of the field. It wasn’t until CVPR 2022 that I had the pleasure of meeting JJ in person, all thanks to our mutual connection, Despoina, who was also a guest on our podcast.

Jeong Joon Park, who recently embarked on his role as Assistant Professor at the University of Michigan CSE, brings a wealth of knowledge and experience to our conversation. After completing his PhD at the University of Washington and post-doctoral work at Stanford, his research is now focused on pushing the boundaries of 3D content generation. His primary emphasis lies in the creation of large-scale, dynamic, and interactive 3D scenes through a fusion of physical and neural representations. Moreover, Jeong Joon actively encourages students to collaborate and contribute to his pioneering research, making him a valuable resource for those looking to explore the frontiers of computer vision, graphics, and AI.

We couldn’t be more excited to feature his exceptional work and insights in our upcoming episode. Stay tuned for a deep dive into the future of 3D scene synthesis with Jeong Joon Park! 🚀🔍 #TalkingPapersPodcast #3DSceneSynthesis #AI #ComputerVision

AUTHORS

Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Xingguang Yan, Gordon Wetzstein, Leonidas Guibas, Andrea Tagliasacchi

ABSTRACT

In this work, we introduce CC3D, a conditional generative model that synthesizes complex 3D scenes conditioned on 2D semantic scene layouts, trained using single-view images. Different from most existing 3D GANs that limit their applicability to aligned single objects, we focus on generating complex scenes with multiple objects, by modeling the compositional nature of 3D scenes. By devising a 2D layout-based approach for 3D synthesis and implementing a new 3D field representation with a stronger geometric inductive bias, we have created a 3D GAN that is both efficient and of high quality, while allowing for a more controllable generation process. Our evaluations on synthetic 3D-FRONT and real-world KITTI-360 datasets demonstrate that our model generates scenes of improved visual and geometric quality in comparison to previous works.

📚StyleGAN