3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation

I am thrilled to share the latest episode of the Talking Papers Podcast where I had the pleasure of hosting Dale Decatur, a brilliant 3rd year PhD student from the University of Chicago’s 3DL lab. Together, we delved into the intricate details of his paper titled “3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation,” recently published in CVPR 2024. The paper introduces an innovative technique, 3D Paintbrush, which allows for the automatic texturing of local semantic regions on meshes through text descriptions.

One of the key contributions of Dale’s paper is the development of Cascaded Score Distillation (CSD), a technique that employs multiple stages of a cascaded diffusion model to enhance the details and resolution of textured areas on 3D shapes. The approach of simultaneously generating localization maps and conforming texture maps not only refines the quality of editing but also streamlines the integration of textured areas into standard graphics pipelines. This advancement marks a significant stride in simplifying the process of editing 3D assets, particularly through text prompts, thus making it more accessible to a broader audience.

As I delved deeper into the paper, I was truly impressed by the clever utilization of generative priors learned from images at different resolutions to supervise the local editing technique. This not only showcases the depth of research conducted by Dale but also highlights the potential impact of his work on the broader research community. It is fascinating to witness the evolution of techniques that leverage the synergy between machine learning and 3D modeling, ultimately pushing the boundaries of what is achievable in the realm of computer graphics.

Reflecting on my interactions with Dale, I am reminded of our initial encounter at CVPR 2023 when he presented his 3D highlighter paper. Our paths crossed again at CVPR 2024, where I had the opportunity to chat with Dale about 3D Paintbrush. It was a moment of realization that this was a conversation that needed to be shared with our listeners. I am eagerly looking forward to witnessing Dale’s future research endeavors and the innovative contributions he will undoubtedly make to the field of computer graphics and deep learning.

AUTHORS


Dale Decatur, Itai Lang, Kfir Aberman, Rana Hanocka

ABSTRACT


In this work we develop 3D Paintbrush, a technique for automatically texturing local semantic regions on meshes via text descriptions. Our method is designed to operate directly on meshes, producing texture maps which seamlessly integrate into standard graphics pipelines. We opt to simultaneously produce a localization map (to specify the edit region) and a texture map which conforms to it. This synergistic approach improves the quality of both the localization and the stylization. To enhance the details and resolution of the textured area, we leverage multiple stages of a cascaded diffusion model to supervise our local editing technique with generative priors learned from images at different resolutions. Our technique, referred to as Cascaded Score Distillation (CSD), simultaneously distills scores at multiple resolutions in a cascaded fashion, enabling control over both the granularity and global understanding of the supervision. We demonstrate the effectiveness of 3D Paintbrush to locally texture a variety of shapes within different semantic regions.

RELATED WORKS

📚3D Highlghter

📚DreamFusion

📚Score Jacobian Chaining

LINKS AND RESOURCES

📚Preprint

📚Paper

💻Project page

💻Code

To stay up to date with his latest research, follow on:

👨🏻‍🎓Personal website

👨🏻‍🎓Google scholar

🐦Twitter

👨🏻‍🎓LinkedIn

This episode was recorded on July 10th 2024

CONTACT


If you would like to be a guest, sponsor or share your thoughts, feel free to reach out via email: talking.papers.podcast@gmail.com

SUBSCRIBE AND FOLLOW


🎧Subscribe on your favourite podcast app

📧Subscribe to our mailing list

🐦Follow us on Twitter

🎥Subscribe to our YouTube channel