Neural RGB-D Surface Reconstruction

In this episode of the Talking Papers Podcast, I hosted Dejan Azinović to chat about his paper “Neural RGB-D Surface Reconstruction”, published in CVPR 2022.

In this paper, they take on the task of RGBD surface reconstruction by using novel view synthesis. They incorporate depth measurements into the radiance field formulation by learning a neural network that stores a truncated signed distance field. This formulation is particularly useful in regions where depth is missing and the color information can help fill in the gaps.

I first met Dejan during my research visit to TUM during my PhD. We spent some long evenings at the office but also managed to play a little bit of volleyball (until someone nearly broke a finger). I am looking forward to catching up in person at CVPR 2022. It was a pleasure hosting him on the podcast.

AUTHORS

Dejan Azinović Ricardo Martin-Brualla Dan B Goldman Matthias Nießner Justus Thies

 

ABSTRACT

In this work, we explore how to leverage the success of implicit novel view synthesis methods for surface reconstruction. Methods which learn a neural radiance field have shown amazing image synthesis results, but the underlying geometry representation is only a coarse approximation of the real geometry. We demonstrate how depth measurements can be incorporated into the radiance field formulation to produce more detailed and complete reconstruction results than using methods based on either color or depth data alone. In contrast to a density field as the underlying geometry representation, we propose to learn a deep neural network which stores a truncated signed distance field. Using this representation, we show that one can still leverage differentiable volume rendering to estimate color values of the observed images during training to compute a reconstruction loss. This is beneficial for learning the signed distance field in regions with missing depth measurements. Furthermore, we correct for misalignment errors of the camera, improving the overall reconstruction quality. In several experiments, we show-cast our method and compare to existing works on classical RGB-D fusion and learned representations.

 

📚NeRF

📚Bundle Fusion

LINKS AND RESOURCES

💻Project website

💻Code

📚 Paper

🎥Video

To stay up to date with Dejan’s latest research, follow him on:

👨🏻‍🎓Dejan’s homepage

🎓Google Scholar

🐦Twitter

👨🏻‍🎓LinkedIn

Recorded on April 4th 2022.


CONTACT

If you would like to be a guest, sponsor or just share your thoughts, feel free to reach out via email: talking.papers.podcast@gmail.com

SUBSCRIBE AND FOLLOW

🎧Subscribe on your favourite podcast app: https://talking.papers.podcast.itzikbs.com

📧Subscribe to our mailing list: http://eepurl.com/hRznqb

🐦Follow us on Twitter: https://twitter.com/talking_papers

🎥YouTube Channel: https://bit.ly/3eQOgwP