Core Concepts
PyTorchGeoNodes, a differentiable module for reconstructing 3D objects from images using interpretable shape programs, enables gradient-based optimization and joint optimization of discrete and continuous parameters of shape programs to efficiently reconstruct 3D objects.
Abstract
The paper introduces PyTorchGeoNodes, a framework that enables differentiable rendering of shape programs designed in Blender. Shape programs are a form of procedural modeling that can represent a variety of 3D shapes from an object category using a computational graph of parameterized geometric operations.
The key contributions are:
- A "compiler" that generates efficient PyTorch code for shape programs, allowing gradient-based optimization of the continuous parameters.
- An optimization method that combines Monte Carlo Tree Search (MCTS) and gradient descent to jointly optimize the discrete and continuous parameters of shape programs for 3D object reconstruction from images.
The authors evaluate their approach on synthetic and real-world scenes from the ScanNet dataset. Their experiments show that the reconstructed shapes match the input scenes well while enabling semantic reasoning about the reconstructed objects, outperforming traditional CAD model retrieval methods.
The differentiable rendering of shape programs enabled by PyTorchGeoNodes opens up exciting research directions, such as self-supervised learning of shape programs from images.
Stats
The average number of vertices and faces in the reconstructed meshes is 64 and 100, respectively, compared to 2760 and 10918 in the ground truth CAD models.
On synthetic data, the mean absolute difference between the reconstructed and ground truth continuous parameters ranges from 0.73 cm for dividing board thickness to 5.65 cm for width.
On the ScanNet dataset, the classification accuracy for discrete shape parameters ranges from 53.84% for sofa legs to 98.46% for the presence of an L-shaped extension.
Quotes
"PyTorchGeoNodes is a framework that enables differentiable rendering of shape programs designed in Blender."
"Our second contribution is to show how to adapt recent methods based on Monte Carlo Tree Search to our problem of jointly optimizing the continuous and the discrete parameters of a shape program."