Core Concepts
A simple and efficient method to transform a non-differentiable rasterizer into a differentiable one, enabling optimization of 3D assets within an existing rendering engine.
Abstract
The paper presents a method to transform a non-differentiable rasterizer into a differentiable one, allowing for the optimization of 3D assets within an existing rendering engine. The key insights are:
Using Stochastic Gradient Estimation to estimate gradients without a differentiable framework. This involves randomly perturbing the scene parameters and computing the gradient based on the difference in the rasterized images.
Estimating gradients on a per-pixel basis rather than the full image. This bounds the dimensionality of the optimization problem and makes the method scalable to scenes with millions of parameters.
The authors implement their method by adding two compute shaders to an existing rasterization engine. The first shader perturbs the scene parameters, the second computes the gradients based on the perturbed and original images. This approach keeps the workflow self-contained, cross-platform, and efficient, while supporting a wide range of primitives such as meshes, textures, volumes, subdivision surfaces, and physically-based materials.
The paper provides a detailed validation of the per-pixel formulation, showing that it significantly outperforms the full-image approach in high-dimensional optimization problems. It also includes a qualitative comparison to the state-of-the-art differentiable rasterizer nvDiffRast, demonstrating that their method can produce similar results while being simpler to implement and integrate into an existing engine.
Stats
The paper does not provide specific numerical data to support the key claims. However, it includes several figures that show the optimization progress and timings for different types of 3D assets.
Quotes
"Our method is simple to implement. Our base differential rasterization component consists of adding ID/UV-buffers to the existing raster targets and two compute shaders."
"Our method keeps the workflow self-contained by bringing the benefits of differentiable rasterization to an existing conventional rasterizer without requiring external dependencies."
"Our method is cross-platform since it uses only conventional graphics API functionalities. This is a significant bonus point for adoption given that existing differential rendering solutions are bound to vendor-specific hardware and/or software."