Meshtryoshka: Differentiable Mesh Rendering for Unbounded Scenes
TL;DR
Meshtryoshka does scene-scale differentiable rendering purely via mesh rasterization, without first starting from volume rendering, triangle soup, etc.
Videos
Optimization Progress
Abstract
Differentiable rendering has emerged as a powerful approach for 3D<br>reconstruction and novel view synthesis. Today, state-of-the-art<br>differentiable rendering methods combine a variety of custom<br>representations of 3D geometry and appearance with specialized<br>renderers. However, most downstream tasks in computer graphics rely<br>on 3D meshes, which provide superior portability, allow for<br>hardware-accelerated rendering, and are at the core of most computer<br>graphics workflows. While prior work has attempted differentiable<br>rendering with mesh representations, these approaches are limited to<br>object-centric scenes and fail to reconstruct large-scale, unbounded<br>scenes.
In this work, we introduce Meshtryoshka, a novel mesh<br>differentiable rendering framework that combines an off-the-shelf<br>triangle rasterizer with a 3D representation that consists of nested<br>mesh shells which resemble a matryoshka doll. In every forward pass,<br>the mesh shells are extracted anew from a 3D signed distance function<br>via iso-surface extraction, and the opacities for each vertex are<br>computed as a function of signed distance. Each mesh shell is then<br>rasterized independently, and the final image is created via alpha<br>compositing. Crucially, mesh vertex positions are updated only<br>indirectly via gradients that flow through the opacity values into<br>the signed distance function, and hence, our method is compatible<br>with off-the-shelf mesh renderers that need not be differentiable<br>with respect to vertex positions.
On object-centric scenes, our method performs competitively with<br>surface-based differentiable rendering techniques. To the best of our<br>knowledge, our method is the first differentiable mesh rendering<br>method that scales to unbounded, real-world 3D scenes, where it<br>yields high-quality novel view synthesis results approaching those of<br>state-of-the-art, non-mesh methods. Our method suggests that it may be<br>possible to solve the differentiable rendering problem without relying<br>on specialized renderers, only using conventional tools from the<br>computer graphics toolbox.
Overview
We transform a standard non-differentiable triangle<br>rasterizer into an effective triangle-based differentiable renderer<br>via several key ideas. First, we parameterize scenes via cubic grids<br>of signed distances and view-dependent colors. We apply differentiable<br>marching cubes on multiple level sets of the signed distance field, to<br>extract a set of nested mesh shells—a Meshtryoshka—from<br>a scene. Each shell is a mesh whose vertices contain color features<br>interpolated from the grid representation.
We render these shells using a two-step pipeline of rasterization and<br>deferred shading, which allows us to utilize an off-the-shelf<br>rasterizer. A rasterizer computes for each pixel the triangle ID and<br>barycentric coordinates of the first ray-triangle intersection. While<br>this operation is non-differentiable, it can be combined with<br>differentiable image-space interpolation to convert<br>per-triangle-vertex values into differentiable per-pixel values. We<br>perform this process on each mesh shell independently, and then combine<br>the results via alpha compositing, using the signed distance values to<br>compute per-shell opacities. As explained in our paper, this<br>formulation is a close approximation of rendering the shells using ray<br>tracing, allowing for gradient-based optimization.
To scale to real-world scenes, we periodically compute an occupancy<br>mask over the cubic grid, allowing us to only store parameters in<br>occupied parts of space. We implement our differentiable marching<br>cubes to work on a sparse grid. We optimize coarse to fine, starting<br>from a dense grid at low resolution, and periodically subdividing. We<br>divide real-world scenes into foreground and background. The foreground<br>is represented as a normal 3D grid, while the background is represented<br>as a set of six truncated frustums resembling a tesseract in 3D. Each<br>frustum composes of frustum-shaped voxels which grow larger in<br>proportion to the distance from the scene center, ensuring that farther<br>regions are represented more sparsely.
Image Comparisons
Drag the slider to compare. Left: 3D Gaussian Splatting · Right: Meshtryoshka (Ours)