Better Gaussian Splatting in Julia<br>Better Gaussian Splatting in Julia<br>use WASD + Mouse or tap on the screen to walk around :)

Amulet of St. Michael in Kyiv, Ukraine

(reconstructed with GaussianSplatting.jl, ~6M Gaussians)

GaussianSplatting.jl 2.0<br>release brings notable quality of life improvements and new capabilities.

Same code, multiple GPU backends

Written entirely in Julia it supports following GPU backends:

AMD GPU (AMDGPU.jl)<br>NVIDIA GPU (CUDA.jl)<br>MacBook GPU(Metal.jl)

This is achieved with<br>KernelAbstractions.jl<br>that allows writing a single kernel that get's compiled to a specific target.

Multithreaded UI

To avoid freezing the app during heavy work<br>(such as JIT compilation of GPU kernels, dataset loading, etc.)<br>the app is now split into two threads:

Frontend: handles the UI, performs OpenGL rendering, dispatches commands to the backend.<br>Backend: performs Gaussian Splatting rendering, training, dataset loading, etc.

In this way, the UI always stays responsive and user can interact with it, even if there are long-standing jobs in the background.<br>It also shows progress bars that something is happening with tips, instead of becoming frozen like before.

The spinner keeps animating while the dataset loads on the backend thread and when JIT-compiling kernels for the first training iteration.<br>UI/UX itself got a big update and is now displays loss plots live during training along with all hyperparameters.

Loss plots and hyperparameters update live while training runs. Markov Chain Monte Carlo Strategy

Besides default cloning and splitting densification strategy,<br>we now support MCMC<br>(3D Gaussian Splatting as Markov Chain Monte Carlo) densification strategy.

It allows precise control of the number of Gaussians in the scene<br>and generally relies less on having a good initialization.

Users can select MCMC during dataset loading or in the code with:

Trainer(<br>rasterizer, gaussians, dataset, opt_params;<br>strategy=MCMCStrategy(; kwargs...),<br>) Depth & Geometry Supervision

To improve reconstructed geometry, we can provide depth priors using off-the-shelf depth estimation models.<br>Depth images should be part of the dataset, under /depths/.png path.

A dataset image and its estimated depth prior using Depth-Anything 3.<br>To enable depth supervision, either toggle it in UI or with OptimizationParams(; use_depth_loss=true).<br>Depth maps then provide supervision during training resulting in better geometry and reduced number of floaters.

Since depth may vary between frames, all depth maps are first refitted against prior point cloud to the same scale.<br>The ones that fail refitting are discarded and not used for supervision.

To further improve geometry and smoothness, we can perform geometry regularization to constrain the shape of the surface,<br>while depth supervision constrains only its location.<br>It can be enabled in the UI or with OptimizationParams(; use_normal_loss=true) and performs two things:

depth normal consistency pins surface orientation: normals derived from the rendered depth map must align with the per-Gaussian normals.<br>flattening surface: flattening each Gaussian along its smalles axis.

A rendered image, depth and normals of the model with geometry regularization enabled. Sky Dome

To help disentangle sky / distant background and reduce floaters, GaussianSplatting.jl now supports Sky Dome,<br>which is a frozen shell of Gaussians at a large radius, rendered in its own pass and composited behind the scene.

Sky Dome in a form of a sphere and a hemisphere.<br>Depending on your environment (e.g. like this fountain above), you may want to prefer hemisphere instead of a sphere,<br>because a full sphere will pull parts of the geometry onto itself, making the ground less opaque.<br>By using hemisphere, the bottom half background is black and does not affect geometry at all.

Comparison of the reconstruction without and with Sky Dome after 3K iterations.<br>With Sky Dome, sky is clearly disentangled from the fountain and does not float around.

The overhead of Sky Dome is negligible (~32K Gaussians) comparing to the rest of the scene (millions of Gaussians).

To further help disentangle the sky, we can use sky segmentation masks obtained from any off-the-shelf sky-segmentation models,<br>which should be part of the dataset under /sky/.png path.

A dataset image and its estimated sky mask.<br>This helps with small details around the edges of geometry, like leaves.

Camera Frustum

Camera frustum visualization got a small quality-of-life update and now shows a miniature picture of<br>the image that the actual camera took.

Camera frustum visualization. Some Other Niceties

List of other improvements in no particular order:

Automatic checkpointing saves checkpoints every N steps in the selected directory.

Custom hyperparameter configuration (learning rate, loss weights, regularization, etc.)<br>can be provided during dataset loading with hyperparameters.toml file<br>(use Load..., Save... buttons).<br>It can also be saved later on if you are...