Peter Sobot's Blog - Using Models to Create Models of New York City

Peter Sobot's Blog

(The above is a video, but click the button to explore this scene in your browser.)

I've been a photographer for most of my life. My personal photo library contains, at time of writing, 134,000 photos going back more than 30 years. I like capturing the moment.

So when I left my job last month - after nearly a decade - and realized I would no longer spend every day on the top floors of 4 World Trade Center with its absolutely incredible view, I decided to take some photos out of the windows.

Around that same time, the wonderful Vincent Woo published his absolutely stunning 3D recreation of Grace Cathedral in San Francisco, using a 3D modelling technique called Gaussian Splatting . I was captivated. As I said my goodbyes to my colleagues, I started taking more photos out the windows. High-quality photos, with my best lenses, with my mirrorless camera, in a near-360º circle around the perimeter of one of the most highly secured office building complexes in America.

For outdoor photogrammetry like Vincent's work, the standard practice is to use a drone-mounted camera to take thousands of high-quality, extremely sharp photos in broad daylight. This office being, well, part of the World Trade Center, a drone was out of the question. I value my freedom.

So, I took more than 1,000 photos in all, by hand, with my trusty Sony a7c's 20mm lens carefully pointing out against the glass, taking care to avoid reflections, and only ever in common areas of the office, well away from desks and people. Time was short, and while I wasn't doing anything illicit or even off-limits, I did want to avoid having to explain myself.

A contact sheet of some of the raw source photos: the same 20mm view out the 71st-floor windows, shot at different times of day, in different weather, and from slightly different spots along the glass. A few frames show the camera's reflection, internal lights, and haze from the glass.

There were a number of challenges, mostly to do with how photogrammetry and Gaussian Splats work best if you have consistent lighting conditions. My shoot times would vary as I was shooting in between meetings. While I could have visited multiple floors to get additional data points, that would have to happen after hours so as to avoid disturbing my colleagues, and this would create datasets with differing lighting conditions. (And on the one convenient weekend day that my badge still worked, it rained.)

And most frustratingly: 3D reconstruction techniques really like clean data. Having images with reflections in them - especially consistent reflections like the reflection of a camera's lens up against triple-pane commercial building glass - wrecks havoc on the algorithm's ability to reconstruct things.

But... I had the data. Now I just needed to process it.

...but wait, what's a Splat?

Right, I should start by explaining what a Gaussian Splat is in the first place.

Traditionally, computer graphics "does 3D" by defining points in 3D space and connecting those points together. These form triangles or quads, and allow 3D software to build up a scene much like a drywaller would install sheets of drywall: one face at a time.

In 2026, the state-of-the-art for capturing realistic world data isn't based on this technique: instead, it trains - yes, like with machine learning - a 3D model of a space, by iteratively creating random "splats" of data in a scene and then trying to fit those splats to the training data (images) provided. Splats can also have spherical harmonics attached, which is really just a fancy way to say "if you look at a splat from different directions, you might see different colours."

A small building model, rendered first as traditional triangles, and then with a few hundred Gaussian splats on the right, drawn with their outlines visible.

This doesn't look better in this example, but when you scale this method up to a huge scene, you can get amazing results:

A great scene of Toronto's Casa Loma, captured from drone footage by Justin Eastman.

There's a step in between capture and training that's also quite important: Structure from Motion, or SfM, which orients these images in 3D space relative to each other to create a vague sense of structure of the scene. This step can also be tricky to get right, and can be a source of persistent problems; but if your SfM data is correct, then it's just a preprocessing step that unblocks training. SfM algorithms include COLMAP, its sibling, GLOMAP, and the closely related hloc.

Much like training an AI model, training a Gaussian Splat can be a bit of an art. There are hundreds of different hyperparameters that can be tuned, multiple different popular software packages to use for training, and multiple algorithms that trade off speed for quality in different ways. I've trained many, many, many AI models in my decade as a machine learning engineer at Spotify, and I'm no...