Zero-copy wgpu rendering inside an Electron app - Murlet

Murlet, a macOS network video recorder (NVR), displays<br>wgpu-rendered video frames straight into Electron without<br>them leaving GPU memory. This contributes to a great local user experience: when<br>a user "scrubs" within a recording, the displayed video is quickly rendered to<br>the screen, giving the sense the output is "locked" to the cursor position.<br>Nice, but how is this implemented? This blog post talks about two different<br>approaches: "hole punching" and Electron's sharedTexture API, and explains why I<br>chose hole punching.

Overview

First, a brief overview of the architecture: Murlet is decomposed into an<br>Electron frontend and a Rust backend. The frontend is a thin React layer that is<br>solely responsible for rendering the UI chrome and calling into the backend. The<br>backend is where the business logic lives: it renders video (using wgpu),<br>records RTSP streams, runs object detection and owns the data model. It is<br>compiled into a shared library that runs within the address space of the<br>Electron main process, and is exposed using NAPI-RS.

("Why Electron?" you might ask. Yep, good question. Basically, I know how to<br>quickly ship with it and I think it's "good enough" for most people. That said,<br>I'd love to explore a native implementation, and this would actually be pretty<br>easy given the current architecture. More on that later.)

Back to business. Let's start by surveying all of our options for displaying<br>video frames into Electron, including those that aren't zero-copy:

Loopback restreaming : serve local RTSP / WebRTC streams from the backend,<br>decode within Electron; requires H.264 or H.265 encode/decode round-trip.<br>Great for uniform local/remote viewing story, terrible for latency.

GPU readback : render to bitmaps on the backend, copy to the frontend,<br>re-render within Electron; adds some latency, burns CPU and GPU.

Hole punching : render to a native NSView peeking through a transparent<br>hole in Electron's contentView. Zero-copy, platform-specific, fiddly. This is<br>what Murlet uses.

Electron's<br>sharedTexture<br>API : this imports shared textures into Electron as VideoFrames. Zero-copy,<br>platform-agnostic, experimental. Murlet does not use this, although I did<br>prototype it. It provides a useful comparison to the hole punching approach.

We're going to skip discussion of the first two options (loopback restreaming<br>and GPU readback), and jump right to the zero-copy options: hole punching and<br>Electron's sharedTexture API.

Hole punching

The hole punching approach stacks a native NSView (the "underlay") beneath<br>Electron's contentView, onto which the backend directly renders video. We then<br>render a transparent div within our Electron UI so that the underlay's video can<br>peek through. (Because the backend lives inside the Electron main process, it<br>can reach Electron's NSWindow and add the underlay.)

Aside: the hole punching technique has a lot of prior art. DirectDraw overlays<br>on Windows and the XVideo extension on X11 both used color keying, where the app<br>painted a specific color where it wanted video displayed. Android's<br>SurfaceView<br>docs say that its dedicated drawing surface "is behind the window holding its<br>SurfaceView; the SurfaceView punches a hole in its window to allow its surface<br>to be displayed." Browsers themselves try to render video using hardware overlay<br>planes, leaving a transparent hole in the main content plane for the video to<br>poke through. You get the idea.

The benefit of this approach is that there's very little coupling between<br>Electron and the underlay; we just have to arrange for the frontend to tell the<br>backend where the transparent div is located. This means that we don't have the<br>performance impact of involving Electron's rendering pipeline for every frame.<br>We also preserve optionality for any future native port. The flip side of the<br>low coupling is that we're responsible for efficiently synchronizing both the<br>location and the content of the transparent div with the backend. This is... a<br>little nuanced.

Let's start with location synchronization. When does the location change?<br>Either during window resizes, or else when the user moves the separator that<br>sits between the video thumbnails and the video player on the search view. If<br>the backend responds too slowly, then we lose the illusion that the UI is a<br>"single pane of glass".

We tackle the problem of location synchronization by simulating the frontend's<br>layout algorithms (CSS grid and flexbox) on the backend, which allows us to<br>minimize communication. Simulation sounds complicated but it's actually<br>straightforward because we have a very simple UI: the live view just has a<br>navbar at the top, and otherwise the video fills the rest of the window. The<br>search view adds a (resizable) panel on the left containing video thumbnails,<br>and again the video fills the rest of the window.

So the div's position is fully determined by a handful of edge distances. For<br>each edge of the transparent div,...