Zero-copy TLS ingress with kTLS and splice(2) for sandboxes — Tensorlake

HomeBlogPricingCareersDocsGitHubSlack communityTalk to FounderDashboard →

◆ ON THIS PAGE 8 MIN<br>Tensorlake Sandboxes are built from the ground up for performance and scale. Our users run them as the environment where agents do their work. They make thousands of API calls into a single sandbox to start, stop, and observe processes; uploading files for agents to operate on; downloading the assets those agents produce.

The infrastructure that authenticates and routes every connection into a sandbox has to get out of the way. Infrastructure overhead should be indistinguishable from talking to the process directly.

Ingress should disappear

We've been rebuilding how traffic enters sandboxes, and the core change is moving the in-dataplane hop from an L7 reverse proxy to an L4 byte-forwarding data path .

On a single large transfer through one host, that change took us from 1.12 GB/s to 2.50 GB/s and cut the proxy's cost from ~0.90 to ~0.49 CPU-seconds per GB . That's 2.2x the throughput for a little over half the CPU per byte moved. The new path does its cryptography in the kernel and stops interpreting the stream.

◆THE CORE IDEATerminate TLS once at the edge for authentication, then move the tenant's bytes through the kernel. Decrypt on the way in, encrypt on the way out, never staged in a userspace buffer, never parsed. The proxy stops parsing HTTP requests, and stops deciding how to respond when the upstream client or the applications running in sandboxes misbehave.

How sandbox networking works today

Traffic ingress starts at an L7 gateway we built on top of Cloudflare's Pingora. It terminates TLS and accepts HTTP/S, WebSocket, and gRPC from users. The gateway extracts the sandbox ID from the request URL (SSH uses a different extraction path off the connection), looks up which dataplane owns that sandbox and on what address, and issues a fresh, authenticated request to a second L7 reverse proxy running on that dataplane, over mutual TLS, speaking HTTP/2. It attaches context to the request headers so the dataplane proxy can verify the call, and the dataplane proxy makes one more request to the application inside the sandbox.

DATAPLANEL7 Ingress GatewayTLS terminate · route lookupL7 Proxyre-auth · re-requestSandboxRoute Tablesandbox → dataplane addrSandbox Schedulerplacement · lifecycleFig 1 — Sandbox networking today: two L7 hops between the user and the app.<br>There are a lot of moving parts, and it works surprisingly well. But as we scaled, and some customers now run thousands of concurrent sandboxes, we found ourselves tuning all the knobs you'd expect on a stack of L7 load balancers: progress and idle timeouts at every hop, connection-abort handling at each stage, graceful shutdown semantics, header-rewriting rules. Every L7 hop also interprets the stream, sniffing bodies, injecting headers, synthesizing status codes when something upstream misbehaves, and each of those behaviors is a place where things can go wrong and become confusing when users troubleshoot failures.

The other reason to change was structural.

Moving the in-dataplane hop to L4

We replaced the dataplane's L7 reverse proxy with an L4 forwarder that does exactly one thing: move bytes between the edge gateway and the application, without understanding them. We also pulled it out of the dataplane process entirely. It now runs as a standalone daemon on each host, with its own lifecycle.

The dataplane's reverse proxy and the process that orchestrates sandboxes were the same binary. But those two jobs change at completely different rates. The orchestrator controls sandbox orchestration, snapshot/restore machinery, and we update it continuously to optimize the dataplane. The data path is boring on purpose: once it correctly forwards bytes, it barely changes. Coupling them meant every routine orchestrator deploy stopped the proxy for a few seconds and interrupted live traffic, so a workspace an agent was actively driving would drop its connections because we shipped an unrelated scheduling change.

Splitting the forwarder into its own daemon aligns each component's lifecycle with how often it actually changes. The dataplane can be restarted, upgraded, or rescheduled as often as we like. The sandbox connections live in the TCP proxy, which keeps working through dataplane restarts. The proxy itself changes rarely, and when it does we can drain it deliberately instead of taking it down. Decoupling the data path from the control plane fixed the upgrade-interruption problem outright.

DATAPLANE HOSTL7 Ingress GatewayTLS terminate · authnL4 Forwarderstandalone daemonkTLS · splice(2)byte meterAppin sandbox · plaintextDataplaneidle timer · lifecycleSchedulersigned routemTLS +preamblebyte countsFig 2 — L4 ingress: one authenticated mTLS hop, then bytes splice straight through the kernel to the app.<br>Two things made this harder than "just forward the bytes":

The forwarder needs...