Accelerating Harbor with Tensorlake. — Tensorlake

HomeBlogPricingCareersDocsGitHubSlack communityTalk to FounderDashboard →

◆ ON THIS PAGE 6 MIN<br>Every Terminal-Bench task ships a reference solution called the oracle that's known to be correct. When you run harbor run --agent oracle, you're not testing a model — you're checking that the environment itself built and booted the way it should.

◆THE BARThe oracle pass rate has to be 1.0 . Since the solution is already correct, any task it fails is a broken environment, not a wrong answer — a score below 1.0 is measuring your harness, not the agent.

Harbor, the underlying framework, builds its environments from Docker images. Cloud sandbox infrastructure is generally built on microVMs or gVisor, so those Docker images get translated to VM or gVisor images and that translation often produces incorrect or buggy environments. If anything about the environment is off — a missing apt package, the wrong Python version, a COPY that didn't land, a service that never started — the oracle fails. And once you switch to a real agent run, you can't tell that kind of failure apart from a mistake the agent actually made.

In the last few weeks we've improved the infrastructure that converts Docker images to VM images. We now import an OCI image and faithfully convert it to an ext4 filesystem, preserving the filesystem metadata the OCI image carries. Runtime metadata — environment variables, users, working directory — is recorded too, so that when a sandbox boots it behaves exactly like a Docker container. In our tests we beat Modal and Daytona on oracle accuracy, and we complete the full suite fastest, thanks to improvements in our sandbox orchestration engine and a faster runtime.

01 · Results: Tensorlake vs. Modal vs. Daytona

“These are oracle runs: the solution is known-correct, so every non-pass is the harness failing, not the agent.<br>— Why oracle pass rate is the only honest baseline

Here's how the three providers compare on Terminal-Bench 2.1 (all 89 tasks). We ran every provider from the same Tensorlake sandbox with identical setup — same Harbor version, same client environment — so the only thing that changed between runs was the sandbox provider itself.

ProviderOracle pass rateWall-clock · 89 tasksTensorlake1.000 · all 89 tasks~37 min · 36m 44s latestModal~0.966 · 86 / 89~1 h · 1h 3mDaytona3 h+ · 3h 35m

Modal loses 3 of 89 environments (about 3%) and takes roughly an hour. Daytona drops more than 30%, well below anything usable, and takes over three hours. The exception breakdowns show where: Modal's losses surface as timeouts; on Daytona, 26 of the 30 failures are EnvironmentStartTimeoutError — the sandbox never even started.

Tensorlake — all 89 tasks pass (mean 1.000), zero exceptions, 36m 44s.<br>Modal — 86 of 89 pass (0.966), 1h 3m. The losses show up as timeouts on a known-correct solution.Daytona — 56 of 89 pass (0.629), 3h 35m. 30 fail on environment errors, 26 of them EnvironmentStartTimeoutError.

02 · Closing the gap: full Docker-image support

We closed that gap with a new TensorLakeEnvironment provider in Harbor. It resolves a task's environment through two high-fidelity paths:

01Prebuilt docker_image — preferred<br>When task.toml declares a docker_image, we import the registry layers into a microVM rootfs once and boot straight from it. The exact published image — no rebuild, no replay.

02Build from the task Dockerfile<br>With no prebuilt image, we build the Dockerfile into a sandbox image once, under a content-hashed name so identical Dockerfiles dedup across runs and hosts.

→FIDELITY ISN'T THE WHOLE STORYModal and Daytona run the real image too, and still didn't hit 1.0. The other half is reliability — bringing every environment up, fast and without timing out. When the transport hiccups mid-run, we retry and reap orphaned sandboxes so a flaky connection doesn't cost you a task.

And we've already done the work for you. We've imported, converted, and registered every Docker image for the full benchmark, so anyone with Tensorlake access can use them directly without building or importing anything. There's still a legacy replay path that boots from minimal and replays the build steps with compat shims, but it's only a fallback if a build fails — not part of the normal flow.

03 · Speed: boot in seconds, not minutes

Because we've already imported, converted, and registered every image, there's no per-run build, import, or pull step. Environments boot in seconds. That pays off most on the expensive tasks:

custom-memory-heap-crash compiles GCC's libstdc++ twice from source — minutes.

qemu-alpine-ssh, qemu-startup, and install-windows-3.11 pull large ISO/VM images and lean on heavy apt installs.

mteb-leaderboard, mteb-retrieve, and hf-model-inference need full Torch + transformers stacks.

crack-7z-hash and make-mips-interpreter build C from source and cross-toolchains.

With pre-published images, each of these turns from a multi-minute build into a...