Asari AI

Speeding up end-to-end inference with self-improving agents

Early results on how our co-inventor agents improve the full AI inference<br>stack using a rigorous self-improving process.

July 28, 2026

By Asari AI

Introduction

We build co-inventors: self-improving agents that design and optimize high-performance infrastructure. Applied to AI inference, they optimized the entire inference stack for two of the largest open-source LLMs: DeepSeek v4 Pro and GLM 5.2 running on NVIDIA B200s with vLLM v0.23. They improved both throughput and interactivity by up to 16% across multiple max-concurrency levels, while preserving model behavior through distribution-level correctness checks. Each concurrency level took the agents about a day to optimize.

As of vLLM v0.26, most of the changes made by our agents have no upstream equivalent. The changes span the entire inference stack, including kernels, schedulers, load-balancers, and configuration settings.

Loading chart...

Throughput and interactivity for an agent-optimized inference stack (blue) and vLLM v0.23 (grey) running on 8 B200s.1 The agents achieved consistent improvements across the board.<br>1 Absolute throughput and interactivity metric values vary with the client-server configuration and the resulting request patterns and batch sizes. At higher concurrency levels, the measured baseline systematically depends on the choice between InferenceX or vLLM bench's default measurement process. Every comparison here is measured against its own matched baseline, and the gains are statistically significant at each level.

A single fast kernel doesn't make a fast system (Amdahl's law), and it doesn't necessarily make a correct one. That's why we optimize the whole stack at once rather than individual components.

Speed only counts if correct model behavior is preserved, and that can be subtle to check. Customizations can quietly alter model outputs, yet still give the same accuracy on standard benchmarks like GSM8K or GPQA. We use stringent distribution-matching checks rather than functional tests or benchmark scores.

Our agents learn over time. Every experiment produces reusable insights that the agents use to improve their invention process across models and workloads. For example, our agents learned to avoid distributed deadlocks that would have cost 44 minutes in unnecessary wait time, generalizing a lesson learned from optimizing DeepSeek v4 Pro applied to GLM 5.2.

What makes optimizing inference systems hard?

Agents must consider millions of lines of code containing tunable parameters and algorithm implementations, plus countless server configuration settings, making the search space enormous. Small changes in the right places can produce large gains, but the hard part is finding them and knowing what to prioritize.

Modern inference systems depend on many interacting factors:

Software: OS, kernels (e.g., PTX, CuTeDSL, CUDA C, and Triton), high-level scripting (e.g., PyTorch), Python glue code, runtime, scheduling, caching, load balancing, interconnect, client-server configurations.

Hardware: GPU, CPU, memory, disk, intra- and inter-node connectivity.

Runtime dynamics: input distributions, caching behavior, memory pressure, throttling, network communication.

Because these systems are highly dynamic, optimizing them requires continuous evaluation and profiling; static code analysis is not enough. But end-to-end evaluations are slow and computationally expensive: a single one can take hours, spanning compilation, server launch, warm-up, and request processing.

As the agents continually iterate on evaluation results, our evaluation system uses a hierarchical decomposition of end-to-end and micro benchmarks to keep feedback loops short.

The invention loop

Our co-inventors run a rigorous discover + build → evaluate → verify → self-improve loop using many agents in parallel. They explore and implement optimization strategies, evaluate whether each speedup is real, verify that it preserves the model's outputs, and distill what they learn for the next run.

Loading diagram...

Discover and build

Our agents propose optimization strategies using dynamic profiling, static analysis, and learned experiences via a self-improvement mechanism. They prioritize optimization targets by weighing potential speedups against the impact on numerical accuracy. This narrows the search space and helps balance exploration versus exploitation.

Evaluate

Our agents get feedback from evaluations in a sandbox that replicates the real production serving environment. There, they test customized vLLM serving stacks, dispatching to their own optimized kernels and code in place of the stock implementations.

For models with hundreds of billions to trillions of parameters running on complex workloads, measurement is slow and inherently noisy. Shifts in batching behavior and hardware effects like GPU power throttling introduce variance in performance evaluations.

An agent optimizing...