An AI-agent cold-tuned our GLM-5.2 serving. Human engineering leveled it up for real production traffic.

Francesco’s Substack

SubscribeSign in

An AI-agent cold-tuned our GLM-5.2 serving. Human engineering leveled it up for real production traffic.

Francesco Parisio, trongvu, danghungdo, and Olalekan Taofeek<br>Jul 14, 2026

Share

TLDR

We deployed an AI autoresearch agent to tune GLM-5.2 serving on our NVIDIA B200 cluster. It doubled throughput in 26 runs. Real production-shape traffic replay broke it at 4-minute TTFTs. We fixed it through a series of manually tuned configuration setup and obtained 18x faster first tokens, 442 tok/s/GPU peak, and pushed the latency/throughput pareto optimal by more than 2.2x in latency and +35% in throughput. While the agent got us 80% there on a cold-start, engineering discipline and production traffic simulations closed the gap.

Introduction

The journey to bring GLM5.2 to our customers within the AI Foundation Service of T-Systems began a few weeks ago when we started experimenting with GLM5 on our brand-new NVIDIA B200 cluster built out in Munich. The excitement of testing brand-new hardware with an open-weights LLM totalling 744B parameters, hosted and served entirely on our own infrastructure, was palpable and, after a few iterations patching missing components on Day-1 of the release, we finally got “something” working. It was prompting, responding, responses seemed reasonable, we could see GPU usage, power spikes and all. Pretty exciting start! After a couple of measurements on our performance, ouch! We realized we still had a long way to go (expected, of course, but it still bites every time). But where to start, exactly?<br>Thanks for reading Francesco’s Substack! Subscribe for free to receive new posts and support my work.

Subscribe

Take for example the topology, i.e. how you decide to parallelize your model over the GPUs in your cluster: a 744B parameters model in NVFP4 (4-bit precision, roughly half-byte per parameter), takes around 372GB of GPU VRAM for weights alone, around 420GB once you include quantization scale factors. Every single B200 has around 183GB of HBM3e VRAM: two GPUs would give you 366GB, which is not enough for the 420GB of weights plus scales, so you need to split the model weights across no less than 3 GPUs. Considering you then need to leave room for the KV-cache pool, CUDA graphs and a bunch of other things, and considering the node is 8 GPUs and you don’t want to break something across 2 nodes (and you want to keep the power of 2 architecture of GPU clusters), then you are left in this case with splitting the model across 4 GPUs at minimum.<br>Also, say you are lucky enough to have for example 32 GPUs: now, you could have 8 replicas of 4 GPUs each introducing tensor parallelism equal 4, or you could have tensor parallelism equal 8 and 4 replicas, or tensor parallelism equal 4, data parallelism equal 2 and still 4 replicas. Then to this, you add inference engine: vLLM, SGLang or TensorRT-LLM? Should we use NVIDIA Dynamo or not? Should we deploy aggregated or disaggregated? While you see we have already piled up quite a few independent variables (say 4 topologies x 3 engines x 2 Dynamo on/off x 2 Agg/Disagg = 48 configurations to explore already...), we are just at the beginning: every engine has several configuration flags, more than 20, of which 10 at least are usually very important. For example, which attention backend, which kernels, how are the kernels compiled, the input sequence length, the maximum number of concurrencies, the memory occupancy, the size of the CUDA graphs to capture etc. etc. You can see how the number of configurations could easily exceed 2^10=1024 (assuming binary values), per engine flags alone!<br>Autoresearch

So, where to start? Of course, brute force would just pile up each one of these configurations, automate benchmarking of each one of those (we’ll talk about this more later), and select the best ones. Considering each benchmark takes several minutes, the brute force approach would have taken weeks of cluster time, making it a dead idea from the start. There are also several tools to run “estimations” from scratch (like aiconfigurator), but on Day-0, you usually don’t have a lot of reliable data, except your own priors in the space. You can also be smart about it and apply Design of Experiments approach, running full or fractional factorial design, or Hadamard design, but even these statistical methods rely on arbitrary models of interactions that you create a-priori. We decided to test a brand-new idea from the community: an autoresearch agent inspired by the work of Andrej Karpathy.<br>The main idea about autoresearch is quite simple: you create a robust harness with verifiable outcome, and an automated procedure to iterate, and with a series of prompts, skills and agents defined (and in this case, a proper security context!), you solve the problem with agentic coding with e.g. Claude Code/Codex/OpenCode and...