Learning to Route: A Shadow-First Contextual Bandit for LLM Model Selection — Avriz

Technical report · AVZ-TR-2026-01

Learning to Route: A Shadow-First Contextual Bandit for Per-Request LLM Model Selection

Avriz Engineering · avriz.io · July 2026 · control-plane/rl_router.py

Abstract<br>Serving every request with a frontier language model wastes an order of magnitude of cost on work a small model handles; serving every request with a small model fails the work that matters. Production LLM platforms therefore route: they pick a model per request. We describe the routing system deployed on Avriz, a managed development-box platform, in three layers: (i) a hand-built LLM judge that classifies each turn's difficulty onto a five-rung model ladder, with a mid-turn escalation valve; (ii) a shadow-mode contextual bandit that learns per-rung reward models online from the platform's existing billing ledger, without storing chat content and without affecting user traffic; and (iii) a flag-gated ε-greedy serving path that lets the learned policy earn real traffic incrementally, bounded by a rung ceiling and measured by an on-policy A/B comparison. The design premise throughout is that observation precedes influence: the learner trains off-policy from decisions the judge already makes, activates only past an explicit data bar, and every step toward autonomy is one configuration flag from rollback. We give the full reward construction, the per-arm estimator and its update rule, the activation and serving math, and the honest limitations of direct-method off-policy evaluation at our scale.

1Introduction

Avriz gives each subscriber an isolated cloud machine with a coding agent on it. Every conversational turn the agent takes is ultimately a call to some large language model, and the platform's managed tier lets it choose which. The price spread across the available ladder is large — wholesale output-token cost spans 12× from the cheapest rung to the most expensive (Fig. 2) — while a majority of real turns ("rename this", "what does this function do?") are handled perfectly by the cheapest rung. Routing is thus a direct lever on both margin and user experience: under-routing a hard request produces a bad answer; over-routing an easy one burns money.

The first version of a router is always a heuristic, and heuristics rot: thresholds hand-tuned in June are silently wrong by September as models, prices and usage change. The interesting question is not "can we write a good routing rule" but "can the platform keep the rule good, from its own traffic, without ever risking that traffic". This report documents our answer, which is deliberately conservative: a learning system that spends its whole early life in shadow.

Three properties drove the design:

Zero-risk learning. The learner must not change a single user-visible decision until a human flips a flag — and even then only for a bounded slice of traffic under a rung ceiling.

Zero new data. Training must use only what the platform already records for billing (token counts, realized cost, status codes) plus derived, content-free features. No chat text is stored by, or recoverable from, the routing system.

Legibility. Every stage — features, prior, reward, estimator, activation — is inspectable in an admin report, and the model itself is a set of eleven-dimensional linear weights a human can read.

2Background and related work

Selecting one action per context and observing a reward only for the chosen action is the contextual bandit problem, the one-step special case of reinforcement learning [8]. The linear-payoff formulation and its optimistic variant LinUCB were established for news recommendation by Li et al. [3]; posterior-sampling alternatives descend from Thompson [1] (see [7] for a modern tutorial), and confidence-based exploration from UCB [2]. Evaluating a new policy from logs generated by an old one is the off-policy problem: the direct method (fit a reward model, evaluate the candidate against it), inverse-propensity weighting, and their doubly-robust combination are analyzed by Dudík et al. [4]; Bottou et al. [5] treat the same machinery as counterfactual reasoning about a production system, which is exactly our setting. Practical system design for production bandits — logging, delayed rewards, gating — is discussed in Agarwal et al. [6] and evaluated at scale in [9]. Cost-aware routing across an LLM cascade was introduced as FrugalGPT [10]; learned routing between model pairs from preference data appears in RouteLLM [11]. Our contribution is not algorithmic novelty — the estimator is deliberately the simplest thing that can learn — but an end-to-end production recipe: reward built from a billing ledger, off-policy training from a heuristic logging policy, an explicit activation bar, and a ceiling-bounded ε-greedy graduation path, all inside a system where routing failures self-heal.

3The live router: judge, ladder, escalation

The unit of decision is a...