Miles: A PyTorch-Native Stack for Large-Scale LLM RL Post-Training – PyTorch

Search

Close Search

Blog

Miles: A PyTorch-Native Stack for Large-Scale LLM RL Post-Training

By Miles TeamJune 30, 2026No Comments

Featured projects

TL;DR

Miles is RadixArk’s open source framework for large-scale LLM RL post-training. It composes SGLang for rollout, NVIDIA Megatron-LM for training, Ray orchestration, and PyTorch-native extensibility behind a small, pluggable trainer, with unified low-precision recipes, MoE-aware rollout/training alignment, fast NVIDIA NCCL/RDMA weight synchronization, observability, and fault tolerance built in — making frontier-scale LLM RL easier to build, reproduce, and operate.

Why Miles?

Reinforcement learning has become a central part of post-training large language models. But as models become larger, transition from dense to mixture-of-experts (MoE), and run across more distributed and specialized hardware (e.g. NVIDIA Blackwell and Hopper series), RL post-training is no longer just a training loop. It is a distributed systems problem.

A modern LLM RL framework needs to coordinate several moving pieces:

Rollout workers must generate samples at high throughput.

Trainers must consume those samples efficiently and compute stable policy updates.

The rollout policy and training policy must stay synchronized.

Large MoE models introduce routing behavior that must remain aligned across rollout and training.

Low-precision recipes need to work consistently across the full pipeline.

Long-running jobs need observability, checkpointing, and fault tolerance from the start.

Miles was built for this setting.

Miles is RadixArk’s open source reinforcement learning framework for LLM post-training. It is built natively on SGLang for high-throughput rollout and integrates deeply with Megatron-LM for scalable training, uses Ray to orchestrate the distributed system, and keeps PyTorch as the common programming and numerical layer throughout the stack.

The goal is simple: make large-scale LLM RL training more composable, reproducible, and easier to scale, while keeping the core trainer small enough for researchers and infrastructure teams to customize.

The Miles Architecture

Miles follows a small-core, many-edges philosophy.

The core training loop is intentionally compact. The pieces that users most often want to change — rollout logic, reward computation, loss functions, sample filtering, metrics, and training-loop hooks — are attached at launch time through user-supplied Python modules. This lets teams adapt the system to new algorithms and production constraints without forking the framework.

Underneath that small core, Miles composes four major systems:

SGLang for high-throughput rollout generation.

Megatron-LM for scalable distributed training.

Ray for cluster orchestration, actor lifecycle, scheduling, and supervision.

PyTorch for models, autograd, distributed primitives, dtype support, extensibility, and profiling.

This composition is important. RL post-training requires generation and training to work together, but the two phases have very different performance profiles: rollout is memory-bandwidth-bound (KV-cache and parameter reads dominate during decoding), while training is compute-bound and communication-heavy. Weight synchronization, sample transfer, checkpoint conversion, routing consistency, and low-precision behavior all need to be handled carefully across the boundary.

The rest of this post walks through how Miles handles each piece of that boundary — orchestration with Ray, scaling with Megatron-LM, extensibility with PyTorch, and what comes out of the box.

Ray: Orchestrating Long-Running RL Jobs

Miles is built directly on the Ray distributed runtime. In a Miles run, every long-lived process is represented as a Ray actor: trainer ranks, SGLang rollout servers, routing proxies, and asynchronous rollout workers all live inside Ray’s actor model.

This gives Miles a natural foundation for cluster-scale RL workloads.

Placing workers on GPUs

Miles uses Ray’s GPU-aware scheduler and placement groups for actor placement, supporting disaggregated (rollout and training on separate nodes) and colocated (rollout and training on the same nodes) layouts via launch-time Ray placement specs. Process placement must be rack-aware to facilitate careful colocation, reserving spare nodes, and key for error isolation, since isolating problems within a rack (e.g., distinguishing a bad GPU from a full rack issue) is not always straightforward.

Moving data across the RL pipeline

Prompts, samples, and updated weights cycle continuously between rollout actors and trainer ranks, and Miles uses Ray actors and tasks to coordinate that flow. For bulk weight transfer, Ray handles the control path while the tensor bytes move over dedicated NCCL/RDMA channels, giving Miles both Ray-level programmability and a fast path for large data.

Supervising long-running jobs

Because a Miles run is a...