Explore distributed inference and training with MLX - WWDC26 - Videos - Apple Developer

View in English

-->

More Videos

About

Summary

Transcript

Code

Explore distributed inference and training with MLX

Scale your machine learning workloads across multiple Macs using MLX. Learn how to tackle interconnect efficiency, large model inference, request batching, and distributed training challenges. Discover how a few Macs on your desk can replace expensive cloud infrastructure for demanding AI workloads.

Chapters

0:00 - Introduction

2:09 - Distributed communication

4:32 - Setting up your cluster

10:33 - Distributed inference and fine-tuning

13:35 - Model parallelism strategies

15:53 - Distributed fine-tuning

18:34 - CLI, Python, Swift, and C++ APIs

20:45 - Next steps

Resources

MLX Swift LM on GitHub

MLX Swift Examples

MLX Examples

MLX Swift

MLX LM - Python API

MLX Explore - Python API

MLX Framework

MLX

HD Video

SD Video

Related Videos

WWDC26

Explore numerical computing in Swift with MLX

Run local agentic AI on the Mac using MLX

WWDC25

Explore large language models on Apple silicon with MLX

Get started with MLX for Apple silicon

Search this video…

Hi, I'm Tatiana, research scientist at MLX team. It's been a remarkable time for local LLMs: models keep getting larger and gaining new amazing capabilities -- becoming smarter and handling harder problems. And as they improve, we use them for more: longer contexts, harder tasks, more complex workflows. Eventually, memory, compute, or bandwidth on a single machine becomes a limitation. In our WWDC 26 video "Run local agentic AI on the Mac using MLX" it is shown how to run AI agents locally. But when you have multiple devices, you can take local AI even further, running larger LLMs or accelerating them through distributed inference and training. Today, we'll take a deep dive into scaling across multiple Macs with MLX, using the hardware right on your desk. We'll start with the command line interface to get models running on your machines, move to the Python API for experimentation, and finish with Swift for embedding these workflows directly into your apps.<br>Let's start! First, we'll look at the full hardware and software stacks to make distributed workloads on Apple Silicon possible. Then we'll put everything together: turn four M3 Ultras into a cluster. We'll walk through every step: choosing the right topology to connect machines, enabling fast communication, and launching distributed jobs.<br>Once the cluster is ready, we'll get to the exciting part: fast and local distributed LLM inference and finetuning. We'll run it with MLX, compare it side by side against a single Mac, and look at how MLX distributes the model across the cluster.<br>Having most examples in command line interface, in the end we'll show how distributed communication is also exposed to you via Python, Swift and C++ APIs. Let's start by looking at distributed communication for Apple Silicon. To send and receive data fast, machines need to be connected with a physical link — an interconnect. On top of that, we also need a transport protocol — a mechanism that pushes bytes from one machine's memory to another's. Starting in macOS 26.2, Remote Direct Memory Access protocol, shortly RDMA, is supported over Thunderbolt 5. RDMA moves data directly from one machine's memory to another's, avoiding most CPU and operating system overhead.<br>RDMA over Thunderbolt gives us the high-bandwidth — low-latency communication we need for distributed workloads. However, alone, it gives us raw data movement between two machines only. Thus, distributed programs need something higher-level — a communication backend which provides communication primitives for sending data between individual machines or coordinating across the entire group. These two operations are building blocks of distributed training and inference. And this is where JACCL comes in.<br>JACCL is an open-source collective communication library built by Apple. It leverages RDMA over Thunderbolt and gives you collective communication primitives for sending data between machines and combining results across the group — without managing any of the low-level transport yourself. And it's not limited to machine learning — any distributed workload on Apple Silicon can be built on top of it.<br>And the final piece of the stack is a machine learning framework that uses the communication backend for distributed inference and training — that's MLX.<br>MLX is an open-source machine learning library built by Apple for Apple Silicon. It leverages JACCL for low-latency distributed communication and provides tools for orchestrating distributed jobs across the cluster. If you're new to MLX, check out our video "Getting Started with MLX on Apple Silicon" from WWDC25.<br>So now we understand the full stack. Let's put it all together and build a cluster — a group of machines that work together on the same task. We will use 4 M3 Ultras.<br>To setup the cluster, we...