Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model — ROCm Blogs

Skip to main content

Back to top

Ctrl+K

ROCm blogs

Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model

Contents

Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model#

July 24, 2026 by Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra, Yonatan Dukler, Gowtham Ramesh, Zicheng Liu, Jialian Wu, Ximeng Sun, Wen Xie, Chaojun Hou, Vikram Appia, Zhenyu Gu, Emad Barsoum.

8 min read. | 1866 total words.

Applications & models

AI/ML

AI

Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra, Yonatan Dukler, Gowtham Ramesh, Zicheng Liu, Jialian Wu, Ximeng Sun, Wen Xie, Chaojun Hou, Vikram Appia, Zhenyu Gu, Emad Barsoum

English

-->

AMD is excited to introduce Instella-MoE, a state-of-the-art fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters. Trained from scratch on AMD Instinct™ MI300X and MI325X GPUs with the AMD ROCm™ software stack, Instella-MoE combines a sparsely activated MoE design with architectural innovations such as Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective connectivity. Instella-MoE delivers competitive performance across a broad suite of benchmarks against both dense and MoE baselines (as shown in Figure 1) including models with comparable or larger active parameter counts, establishing it as one of the strongest fully open language models at its scale.

Instella-MoE advances AMD’s broader effort to train advanced language models fully on AMD hardware and software with open tools and open research practices. Trained end-to-end with the open-source Primus and Miles frameworks, this model highlights the scalability and efficiency of AMD hardware and software stack for large-scale MoE training and inference, spanning from pre-training to post-training. Beyond strong model quality, it incorporates architecture- and system-level innovations that improve both training and serving efficiency.

Figure 1: Pre-trained and Post-trained Instella-MoE model performance compared with other similar size state-of-the-art models.#

In line with AMD’s commitment to open source, we are releasing all artifacts for the Instella-MoE model here, including model weights from every training stage, training configurations, data mixtures, and code. By making these resources openly available, we hope to help the AI community collaborate, reproduce results, and accelerate innovation in open language models.

This blog introduces the Instella-MoE model and then describes the architecture and multi-stage training pipeline behind the model. The release includes checkpoints from the following Instella-MoE training pipeline stages as shown in Table 1 below:

Model

Stage

Description

Instella-MoE-16B-A3B-Pretrain (Link)

Pre-training

MoE base model trained from scratch on a large and diverse training corpus.

Instella-MoE-16B-A3B-Midtrain (Link)

Mid-training

Pretrained model further trained on high-quality data mixtures to refine key capabilities.

Instella-MoE-16B-A3B-Base (Link)

Long-context

Long-context training to extend the model’s ability to process and reason over longer sequences. We use this as our final base checkpoint .

Instella-MoE-16B-A3B-SFT (Link)

SFT

Base checkpoint extended via supervised fine-tuning (SFT) to enable instruction following and chain-of-thought reasoning capabilities.

Instella-MoE-16B-A3B-DPO (Link)

DPO

Direct preference optimization (DPO) on contrastive preference data to improve model performance.

Instella-MoE-16B-A3B-Think (Link)

RL

Final thinking checkpoint refined with reinforcement learning (RL) to further strengthen instruction following and overall response quality.

Table 1: Instella-MoE-16B-A3B models and training stages.

Takeaways#

Instella-MoE is a new state-of-the-art fully open Mixture-of-Experts language model developed by AMD, with 16 billion total parameters and 2.8 billion active parameters per token, trained from scratch on AMD Instinct™ MI300X and MI325X GPUs.

The Instella-MoE model checkpoint release spans every stage of the model training pipeline, including pre-training, mid-training, long-context extension, SFT, DPO and RL.

Built entirely on the AMD ROCm™ software stack on top of the Primus training and Miles RL frameworks, Instella-MoE incorporates cutting-edge architecture and systems innovations—including Gated Multi-head Latent Attention (Gated MLA) and extreme communication-computation overlap through FarSkip-Collective—for efficient large-scale training and inference on AMD hardware.

Fully open and accessible: we provide our complete training recipe across all training stages, including training frameworks, data mixtures, intermediate checkpoints and inference code.

Instella-MoE Model Architecture#

The Instella-MoE architecture is a decoder-only MoE design built with Gated Multi-head Latent...