BYO NOS: Building a maintainable, SRE-friendly switch — igloo's blog

BYO NOS: Building a maintainable, SRE-friendly switch

May 28, 2026<br>Introduction

I've got packets, they need moving.

For the last several years, I've used an Arista 7050QX2-32S as my core switch. It's a capable device that has served admirably. Unfortunately, it's also getting a little long in the tooth, having come out originally in 2016 and reached end of life in 2025.

Good alternatives are hard to come by. To understand why, we first need to understand the hardware we're working with.

From the perspective of hardware, the control plane of a network switch typically isn't complicated. In loose terms, the switch fabric is an extremely large - and expensive - chunk of silicon grafted onto the side of a reasonably low-power server from the era.

Arista's devices are built atop a Linux base with kernel modules and userspace extensions. While a traditional CLI is available, some operations inside of the base Linux system will be copied into the ASIC. For example, if an IPv4 or IPv6 route is inserted into the kernel, it'll be interpreted by Arista as a kernel sourced route (with a 'K' route origin in show route) and will be offloaded into hardware. As a result, running alternative routing daemons (such as BIRD) on Arista hardware is quite trivial!

When I first acquired the hardware, I quickly took advantage of Arista's generous functionality - I've built considerable infrastructure around BIRD, non-SNMP provided metrics, and the ability to monitor system health like any other server.

Unfortunately, the open and seemingly accessible nature of Arista's hardware can act almost like vendor lock-in, as no other vendors that I could find have built such an extensive ecosystem around supporting third party, off-the-shelf daemons on their hardware. So what other options are there?

Whitebox switching

Some years ago, hyperscalers started to build their own switches. These devices (generally) didn't ship with bespoke ASICs, instead opting for an in-house software implementation running against existing silicon. While this shifted the burden of building and maintaining a NOS onto the hyperscalers, it simultaneously enabled them to control the underlying software running inside their NOS. Gone were the days of long-standing software bugs that a vendor refused to fix, replaced instead with bespoke control planes that made configuring switches feel more like interacting with a generic microservice.

The trend didn't stay internal. Publicly available, non-vendor developed NOSes began to pop up, notably including SONiC and DENT.

SONiC was originally developed by Microsoft and utilizes ASIC-specific, typically black-box blobs to support each individual vendor/SKU it runs on; as a result, support is largely limited by which vendors (or vendor licensees) have taken the time to develop the magic binary. While it's got support for my existing hardware, it's got one major problem: like many open source projects, support for various functionality is heavily dependent on individual need. It's quite evident that Microsoft had a need for Layer 3 functionality, while Layer 2 functionality has been spotty and prone to de-syncronization on all hardware I've used it on. Additionally, some functionality - like IP route insertion from the kernel - isn't available.

In contrast to SONiC, DENT does not take advantage of a project-specific abstraction layer; instead, it acts as a NOS-like interface on top of a generalized, vendor-provided, kernel level abstraction layer called switchdev.

Switchdev offloads familiar Linux functionality into the ASIC. Like Arista, functionality exists to copy IPv4 / IPv6 routes. Unlike Arista, additional common functionality - such as ACLs and gathering of interface metrics - can be performed via standard tooling. DENT is functionally a pre-built, NOS-like set of tools built to manage a Linux host.

While DENT itself didn't offer the exact tooling I was after, switchdev presented an opportunity.

Selecting the hardware

DENT - and by extension switchdev - has a relatively small supported hardware list. Looking down the list, we see three major ASIC families that have some form of switchdev support: Marvell's Alleycat3, Marvell's Aldrin 2, and Mellanox's Spectrum. After a bit of research, I managed to pick up a Mellanox SN2010 - a Spectrum-based, switchdev-supporting chassis - for relatively cheap.

Opening it up, we see a fairly simple layout:

The concept of a "small PC that grew an ASIC" is fairly obvious; the computer itself is a tiny daughterboard!

In addition to the switching hardware, the chassis contains two unshielded power supplies and 4 internal fans. If you'd rather have the opposite direction of airflow, the fans can be turned around with three screws in a few minutes - not bad for a fully internal config.

Apart from the front panel SFP/QSFP ports, the device has 3 administrative ports: a RS232 serial port, a RJ45 Intel...