Connecting my illumos box to a Japanese ISP | skalski.dev

I moved an Ethernet cable from my router to an OmniOS box that I call fridge. The other end was connected to the modem provided for my Docomo Hikari line. This was meant to be the last test of a Rust networking project I had worked on during weekends since the end of March. I had written a daemon that could manage a native DS-Lite tunnel on Linux and illumos. If it went well, the box would connect directly to the ISP and provide IPv4 connectivity through that tunnel.

The modem lights looked normal. OmniOS received router advertisements and installed a default IPv6 route on ixgbe1. It did not, however, receive an address that could be used to reach the Internet.

$ ipadm show-addr<br>ADDROBJ TYPE STATE ADDR<br>lo0/v4 static ok 127.0.0.1/8<br>ixgbe1/v6 addrconf ok fe80::ae1f:6bff:fe6c:5ca7%ixgbe1/10

$ netstat -nr -f inet6<br>Destination/Mask Gateway Flags If<br>default fe80::be4a:56ff:fe1c:3c10 UG ixgbe1

There was a route, but only a link local address to use as its source. Restarting the modem and recreating the IPv6 interface changed nothing. Four months earlier I would probably have assumed that the tunnel was the missing part. By this point I had also implemented the provisioning protocol used by my ISP, an SMF service, and an OmniOS package. The cable test exposed a more basic gap.

I started the project because I wanted to learn more about the illumos network stack. I had also been looking for a Rust project that would let me work directly with operating system interfaces. An article by Ryan Goodfellow about the life of an IPv6 address on illumos had stayed in my mind. It follows one address through ipadm, libraries, doors, STREAMS, and the kernel. I wanted a problem that would make me visit some of the same layers.

There was also a practical goal. fridge already runs services on my home network. If it could terminate the ISP connection as well, I could use it as the entry point to that network.

Calling it my ISP connection hides some Japanese telecom layering. The physical access runs over NTT's FLET'S network. My retail contract for that line is Docomo Hikari, while ASAHI Net provides Internet connectivity on top of it using native IPv6 over IPoE and IPv4 over DS-Lite. All three names therefore describe parts of the same connection, but they do not play the same role.

DS-Lite exists because an ISP can provide native IPv6 without assigning a public IPv4 address to every customer. The customer side component is called a B4. It takes an IPv4 packet and places it inside an IPv6 packet addressed to an ISP endpoint called the AFTR. The AFTR removes the outer packet and performs IPv4 network address translation before sending the traffic to the Internet. RFC 6333 describes the architecture.

Applications on the customer network still see ordinary IPv4. Their packets reach the B4, which uses the reserved 192.0.0.0/29 range for the point to point side of the tunnel. The outer IPv6 packet travels over the access network to the AFTR. Replies follow the same path in reverse. Once the tunnel exists, the daemon is no longer involved in moving each packet. Encapsulation and forwarding remain inside the operating system network stack.

The B4 does not need to implement packet forwarding itself. Both Linux and illumos already know how to create an IPv4-in-IPv6 tunnel. The first useful experiment was therefore small. I created an iptun link on OmniOS, configured a matching Linux host on the same network as a temporary AFTR, assigned the reserved DS-Lite IPv4 endpoints, and installed a route. IPv4 pings crossed the tunnel in both directions.

That result seemed to remove the largest risk because the kernel data path worked. The remaining task looked like turning a few administrative commands into a daemon that could discover the two IPv6 endpoints and keep the tunnel in the desired state. I treated the tunnel as the uncertain part and ISP provisioning as information that some existing service would eventually hand to it. Building the tunnel daemon followed that plan, while finding a reliable source for the ISP information became the rest of the project.

I did not want the program to run dladm, ipadm, and route as subprocesses. Calling the underlying interfaces would teach me more, avoid parsing command output, and make it possible to observe and reconcile state through the same APIs. The Linux backend used netlink. The illumos backend used libdladm to create the tunnel link, libipadm to create its IP interface, socket ioctls to configure its addresses, and a PF_ROUTE socket for the IPv4 default route.

This choice made the Rust side more interesting than a sequence of command invocations. Some illumos interfaces are regular C library calls, while others exchange structures whose layout comes directly from system headers. Each foreign call required deciding which values could be represented safely in Rust, how long pointers remained valid, and which resources needed...