Self-hosting email the hard way from your own routable IPv4 block up | Anil Madhavapeddy

>_ Anil Madhavapeddy

>_esc<br>PapersNotesProjectsIdeasTalksLinks<br>Type to search across all content

↑↓ nav↵ openesc close

Anil MadhavapeddyFresh from rewilding the web, I updated the Recoil self-hosting<br>infrastructure that Nick Ludlam and I have run since 1997. Most exciting, this is now<br>'email the hard way' that includes getting our very own dedicated IPv4 allocation routed thanks<br>to my buddy Thomas Gazagnaire helping out from France!

This post will be quite technical, aimed at those interested in building their own email stack. We'll talk about why bother running your own email; receiving email (covering IP reputation, getting our own IPv4 allocation, stopping bots, and Sieve delivery); sending email reliably with SPF, DKIM, DMARC and SRS; user access via Dovecot IMAP and Roundcube webmail; and finally what's left to do and a reflection on future research ideas.

1 Why run your own email?

For someone just getting into systems and networking, it's a hugely educational<br>experience. Running my own servers has been how I've learnt how the Internet<br>works, and how I got into open-source back in the day, by installing OpenBSD and fixing bugs in PHP!

More broadly, self-hosting is important for sovereign access to your own data<br>as the web steadily consolidates among a few<br>players. A 2023 analysis<br>showed that two companies can mostly read all of your email traffic:

So all in all, the answer to the question of who can read your email pretty<br>much boils down to -- yep -- "Google and Microsoft". Even if your domain<br>doesn't use one of their mail servers, chances are that whoever you are sending mail to does.<br>-- MX diversity, Jan Schaumann, 2023

Email is right at the centre of our digital lives; consider just how many online<br>accounts you could reset if your email got hijacked or phished. In many ways, it's the online service that connects up all the other ones.

1.1 Should you run your own email?

Hosting your email is a fair bit of work, but that's spread over a long period of time as you keep an eye on it. Nick Ludlam and I started our hosting adventure back in 1998 or so when we worked at NASA for the summer. The first time we did a major server move back in 2002, here's Nick and Chris Luke<br>loading up our second server into a dodgy white minivan to host in Easynet, right outside London's first Internet cafe.

From Cyberia outside Fitzrovia over to Brick Lane and Easynet hosting! (2002)<br>If you do decide to have a go for yourself, you'll need to find stable Internet hosting (your own home network isn't the best choice,<br>for reasons we'll see later in this post), and also build up a reputation. Luckily, finding stable Internet is much easier these days than it was in the late 90s, and we use Mythic Beasts who are excellent, reliable and local.

I'll explain in this post how we obtained our own dedicated IPv4 address block<br>to help build up a high deliverability index for our personal email that's<br>entirely independent! While email looks like one service to most users, it<br>is actually three separate activities: email receipt, email<br>submission, and email access. I'll dive into how each of<br>these work on Recoil now, in case it's useful for your own setup.

2 Email receipt

Each domain (like recoil.org) on the Internet runs an SMTP server that allows<br>it to receive email from other servers. You can query this server for any<br>domain by looking up its 'MX' DNS record:

$ host -t mx recoil.org<br>recoil.org mail is handled by 10 pork.recoil.org.

This indicates that the Internet host pork.recoil.org must accept connections from any server<br>on the internet that wishes to deliver email to @recoil.org.<br>The core difficulty is that SMTP (as<br>designed in the more trusting 1980s) doesn't mandate a built-in proof of trust.<br>Anyone can claim to be anyone else, and the 'sender' in the email we receive<br>can be trivially forged.

The IETF's response was to accrete a stack of checks over the years that an<br>email sender must pass, or be filtered by the recipient. If we mess these<br>identity checks up, then our email won't get delivered reliably across the<br>Internet and the service won't be very useful.

2.1 IP reputation and denylists

Spam was cheap to send from anywhere on the Internet, and so naturally grew as<br>the wider network gained adoption. Paul Vixie came up with a DNS-based<br>blocklist back in<br>1997. Since then, many more have sprung up, operated by organisations like<br>Spamhaus, Spamcop and<br>Barracuda that aggregate reports about<br>botnets, compromised hosts and spammers into lists that any email server can<br>consult.

The DNS blacklist/whitelist protocol (RFC 5782) is super simple and can be queried right from your command line:

$ dig 2.0.0.127.zen.spamhaus.org +short # testing address<br>127.0.0.2<br>127.0.0.10<br>127.0.0.4

That's a localhost testing address, but the presence of a DNS record in the RBL<br>means that that server is suspect. This is where having control of...