How to increase OpenBSDs Resilience to Power Outages - ZLHGO

Page content

Most of the OpenBSD systems I am in charge of are deployed in data centres, powered by UPSs which provide them with electrical power during periods of public grid power outages. But there is also a number of OpenBSD systems I administer, which are deployed in much less favourable conditions; where frequent power outages last longer than UPS batteries do, or where there are no UPSs at all (such as branch office routers in godforsaken places where having electricity and Internet access at all is considered a lucky circumstance). These latter systems are likely to have high rate of unclean shutdowns caused by prolonged or unexpected power outages, which in turn increase the probability of their inability to boot without human intervention. This article describes steps to make OpenBSD system more resilient to unexpected power outages by minimising the possibility of inconsistent file systems after unclean shutdowns, which is achieved by mounting all disk partitions in read-only mode. Filesystems which have to be writable - /var, /dev and /tmp - are mounted as writable memory file systems.

A few words of caution

This is “works for me” kind of tutorial. Although I use described setup on more than 30 production instances, that doesn’t mean it will suit your purpose as well. Primary goal of this article is to educate readers about some of the less-known functionalities of OpenBSD and deepen their understanding of underlying components so they can become more creative with their setups. Keep in mind that - talking in LEGO terms - being a master builder who doesn’t rely on standard templates means you’re mostly on your own when something goes bad with your design. Keeping /var in memory file system has been characterized as “wierd tweak” by Theo de Raadt, founder and leader of OpenBSD. He also said “we are not writing software for your crazy tweak”. Truth to be told, I don’t take any offense in this, as both statements are correct. Quite the contrary, I am flattered to be recognized by OpenBSD leader as someone who does wierd and crazy tweaks.

Now, some OpenBSD developers are more supportive to non-standard setups - for example my question on (AFAIK unofficial) #OpenBSD IRC channel regarding tcpdump(8) not being able to start with read-only / filesystem has been taken into account and identified as unveil(2) bug. This lead to kernel patch which fixed the problem in OpenBSD 6.6. Thanks (I guess) @semarie, @beck, possibly others, who did not dismiss this with “read-only / is not supported”. On the other hand, some OpenBSD developers are less supportive to non-standard setups. For example, when informed that syspatch(8) has problem patching stuff in /var mounted as memory file system, its creator and maintainer @ajacoutot replied he “will make that check stronger so that syspatch fails right away on MFS-mounted /var” (no worries, I have already worked around this as you will be able to read below).

Taking all above into account, please think twice before asking OpenBSD developers and users for help when using setup described in this article. Keep in mind that number of people using read-only / and /var on mfs is insignificant compared to whole OpenBSD user base. Problems you are encountering could be, and most probably are, problems with your way of using OpenBSD, and not problems with OpenBSD itself. This doesn’t mean you must never ask anything on OpenBSD mailing lists if you use described setup, only that it would be fair to declare which on-disk file systems you keep read-only, and which ones you mount as memory file systems.

Hardware and software used

I use PC Engines apu4c4 router board because of the following characteristics:

It features 4 GbE ports which is ideal for my branch offices - two ports are for two different ISPs, third port for local LAN, fourth for pfsync(4) interface in high availability setups which include carp(4).

Its CPU - 1 GHz quad Jaguar core with 64 bit and AES-NI support - does good job encrypting IPSEC flows

It has 4Gb of RAM, which is more than enough for basic routing and firewalling tasks, but also for memory file systems

As for local SD card storage, I use 32Gb SanDisk Extreme PRO. It’s relatively pricey, but my experiments with lower quality SD cards resulted in prolonged branch office downtime and 500 kilometer trip, which in the end proved much more expensive than initial savings.

I am connecting to PC Engines apu4c4’s serial interface from ThinkPad T440 via DIGITUS USB 2.0 serial adapter, which runs FreeBSD 12.1-RELEASE as main OS (although I sometimes boot to OpenBSD as well as - yes, I confess - Windows 10 thanks to GPT, UEFI and ReFind). My primary desktop environment is Xfce, and I am using GNU Screen to connect to PC Engines apu4c4’s serial console.

I use dd to flash install68.img...