Block-layer error injection [LWN.net]
LWN<br>.net<br>News from the source
Content Weekly Edition<br>Archives<br>Search<br>Kernel<br>Security<br>Events calendar<br>Unread comments
LWN FAQ<br>Write for us
Edition Return to the Front page
User:<br>Password: |
Log in /<br>Subscribe /<br>Register
Block-layer error injection
LWN.net needs you!
Without subscribers, LWN would simply not exist. Please consider<br>signing up for a subscription and helping<br>to keep LWN publishing.
August 12, 2026
This article was contributed by Haris Iqbal
Storage code has to cope with hardware that fails in inconvenient<br>ways, but coaxing a healthy disk into producing those failures on<br>demand, for testing, is usually not possible. The kernel<br>provides several ways to inject block-layer I/O errors, but none of those can select the<br>operation to fail, pick the status code to return, or target a disk<br>directly without employing a stacked device on top. Use of a stacked device means<br>the test runs against the mapper device, not the disk it was meant to<br>exercise. A patch<br>series from Christoph Hellwig adds a configurable error-injection<br>interface that does all three things that the current error-injection code<br>lacks, controlled by a per-disk debugfs<br>file.
Existing methods
The kernel has had a form of block-layer fault injection
since 2006, when Akinobu Mita added fault-injection<br>infrastructure. The block-layer portion (called fail_make_request) exposes<br>debugfs knobs for controlling the probability,<br>interval, and number of times that a fault will be injected.<br>Its shortcoming is that it treats every request the same. It cannot<br>tell a read from a write or a discard, cannot restrict failures to a<br>range of sectors, and can only fail a request with<br>BLK_STS_IOERR. The failure happens in<br>submit_bio_noacct(),<br>before the request reaches the driver. The filesystem or other kernel<br>code that submitted the
bio structure representing the request sees -EIO. If nothing<br>retries the request on the way up, user space sees EIO from the<br>system call. A single<br>status is limiting for a subsystem in which a media error, a transport<br>error, and a timeout each take a different path through the recovery<br>code.
A second fault-injection mechanism,<br>should_fail_bio(),<br>was added by Howard McLauchlan in 2018 as a hook for BPF programs.<br>That function is annotated with
ALLOW_ERROR_INJECTION(), so the kernel's error-injection<br>framework allows BPF programs to override the return value.<br>Before that change, the bio<br>submission path called should_fail_request()<br>to determine whether an error return should be injected<br>and passed only the disk device and a byte count. It now calls<br>should_fail_bio() instead, which takes the bio and<br>calls should_fail_request(), so the older feature behaves as<br>before until a BPF program overrides the call. Such a<br>program can read the bio and decide, for each request, whether<br>to fail it based on the operation type or the sectors involved. That<br>gives should_fail_bio() the selectivity that<br>fail_make_request lacks.
The approach has a limitation, though. BPF programs<br>can only replace a return<br>value. A program can select which<br>bio to fail, but not how it fails: submit_bio_noacct()<br>ignores the value returned and completes the bio with<br>BLK_STS_IOERR either way.
The device-mapper subsystem offers the other common approach, in the<br>form of targets built to fail I/O<br>requests. The simplest, dm-error, maps a region that returns<br>an error for every request it receives. Like<br>fail_make_request, it can produce only BLK_STS_IOERR,<br>and it fails every command routed through the target regardless of<br>operation or sector. To inject failures into I/O<br>requests made to an existing device, a dm-error target must be<br>stacked over that device. The test is then directed at the resulting<br>mapper device, not the device that needs testing.
The<br>dm-flakey<br>and dm-dust<br>targets are more configurable, and can, for example, fail I/O requests intermittently or emulate<br>individual bad sectors. They still share the limitations that matter<br>here: no real choice of error status and no good handling of commands<br>other than reads and writes (such as zone operations or discards).<br>Being device-mapper targets, they are also<br>bound by the alignment rules that device-mapper imposes on the sector<br>ranges a target may cover. dm-dust does not support<br>zoned devices at all, while dm-flakey, like any zoned-capable<br>target, can only confine failures to a whole-zone-aligned region, since<br>a target's range must begin and end on a zone<br>boundary.<br>And, like dm-error, they require a stacked block device on top<br>of the device under test.
The proposed mechanism
Hellwig's proposal, enabled through the new<br>CONFIG_BLK_ERROR_INJECTION Kconfig option, takes a more direct<br>route. When it is turned on, the block layer creates an<br>error_injection file in debugfs,<br>under /sys/kernel/debug/block/, for every registered<br>gendisk.<br>Reading the file lists the<br>injection entries in effect for that disk; writing to it either adds a<br>rule or, with a removeall command, clears...