Tracking down a Zsh history data loss bug 🐞 - Michael Stapelberg

Tracking down a Zsh history data loss bug 🐞

published 2026-08-09

in tag

debug

Table of contents

For many years, I sometimes discovered that commands I was sure I had run were<br>no longer present in my Z shell history<br>file (~/.zsh_history). In this article, I will show you how I tracked down the<br>bug. Spoiler: ultimately, patching Zsh to make it crash loudly and analyzing the<br>crash’s core dump was the winning strategy!

The good news first

Zsh 5.9.2 (released July 12th, 2026) contains a fix for this issue — open it<br>after reading this investigation to not spoil the fun.

Spoiler: link to the upstream fix<br>Zsh fix 53454

The symptom

Occasionally, I noticed that commands I knew I executed the day before were<br>not findable in my shell history, meaning pressing Ctrl+R for backward history<br>search yielded no results. Whenever I noticed this, my shell history file<br>contained only very old entries, with years of newer entries missing.

The first few times this happened I just restored my shell history from my daily<br>backup and did not bother investigating any further. But the issue kept happening.

I noticed that there was no visible corruption in the .zsh_history (no<br>non-printable characters or incomplete lines of text), and that the number of<br>lines in the file was not always the same.

What was not clear to me was whether it was Zsh itself, or some other program,<br>or perhaps the combination of multiple zsh(1)

processes that caused the issue.

My Zsh history config

I set the following history-related options in my ~/.zshrc:

# Load 4000 lines of history (for Ctrl+R backward search), but save O(∞)<br>HISTSIZE=4000<br>HISTFILE=~/.zsh_history<br>SAVEHIST=10000000

# Do not save (adjacent) duplicate entries<br>setopt HIST_IGNORE_DUPS

# Append history entries to `~/.zsh_history` when commands are run.<br>setopt INC_APPEND_HISTORY<br># …but do not share history (enabled by default in NixOS’s /etc/zshrc).<br>unsetopt SHARE_HISTORY<br>In practice, this means my shells are separate sessions that all stream their<br>commands into a shared ~/.zsh_history. The history is intentionally not<br>shared, so when I want to access entries that another shell wrote, I explicitly<br>run exec zsh.

Tracing the behavior

When I asked for help on Mastodon in December<br>2024 (mostly in the hope that<br>somebody else already encountered and diagnosed this problem), one suggestion I<br>got was to use file system change monitoring mechanisms like inotify or fsevents<br>to find the culprit that truncates (or changes?) the Zsh history file.

The next sections walk through the available options on Linux which I tried.

inotify

The inotify(7)<br>Linux kernel subsystem is one of the<br>oldest file system change monitoring APIs available in Linux (released 2005). To<br>get a good understanding of how Zsh modifies the history file, it is not<br>sufficient to monitor just .zsh_history:

midna ~ % inotifywait --monitor .zsh_history<br>Setting up watches.<br>Watches established.<br>.zsh_history OPEN<br>.zsh_history ACCESS<br>.zsh_history ACCESS<br>[…]<br>.zsh_history ACCESS<br>.zsh_history CLOSE_NOWRITE,CLOSE<br>.zsh_history ATTRIB<br>.zsh_history CLOSE_WRITE,CLOSE<br>.zsh_history DELETE_SELF<br>^C<br>The file is opened, accessed (= read) and then… deleted?!

By monitoring the containing directory, we see the whole picture:

midna ~ % inotifywait --monitor ~<br>/home/michael/ OPEN .zsh_history<br>/home/michael/ ACCESS .zsh_history<br>/home/michael/ ACCESS .zsh_history<br>[…]<br>/home/michael/ ACCESS .zsh_history<br>/home/michael/ CLOSE_NOWRITE,CLOSE .zsh_history<br>/home/michael/ CLOSE_WRITE,CLOSE .zsh_history<br>/home/michael/ OPEN .zsh_history<br>/home/michael/ CLOSE_WRITE,CLOSE .zsh_history<br>/home/michael/ OPEN .zsh_history<br>/home/michael/ ACCESS .zsh_history<br>/home/michael/ CLOSE_NOWRITE,CLOSE .zsh_history<br>/home/michael/ CREATE .zsh_history.new<br>/home/michael/ OPEN .zsh_history.new<br>/home/michael/ ATTRIB .zsh_history.new<br>/home/michael/ MODIFY .zsh_history.new<br>/home/michael/ CLOSE_WRITE,CLOSE .zsh_history.new<br>/home/michael/ MOVED_FROM .zsh_history.new<br>/home/michael/ MOVED_TO .zsh_history<br>/home/michael/ CLOSE_WRITE,CLOSE .zsh_history<br>So Zsh reads the old history file contents, writes them to a new file, then<br>renames the new file over the old one, thereby deleting the old one. Now it<br>makes sense!

Unfortunately, we do not see the process IDs (PIDs) of the responsible process<br>for the file system event, not even with the sibling utility fsnotifywait(1)<br>, which uses fanotify(7)<br>, an API that does provide this information! I checked, and the<br>kernel does send the PID, but fsnotifywait does not display the PID.

fatrace

Luckily, there is fatrace(8)<br>, which does display the<br>process name and PID.

Here is what Zsh’s history rewriting looks like with fatrace(8)

zsh(197994): CWO /home/michael/.zsh_history<br>zsh(197994): O /home/michael/.zsh_history<br>zsh(197994): R /home/michael/.zsh_history<br>zsh(197994): R /home/michael/.zsh_history<br>[…]<br>zsh(197994): R /home/michael/.zsh_history<br>zsh(197994): C...