Watch Out for Data Loss When Redirecting to a File in a Shell Script

Watch Out for Data Loss When Redirecting to a File in a Shell Script — Nick JanetakisLearn Docker With My Newest Course Dive into Docker takes you from "What is Docker?" to confidently applying Docker to your own projects. It's packed with best practices and examples. Start Learning Docker →

Updated on July 7, 2026 in #linux Watch Out for Data Loss When Redirecting to a File in a Shell Script

There's a number of subtle things going on when you redirect output to a file, here's how to do it safely and verify it with strace. Quick Jump: Prefer video? Here it is on YouTube. I’ve written 10,000+ lines of shell scripts over the years and I can’t believe I only learned about this recently. Let’s say you have a normal shell script. I pretty much always set these options to protect myself. They’ll make your script exit if any commands you run encounter an error, if a pipeline fails or if a referenced variable is unset: #!/usr/bin/env bash

set -o errexit set -o pipefail set -o nounset

echo "Hello"

Easy right? The expected output is “Hello” which is what we get. What about echo "Hello" > results? Also easy, it will create a new results file in the directory where you ran this script with the contents of “Hello”. This is shell syntax to redirect output to a file.

Showing the Problem Without looking it up or trying it, take a guess what happens if you put this line at the bottom of the script. Don’t just run it in your terminal, since the same options aren’t set: echo "Hello" | awk '{system("sleep 5"); print}' >results

To give you a hint, the awk command is just sleeping for 5 seconds. We’re using that instead of sleep 5 because the sleep command doesn’t accept STDIN where as awk does. I don’t know about you but initially I thought this would happen: “Hello” would be printed to STDOUT It would get piped to awk which accepts STDIN awk would sleep for 5 seconds and print “Hello” to STDOUT A results file would be created with the contents “Hello” The above mental model is how most pipelines are read. It flows from left to right. What Really Happens The file descriptor for the results file is created immediately and a 0 byte file is createdYou can verify this by running the command and catting the file in terminal #2

echo prints “Hello” and pipes it into awk awk receives this input and begins executing its system command awk sleeps for 5 seconds awk prints “Hello” to STDOUT which is already connected to the open results file, this yields our expected text in the file The important takeaway is as soon as this pipeline starts to run the file you’re redirecting to can be considered truncated to a 0 byte file and there’s 3 possible states this file can be in after the script is called: All commands finish successfully and it’s what you expect One of the commands fails and the scripts exits with a partially written file The script exits before processing has occurred (power outage) and the file is empty What If You Use Tee Instead? If you’re comfortable writing shell scripts, you might think about trying: echo "Hello" | awk '{system("sleep 5"); print}' | tee results 1>/dev/null

This has the same problem as > since the results file will get immediately created. Technically when you run the above, echo, awk and tee are being run in parallel. Then tee is like “oh, I need to write to the results file, let me open a file descriptor for that”. At this point awk can start streaming data into the file at its discretion. Why This Is a Problem It comes down to potential unexpected data loss. Here’s 1 of many examples: Imagine you’re doing something like a database backup. It’s not unreasonable to think you’d run your database dump command (pg_dumpall, etc.), maybe pipe it into gzip and then redirect it to a file. If you’re keeping 1 day’s worth of backups by overwriting the existing file, if your current day’s backup command fails to run for some reasons, you could end up with a partial backup that overwrote yesterday’s valid backup. In a disaster scenario where you have to restore, you suddenly have data loss.

Using strace to Pry Deeper The more I use native Linux on the desktop, the more I find myself interested in lower level details. I wish I had unlimited time: $ strace --follow-forks -e trace=openat ./demo

If you run the above with either version you’ll see this line early on which demonstrates a truncated 0 byte file is being created: [pid 1964503] openat(AT_FDCWD, "results", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3

Also, while no timestamps are shown, when you run the above command with the tee version you’ll see something like this pop out at about the same time: strace: Process 1966073 attached strace: Process 1966074 attached strace: Process 1966075 attached

This demonstrates all...

Watch Out for Data Loss When Redirecting to a File in a Shell Script

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills

The Anatomy of an AI-Native Org