The occasional `ECONNRESET` (part 1/2)
blog - git - desktop - contact
The occasional ECONNRESET (part 1/2)
2026-05-05
Two services running on the same machine. One of them opens a listening<br>TCP socket bound to localhost, the other one connects to that. They<br>exchange data. Every now and then, the service that initiated the<br>connection gets an ECONNRESET while reading data from the socket -- but<br>no other errors show up in the logs, no crashes, nothing. What's going<br>on?
A reproducer in the "lab"
What tcpdump sees
What strace ./server sees
What strace ./client --spam sees
A first hypothesis
The real-life scenario
Next steps
Go on to part 2.
A reproducer in the "lab"
Let's start with the "server", i.e. the service that opens the listening<br>socket.
The following program does just that: Create a new TCP socket, wait for<br>connections, fork into a new process for each request. There's not much<br>of a "request", though: The server simply dumps 600'000 x bytes to the<br>client upon connection.
The number 600'000 bears some meaning: It needs to be large enough to<br>trigger the behavior that I want to show. 600 bytes, for example,<br>probably won't work.
server.c
Now on to the client: It connects to port 8125 on localhost where our<br>server is waiting, and then it calls recv() until EOF or error. We'll<br>get to the --spam flag in a second.
client.c
Also:
Makefile
Let's run the two programs:
[terminal1]$ ./server
[terminal2]$ ./client<br>Read 600000 bytes, final return value was 0, errno was 0
Nothing spectacular.
But let's use the --spam flag:
$ ./client --spam<br>Read 600000 bytes, final return value was -1, errno was 104<br>Connection reset by peer
$ ./client --spam<br>Read 256000 bytes, final return value was -1, errno was 104<br>Connection reset by peer
$ ./client --spam<br>Read 351232 bytes, final return value was -1, errno was 104<br>Connection reset by peer
$ ./client --spam<br>Read 351232 bytes, final return value was -1, errno was 104<br>Connection reset by peer
$ ./client --spam<br>Read 351232 bytes, final return value was -1, errno was 104<br>Connection reset by peer
$ ./client --spam<br>Read 256000 bytes, final return value was -1, errno was 104<br>Connection reset by peer
$ ./client --spam<br>Read 600000 bytes, final return value was -1, errno was 104<br>Connection reset by peer
--spam causes the client to first send some data to the server before<br>it tries to receive data. And appararently this causes the connection to<br>break at some point: The client's recv() sees a return value of -1<br>and errno gets set to 104 = Connection reset by peer.
What tcpdump sees
First, what's "on the wire"?
Okay. So there actually is a TCP RST. Could have also been a<br>programming error or misinterpretation on my part.
What strace ./server sees
The RST originates from the server, so let's attach strace and see<br>what we get:
19:59:03.420432 accept(3, NULL, NULL) = 4<br>19:59:05.652715 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe43484fa10) = 239546<br>[pid 239546] 19:59:05.652831 ...<br>[pid 239546] 19:59:05.652959 mmap(NULL, 602112, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0<br>[pid 239546] 19:59:05.652980 ) = 0x7fe43456d000<br>[pid 239546] 19:59:05.653235 sendto(4, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 600000, 0, NULL, 0) = 600000<br>[pid 239546] 19:59:05.653474 close(4) = 0<br>[pid 239546] 19:59:05.653553 exit_group(0) = ?<br>[pid 239546] 19:59:05.653667 +++ exited with 0 +++
No crash. It forked and used sendto() to dump all the data to the<br>client. Then it quit.
Also note that sendto() returned the full 600000, so from the<br>perspective of this program, "all data got sent" (there's a footnote,<br>obviously, as the manpage explains: "Successful completion of a call to<br>sendto() does not guarantee delivery of the message. A return value of<br>-1 indicates only locally-detected errors.").
In fact, there is no difference here whether you use --spam on the<br>client or not.
What strace ./client --spam sees
19:59:05.652518 connect(3, {sa_family=AF_INET, sin_port=htons(8125), sin_addr=inet_addr("127.0.0.1")}, 16) = 0<br>19:59:05.652649 sendto(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 100, 0, NULL, 0) = 100<br>19:59:05.652805 recvfrom(3, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 4096, 0, NULL, NULL) = 4096<br>19:59:05.653382 recvfrom(3, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 4096, 0, NULL, NULL) = 4096<br>...<br>19:59:05.654440 recvfrom(3, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 4096, 0, NULL, NULL) = 4096<br>19:59:05.654473 recvfrom(3, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 4096, 0, NULL, NULL) = 1024<br>19:59:05.654506 recvfrom(3, 0x55d23c5b5010, 4096, 0, NULL, NULL) = -1 ECONNRESET (Connection reset by peer)<br>19:59:05.654575 write(1, "Read 128000 bytes, final return "..., 60) = 60<br>19:59:05.654694 write(4, "Connection reset by peer\n", 25) = 25<br>19:59:05.654725 close(4) = 0<br>19:59:05.654750 close(3) = 0<br>19:59:05.654783 exit_group(0) = ?<br>19:59:05.654864 +++ exited with 0 +++
Nothing out of the ordinary here, either. We...