Parallel Reconstruction of Lawful TLS Wiretapping |<br>REMY HAXParallel Reconstruction of Lawful TLS Wiretapping<br>May 30, 2026<br>11 mins read<br>Let's Encrypt<br>Reverse Engineering

Transport Layer Security (TLS) is the protocol involved in getting the lock icon to appear in your browser next to the URL. Under the hood it uses a bunch of really cool numbers for encryption. Some numbers are considered private and need securing; some are considered public and are fine for sharing. You can mix your numbers with other people&rsquo;s numbers in such a way that you can verify a chain of trust. Ultimately, at the top of this chain there has to be an entity or entities that are implied to be trustworthy, so that the links further down the chain of numbers can inherit that trust. This is the role of a root Certificate Authority (CA) at the top (root) of the chain.<br>There is, of course, a lot of nuance and detail missing from this high-level explanation of TLS and CA trust, but rest assured that understanding how things are supposed to work bears little influence on the ability to simply do things anyway.<br>As a baseline, TLS wiretapping (presumably lawful) with root-CA-signed certificates is a thing that both happens and verifiably has happened.<br>Encrypted traffic interception on Hetzner and Linode targeting the largest Russian XMPP (Jabber) messaging service (2023)<br>This being a fact rather than a conspiracy theory tends to upset people. Meanwhile, if you understand the mechanics at play, it&rsquo;s objectively very funny that someone likely forgot to renew the TLS certificate for a lawful intercept, resulting in a huge warning page for users and ultimately prompting the detailed investigation seen in the link above. It&rsquo;s a rather amusing way to burn an operation.<br>In this blog, we&rsquo;ll exercise the benefit that hindsight is 20/20 and further suspend our expectations of how TLS is supposed to work. We&rsquo;ll take a look at the analysis, the recommendations, and the factors in the larger system that was the year 2023, to attempt to answer how it actually could have worked, with a demonstration.<br>Analysis<br>The analysis blog on valdikss.org is extremely detailed, which is particularly useful since these things are almost exclusively only ever seen when an operational mistake occurs. I can&rsquo;t &ldquo;read&rdquo; in the traditional left-to-right sense; it&rsquo;s more like a smattering of a word cloud, and I&rsquo;ve got 30+ years of experience correctly guessing the order. Allow me to demonstrate the value of that visually as we read through an analysis.

Big things are easy to guess the relevance of, but if the mystery were obvious it wouldn&rsquo;t be a mystery. The devil&rsquo;s in the details, and acme.sh (with the arrow pointing to it) is very small.<br>When you process information this way, you lose the ordering of relevance. I typically skim a document deliberately, looking for numbers indicative of time so I can put it together in my head.<br>DateEvent18 Apr 2023Unknown actor begins issuing SSL/TLS certificates25 Apr 2023 - 03 Nov 2023Other stuff happensThe core takeaways: look for events around April 18th, 2023 that may involve acme.sh (that&rsquo;s a pretty clear missing piece) and note the things that happened afterward.<br>acme.sh<br>Remember that chain-of-trust example from the start of the article? ACME is a protocol used to establish trust for the issuance and renewal of TLS certificates from certificate authorities. acme.sh is a shell-script executable that helps automate that process using the ACME protocol. acme.sh is what was running on the jabber.ru server to facilitate their TLS certificate renewals. Typically these run on a timer that calls out and renews a certificate before it expires.<br>Notable events related to acme.sh around April 18th, 2023 include a remote code execution vulnerability disclosed on June 8th, 2023, eventually assigned CVE ID CVE-2023-38198. A patched release was available on Jun 9th, 2023. The jabber.ru server, running acme.sh on April 18th, would have been using a version vulnerable to this exploit.<br>That seems potentially relevant!<br>CVE-2023-38198<br>In the GitHub issue that first disclosed the vulnerability, it was noted that this was being abused by a certificate authority , &ldquo;HiCA&rdquo;, to&mldr; issue a certificate. Across all the observed activity in the GitHub issue, you&rsquo;ll see the chaos that is shell interpolation and dancing around forbidden/filtered characters to do the desired thing the wrong way.<br>There are lots of useful goodies there, but even the examples given are in fact &ldquo;broken&rdquo; for the purpose of reproducing this vulnerability. The nature of the vulnerability is such that the crux of the issue lies between the data on the wire and the representation of that data when processed by the ACME client, including for logging/debug purposes. So while what we see in the debug logs will be close to the original, there has to be stuff missing.<br>@mholt - It turns out...