Static Devirtualization of Tencent VM<br>search icon

to navigate

to select

ESC to close

Static Devirtualization of Tencent VM

Windows

July 31, 2026<br>Table of ContentsPrerequisite Material<br>https://k0mkc.hatenablog.com/entry/2026/07/29/042406<br>https://arxiv.org/abs/2603.18355<br>https://arxiv.org/abs/1909.01752<br>Legal Disclaimer<br>This work constitutes reverse engineering, decompilation, and de-virtualization performed solely for the purpose of achieving interoperability with ACE-protected software on Linux and Proton environments. Such activities are undertaken in good faith under applicable exceptions to copyright law, including those recognizing the right to reverse engineer software for interoperability purposes.<br>Any sharing has been limited exclusively to trusted third parties for independent technical validation. The original protected software remains subject to copyright and the protections of the Digital Millennium Copyright Act (DMCA) and other applicable laws.<br>Introduction<br>Over the course of several months we have noticed an increased interest in Tencent VM obfuscation. We have had complete static devirtualization of this VM obfuscation for quite some time now and have noticed others have achieved similar deobfuscation results. As AI continues to advance it has brought to light the gap between strong obfuscation and simple obfuscation. We have long held the belief that the classic style of “Virtual Machine” obfuscators are not strong to an attacker that has a flexible lifting and recompilation framework. Many companies have attempted to build inhouse obfuscation solutions, most of which are not very strong and are being deobfuscated by private entities. In this article we will explain in great detail the Tencent virtual machine obfuscation, how it supports CET and SEH, and how it is weak against guided symbolic evaluation.<br>In our last article, Static Devirtualization of Themida, a reader asked about coverage statistics and other versioning related questions. Although we do not have version information for Tencent VM, we will provide detailed information about the static devirtualization coverage. A third party (Daax from secret club) has been provided with these files and can independently confirm the cleanliness and coverage of the output. A later section in this article will cover that more formally.<br>Virtual Machine Architecture<br>The Tencent virtual machine obfuscation is structured so that the original function entry point jumps into the .tvm section. Stack space is allocated for the VM context and all GPRs and EFLAGs are saved into this context (first on the stack and then moved into the VM context). The VM is highly reminiscent of VMProtect in this manner, as VMProtect pushes all GPRs to the stack and then the first few VM handlers pop them from the stack and store them in the VM context area. Execution threads in and out of a shared virtual machine dispatch loop. Exits happen to execute calls and boxed instructions.

Boxed Instructions<br>The Tencent VM models only a subset of AMD64. Anything outside that subset is handled by what we refer to as a boxed instruction: the VM performs a context restoration, executes the original instruction natively. At that moment the machine state is indistinguishable from what the unvirtualized function would have produced, so the instruction executes with correct semantics without the VM having to model it at all. Once the instruction retires, the VM captures register state back into the VM context and resumes dispatch. More usefully, every boxed instruction is a point where the VM is forced to materialize real guest state, which makes it a reliable sink point. We rely on exactly this property when recovering the original stack frame size. It is also important to note that a way to execute native instructions in virtualized functions is a practical necessity. Consider for example instructions like CPUID which would be impossible to model inside the VM without actually executing the CPUID instruction. Not to mention other architectural instructions like RDMSR, WRMSR etc.

CET Compatibility<br>Entering the dispatcher with a CALL creates a problem on hardware with Control-flow Enforcement Technology. Under CET a CALL pushes the return address onto a shadow stack in addition to the data stack, and a RET pops it and compares the two. The VM&rsquo;s calls into the dispatcher never return. Left alone, the shadow stack would grow monotonically for the lifetime of the virtualized function and eventually fault.<br>Tencent VM handles this at runtime rather than at protection time. It executes RDSSPQ to read the current shadow stack pointer. RDSSPQ is encoded in the hint-NOP space, so on a processor or in a process without shadow stacks enabled it retires as a NOP and leaves its destination register untouched, zeroing the register beforehand and testing it afterward is therefore a feature check that is correct on old and new hardware alike and costs nothing on either.

When the check...