Applied Reverse Engineering Series - Reverse Engineering

On

July 24, 2019

By<br>daax

Applied Reverse Engineering Series

Posts

Series Overview

This series is intended for readers who are interested in reverse engineering but have only opened a debugger a handful of times. If you have trouble with certain concepts of reverse engineering, tooling, disassembly or debugging then you’ve come to the right place. Starting from the ground up we’ll work our way to advanced topics that aid in automating the reversal process such as heuristic analysis using a disassembly engine and return-oriented programming. If you’re new it’s recommended you start from the first article and work your way through the series, as it’s meant to guide you through the intricacies of the architecture and operating system structures. This series does expect the reader to have prerequisite knowledge of a native programming language such as C, C++, Rust, etc. Native meaning compiled to a native machine language, as opposed to interpreted. I do not cover reverse engineering Java Byte Code . If you don’t have a background in a compiled programming language this series may be confusing and esoteric. Otherwise, you’re in good hands!

This series is written for reverse engineering on a 64-bit Windows OS . Windows 10 will be the OS that the author is working in, and all examples will be relevant to Windows 10 and the Intel64/AMD64 architecture. You’ll certainly be able to take what you learn from this series and apply it to other architectures and operating systems, however, you’ll have to adapt to any changes present on those platforms. Also worth noting that I will address 64-bit Assembly in detail with a small subsection regarding 16-bit and 32-bit assembly to help solidify the readers’ understanding of x64 Assembly.

All that being said, if you’re familiar with reverse engineering and interested in a specific topic then feel free to skip around, and visit the sections you find most interesting! It’s by no means linear, but if you’re starting out then going in order will be much less confusing.

Note: The documentation referenced will be the Intel and AMD SDM, among other books, articles, and blogs.

I’ve decided on this series that, in order to reduce the length of my articles, I’m going to cover topics in their own separate post. They will be linked here so they’re easy to find from the main navigation bar on the left side of the site.

Index

— The Foundation

ð Basic Architecture – This article will cover general-purpose registers, RFLAGS, data types, and a few examples of assembly at work using status flags.

ð The Stack – Addresses the stack layout, stack operations, the purpose of a stack, calling conventions, alignment, and stack faults.

ð Exceptions and Interrupts – The basics of exceptions, how software-generated and hardware generated exceptions are handled, the most common exceptions, SEH, VEH, and the role of the OS.

ð Accelerated Assembly Part 1 | Part 2  – Covering x64 assembly from simple operations, conditional compares and jumps, to bit shifting and stack manipulation.

— High-Level Structure Redefinition

Following learning how to identify key instruction sequences you’ll refine your skills by learning how to identify structure accesses in IDA Pro and x64dbg . You’ll learn how to take the low-level operations and construct a high-level view of the structure being modified. We won’t be using a decompiler until after we’ve covered how to do it from the plain and dry dead-listing . Once we get to using the decompiler I’ll illustrate how to transfer that high-level reconstruction to IDA to build a more accurate and understandable representation of what the code is doing.

— Heuristic Analysis

Learning heuristic analysis and how to write your own analytical processes can make reverse engineering much easier by determining how a specific process interacts with the OS and executes in general. It can reduce the amount of time required to hunt for pointers after updates, or aid in decryption during runtime operations. In this section, now having a background on reverse engineering, we’ll leverage a disassembly library to perform heuristic analysis on an example application. We’ll develop a tool that uses heuristics to automate the identification of code sequences, and perform behavioral analysis based on how the application interacts with the OS. Knowing how to perform heuristic analysis on a target is an invaluable skill for any reverse engineer.

— Hooking Techniques

There’s a crazy amount of ways to hook something, so in this article, we’ll cover the 3 major techniques used for hooking functions in applications. If you’re unsure what a hook is, don’t worry we’ll address that too as well as it’s purpose. We’ll discuss a more advanced hooking technique provided with an example and further reading material for those interested. In the section covering reverse engineering classes and the vtable we’ll introduce one more hooking...