Tuesday, March 23, 2010

Reverse Engineering Through Inline Hooking

Reverse Engineering techniques are generally divided into two broad categories:
1. Static Analysis
2. Dynamic Analysis

Static Analysis
-Techniques which do not involve running the code
-Disassembly, file structure analysis, strings, etc.

Dynamic Analysis
-Techniques which involve running the code
-Behavioral analysis

Approaches to Dynamic analysis involve:
-Network Monitoring
      Isolated Physical Networks
      Virtual Networks
-Hardware Emulation
      Norman Sandbox, etc.
-Kernel-Level Monitoring (SSDT hooks)
      Sysinternal Process Monitor

Kernel-Level Monitoring

      Captures every system call
      Can’t be avoided from userland
      Only captures functions implemented as system calls
      Not every important function call in the Win32 API is implemented as a system call
      Tools don’t differentiate between process housekeeping and calls from usercode
      Calls to internal DLL’s cannot be observed

Process Monitoring via Debugging
      Debugger can trap any function call, not just system calls
      Trapped calls are more likely to be highly relevant to the program’s operation
      Have to act as a debugger
      Susceptible to countless anti-debugging techniques

Inline Hooks
      Can trap any function call, not just system calls
      Trapped calls are more likely to be highly relevant to the program’s operation
      Not operating as a debugger
      No device driver required
      Hard to implement
Implementing Inline Hooks
1. Find a function of interest
2. Disassemble the beginning of the function
3. If possible, overwrite the beginning bytes of the function with a jump or call instruction
4. Implement a handler for the hooked function

What to do with hooked functions?
Observe and Report
      Collect data about the current function call by gathering data from stack and report to console
      Execute any instructions overwritten from the hook
      Jump back to the next instruction in the hooked function
Intercept and Emulate
      Perform a specified action instead of calling the intended function

Running your own Sandbox
-Trap gethostbyname() to always return a fixed IP address.
-A pseudo-handle interface to allow fake reads and writes to files and netwok sockets. Trap connect() to connection to a pseudo-socket. CreateFile(), ReadFile(), WriteFile(), etc.

API Thief Tool (by mandiant.com)
-Launches target process in a suspended state
-Injects a DLL into the process.
-The Injected DLL hooks all Win32 API functions before the target process is resumed
-API Call monitoring can be used simply with a process monitor-style console
-Embedded python can be used to write custom handlers for specific hooked functions