API Hooking with Detours Cookbook (Part 1)

Introduction

The subject of Windows API Hooking has been addressed in many tutorials and it is not our intention to once again present the theory behind Windows API Hooking. If the reader would like to review that theory an excellent presentation of the topic can be found here.

Why would one want to hook the Windows API? One reason that readily comes to mind is to be able to monitor what an application is doing. In our case we will be hooking a few functions from the Windows 2 Sockets library which is used to create network capable applications. These applications can either be a server based application or a client based application. In our case we will be looking at our favourite client application which just happens to be the web browser.

Compiling the Detours Library

The Detours Library as defined on the Microsoft Detours web site as follows:

"Detours is a library for instrumenting arbitrary Win32 functions on x86, x64, and IA64 machines. Detours intercepts Win32 functions by re-writing the in-memory code for target functions. The Detours package also contains utilities to attach arbitrary DLLs and data segments (called payloads) to any Win32 binary.

Detours preserves the un-instrumented target function (callable through a trampoline) as a subroutine for use by the instrumentation. Our trampoline design enables a large class of innovative extensions to existing binary software."

The Detours Library is released in source code form and in order to use it we will have to compile the library. We will use the nmake application that comes with Microsoft Visual C++ 2005 Express to achieve this.

In order to move on we will need to download the Detours Library from the Microsoft Detours web site. In addition we will also require Microsoft Visual C++ 2005 Express and the Windows Server 2003 R2 Platform SDK Full Download .

After you have completed the downloads the next step is to install install Microsoft Visual C++ 2005, the Windows Server 2003 R2 Platform SDK, and the Detours Library.

In order to compile the Detours Library we will have to first of all open the "Visual Studio 2005 Command Prompt" as depicted.



Once you are at the command prompt change to the "C:\Program Files\Microsoft Platform SDK for Windows Server 2003 R2" directory. Type "setenv.cmd" to create the environmental variables that are required to use the Windows Server 2003 R2 Platform SDK. The Windows Server 2003 R2 Platform SDK provides components that are required to successfully compile the Detours Library.



The next step is to change to the "C:\Program Files\Microsoft Research\Detours Express 2.1" directory and type nmake. You will now be able to successfully build the Detours Library.




Hooking the Windows Sockets Functions

As was said earlier the Windows 2 Sockets library is used to create network aware applications.

The network aware applications typically use the WSASend() or the send() function calls to send data over the network and the WSARecv() or the recv() function calls to receive data from the network.

The network aware applications can also use the gethostbyname() function call to look up a host via name. In other words we would use the gethostbyname() function call to look up www.goolgle.com and get back the ip address for the server www.google.com. This is done because in Windows Sockets programming we will be using the ip address to communicate between hosts on the network and not the names of the hosts.

In this example we will be using the Detours Library to intercept and print the data sent by the following Windows 2 Sockets functions and these are:

1. gethostbyname()
2. WSASend()
3. send()
4. WSARecv()
5. recv()

The interception of these Windows 2 Sockets functions requires the creation of a DLL that will be injected into the target process. We will be using the withdll utility that comes with the Detours Library to inject our DLL into the target web browser process.

The DLL can be created using Microsoft Visual C++ 2005 Express. However before you start creating the DLL you must follow these instructions on how to integrate the Windows Server 2003 R2 Platform SDK with Microsoft Visual C++ 2005 Express.

From the tools menu in Visual C++ Express select Options. From the Options dialog box, expand the Projects and Solutions node and do the following:

• Add the path C:\Program Files\Microsoft Research\Detours Express 2.1\include to the Include files
• Add the path C:\Program Files\Microsoft Research\Detours Express 2.1\lib to the Library files

The DLL can be created by choosing File->New->Project from the the Visual C++ 2005 Express File menu option. Give the Project a name and then click the OK button.



In the new window select Application Settings and click on the DLL option. Click the Finish button.



When a DLL is injected in a process the DllMain() entry point is called along with the reason for the call. In our case we are interested in two reasons for calling the DllMain() entry point and these are DLL_PROCESS_ATTACH, and DLL_PROCESS_DETACH.

The DLL_PROCESS_ATTACH will be used to inject our code into the web browser process and the DLL_PROCESS_DETACH will be used to remove our code from the web browser process.

In our project the DllMain() entry point will look something like this:

BOOL WINAPI DllMain(HINSTANCE hinst, DWORD dwReason, LPVOID reserved)
{
LONG error;

if (dwReason == DLL_PROCESS_ATTACH) {

DetourRestoreAfterWith();

DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourAttach(&(PVOID&)Real_gethostbyname, Hook_gethostbyname);
DetourAttach(&(PVOID&)Real_WSASend, Hook_WSASend);
DetourAttach(&(PVOID&)Real_send, Hook_send);
DetourAttach(&(PVOID&)Real_WSARecv, Hook_WSARecv);
DetourAttach(&(PVOID&)Real_recv, Hook_recv);

error = DetourTransactionCommit();

if (error == NO_ERROR) {
OutputDebugString(TEXT("testhooks.dll: Attach to functions called: \n"));
}
else {
OutputDebugString(TEXT("testhooks.dll: Error attaching to fucntions: \n"));
}



}

if (dwReason == DLL_PROCESS_DETACH) {

DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourDetach(&(PVOID&)Real_gethostbyname, Hook_gethostbyname);
DetourDetach(&(PVOID&)Real_WSASend, Hook_WSASend);
DetourDetach(&(PVOID&)Real_send, Hook_send);
DetourDetach(&(PVOID&)Real_WSARecv, Hook_WSARecv);
DetourDetach(&(PVOID&)Real_recv, Hook_recv);

error = DetourTransactionCommit();

OutputDebugString(TEXT("testhooks.dll: Detach from functions called: \n"));

}

return TRUE;
}

In the DLL_PROCESS_ATTACH we use DetourAttach() to replace the real gethostbyname() function with our hooked gethostbyname() function. In the DLL_PROCESS_DEATCH we use DetourDetach() to reverse the hooks.

We also need to define the functions that we need to hook and these are defined as follows:

// Functions to be Hooked/Detoured

hostent* (WINAPI __stdcall * Real_gethostbyname)(const char* a0) = gethostbyname;

int (WINAPI __stdcall * Real_WSASend)(SOCKET a0,
LPWSABUF a1,
DWORD a2,
LPDWORD a3,
DWORD a4,
LPWSAOVERLAPPED a5,
LPWSAOVERLAPPED_COMPLETION_ROUTINE a6)
= WSASend;

int (WINAPI __stdcall * Real_send)(SOCKET a0, CONST char* a1, int a2, int a3) = send;

int (WINAPI __stdcall * Real_WSARecv)(SOCKET a0,
LPWSABUF a1,
DWORD a2,
LPDWORD a3,
LPDWORD a4,
LPWSAOVERLAPPED a5,
LPWSAOVERLAPPED_COMPLETION_ROUTINE a6)
= WSARecv;

int (WINAPI __stdcall * Real_recv)(SOCKET a0, char* a1, int a2, int a3) = recv;

What we have done here is to create our own definitions for the real gethostbyname(), WSASend(), send(), WSARecv(), and recv(). The Detours Library will actually replace the calls to these functions by calls to our hook_gethostbyname(), hook_WSASend(), hook_send(), hook_WSARecv(), and hook_recv() in the DLL_PROCESS_ATTACH.

The other thing that we need to ensure is done is that we include the correct headers and libraries and this should be as follows:

#include "stdafx.h"

#if _MSC_VER >= 1300
#include
#endif
#include
#include "detours.h"


#ifdef _MANAGED
#pragma managed(push, off)
#endif

#pragma comment(lib, "detours.lib")
#pragma comment(lib, "detoured.lib")
#pragma comment(lib, "ws2_32.lib")

The winsock2.h and the detours.h headers need to be included in the project. We also need to include the detours.lib, and the detoured.lb libraries as these are required to correctly use the Detours Library. We need the ws2_32.lib library in order to correctly hook the Winsock 2 Sockets fucntions.

The complete code listing for the DLL is as follows:

// hooktest.cpp : Defines the entry point for the DLL application.
// This application is used to hook the following Windows 2 Sockets functions:
// 1) gethostbyname()
// 2) WSASend()
// 3) send()
// 4) WSARecv()
// 5) recv()
// written by zenerview [at] gmail[dot]com


#include "stdafx.h"

#if _MSC_VER >= 1300
#include
#endif
#include
#include "detours.h"


#ifdef _MANAGED
#pragma managed(push, off)
#endif

#pragma comment(lib, "detours.lib")
#pragma comment(lib, "detoured.lib")
#pragma comment(lib, "ws2_32.lib")

#define DATA_BUFSIZE 32768


/*
Fake export function
*/
extern "C" __declspec(dllexport) void fakeexportfunction()
{
return ;
}

// Functions to be Hooked/Detoured

hostent* (WINAPI __stdcall * Real_gethostbyname)(const char* a0) = gethostbyname;

int (WINAPI __stdcall * Real_WSASend)(SOCKET a0,
LPWSABUF a1,
DWORD a2,
LPDWORD a3,
DWORD a4,
LPWSAOVERLAPPED a5,
LPWSAOVERLAPPED_COMPLETION_ROUTINE a6)
= WSASend;

int (WINAPI __stdcall * Real_send)(SOCKET a0, CONST char* a1, int a2, int a3) = send;

int (WINAPI __stdcall * Real_WSARecv)(SOCKET a0,
LPWSABUF a1,
DWORD a2,
LPDWORD a3,
LPDWORD a4,
LPWSAOVERLAPPED a5,
LPWSAOVERLAPPED_COMPLETION_ROUTINE a6)
= WSARecv;

int (WINAPI __stdcall * Real_recv)(SOCKET a0, char* a1, int a2, int a3) = recv;


// Functions that replace Hooked/Detoured version

hostent* WINAPI __stdcall Hook_gethostbyname(const char* a0)
{
struct hostent* rv;
struct in_addr realaddr;
char hookmsg[DATA_BUFSIZE];
wchar_t dbgmsg [sizeof(hookmsg)*2];
__try {
rv = Real_gethostbyname(a0);
} __finally {

memset(&realaddr, 0, sizeof(struct in_addr));
realaddr.s_addr = *(u_long *) rv->h_addr_list[0];
_snprintf_s(hookmsg, sizeof(hookmsg), "Real_gethostbyname(,%s,,) -> %s\n", a0, inet_ntoa(realaddr));
hookmsg[sizeof(hookmsg) - 1] = 0;
MultiByteToWideChar(CP_ACP, 0, hookmsg, sizeof(hookmsg), dbgmsg, sizeof(dbgmsg));
OutputDebugString(dbgmsg);

};
return rv;
}


int WINAPI __stdcall Hook_WSASend(SOCKET a0,
LPWSABUF a1,
DWORD a2,
LPDWORD a3,
DWORD a4,
LPWSAOVERLAPPED a5,
LPWSAOVERLAPPED_COMPLETION_ROUTINE a6)
{
int rv = 0;
char hookmsg[DATA_BUFSIZE];
wchar_t dbgmsg [sizeof(hookmsg)*2];
__try {
rv = Real_WSASend(a0, a1, a2, a3, a4, a5, a6);
} __finally {

_snprintf_s(hookmsg, sizeof(hookmsg), "%p: Real_WSASend(,%s,,) -> %x\n", a0, a1->buf, rv);
hookmsg[sizeof(hookmsg) - 1] = 0;
MultiByteToWideChar(CP_ACP, 0, hookmsg, sizeof(hookmsg), dbgmsg, sizeof(dbgmsg));
OutputDebugString(dbgmsg);
};
return rv;
}

int WINAPI __stdcall Hook_send(SOCKET a0, char* a1, int a2, int a3)
{

int rv = 0;
char hookmsg[DATA_BUFSIZE];
wchar_t dbgmsg [sizeof(hookmsg)*2];
DWORD threadid = GetCurrentThreadId();
__try {
rv = Real_send(a0, a1, a2, a3);
} __finally {
if (rv == SOCKET_ERROR) {
int err = WSAGetLastError();

_snprintf_s(hookmsg, sizeof(hookmsg), "%p: Real_send(%d,,,) -> %x (%d)\n", a0, threadid, rv, err);
hookmsg[sizeof(hookmsg) - 1] = 0;
MultiByteToWideChar(CP_ACP, 0, hookmsg, sizeof(hookmsg), dbgmsg, sizeof(dbgmsg));
OutputDebugString(dbgmsg);
}
else {

_snprintf_s(hookmsg, sizeof(hookmsg), "%p: Real_send(%d,%s,,) -> %x\n", a0, threadid, a1, rv);
hookmsg[sizeof(hookmsg) - 1] = 0;
MultiByteToWideChar(CP_ACP, 0, hookmsg, sizeof(hookmsg), dbgmsg, sizeof(dbgmsg));
OutputDebugString(dbgmsg);
}
};
return rv;
}

int WINAPI __stdcall Hook_WSARecv(SOCKET a0,
LPWSABUF a1,
DWORD a2,
LPDWORD a3,
LPDWORD a4,
LPWSAOVERLAPPED a5,
LPWSAOVERLAPPED_COMPLETION_ROUTINE a6)
{
int rv = -1;
char hookmsg[DATA_BUFSIZE];
wchar_t dbgmsg [sizeof(hookmsg)*2];

__try {

rv = Real_WSARecv(a0, a1, a2, a3, a4, a5, a6);
} __finally {
if (rv == 0) {

_snprintf_s(hookmsg, sizeof(hookmsg), "%p: Real_WSARecv(,%s,,) -> %x\n", a0, a1->buf, rv);
hookmsg[sizeof(hookmsg) - 1] = 0;
MultiByteToWideChar(CP_ACP, 0, hookmsg, sizeof(hookmsg), dbgmsg, sizeof(dbgmsg));
OutputDebugString(dbgmsg);

}

};
return rv;
}

int WINAPI __stdcall Hook_recv(SOCKET a0, char* a1, int a2, int a3)
{
int rv = 0;
char hookmsg[DATA_BUFSIZE];
wchar_t dbgmsg [sizeof(hookmsg)*2];
DWORD threadid = GetCurrentThreadId();

__try {
rv = Real_recv(a0, a1, a2, a3);
} __finally {

_snprintf_s(hookmsg, sizeof(hookmsg), "%p: Real_recv(%d,%s,,) -> %x\n", a0, threadid, a1, rv);
hookmsg[sizeof(hookmsg) - 1] = 0;
MultiByteToWideChar(CP_ACP, 0, hookmsg, sizeof(hookmsg), dbgmsg, sizeof(dbgmsg));
OutputDebugString(dbgmsg);
};
return rv;
}




BOOL WINAPI DllMain(HINSTANCE hinst, DWORD dwReason, LPVOID reserved)
{
LONG error;

if (dwReason == DLL_PROCESS_ATTACH) {

DetourRestoreAfterWith();

DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourAttach(&(PVOID&)Real_gethostbyname, Hook_gethostbyname);
DetourAttach(&(PVOID&)Real_WSASend, Hook_WSASend);
DetourAttach(&(PVOID&)Real_send, Hook_send);
DetourAttach(&(PVOID&)Real_WSARecv, Hook_WSARecv);
DetourAttach(&(PVOID&)Real_recv, Hook_recv);

error = DetourTransactionCommit();

if (error == NO_ERROR) {
OutputDebugString(TEXT("testhooks.dll: Attach to functions called: \n"));
}
else {
OutputDebugString(TEXT("testhooks.dll: Error attaching to fucntions: \n"));
}



}

if (dwReason == DLL_PROCESS_DETACH) {

DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourDetach(&(PVOID&)Real_gethostbyname, Hook_gethostbyname);
DetourDetach(&(PVOID&)Real_WSASend, Hook_WSASend);
DetourDetach(&(PVOID&)Real_send, Hook_send);
DetourDetach(&(PVOID&)Real_WSARecv, Hook_WSARecv);
DetourDetach(&(PVOID&)Real_recv, Hook_recv);

error = DetourTransactionCommit();

OutputDebugString(TEXT("testhooks.dll: Detach from functions called: \n"));

}

return TRUE;
}

#ifdef _MANAGED
#pragma managed(pop)
#endif



We can now go ahead and compile our DLL.

In order to inject the hooktest.dll into a web browser process we will use the withdll utility. This can be found in the "C:\Program Files\Microsoft Research\Detours Express 2.1\bin\" directory. We will also require the detoured.dll which is also in that directory as well.

The withdll utility requires that we create an export function in our dll or it will not load in the web browser process. We have created such a function in our DLL and it is as follows:

/*
Fake export function
*/
extern "C" __declspec(dllexport) void fakeexportfunction()
{
return ;
}

In our versions of the hooked functions we use the OutputDebugString() function call to display the data that is either sent or recived by the web browser. We will be using the DebugView application to see the output messages from the OutputDebugString() function.

Open the command prompt and type the following command "C:\Program Files\Microsoft Research\Detours Express 2.1\bin\withdll" -d:C:\Devel\hooktest\debug\hooktest.dll "C:\Program Files\Mozilla Firefox\firefox.exe" .

This will start up the firefox browser and inject our hooktest.dll and the detoured.dll in firefox web browser. The output of the gethostbyname(), WSASend(), and the WSARecv() fucntion calls are shown in the DebugView application below.