Monday, September 21, 2009

Useless mov edi,edi in the Prologue

The seemingly useless statement is used to enable hot patching (patching without stopping the component). The 2-byte instruction can be changed to a short jmp operation (within a range of 127 bytes in either direction). To extend the jmp target, NOP statements are generated before the function labels so that a long jmp statement could be patched in:

xor eax,eax
jmp xyz
nop
nop
nop
nop
nop
func-abc:
mov edi,edi
push ebp
mov ebp,esp
:

Frame Point Omission

FPO is an optimization technique. ebp is used as a general purpose register rather than the stack frame base pointer. Execution is sped up by the availability of this additional register.

Call Convention

Stdcall pushes the argument from right to left onto the stack. The called function is responsible to remove the parameters passed in by decrementing the esp by the length of the parameters. Cdecl call convention differs from Stdcall by having the calling function to remove the argument passed from the stack. Stdcall is preferred because the clean up is done one place (no mater how many times it is being called), which is simpler. Cdecl is used for C/C++ because they support variable number of parameters for function call. As the called function will not know the number of parameters beforehand, the clean up has to be performed by the calling function instead.

Linker generates special name for different call conventions. For Stdcall, function name will be prefixed by "_" and appended by "@", follow by the number of bytes of stack space required. For Cdecl, function name is prefixed by "_".

Fastcall uses ecx and edx to pass the first 2 argument. Clean up is by the called function, similar to Stdcall. Function name is prefixed by "@", appended by "@" and followed by the number of bytes of stack space required.

Thiscall passes this point via exc and the rest of arguments on the stack. Clean up is by called function.

Stack Frame

Before calling a function, the caller will first reserved space for the parameters in the stack. For example, assuming the parameters occupies 20 bytes:

sub esp,14h

Following this is a series of mov statment to move the parameters to the stack using offset with ebp. For example,

mov dword ptr [edp-14h],3
:
:

Then the call operation is used to jump to the function. Call will push the eip onto the stack (esp will advance as a result).

At the beginning of the function, the compiler generates a stack frame using the frame base pointer register ebp. The function prologue saves the current ebp onto the stack before setting up a new stack frame:

mov edi, edi
push ebp
mov edp, esp

As a result, the ebp of the new frame points to the old ebp value (the last frame base). The ebp is then used to access the parameter (positive offset) and local variables (negative offset).

At this stage, the call stack contains the following (growing downwards):

parm1
parm2
:
return address
saved ebp of caller
local variable1
local variable2
:

When the function finishes, the epilogue restore the previous stack frame

add esp,14h ; clean up the parameter stack space assuming this is stdcall
mov esp, ebp
pop ebp
ret