The given method is intended for protection against attacks with substitution of the function return address. The essence of its work follows from the reason of occurrence practically all buffer overflow vulnerabilities. This reason consists in use of vulnerable C string functions.
For example, the following functions: strcpy, strcat, sprintf, gets and others are potentially vulnerable. There is recommendations to replace the specified functions on similar with the length control (strncpy, strncat, snprintf). The given method provides replacement of the given functions in a binary file but not in source code. Let's note, that the string length control functions have one additional argument - maximal string length that is used to boundaries out check. However, the correctness of this argument is supervised by the programmer, which easily can be mistaken. Besides the additional argument will require to change calls of all specified functions, that for binary file is practically not sold.
Basing on specified the new algorithm of substituted functions was developed which allows to detect buffer overflow without use of additional arguments. It is algorithm is based on a standard prologue of the subroutines having local variable. So for function of the following kind:
void func(void)
{
char buf[30];
gets(buf);
}
The following prologue will be generated:
push ebp
mov ebp,esp
sub esp,30
Where sub esp, 30 - reserves the buffer for a string in stack, and ebp - contains local variable frame pointer.
Subsequently reference to buf occurs by means of the ebp register,
for example: lea eax, [ebp - 30] - putss the buf address in the eax register.
It is obvious, that stack will have structure shown in the following figure.
Stack structure for function with local variable
Thus, proceeding from contents of the local variable frame base, i.e. contents of the EBP register, it is possible to define an arrangement of the return address:
The return address = [EBP + 4]
The knowledge of the address of return will allow to check up its integrity till execution of usual vulnerable functions. So, we receive the following algorithm of work of substituted function:
1. Get and save return address
2. Execute required vulnerable function
3. Get the return address
4. Check up integrity of the return addresses
In case of integrity conflict the message of protection infringement is generated. Note, that overflowed buffer can be situated not in the current local variable frame but in one of previous.
Stack structure with the several local variable frames
In such case it is necessary to pass chain of the frames and to choose the first frame, which base has the address the greater addresses of a string that is parameter of vulnerable function. For example, for a situation represented on figure, the address of the current frame is less than the address of the string, so the following frame is necessary to choose and etc. Let's note also, that some compilers optimize a code so, that frame base register is not used, but the relative offsets based on current stack pointer are applied. Therefore for a code optimized under such circuit the given method will not function.