Tuesday, June 26, 2012

Who's fault is this?

Today I ran into an interesting phenomenon when writing a test application in Visual C++. The phenomenon appears to be a bug in Visual Studio 2008 optimizer, which seems to lose an update of a variable. Of course, it is equally possible that the bug is in my code, which does some really ugly stuff.

Consider the following function:

static Status test(int (__cdecl *f)(void *), void *arg,
    Status errstatus)
    Status stat = errstatus;

    __try {
        (void) f(arg);
    } __except((GetExceptionCode() == EXCEPTION_GUARD_PAGE) ?
        ((stat = SUCCESS), -1) : 1) {
        return (Status) -errstatus;

    return stat;
The idea of function test() is to return SUCCESS if and only if function f() triggers a guard page exception (and nothing else). The problem is that it does not work when using more aggressive optimizations.

Let's have a look at this on the assembly language level. Here are the relevant parts of function test(). At its beginning, we see register ESI loaded with the errstatus argument:

011C12C3  mov         esi,dword ptr [ebp+10h]
    Status stat = errstatus;
011C12C6  mov         dword ptr [ebp-1Ch],esi
The compiler moves the value to the stat variable, which lives at address EBP - 0x1c on the stack. The next interesting place to look at is the piece of code which updates stat in the case of the guard page exception:
011C12FF  cmp         eax,80000001h
011C1304  jne         11C1311h
011C1306  mov         dword ptr [ebp-1Ch],7D00h
011C130D  or          eax,0FFFFFFFFh
011C1310  ret             
011C1311  mov         eax,1
011C1316  ret
When the exception code, now stored in EAX is EXCEPTION_GUARD_PAGE, the stat variable on the stack is updated to 0x7D00 (SUCCESS). Because the __except expression evaluates to -1, execution of f() is resumed where it was previously interrupted. We'd expect f() to return to test() and test() to return the value now stored in stat. But looking at the end of function test() in the assembly, this is not how the compiler sees it:
    return stat;
011C12E1  mov         eax,esi
Instead of reading the stat variable from the stack, where the correct value is, the compiler returns the old value it previously loaded to ESI.

My impression is that either I am unintentionally doing something illegal, such as updating a local variable from the filter expression, or the compiler contains a bug which does not take the exception handling code into account when performing optimizations.