Integer Overflow and operator::new

  • Article

As Raymond Chen pointed out last year (https://blogs.msdn.com/oldnewthing/archive/2004/01/29/64389.aspx), there is a potential integer overflow when calling operator::new. The C++ compiler in Visual Studio 2005 automatically generates defensive code to mitigate this potential vulnerability.

 

Code like this:

 

class CFoo {

public:

      CFoo() {m_p = NULL;}

private:

      void *m_p;

};

void *func(size_t count) {

      if (!count) return NULL;

      return new CFoo[count];

}

 

When compiled with VS2005, yields output like this:

 

; _count$ = esi

; 14 : if (!count) return NULL;

  00000 85 f6 test esi, esi

  00002 74 36 je SHORT $LN4@func

; 15 : return new CFoo[count];

  00004 33 c9 xor ecx, ecx

  00006 8b c6 mov eax, esi

  00008 ba 04 00 00 00 mov edx, 4

  0000d f7 e2 mul edx

  0000f 0f 90 c1 seto cl

  00012 f7 d9 neg ecx

  00014 0b c8 or ecx, eax

  00016 51 push ecx

  00017 e8 00 00 00 00 call ??2@YAPAXI@Z ; operator new

 

As you can see (assuming you can read assembly code!) the multiply occurs (mul edx) and then the CL register is set or not set depending on the value if the overflow flag. Now here’s the funky part, ECX will end up being 0x00000000 or 0xFFFFFFFF, and because of the next operation (or ecx) the ECX register will either be 0xFFFFFFFF or the value held in EAX, which is result of the initial multiply. This is then passed to operator::new which will fail in the face of 2^N-1 by returning NULL or throwing an exception.

 

It’s better to fail, and potentially crash the application, than have an integer overflow that may lead to code execution.

 

Of course, the correct solution is get the code right in the first place by performing good arithmetic hygiene, and in this case, restrict the maximum number of objects you’ll create:

 

void *func(size_t count) {

      if (!count || count > MAX_FOO_ALLOWED) return NULL;

      return new CFoo[count];

}