Someone pointed out that it isn’t sufficient to check for whether the pointer math wrapped, but that we also need to check that the resulting pointer is in our buffer. They then came to the possibly erroneous conclusion that really all you had to do was to check whether the resulting index was in range. The real problem with this is that there’s so many different scenarios that I don’t see a one size fits all technique.
Depending on the code involved, it may not be sufficient to just know you’re in the buffer. For example, bitmaps have 2 different ways to determine where to start drawing (scan0). Top left corner is easy – it’s 0,0. Bottom left corner is harder – it’s pixel 0 of the last line. So if I calculated scan0, and it ended up in some random place in the buffer, and then I started drawing up from there, I’d eventually start drawing my bitmap header and pieces of the heap and stack. This might lead to ways to overcome ASLR or other mischief.
Where the comment is absolutely correct is that if we’re just doing a simple de-reference (get the nth element), then all we have to do is determine if n is somewhere in the array, and there’s no need to do any complicated pointer math checking. If 0 <= n < max_elements, then we’re in great shape. One would hope that this isn’t the sort of code where we’d be checking if the pointer wrapped anyway. In other cases, I might want to do something more complicated, like calculate where to access some element given a starting point of a variable length header, and the element is in some structure following that header. To use a contrived example, figure out the pointer to the SID contained in the 5th ACE of a SECURITY_DESCRIPTOR’s DACL – ick. This actually brings up a reasonable example – say someone stored a self-relative security descriptor in a file (Excel does this), and you’ve found out that passing a “malformed” security descriptor to random Windows APIs resulted in a non-exploitable crash. Now go write code to completely validate a security descriptor and all of the associated sub-structures. You have 4 possible elements following the header, and you have to check not only that each of them start in the buffer, but also that none of them go outside the buffer, and worse yet, that none of them overlap. This is quite tricky, since a DACL is a variable sized struct that contains a variable number of ACEs, which are also variable in size, and the ACEs contain SIDs, which is why they vary in size, and so on. Got to be quite a bit of code. The Windows APIs along the lines of IsValidSecurityDescriptor() are of very limited help in this area.
While there are some simple cases that can be done easily, it is a hard requirement that pointer math must be mathematically correct AND the result makes sense in terms of the buffer you’re dealing with. And as we now know, we also have to worry about doing it in such a way that we don’t run into undefined behavior according to the language.