GetWindowLongPtr(hwnd, GWLP_WNDPROC) [or GetWindowLong(hwnd, GWL_WNDPROC) if you haven't yet made your code 64-bit compatible] is supposed to return the current window procedure. Why do I sometimes get wacko values?
Because sometimes "you can't handle the truth".
If the current window procedure is incompatible with the caller of GetWindowLongPtr, then the real function pointer cannot be returned since you can't call it. Instead, a "magic cookie" is returned. The sole purpose of this cookie is to be recognized by CallWindowProc so it can translate the message parameters into the format that the window procedure expects.
For example, suppose that you are running Windows XP and the window is a UNICODE window, but a component compiled as ANSI calls GetWindowLong(hwnd, GWL_WNDPROC). The raw window procedure can't be returned, because the caller is using ANSI window messages, but the window procedure expects UNICODE window messages. So instead, a magic cookie is returned. When you pass this magic cookie to CallWindowProc, it recognizes it as a "Oh, I need to convert the message from ANSI to UNICODE and then give the UNICODE message to that window procedure over there."
As another example, suppose that you are running Windows 95 and the window was created by a 32-bit component, but a 16-bit component calls GetWindowLong(hwnd, GWLP_WNDPROC). Again, the raw 32-bit window procedure can't be returned since the message needs to be converted from 16-bit to 32-bit. (And besides, a 16-bit program can't jump to a 32-bit flat address.) So instead, again, a magic cookie is returned which CallWindowProc recognizes as a "Oh, I need to convert the message from 16-bit to 32-bit and then give the converted message to that window procedure over there."
(These conversions are known as "thunks".)
So remember, the only things you can do with the values you get from GetWindowLongPtr(hwnd, GWLP_WNDPROC) are to (1) pass them to CallWindowProc, or (2) pass them back to SetWindowLongPtr(hwnd, GWLP_WNDPROC).