If you are encountering an empty message box followed by a browse folder dialog in for MLibrary::initialize Open Maya SDK 2008, then the chances are that you can resolve the problem by defining the MAYA_LOCATION environment variable.
Define the MAYA_LOCATION environment variable to point to the Maya installation directory (for e.g. C:\Program Files\Autodesk\Maya2008). The MLibrary::initialize function typically tries to load the dependent Dlls and resources from the sub-directory structure pointed to by MAYA_LOCATION.
Note that you might need to restart the Visual studio (or reboot the machine depending on your OS) to let the updated environment variable value reflect in the application.
Also note that the Maya SDK libraries are release version libraries and are linked with release version of CRT and MFC. You need to be aware of this if you are building your plug-in/application in debug mode. Because in debug mode your plug-in/app code gets linked with debug versions of CRT/MFC, while the linked-in Maya libraries (such as Foundation.lib etc…) are linked with the release versions – which could cause potential memory access violation errors as described at http://msdn2.microsoft.com/en-us/library/ms235460(VS.80).aspx
Finally, if you are trying to build Maya SDK 2008 applications with Visual Studio 2008 then the first thing you would notice is the linker error
LINK : fatal error LNK1104: cannot open file ‘mfc80.lib’
You will not have this error if you have Visual Studio 2005 MFC libraries installed on the machine. This is because mfc80.lib is part of VS2005 and Maya SDK2008 libraires are linked with that version of VS and not with VS2008. You can avoid this error by using preprocessor definition MLIBRARY_DONTUSE_MFC_MANIFEST as indicated at http://www.jb101.co.uk/2008/01/21/maya-2008-sdk-using-mlibrary-when-you-dont-have-the-mfc-libraries-installed/
And when building the AnimEngine application you might encounter “Values.h missing” error. #Including float.h inplace of values.h should resolve it.