Why CryptCATAdminCalcHashFromFileHandle fails with a seemingly unexpected error code

  • Article

CryptCATAdminCalcHashFromFileHandle can fail when evaluating a file path to an executable while setting the last error code to 0x800700c1.  It’s an HRESULT instead of just an error code, but the relevant portion of the error code is 0xc1 (193L).  That is the error code for ERROR_BAD_EXE_FORMAT.  In some cases, this will happen even though the executable can run without an error.  So why would this function return ERROR_BAD_EXE_FORMAT when directly executing the file works without issue?

There are actually several reasons why this error can be returned, and only some of them would actually prevent an executable from being able to run on an unrestricted system.  All of them come from the portable executable (PE) header of the file.  The types used below can be found in the Windows SDK, primarily in winnt.h.  Here are most of the reasons:

  • PIMAGE_DOS_HEADER->e_magic is an invalid value
  • PIMAGE_DOS_HEADER->e_lfanew is an invalid value
  • PIMAGE_NT_HEADERS->Signature is an invalid value
  • PIMAGE_NT_HEADERS->FileHeader.SizeOfOptionalHeader is an invalid value
  • PIMAGE_NT_HEADERS->FileHeader.Machine is an invalid value
  • PIMAGE_NT_HEADERS->OptionalHeader.Magic is an invalid value
  • PIMAGE_NT_HEADERS->OptionalHeader.FileAlignment is an invalid value
  • Any of the populated members of PIMAGE_NT_HEADERS->OptionalHeader.DataDirectory have invalid values
  • The certificate directory (IMAGE_DIRECTORY_ENTRY_SECURITY) has an offset that puts its data in an invalid location; see https://msdn.microsoft.com/en-us/windows/hardware/gg463180 for more details on what the standards for that are.

If you encounter this error on an executable file, these can be identified manually by looking at the output of a couple of tools.  If there’s a problem with the IMAGE_DIRECTORY_ENTRY_SECURITY section, then running SignTool.exe verify /v filename will output “SignTool Error: File not valid: filename”.  The rest of them can be identified by looking through the output of dumpbin.exe.  Dumpbin is available through Visual Studio and SignTool is available through the Windows SDK.

Follow us on Twitter, www.twitter.com/WindowsSDK.