Installation of VS fails on CA_SetPidProps

This was an interesting one.  I saw two instances of this during VS Beta 2.  It was rare enough that debugging it was nearly impossible.  Unfortunately, the dev on the MS security team that was helping me debug this was pulled back to working on Vista and wasn’t able to find out what the real problem or solution was.

When installing VS, VS fails and in the logs, it fails on the custom action CA_SetPidProps.  Debugging this, it turns out the CryptProtectData API (a Windows Security API) is failing.  During the installation process, we need to encrypt some data and if that fails, VS setup will fail.

Our Product Support Technician just ran into another repro of this issue.  After a week, many phone calls, emails, and a lot of frustration and collaboration with three different teams, he was able to figure out a workaround.

If you run into this issue, look in your c:\windows\system32\microsoft folder.  If there’s a zero byte file in that directory called Microsoft, delete it and then reboot your machine.  Windows will then write the right data and your Crypto APIs should function normally after that.

Huge Kudos to Will for figuring out a workaround.  We still don’t know how the operating system gets into this funky state but atleast we now know of a workaround.




Comments (3)

  1. ohhhmed says:

    I had this problem too. Just after 2005 hit the market, I installed it without any problems. But for some reasons I uninstalled it. Recently, I had the use of 2005 and  I tried reinstalling it, but it kept failing on SetPidProps. I tried stopping services, cleaning temp dirs, deleting VS8 registry keys and so on. Nothing worked, until I for some reason suspected UltraVNC to be the bugger. I uninstalled it and at the moment of writing I’m installing 2005 again :).

    Btw, I didn’t have this zero byte file in that folder described.

  2. DrWatson says:

    Problem finally solved :)

    I had to debug with OllyDBG the CA_SetPidProps custom action down to the CryptProtectData call, to verify that the parameters were correct.

    The parameters *are* correct, and are safe for windows 2000 too (there is the description parameter, optional for winxp+ but mandatory for win2k).

    The problem, in my case, was that the ProtectedStorage service was disabled, but i didn’t spot the problem until I checked the parameters in the MSDN web site, that (not clearly!) mentioned some "protected storage location". So I recalled the name of a service I usually disable, looked for it in the registry and re-enabled it.

    No need to say that now it works, and if ProtectedStorage is disabled VS2005 won’t boot with the classical "Invalid license data" message box.

    Hope it’s useful 😀

  3. raymon says:

    操作结束 12:13:01: CostInitialize。返回值 1。

    MSI (s) (54:E0) [12:13:01:718]: Note: 1: 2235 2:  3: ExtendedType 4: SELECT `Action`,`Type`,`Source`,`Target`, NULL, `ExtendedType` FROM `CustomAction` WHERE `Action` = 'CA_SetPidProps'

    MSI (s) (54:C0) [12:13:01:718]: Invoking remote custom action. DLL: C:WINDOWSInstallerMSI1F4.tmp, Entrypoint: SetPidProps

    操作启动 12:13:01: CA_SetPidProps。

    07/16/10 12:13:01 DDSet_Status: LANGID: 2052

    07/16/10 12:13:01 DDSet_Entry: SetPidProps started

    07/16/10 12:13:01 DDSet_Status: PIDSEQ:

    07/16/10 12:13:01 DDSet_Status: PIDRET:

    MSI (s) (54:C0) [12:13:01:734]: Leaked MSIHANDLE (178774) of type 790531 for thread 332

    MSI (s) (54:C0) [12:13:01:734]: Note: 1: 2769 2: CA_SetPidProps 3: 1

    07/16/10 12:13:01 DDSet_Status: EDITIONTYPE:

    DEBUG: Error 2769:  Custom Action CA_SetPidProps did not close 1 MSIHANDLEs.

    内部错误 2769。CA_SetPidProps, 1

    操作结束 12:13:01: CA_SetPidProps。返回值 3。

    操作结束 12:13:01: INSTALL。返回值 3。

    Property(S): REDISTFOUNDVERLANG = #1

    Property(S): REDISTFOUNDVER = #1