Update for VSTO 2005 SE released

  • Article

We have recently released an update for VSTO 2005 SE design time support on Vista. The KB article is available here and does not explicitly say what exactly was broken or what fix has been made – it simply states the symptoms.

On a Windows Vista-based computer, add-ins that are created by using Microsoft Visual Studio 2005 Tools for the Microsoft Office System Second Edition (VSTO 2005 SE) do not run in Microsoft Office 2003 Professional. This problem occurs when you try to debug one of these add-ins in Visual Studio 2005.

In KB article it is difficult to state that the issue in hand was quite interesting to deal with and reflects on certain aspects of deploying COM applications on Vista. So, let's start looking at the issue at hand and then I will generalize with more broad deployment insights.

First of all, Visual Studio writes certain registry keys when VSTO 2005 SE add-in is compiled. The goal of writing those registry keys is:

  1. Add a registry entry that tells Office there is an add-in it needs to instantiate.
  2. Register a COM component that Office will need to instantiate when initializing the add-in using standard CoCreateInstance API (this part is only true for Office 2003 since Office 2007 can natively recognize and instantiate VSTO add-ins).

Being good citizens we have decided that we do not want to require from our developers to run Visual Studio with administrative privileges hence those registry entries are all written under HKCU registry hive. Specifically the COM component related registry entries are written under HKCU\Software\Classes registry hive.

This registration model works quite well on Windows XP as well as on Windows Vista. The problems begin when the hosting process (e.g. Outlook, Excel, Word etc) is being run with elevated privileges or when User Access Control (UAC) is turned off. On Windows Vista this means that process receives High Integration Level. Junfeng Zhang points that on Vista COM is blind to HKCU\Software\Classes hive when IR is greater than MEDIUM. So, VSTO add-ins failed to run when host processes where executed in elevated mode.

Now, why would anyone run Excel in elevated mode while debugging it? The problem is that elevation happens unintentionally – the recommendation for running Visual Studio 2005 on Vista is to elevate it. Hence, any processes spawn by Visual Studio 2005 (which happens when you process F5 or CTRL+F5) and lack Vista activation manifests (and Office 2003 is old enough to have no idea what these manifests are) will be executed in elevated mode as well.

So, back to the update for VSTO 2005 SE this article started with. To address the problems above we have changed the location we are writing the COM-related registry keys to. During compilation we will attempt to put the registry keys under HKLM\Software\Classes and, if it fails due to insufficient privilege, will resort back to HKCU\Software\Classes. That's all this update is about.

Few words of caution, we did not actually make any modifications to the VSTO add-in satellite setup project. The way we still construct the setup project – it will always write COM-related registry keys under HKCU\Software\Classes. This may break the add-in on Vista for Office 2003 applications for the reasons described above. Hence you might want to modify what you get by default and move the COM-related registry keys under HKLM hive.

Additionally, I would caution those who use Windows Installer against putting their registry keys under HKEY_CLASSES_ROOT (HKCR) hive in general. This hive is a virtual one and is essentially a merged HKCU\Software\Classes and HKLM\Software\Classes hives. Windows Installer chooses which one of the two hive to use depending on the type of the installation – it will write into HKLM for machine-wide installations and into HKCU for per-user installs. And as we now know, the latter will not work well on Vista when running in High Integrity Level.