New for Visual Studio 2008 - Code Analysis Policy improvements

One not very well known feature that lights up when connected to a Team Foundation Server, is the ability to enforce that Code Analysis be run before every check-in. Called a Code Analysis Policy, this feature allows your team to find and fix Code Analysis warnings earlier in the product cycle rather than later, where code changes are riskier and more expensive to make.

To add a Code Analysis Policy to a Team Project, see the following topic on the MSDN Library, How to: Add Check-In Policies.

In Visual Studio 2005, while a useful feature, there were a few usability issues that customers ran into time and time again:

1. Migrating the Code Analysis settings stored in the policy on the Team Foundation Server to the individual projects was confusing. At times, I literally sat in amazement as I watched my own colleagues new to Code Analysis Policy struggle to perform the (what should have been an easy) task of finding a menu item.
2. Once the settings had been migrated, users were confused as to why the settings in the individual projects didn't match exactly what was stored in the project. For example, rules turned off in the policy, were still turned on in the project.

Our team actually knew about the confusion that issue 2 would cause when Visual Studio 2005 shipped. Surprisingly as it sounds, this behavior was actually the lesser of two evils. With the time that was available in the ship cycle they knew that they had two distinct choices, either a) have the policy completely override the settings stored in the project, or b) merge the policy settings with the project settings. Because a) prevented the scenario of allowing users to turn on more rules than the policy specified (in effect being stricter than the policy), the later was chosen as the preferred behavior.

Issue 1 was somewhat of a surprise. Code Analysis Policy along with the rest of Code Analysis was a late edition to Visual Studio 2005, probably too late to get any real user feedback and perform any actionable usability studies. We didn't hear about the troubles users were having with this until after the product had already shipped.

Once Visual Studio 2008 planning began, improving both of these issues was one of our top priorities, so we set about a feature crew to fix it.

Code Analysis Policy Failure Details dialog

Clearly users were first of all struggling to actually perform the migration; they were told by the Policy Failure that their projects settings were out-of-date, but not actually how to go about updating them. Even if they somehow figured out that a menu item might do this, looking intuitively on the individual project's context menus didn't help - the menu item that performed the job, Migrate Code Analysis Policy Settings to Solution (what a mouthful!),was hidden under File -> Source Control and on the solution's context menu.

The first thing we did to solve this confusion was to add a new dialog (displayed when double-clicking on a Code Analysis policy failure) that provided extra information about the situation they were in, what projects were out-of-date and how to go about solving it.

We also chose to tell them at this time that their build date was out-of-date - something that we previously only told them after they had updated their project settings and then attempted to check-in again (breaking their workflow).

New Policy Menu Items

The way to solve the second issue was easy; provide the user a choice:

Some users preferred policy to win out and completely overwrite the local project settings, whereas, others wanted to be able to specify additional rules above and beyond what was specified in the policy (ie the current Visual Studio 2005 behavior). The new Analyze menu (which I'll talk about in a future post), now provides two menu items, Replace with Check-in Policy and Merge with Check-in Policy, for performing both of these actions, respectively.

Hopefully these changes will make using Code Analysis Policy a little more pleasant experience in Visual Studio 2008. The new dialog and menu items themselves made into Beta 2 (albeit with slightly different text and names), so install it today and tell what you think.