"Open-source projects certified as secure" – huh?

I really got a chuckle out of this news item, especially this line:

“Coverity, which creates automated source-code analysis tools, announced late Monday its first list of open-source projects that have been certified as free of security defects.”

So we finally have the security silver bullet!

Run this tool on your code, fix the bugs, and you’re secure (and maybe unbreakable?!) I don’t think so.

There are three big problems with this line of thought:

  1. First, the security bugs found are only the security bugs found by the tool, and that list is always smaller than the list of all bugs.
  2. Second, it assumes that any new code or code changes are bug free. Which may or may not be true. In my experience, it is rarely true that new code is utterly bug free if you don’t take a holistic, process-oriented view to security.
  3. Third, and this is probably the most important, at best the tool understands a subset of today’s vulnerabilities; that could all change tomorrow when a new class of vulnerability or a subtle variant is found.

The last point is important; security is a constantly evolving environment, and that's why we update the SDL regularly, to improve the process as we learn of new threats and design new defenses and mitigations.

Tools are very useful, we build a lot of tools, and use them all the time here at Microsoft. Some of those tools have found their way into our SDKs and Visual Studio so our customers can use them too. But I would never claim that these tools make code "free of security defects."