My debugger had gotten slow recently. Really slow. Slow as in the "takes 20 seconds to hit a breakpoint" slow.
After conversing with some of the debugger devs, it was discovered that there was an invalid directory in the "debug source files" list. One of my co-workers keeps the source for some code that we need on his box, and VS dutifully recorded that location to make my experience better. Unfortunately, when the new version of that component showed up, the directory was no longer there, and VS kept timing out trying to see if the file it wanted was there.
The list is stored as a solution property. Right click on the solution, choose "debug source filse", and you'll find a list of locations that the debugger will search. Remove any cruft from this, and things get back to normal.
This happened to me on Whidbey Beta 1. I'm not sure if it's present in Everett. The debugger team is thinking of (I'd say "planning to", but I don't want to speak for them...) adding some code to stop trying the same non-existant location over and over.