Help Make Visual Studio Faster


One of the most difficult things about our job is trying to decipher why Visual Studio is slow for a customer.  Often it starts with a vague complaint (e.g. “Visual Studio is sluggish”) which we then have to narrow down to a particular action that’s slow, and try and get a profile.  Then we have to look through a large profile and figure out which code is running slower than it should be and why.  Sometimes the problem is CPU intensive, sometimes it’s disk or network, sometimes it’s a different program altogether that just happens to be slow.  I know the process is just as frustrating for customers, who have to try and figure out just where it is slow and get us a profile.  And of course, if VS is just a little bit slow, then it’s often much easier to live with it than go through the hassle of trying to help us isolate it.

That’s why I’m pleased to announce that we now have a better approach.  Visual Studio Perf Watson is now available for download and use with VS2010 SP1.  If you download and install Perf Watson it does everything for you (and us).  Here’s how it works:

  1. Perf Watson monitors VS to make sure it is responsive.
  2. If VS goes unresponsive for more than 2 seconds, Perf Watson grabs a stack frame.
  3. Perf Watson then times how long it takes for VS to become responsive again.
  4. Perf Watson then takes the stack information along with the total time of the hang and sends it to Microsoft.
  5. We take the data and load it into a database.
  6. We analyze the data to see which call stacks are causing the longest hangs that impact the most customers, and then we log bugs on those.
  7. Since we have call stacks, the bugs tend to be very actionable.
  8. Since the data comes from real customers, with hit counts and severity information, we have no trouble prioritizing these performance hangs appropriately – we know exactly what they’re costing you.

We’ve been using Perf Watson internally for a while now, and we’ve been able to identify and fix a lot of problems just based on our internal product usage.  We’ve also seen some nice correlation between issues raised by Perf Watson and issues found through other analysis.  This correlation helps us know that we’re on the right track and gives extra weight to getting these issues resolved.

But our usage patterns aren’t the same as yours.  Everyone uses VS a little bit differently.  That’s why we’re excited to put Perf Watson in your hands so that we can get an accurate picture of the performance issues you’re running into, with strong metrics and data to back up the work we need to do.

I hope you’ll decide to download and install Perf Watson. 

Regards,

David Berg
DDPE

 

 

 

 

Comments (20)

  1. Juan Zamudio says:

    Very Nice, extension installed.

  2. Asher says:

    2 seconds????? should be 250-300msec – this is acceptable response time. not 1 second, not 2. 300msec MAX

  3. Graham says:

    I aspire to get 2-3 seconds. 10-20 is often good for me. Extension being installed.

  4. David Berg says:

    @Asher.  It depends on what you're doing.  When you are typing / editing, then I would agree: 100ms is optimal, with 250-350ms being the top end.  However, when selecting options that bring up dialog boxes, do builds, rearrange the screen, etc. – then 400-500ms is generally very good, with 1-2 seconds being acceptable and 4+ seconds being unacceptable.  Since Perf Watson isn't currently able to discern the difference between various system activities, we set the target at 2 seconds, which is the outer threshold of acceptability.  This allows us to initially focus on the more serious issues.  Over time we will look at how we can tighten this down, especially in typing scenarios where we really want typical response times under 100ms.

  5. Judah Himango says:

    I posted about this tool in the Code Project forums: http://www.codeproject.com/Lounge.aspx

    It quickly devolved into a whine-fest about how MS Connect sucks, and how MS doesn't listen to user feedback. Might be worth your time reading the above linked thread.

  6. Matt Fenelon says:

    It'd be great if you could tell the extension that VS is acting slow. It could then start it's process of recording and sending data back to Microsoft. One of the problems I suffer a lot from is how slow VS responds to key presses at times. As an IDE, the performance of typing should be a keen concern.

  7. David Berg says:

    @Matt.  Noted.  We'll be considering something like this as we look at addressing keyboard response rates (as discussed above).

  8. PleaseFixYourBugs says:

    I just found this post.

    The topic of performance of VS2010 is very dear to my heart. The performance issues were one of the two major reasons our team has given up on using VS2010 as an IDE and has moved back to VS2008. That's after VS2010 SP1, which we have been thoroughly disappointed with.

    I hate to say it, but this:

    "I posted about this tool in the Code Project forums: … It quickly devolved into a whine-fest about how MS Connect sucks, and how MS doesn't listen to user feedback."

    …was totally deserved. Me and my team have stopped reporting bugs on Connect, because this goes nowhere, the reports are being left unattended for months, many are frivolously closed as "won't fix" or "by design", etc.

    Now, I am going to install and use the add-in even though VS2010 is no longer my primary IDE. I frankly don't believe that will change anything, having this update is a far cry from having a functioning system where customers report bugs, and developers fix these bugs and publish product updates in a timely manner (Connect fails on all three counts: the updates to VS came with year-sized intervals between them, a great deal of reported bugs are never fixed, consequently many of your customers have stopped reporting bugs to you), but I will still do it as an act of good faith.

  9. David Berg says:

    @PleaseFixYourBugs.  Yes, we can do better on Connect bugs.  The good thing about Connect bugs is that they feed directly into our bug system.  This means they get just as much visibility as any other bug in our system and teams can't just ignore them.  It also means there is a lot of pressure to resolve them, and unfortunately, sometimes teams resolve them won't fix, or by design without adaquately explaining why (although our bug system actually asks them if they responded to the customer before allowing them to resolve the bug, they are supposed to at least provide a good reason).  

    The down side about any external bug is that often what reproduces often and everytime for a customer, doesn't reproduce at all for us due to differences in the environment, projects, or even work styles.  When that happens on an internal bug, we can walk next door and see the repro, that's much tougher when the customer is thousands of miles away (yes, we can and have done remote sessions, but it takes time to setup, and sometimes we have to go through legal to get NDAs signed or jump through other hoops to protect the customer's intellectual property.)

    The big advantage of Perf Watson is that it gives us call stacks, counts, and durations – so we know how many customers are hitting the hang, how bad it is, and where we are in the code when it happens.  With most performance bugs that come in through connect it takes hours of work to figure this out, and that only tells us about one customer, which makes it hard to prioritize.

    We put out fixes more than once a year, but they get less visibility.  They're called QFE's (Quick Fix) or HotFixes and they just address a single problem.  They haven't been through a full test pass though (which takes 100 people a month), so we can't send them out to everyone.  We also have GDR's (General Release) which are more generaly available and much better tested, and SP's (Service Packs) which are majore releases.  You can get a list on connect here: connect.microsoft.com/…/Downloads.

    If you get a chance, I'd love to know what parts of VS2010 performance you're seeing as worst than VS2008, and what types of projects you're working on.  My expectation is that it's probably general slowness caused by increased memory usage, but if it's not, it would be nice to know.

  10. PleaseFixYourBugs says:

    "If you get a chance, I'd love to know what parts of VS2010 performance you're seeing as worst than VS2008, and what types of projects you're working on."

    Sure.

    We are working with relatively large projects, the typical project size is 2 million lines of code, all C++. No Boost or similar libraries. No generated code. Precompiled headers are turned on.

    Our typical hardware is Core i7, 8+ GB RAM, very fast disk (10000 RPM HDD + SSD), latest or next-to-latest graphics card (we need powerful GPUs). Windows 7 x64. No add-ins (well, I use your add-in on one of the machines now).

    A general observation with VS2010 is that in many cases, the performance of a particular function in VS2010 is about the same as in VS2008 *right after you launch VS2010 and open the project*, but quickly degrades after you work with the project for some time (10-20 minutes of editing / compiling / debugging is all it takes for the effect to become visible, 2-3 hours and the performance is nuts).

    Main areas where VS2010 performs much worse than VS2008:

    1. Editor.

    I can outtype VS2010 pretty much right after I load the project. There is a visible lag between the moment I type a key and the moment I see the corresponding character on the screen. This lag gets progressively worse the more I work with VS2010. I couldn't outtype VS2008.

    After about an hour of working with the project, pressing F3 in the window showing a CPP file (Edit.FindNext, no regular expressions, normal line lengths, the text fragment being searched for is in the next three or four lines of text from the cursor) might take 3-5 seconds, sometimes more than that. I have never seen this in VS2008.

    2. Debugger and debugger / editor interaction.

    After some time of working with the project, placing a breakpoint in a regular function might take 5-10 seconds. Placing a breakpoint in a template function with multiple overloads, of course, takes much longer. Same for stepping through the code. This was not the case with VS2008.

    The vast majority of our code is raw, vanilla C++. We do, however, have some code in C++/CLI. Sometimes we have to debug that code and for this we switch the debugger type from "Native Only" to "Mixed". The performance of the debugger in mixed mode is much, much, MUCH worse than the performance in native mode, with our code we are talking about at least 5x difference, even before we place any breakpoints into the C++/CLI portion. This was not the case with VS2008.

    3. Intellisense.

    I stopped waiting for autocomplete suggestions from Intellisense, they don't appear fast enough. It is faster to type the relevant name manually. Sometimes, when I am unsure about the name I am typing, I might let Intellisense finish. In roughly half of these cases, Intellisense displays nothing (a problem, but likely not related to performance), but when it actually displays something, it takes in the range 10 seconds. This was faster in VS2008.

    The usual performance of F12 (Edit.GoToDefinition) is 5-10 seconds. This is already unacceptable, but, again, it gets worse, much worse, up to 20 seconds and more, the more you work with the project. Given that in many cases F12 either does not do anything or takes you to a wrong place, some of my colleagues have unbound that key so that they don't accidentally hit it. This was faster in VS2008.

    There are other areas, but these are the main ones.

  11. PleaseFixYourBugs says:

    Also, I have browsed the list of QFEs. I do not see any items related to the performance of VS2010 and I see very few items related to the functionality of VS2010 IDE with respect to C++ development (I think three in total, all of them are in SP1). So, QFEs are a fine idea, but as of today, you just don't have QFEs for problems we are experiencing.

  12. PleaseFixYourBugs says:

    @David Berg

    "If you get a chance, I'd love to know what parts of VS2010 performance you're seeing as worst than VS2008, and what types of projects you're working on."

    Do the above notes help or do you want me to be more specific / provide additional details? Do any of the items I list surprise you?

  13. PleaseFixYourBugs says:

    Another four weeks.

    No reply from Microsoft.

    Life as usual.

  14. Zola says:

    hahahahahahhahaha I just tried Visual Studio 2010, the UI is so slow, what a joke

  15. PleaseFixYourBugs says:

    Decided to pay this old link one last visit to see if anyone from Microsoft actually read what I wrote in response to their question…

    Of course, nobody read anything.

    It's been 3 months.

    Why ask questions if you are not interested in answers? Sigh.

    In case someone from Microsoft reads this later, perhaps you might want to know that me and my team are fed up with the abomination that is VS2010, we don't believe you will really fix the numerous performance issues in vNext (the developer preview did nothing, there are bad signs on blogs and forums, etc), so, ultimately, we have decided to move to another compiler. We used to like Visual C++, but you've ruined the product, making it next to unusable on all but smallest C++ projects. The move to WPF in VS2010 was the last huge straw.

    Farewell.

  16. David Berg says:

    @PleaseFixYourBugs

    Sorry for the lengthy delay, the whole performance team has been heads down reworking our internal performance testing infrastructure and working on performance and memory improvements for the next major release.

    The Perf Watson data we've been collecting has been a big help in allowing us to see where major hangs are coming from and fix many of them.  We've also come up with better tests that help us identify typing delays and hangs, this includes background work that goes on during idle and can sometimes block the UI thread and create delays.

    I can't talk about all the details yet, but I should be able to provide more information once we ship the next release.

  17. Mike Nacey says:

    Err. Perf Watson fires on every solution load (which is slow), but my IDE is now sitting at "Not Responding" with no cameo by Perf Watson. So for your/our edification, Perf Watson can detect slow things, but not when VS becomes "Not Responding".

  18. David Berg says:

    @Mike,

    Thanks for your help.  You're right.  Perf Watson will detect the hang and fire about 2 seconds in to collect a stack; however, it waits for VS to come back so it can use the total time to weight the hang.  If it never comes back, then it probably won't complete the cycle.  I'll forward your note to the Perf Watson owner.

  19. Andrea Mantler says:

    Does Perf Watson work with VS 2012?

  20. David Berg says:

    No, the current version of Perf Watson posted in the VS Gallery does not work with VS 2012.  We are working on some new tools though which will improve on the quality and action-ability of information we can collect while reducing the system overhead.

    In the meantime you can report performance (and other) issues using the VS 2012 Feedback tool which can be downloaded from here: visualstudiogallery.msdn.microsoft.com/f8a5aac8-0418-4f88-9d34-bdbe2c4cfe72

    Thanks for asking!