Visual Web Developer performance numbers
About a year ago, right before Beta 1 release, I blogged about Visual Web Developer performance numbers for Beta 1. How do we fare these days?
| Scenario | 40512 | 50516 |
| Add a new web form (cold) (s) | 11.01 | 7.05 |
| Cold F5 (PortalVBSDK) (s) 256MB | 35.62 | 36.27 |
| Open DeskTopDefault.aspx (cold) (s) | 10.67 | 8.65 |
| Load Web Site (Cold) (s) | 4.71 | 3.35 |
| Warm F5 (s) 256MB | 2.29 | 4.85 |
| Devenv ShoppingCart.aspx - cold (s) | 15.22 | 15.64 |
| Devenv ShoppingCart.aspx - warm (s) | 5.98 | 5.83 |
All measurements were done on P3 600MHz, 128MB RAM running Windows XP Professional. 40512 is build dated 5/12/2004 and 50516 is 5/16/2005 respectively. See Scott’s blog entry for more information on VS build numbering. In case you don’t know what cold and warm mean: cold means fresh machine after reboot so OS page cache does not contain any VS pages while warm means VS was run once so OS page cache still contains VS code pages.
Couple of scenarios appears to be tad slower, but still within tolerance. It is pretty much impossible to get exactly same time even between runs on the same build; times that you see are averaged across several runs. Warm F5 regressed significantly, but we are working on it.
What else do we measure in addition to the Beta1 scenarios above? We added scenarios like View In Browser (i.e. run without debugging), Build Web, First switch to Design view and Open C# code beside file. All scenarios are measured as cold and warm. I am not giving performance numbers for the new scenarios yet since a) we are still working on them and b) we are moving from measuring on 128MB machine to measuring on 192MB machine so target times will change.
Why are we changing from 128MB to 192MB? First, 192MB machine better represents typical 256MB box running a few application besides VS such as antivirus, e-mail client or a Web browser. We are no longer targeting 128MB machine since under low memory conditions most scenarios appear to be I/O bound. Second, optimizing too much for low memory conditions may actually reduce performance on higher end machines.
Debugging scenarios always have been measured on 256MB machine since debugging requires loading VS debugging engine, spinning up Web server as well as launching the browser.