Reduced Out of Memory Crashes in Visual Studio “15”

This is the third post in a five-part series covering performance improvements in Visual Studio “15” Preview 5. The previous 2 posts talked about faster startup, and shorter solution load times in Visual Studio 15″.

Visual Studio is chock-full of features that millions of developers rely on to be productive at their work. Supporting these features, with the responsiveness that developers expect, consume memory. However, in Visual Studio 2015, in certain scenarios the memory usage grew too large. This led to adverse impact such as out-of-memory crashes and UI sluggishness. We received feedback from a lot of customers about these problems. In VS “15” we are tackling these issues while not sacrificing the rich functionality and performance of Visual Studio.

While we are optimizing a lot of feature areas in Visual Studio, this post presents the progress in three specific areas – JavaScript and TypeScript language services, symbol loading in the debugger, and Git support in VS. Throughout this post I will compare the following two metrics for each of the measured scenarios, to show the kind of progress we have made:

Peak Virtual Memory: Visual Studio is a 32bit application, which means the virtual memory consumed can grow up to 4GB. Memory allocations that cause total virtual memory to cross that limit will cause Visual Studio to crash with an “Out of memory” (OOM) error. Peak Virtual Memory is a measure of how close the process is to the 4GB limit, or in other words, how close the process is to crashing.

Peak Private Working Set: A subset of the virtual memory which contains code that the process executes or data that the process touches, needs to be in physical memory. “Working set” is a metric that measures the size of such physical memory consumption. A portion of this working set, called “Private Working Set”, is memory that belongs to a given process and that process alone. Since such memory is not shared across processes, their cost on the system is relatively higher. Measurements in this post report the peak private working set of Visual Studio (devenv.exe) and relevant satellite processes.

JavaScript language service

Over a third of Visual Studio developers write JavaScript (JS) on a regular basis, making the JS language service a component that is loaded in a significant number of Visual Studio sessions. The JS language service provides such features as IntelliSense, code navigation, etc., that make JS editing a productive experience.

To support such productivity features and to ensure they are responsive, the language service consumes a non-trivial amount of memory. The memory usage depends on the shape of the solution, with project count, file count, and file sizes being key parameters. Moreover, the JS language service is often loaded in VS along with another language service such as C#, which adds to the memory pressure in the process. As such, improving the memory footprint of the JS language service is crucial to reducing the number of OOM crashes in VS.

In VS “15”, we wanted to ensure Visual Studio reliability is not adversely impacted by memory consumption regardless of the size and shape of the JS code. To achieve this goal without sacrificing the quality of the JavaScript editing experience, in VS “15” Preview 5 we have moved the entire JS language service to a satellite Node.js process that communicates back to Visual Studio. We have also merged the JavaScript and TypeScript language services, which means we achieve net memory reduction in sessions where both language services are loaded.

To measure the memory impact, we compared Visual Studio 2015 Update 3 with VS “15” Preview 5 in this scenario:

  • Open WebSpaDurandal solution. This is an Asp.Net sample, which we found to represent the 95th percentile in terms of JS code size we see opened in VS.
  • Create and enable auto syncing of _references.js
  • Open 10 JS files
  • Make edits, trigger completions, create/delete files, run the format tool

Here are the results:

Chart 1: Memory usage by the JavaScript language service

Peak virtual memory usage within Visual Studio is reduced by 33%, which will provide substantial relief to JS developers experiencing OOM crashes today. The overall peak private working set, which in Preview 5 represents the sum of the Visual Studio process and our satellite node process, is comparable to that of Visual Studio 2015.

Symbol loading in the debugger

Symbolic information is essential for productive debugging. Most modern Microsoft compilers for Windows store symbolic information in a PDB file. A PDB contains a lot of information about the code it represents, such as function names, their offsets within the executable binary, type information for classes and structs defined in the executable, source file names, etc. When the Visual Studio debugger displays a callstack, evaluates a variable or an expression, etc. it loads the corresponding PDB and reads relevant parts of it.

Prior to Visual Studio 2012, the performance of evaluating types with complex natvis views, was poor. This was because a lot of type information would be fetched on demand from a PDB, which would result in random IOs to the PDB file on disk. On most rotational drives this would perform poorly.

In Visual Studio 2012, a feature was added to C++ debugging that would pre-fetch large amounts of symbol data from PDBs early in a debugging session. This provided significant performance improvements when evaluating types, by eliminating the random IOs.

Unfortunately, this optimization erred too much on the side of pre-fetching symbol data. In certain cases, it resulted in a lot more symbol data being read than was necessary. For instance, while displaying a callstack, symbol data from all modules on the stack would get pre-fetched, even though that data was not needed to evaluate the types in the Locals or Watch windows. In large projects having many modules with symbol data available, this caused significant amounts of memory to be used during every debug session.

In VS “15” Preview 5, we have taken a step towards reducing memory consumed by symbol information, while maintaining the performance benefit of pre-fetching. We now enable pre-fetching only on modules that are required for evaluating and displaying a variable or expression.

We measured the memory impact using this scenario:

  • Load the Unreal Engine solution, UE4.sln
  • Start Unreal Engine Editor
  • Attach VS debugger to Unreal Engine process
  • Put Breakpoint on E:\UEngine\Engine\Source\Runtime\Core\Public\Delegates\DelegateInstancesImpl_Variadics.inl Line 640
  • Wait till breakpoint is hit

Here are the results:

Chart 2: Memory usage when VS Debugger is attached to Unreal Engine process

VS 2015 crashes due to OOM in this scenario. VS “15” Preview 5 consumes 3GB of virtual memory and 1.8GB of private working set. Clearly this is an improvement over the previous release, but not stellar memory numbers by any means. We will be continuing to drive down memory usage in native debugging scenarios during rest of VS “15” development.

Git support in Visual Studio

When we introduced Git support in Visual Studio, we utilized a library called libgit2. For various operations, libgit2 maps the entire git index file into memory. The size of the index file is proportional to the size of the repo. This means that for large repos, Git operations can result in significant virtual memory spikes. If VS was already under virtual memory pressure, these spikes can cause OOM crashes.

In VS “15” Preview 5, we no longer use libgit2 and instead call git.exe, thus moving the virtual memory spike out of VS process. We moved to using git.exe not only to reduce memory usage within VS, but also because it allows us to increase functionality and build features more easily.

To measure the incremental memory impact of a Git operation, we compared Visual Studio 2015 Update 3, with VS “15” Preview 5 in this scenario:

  • Open Chromium repo in Team Explorer
  • Go to “Changes” panel to view pending changes
  • Hit F5 to refresh

Here are the results:

Chart 3: Incremental memory usage when “Changes” panel in Team Explorer is refreshed

In VS 2015, the virtual memory spikes by approximately 300MB for the duration of the refresh operation. In VS “15”, we see no measurable virtual memory increase. The incremental private working set increase in VS 2015 is 79MB, while in VS “15” it is 72MB and entirely from git.exe.

Conclusion

In VS “15” we are working hard at reducing memory usage in Visual Studio. In this post, I presented the progress made in three feature areas. We still have a lot of work ahead of us and are far from being done.

There are several ways, you can help us on this journey:

  • First, we monitor telemetry from all our releases, including pre-releases. Please download and use VS “15” Preview 5. The more usage we have with external sources in day to day usage scenarios, better the signal we get and that will immensely help us.
  • Secondly, report high memory (or any other quality) issues to us using the Report-a-problem tool. The most actionable are reports that help us reproduce the issues at our end, by providing us with sample or real solutions that demonstrate the issue. I realize that is not always an option, so the next best are reports that come attached with a recording of the issue (Report-a-problem tool lets you do this easily) and describe the issue in as much detail as possible.
Ashok Kamath, Principal Software Engineering Manager, Visual Studio

Ashok leads the performance and reliability team at Visual Studio. He previously worked in the .NET Common Language Runtime team.