Response to the feature poll

There were over 80 responses to my recent post asking for feedback on where MSBuild should be heading (if the graph doesn't appear, it's here).

Thank you all for your thoughtful allocations! Let's go through each one in decreasing order of "investment".

1) Debugger. Debugging by gazing at diagnostic logs and putting in <Message> tasks is less than ideal, and I've probably done this more than anyone, but I would never have expected that a debugger would be the #1 request. A while back, one of us hacked up a proof of concept for debugging MSBuild in VS by using this technique, and it worked rather well. Apparently this is a perfectly supported route, and presumably much easier than writing a new debugging engine for VS. If and when we support this publicly, it would most likely be by this route. I hope we can.

2) Converting other project types. Deployment (.vdproj) was the highest mentioned by far, with some other mentions of Biztalk. Some good news here. WiX support integrated into Visual Studio has just appeared in the CTP of Visual Studio Rosario. It's based on the existing Votive plugin for Visual Studio, “productized” into the Visual Studio box. Rob Mensching has some detailed information. Of course, .vdproj isn't the same as WiX. I've passed on the feedback from this blog to the owners of this effort (who also own .vdproj) so they can see how many people would love to see .vdproj be MSBuild based. If you grab the CTP you can let them know exactly what you want via opening Connect issues -- Jeff Beehler's blog has information about how to do this. Open issues there so they get feedback, and vote them up the list :-)

3) Converting the .sln file. No surprise this is a big deal. Visual Studio itself is built by MSBuild, and for that purpose we didn’t use .sln files – we use files named "dirs.proj" to represent nodes in the build tree. Each dirs.proj does little more than list the projects and dirs.proj’s below it, and import simple build rules that use the <MSBuild> task to continue the tree traversal. Building Visual Studio itself from a huge solution file would be impossible. So we’re well aware how much pain they cause outside Microsoft. The hard part here from the MSBuild point of view is getting VS to read and write these dirs.proj files instead of .sln files.

4) Visualization. This was the other big surprise for me as I figured that you would say "complete more of the basics first". Also, of the items on this list, this would be one of the easiest for folks outside of Microsoft to accomplish. A couple months ago a couple of us spent a week and prototyped a WPF visualizer. Chris wrote a SQL logger that listened to just the regular, standard logger events and wrote to a database using the free SQL Server Compact Edition. Cliff wrote a visualizer based on the free GLEE graphing package which queried the database. It had two views -- bubble and line of target dependencies, and bubble and line for project dependencies. We built parts of VS using multiprocessor support (“/m” switch) with this logger attached, then pointed the visualizer at the database and we could see, for example, projects causing bottlenecks in the build. Because the visualizer was based on a database, there was some UI to do simple queries, like color all projects with names matching pattern X. Of course, this could go a lot further. What sort of visualizations would you want to see?

5) Native code/VC support. More good news here! Both command line and IDE support is now planned for the next release of Visual Studio after Visual Studio 2008 ("VS10"). Some of our friends downstairs in VC have been working on it for a little while now. As more details emerge we'll pass them on. (Putting my “platform” rather than “Visual Studio” hat on, you may have noticed that earlier this year, a competing IDE product began to offer native code build based on the MSBuild platform.)

6) Distributed build. Inside Microsoft, both the Visual Studio build team (the team that builds VS itself) and the Windows build team have been asking for this, amongst others. Six months ago one of us did a proof of concept of MSBuild based distributed build running on some large subtrees of the VS code and got near linear scalability in some cases - better than linear up to 4 machines in one particularly IO-bound subtree. We have a good idea how we would do this, but it isn't yet scheduled to happen. This seems to be a feature aimed more at the "build lab" than the developer. On the developer's machine, we typically aren't making full use of the cores she already has, and the kinds of builds she does tend to be smaller and perhaps less parallelizeable at the project level and the transmission and provisioning overhead would be significant.

7) Inline tasks. This is one of my personal favorites. Rather than say "go write and compile and deploy a task" I'd like to say "write some Powershell/batch/jscript and paste it in here and use it like a task elsewhere". Again, we've done a proof of concept. One particular kind of inline task we use internally we call "DataDrivenToolTask". You probably know that tasks that wrap a command line tool generally derive from ToolTask. These all have to do the same tiresome job: map friendly properties to command line switches. When we created tasks to wrap the native compiler and linker for building Visual Studio internally, this code became unwieldy. Yet it each mapping is a simple variation of the other. So we threw out the old tasks and created XML files that stated the tool name, defined mappings between property names and switches, described the type of the switch parameter, indicated how the switch is negated, what it could legally be combined with, and so on; then we wrote a generator tool that created the task from these. (Actually it creates a partial class, deriving from ToolTask, which we can combine with a little special code for that tool if we need to.) This turned out to work very well. What I would like to see is the ability to put this XML directly under the UsingTask element, so the task is created dynamically. That's probably not something anyone would do for something as huge as CL, but for tools with just a handful of parameters - think regsvr32 - one could make a nice typesafe readable task and pass it around to your friends by simply pasting it into an email. It's my hope that this would be one of our first types of inline tasks.

8) Multiproc performance. We're embarking on a significant rewrite of the "back end" of the MSBuild engine in VS10 and expect that we'll be able to improve the rather "basic" multiproc scheduling that MSBuild does in v3.5/Orcas: both to improve performance on existing machines and work better on the many core machines that will become increasingly common over the next few years. Our expectation is that this refactoring will make extending to distributed build more straightforward in a future release.

Skipping forward a few (the missing #10 probably should have been "add more built-in tasks") I'd like to mention ..

11) Extensible dependency checking. This was a bit confusingly described. There are really two parts to this: support transitive dependencies, and provide an extensibility point for arbitrary dependency checking plugins (such as an assembly interface comparer). The transitive dependency support is really essential for reasonable support of native code. C/C++, IDL, and .RC files amongst others all pull in files indirectly via #include and it's not feasible or reasonable to expect a developer to describe in their build process the full transitive closure of files that are read by their build in all circumstances: yet to do a trustworthy incremental build MSBuild must check the timestamps of all these files. This problem actually arises in many situations, with arbitrary file types. We have been solving this problem internally for the build of Visual Studio itself using a technology we created called Tracker, and it's proven itself to be solid and reliable. The other part is the extensibility point, which is perhaps what most people were actually voting on, and seems less urgent.

12) Assembly versioning support was suggested. Many of us have used Neil’s AssemblyInfoTask. Again, we hear you and there's good news. TeamBuild is planning to have a similar, but official and supported solution to this shipping in Rosario.

14) Object model improvements weren't on my list but probably should be: they were suggested by several people. Today the MSBuild object model is somewhat clunky and certainly incomplete. It was created to do just exactly what Visual Studio needed for VB and C# projects and no more. It mixes up the evaluated and unevaluated states amongst other problems. We expect to considerably improve this OM - not least because VC in VS10 will need it improved – to make it truly generic and powerful. (I already have a small group of people interested in giving detailed feedback on the design, but if you'd like to join them, please send me an email at msbuild@microsoft.com).

My caveat again -- nothing is set in stone until it appears in the final box, (or else is officially announced!) - resources and priorities change, risks to the product are managed, customer feedback sometimes shifts our direction directly or indirectly. In particular, work we need to do to support larger Visual Studio features has to happen. I hope though that you'll get value from my blogging candidly about our ideas and likely plans, while you realize we can’t make firm commitments to particular features at this stage. Enough caveats :-)

Any thoughts?

Dan

"This posting provided AS-IS, with no warranties"