My philsophical musings about building setup for software.


Mike Gunderloy, who has written a couple articles about the WiX toolset, posted a comment on a previous blog entry.  In the comment, he suggests that splitting a setup project into fragments only moves the problem but doesn’t solve it.  Thus, he argued that adding a tool to generate the fragments has value.  I still didn’t agree.  Then I realized that I disagreed because I have some philosophies about how to build software and setup for software that I’ve never posted here.  So, I thought I’d share those philosophies today.  Please note that these are guidelines that I use when discussing setup build processes with other people not hard rules. 


The developer that wrote the feature knows best what needs to be authored into setup.


This part of my philosophy is based on the fact that the person that knows the most about a particular resource is the developer that wrote the resource.  Seriously, if the developer who wrote the code doesn’t know where his or her files need to be installed or doesn’t know what registry keys are necessary to make the feature work (or whatever other resources are necessary) who does?


Yes, there have been (too many) cases in my career where the developer who wrote the code said, “Uhh, I don’t know what my code depends on.”  However, in those cases it was pretty easy to look at him or her (or his or her manager) and ask, “Well, uhh, shouldn’t you?”  In every case, they went off and figured out what was necessary to get their software “in the box”.


Fundamentally, if developers don’t know what their dependencies are there is very little chance their project will have a solid security, performance, or deployment story.  Area experts may be necessary to help developers work through complicated issues but, in general, developers must be aware of what their software is doing.


 


Setup authoring is a part of the development process.


Every team I have interacted with (the grand majority of Microsoft and some smaller companies), the developers on the team are expected to add their source files to the makefile before checking in new files to the project.  Yet, many of those teams have people who are solely responsible for adding new files to the setup project.  In many cases, one set of developers write registry keys into the code for SelfReg (this is evil, and I’ll explain why one day) and the setup developers have to reverse engineer the registry keys out of the built executables to author setup.  Why didn’t the original developer just author the registry key into setup in the first place?


There was a point in time where it was arguably difficult to distribute the authoring to all developers when you had to buy custom tools that didn’t fit well into the development process.  However, today there are a few alternatives out there (like the WiX toolset) that allow setup to be treated like source code.  Now there are no excuses I’ve heard that hold water why setup authoring for the majority of the resources in setup cannot be distributed across the developers in the organization.  Doing that distribution leaves only the “look and feel” and integration between the individual pieces of setup to the core setup developers.  All other excuses have always been just whinging (as Peter might say).


This part of my philosophy and the one above are the reasons I disagree when Mike says that “breaking up the package into multiple source fragments pushes the problem back one level, but doesn’t necessarily solve it.”  Breaking up your setup in to multiple text files enables developers to maintain setup authoring the same way they maintain all the other code that is part of a project.  And that brings me to my final point.


 


Text files should be the only inputs into the build process.


Over the years, developers have created fairly significant processes for tracking, merging, and reverting changes to text files.  For example, Concurrent Version System (better known as CVS) is used heavily by SourceForge.net can show you the individual lines changed in a text file over time.  For a demonstration, take a look at Compiler.cs from v1.6 to v1.7 in CVS at SourceForge.net for WiX.  It is not generally possible to visualize the differences between two revisions of a binary file.


Every input into the build process that is manipulated by a human will eventually hose the process at least once.  Being able to text search (using even the simplest tools like grep) for the exact change that caused the problem significantly improves build throughput.  Binary files usually hide the information by requiring custom tools to be opened and queried to find the break.


Of course, once you find the break it is really nice to be able to fix the problem by launching your favorite text editor and tweaking the line(s) of code with the fault.  Requiring a tool to be installed on the build machines increases your impedance to fixing the issue.


Ultimately, keeping binary files out of your build process simplifies your life.  That fact is why I disagree with Mike when he suggests it makes sense to “put one more tool into the chain, something with a friendly interface that could spit out the source fragments as needed? I could see doing that with Access/Excel, among other things.”


Note I do think it is reasonable for developers to use tools that help generate text files.  Those text files then can get checked into source control and built as part of the standard process.  However, the tools need to generate human friendly text files.  Text files that can only be modified with a custom tool are only half a step better than raw binary files.


 


This is the philosophy that I developed during my tenure in Office and have promoted across the company for a couple years now.  Many teams have had an incredible amount of success with their setup processes when following these guidelines.  There have been many cases where I was asked by other teams to talk about the philosophy.  At the end of my talk I always left them with, “If you have someone resisting change, have them contact me.  I’ll be happy to talk to him or her to discuss how successful other teams have been with this process and see if there isn’t some way to address their concerns.”  I’d be happy to do the same for you.


 


PS:  This philosophy has worked out extremely well for me, but if you have a unique situation that you think wouldn’t work using these guidelines, I’d be very interested to hear about them.  I’m always looking for alternative views (as long as people aren’t just whinging <smile/>).

Comments (27)

  1. "Note I do think it is reasonable for developers to use tools that help generate text files." We’re not disagreeing – at least not about that, or about the need for having text files be the only input into the process.

    But isn’t your registry key example the perfect example of why it makes sense to provide targeted setup-authoring tools to every developer? The developer should absolutely know "my piece is X, and it depends on Y and Z, and it needs these registry keys." But I think the team’s toolmaker (or whoever) can profitably provide a tool that translates that knowledge into the right XML input for WiX (or whatever other process the team is using), thus avoiding any chance that a developer who isn’t up on setup will write SelfReg code.

    So, to sum up where I’m coming from: setup should be a distributed activity, but it’s not realistic to expect every developer to know the ins and outs of the setup rules from the start, and that’s where some tool can come in – to take the knowledge that the developer has about their own bits and translate it into a proper chunk of the setup source.

  2. James Hancock says:

    All of this is wonderful, but as the SQL Team proves (MSDE installs integrated into applications and MSMs anyone?) the core thing that MS doesn’t seem to get (well the Office team does…) and continually screws up with Windows Installer is that the end user should be presented with at most one choice (unless they choose custom) and EVERYTHING should install automatically and transparently without the user having to know anything else.

    That means that I should have the tools from MS to build a single installation that will install the correct IE version, the correct MSDE version (and upgrade if applicable), the correct Jet version, the correct MDAC version, the correct OS Service Pack, the correct OS hotfixes too!, and install my application all at once automatically without the user knowing that separate things are being installed.

    The SQL Server teams’ solution is to do what the VS.net installer team has done and install pre-requisits. This just confuses the average user and is the surest way to prove that your installation tool is broken when you can’t integrate the install in one simple package that allows everything to just work. (MSMs were the way to go, but then Windows Installer is busted…)

    MS needs to realize this. Yes, it’s great to have click once. But that works for a VERY limited set of senarios. Yes, it’s great to have Windows Installer that doesn’t break the OS and provides roll backs etc.

    BUT! Windows Installer is broken, it was broken the second it came out of MS. Merge Modules are the equivalent of DLL hell because they MERGE with the install. They should run like chained MSIs and not screw with any of the other installers in the package and prevent older versions of files from being installed, all automatically.

    Combine that with the setup script being XML based and encrypting the contents so that things like MSDE SA passwords can’t be found out simply by editing the MSI in Orca and you’d have something usefull.

    Make it so that we can script against it using C# or any other .net language because it automatically installs .net first, and you’d have something truely impressive.

    Put that all into VS.net so that we have a real tool instead of the play toy that’s in Vs.net, and you’d impress the hell out of me. (and I would be eternally greatful for getting rid of Wise and Installshield, because they are CRAP)

    Yes, the symantecs of getting an install ready as a developer is of great importance, however if the end user gets a bad taste in their mouth becasue they had a difficult and confusing time installing your application, you just lost your first chance to have a return customer and your first chance to resell them that it was a good idea to buy your software in the first place.

    Yes, Linux is even worse, but that’s no excuse. The Windows Installer people should go back to the drawing board and start again with the things that I’ve listed in mind. MSI is barberic and crude (and yes, I can say this because I’ve been building install programs with it since the beta of V1) and wasn’t properly implimented for even the most basic tasks from day one.

  3. Riko Eksteen says:

    Rob, what an amazing post. I wish I could force the people in my company to read some of your blog entries, because they are so much more than philosophical musings!

    I have a presentation to management about the ins and outs of MSI soon, and I think I will include these principles in there (along with "What is setup" and the two axioms :-)). We are busy (or plan to) inform management about setup in general and MSI in particular, as I desperately want to get away from the current "ask the setup developer – he/she can make it work" scenario.

    The more I work with MSI, the more I realize how closely the install should be integrated with the rest of product development. The problem is getting management to realize this too. It is so convenient for them to just "hand it off" to the setup developer. Only problem is, when using MSI, you can’t just "make it work" – you have to know the limitations and advantages and rules!

    And WiX is the perfect tool to do this with (the more I work with it, the more I like it), if there can be some way to simplify file handling etc. for developers. Luckily, as you pointed out in an earlier blog entry, some tools are being developed (and I’m playing around with this myself).

  4. David Levine says:

    I agree that every developer should know what requirements they are imposing on the system, but I don’t necessarily agree that they should also be the ones to directly make changes that affect the build and installation process.

    Individual developers are not always aware of the requirements/limitations of the build machine and its environment, and they often make changes and use settings on their developmnent machine that absolutely should not be made on the build machine. The build machine is kept as pure as possible to avoid contamination, but often a developer’s machine is a jumbled collection of components, and is geared towards development and debugging, not building release products. What works on a developer’s machine is unlikely to work on the build machine.

    Another issue has to do with integrating the install into a larger install package with prerequisites; this is often a non-trivial task as several other writers have pointed out.

    Another issue has to do with uninstalling the product. This sometimes involves custom code as not all assemblies (dynamically generated) are known to MSI and so cannot be automatically uninstalled.

    Another issue involves upgrades… there are limitations and bugs in MSI that can cause havoc with an upgrade – unless you are aware of the issues you might wind up executing the wrong version assembly.

    On small projects managing this isn’t a big deal, but on larger projects changes should be discussed with the setup person/team so that issues can be resolved before they become problems.

  5. Mike,

    As noted in my blog entry, I’m completely comfortable with having development environments that help build WiX files to ease a developers’ life. I use VisualStudio today to do all my WiX authoring because it will do "intellisense" on the WiX schema. That saves me a lot of time. For example, with VisualStudio, writing a registry key is trivial because I type "<Reg" + space, then "I" + "=" then "K" + "=" + select from dropdown list then "P" + "=" then "N" + "=" then "V" + "=" and I end up with something like:

    <Registry Id="" Key="HKCR" Path="" Name="" Value=""/>

    Hopefully, any Windows developer could figure out what to do with that. Granted, I am expecting developers to spend a little bit of time understanding the basics of setup just like I am expecting developers to know a little bit about programming secure code and “not slow” code. If tools can help that learning curve and make life better, that’s great.

    Also, I’m not arguing that every developer should need to know the installation in depth. Every team will need an area expert that they can leverage to take care of the more complicated things like integrating all the smaller pieces into the larger whole (or "wholes" if there are multiple SKUs). The interesting thing is that even on large projects your core setup team can be very small if you distribute authoring across all the developers.

  6. James,

    I agree with everything you have said except when you compare Merge Modules to "DLL Hell". (Merge Modules and "DLL Hell" are very different problems from each other, but your issues about Merge Modules being integrated into your setup are very valid. <smile/>). Everything you have said here is something I have had issues with in my tenure on Office and have brought up with the Windows Installer team. In the meantime, I am here trying to blog about the Windows Installer to at least make some things a bit more tolerable (or better understood).

    Also, I’m surprised you didn’t pick the number one issue that the Windows Installer was broken from day one: http://blogs.msdn.com/robmen/archive/2003/10/18/56497.aspx

  7. David,

    My point about developers understanding dependencies should take care of your build machine issues. If a developer creates a dependency on a tool that is not in the build why is it acceptable for the developer to check in his or her code? Seriously, if developers don’t understand what is in the build lab (and I agree the build lab should be as minimal as possible) how do you prevent them from checking in stuff that breaks the lab all the time? Having developers author setup (or not) wouldn’t affect this.

    All the other issues you raise are things that I would expect the core setup team to manage. Understanding and debugging those cases is not something every developer needs to do. Area experts (that know everything about conditionals in MSI, upgrades/patching, and CustomActions and such) will always be needed. However, I don’t believe you spend your area experts’ time doing work that can (and should) be distributed across all the developers.

    I think your last point argues for code reviews by the area experts. I think code reviews is a great idea for setup, security, and performance. However, that does not mean that the setup authoring should not be distributed across your development organization. In fact, over time, I think you will find that a number of developers will become setup area experts on their own.

  8. Reid Gustin says:

    James: When you said "Linux is even worse, but that’s no excuse", you hit on one of my favorite arguments, and I completely agree. It shows up on Slashdot all the time, by the way (though typically as "Yeah, Linux has a security hole, but Microsoft is worse").

    Anyway, you’re absolutely right. As we get to the point in software where tools do everything they need to do, we can start comparing them to competitors and worrying about whose solves the problem better. But until that point, there needs to be some fundamental idea of correctness that we shoot for. And, of course, the Windows Installer has some places that need fixing that are absolutely fundamental to the way it works (see Rob’s entry on component rules).

    Anyway, I don’t really have anything relevant to add, but I wanted to pounce on that statement and say "me too"!

  9. Reid Gustin says:

    James: Actually, I do have something relevant (mostly… I’m probably nitpicking) to add. You mention SQL server as a particularly bad offender of the "setup should be simple" mantra, and you say that an average user gets confused. But why, exactly, is an average user installing SQL server? It seems like that would be an automatic contradiction. If it’s a developer installing it to write code against, they’re really going to have to understand what they’ve installed, or they won’t know the requirements of their code.

    It’s very similar with Oracle (I used to run Solaris boxes and networks at school). Yes, it’s a giant pain to get Oracle running, and you need (or needed at the time) to write some scripts to get things working. How do they expect an average user to get Oracle installed? Well they don’t, of course.

    Now I personally believe that SQL Server is easier to install than Oracle, but that it could be made easier. But I still don’t understand why your average user would want or need to do so.

  10. Jeff says:

    I am the lead developer at an ISV, which makes software used by small businesses which do not have a particularly technical orientation. Our product consists (among other things) of a client application, a middleware layer, and a sql server. Our customers need to install this without our supervision. The middleware layer uses SQLDMO to discover sql servers, so we have to include that also.

    Ideally, I should be able to use some sort of MSDE redistributable installer to make a new sql server instance on the machine, prompting the user for their sa password as part of the install. Then I could create my sample database as part of my app’s install.

    This is really, really hard to do in the current state of affairs. We ship a separate MSDE installer, and hope that our users read the readme file.

    In a perfect world, MSDE would be installed with the OS, along with MSMQ, but convincing the powers that be that _that_ should be on every desktop is beyond my ability.

  11. James Hancock says:

    Reid: MSDE is a freely distributable version of SQL Server designed to replace Access Databases. This is a good thing as Access Databases are horrible and horrible for security (SP8 of Jet4!!!!) But the install procedure is horrible for it. They released merge modules and it’s been impossible (from day one) to install an application with the merge modules that were supplied with SQL Server 2000 and they’ve gotten worse. I learned most everything I know about Windows Installer by hacking the MSDE 2000 merge modules and fixing all of the bugs that MS had in the them.

    Then came blaster and MS decided that MSMs weren’t the way to go. (rightfully) The reason? Because MSMs require a patch to the original application to patch the MSM install thus you can’t just download SP3 for MSDE 2000 and upgrade, you have to run a Windows Installer patch against the installing application which of course is so very very very broken in Windows Installer and incredibly hard to make, that most developers don’t bother.

    But the solution wasn’t to fix Windows Installer by making MSMs work as modules that can be upgraded and patched and chained or to release an MSI installer of MSDE that can be properly chained from your install, it was to create the MSDE installation pack that creates a window that installs Pre-requisits (without any source code so that we can add things like IE easily damnit!) and then once that’s done and it’s decided if it should restart, then installs the application. While this solution was just fine for VS.net because you’ve got a boat load of geeks using VS.net, it isn’t fine for joe user installing a client application that happens to need a database.

    If MS took all of the time that their various divisions implimented work arounds for stuff that is entirely broken in Windows Installer and instead invested in fixing Windows Installer in the first place, they would have saved a couple of million of dollars I’m sure and made people like me much happier. (i.e. I would still have hair at 27 and wouldn’t rather eat rocks than edit our installation for our software!)

    Jeff:

    It is possible to make an installer that does all of the stuff right with the merge modules. I have ours so that it actually detects if there is SQL server on the machine and leaves it alone. If it isn’t, then it prompts of the user wants to use another remote machine or install MSDE natively. If they want to use MSDE it asks for password info etc. If not, it asks for the connection information for the remote/local SQL Server instance. The trick is to remove the actions in the MSMs that check for existing installations of MSDE and treat it as an error condition if it is there already. (which isn’t an error condition) Check out http://www.installsite.org for more information as I’ve posted all of my hacks and work arounds and violations of the DMCA on there for everyone else’s benefit :)

    Which brings me to another point that many groups don’t understand in MS! The existance of the correct version of a component on the system IS NOT AN ERROR CONDITION! Just cause you don’t have to execute your install, doesn’t mean that it’s a bad thing, it just means that the end result is the same and you’re happy.

    MDAC people tried to release an MSM once. It took me 5 months of complaining to convince them that MDAC 2.7 being installed on Windows XP machines was not an error condition for an MDAC 2.7 MSM and should just continue the install normally exiting with an error code of SUCCESS. Once I finally convinced them of this, they promptly canceled the MSM project and left it to Wise and Installshield to fix their mess.

    Then there’s the MSDE people putting in error conditions for existing MSDE instances (even if the instance isn’t the same as the one you’re trying to install!) amoung other things. If it’s there great! If it’s there and it isn’t the right version, then upgrade. If it isn’t there at all, then install it! There is no other conditions.

    Rob: Yes You’re so right about the component issues… I just try and ignore that one as much as I can :) If I didn’t I would really be bald at 27!

    In all of this I have talked to VPs, the head of the SQL division, 15 different people in the Windows Installer, and SQL Server Installer groups and even written up an analysis of the problems, and how to fix them giving specifics. Hell I even uploading on my dialup connection my fixed merge modules, thus giving MS all of the info necessary to charge me under the DMCA just so that they could fix the damn things and post them so that others could benefit from not having broken stuff.

    I further went and showed them how they could fix the MSDE installer’s MSI so that it could be chained correctly with other installers, thus fixing the patching problem and solving everything so that the user saw a transparent installation process that just works without any other information required by the user.

    Apparently everything I sent them has been completely ignored and we now have the completely STUPID, INANE, LAME solution that they have posted at this point. When I point out the major issues with their solution and how they’re treating MSDE as the same thing as MDAC which can be left on the clients computer, they promptly go and write more code to make it roll back on failure to install your app, but then leave 3 other holes where the damn thing gets left on the client computer eating up resources for no good reason (MSDE takes up huge amounts of memory when running, so you don’t want it there if it doesn’t have to be.)

    Yes, as you can tell, I’m anoyed. Almost anoyed enough to impliment my .net native installer and make it public source so that people can have a real install program that they can program like any other winforms application and handles all of the rollback etc. that Windows Installer does amoung other things…. but that’s for another day when I have lots of free time…

  12. Amen, James. I’ve been there right with you since 1998.

    The only thing I ask is that you think twice about creating your own installation engine. Multiple installation engines has the capacity for truly hurting our customers. There are so many interoperation problems with multiple installation engines that we are better off getting one good engine in the platform than fragmenting the world. I lived in a world with several installation engines and customers were getting completely hosed. I don’t want to go back there.

    Your well-informed and completely appropriate "rant" here provides yet another example (that I have been throwing around for years) of things that need to get fixed.

    Just know that you’re not alone. <smile/>

  13. Reid Gustin says:

    James: I’ll second the amen. Also, I clearly didn’t understand the scenario you were saying was broken. What you described is definitely not the way that should work.

    On my first read through your original post, I thought you were writing about a stock SQL Server install, rather than merged into your own. Apologies for the misunderstanding.

  14. James Hancock says:

    BTW, I’m in the Windows Installer 3.0 Beta… none of this has been addressed. (since the 3.0 specs have been publically released I think I’m find to disclose this?)

  15. James,

    Yes, the What’s New page for MSI 3.0 can be found here: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/what_s_new_in_windows_installer_version_3_0.asp

    And you are correct, none of the issues listed in this thread have been addressed in the next version of the Windows Installer.

  16. James Hancock says:

    whew! :)

    Now if only they would fix all of this stuff… We don’t need a new version just for easier updates.

  17. David Levine says:

    Rob,

    It’s not that developers are irresponsible, but some are better then others, and the natural state of any system tends towards chaos. It’s human nature to forget things (pilots need checklists!). One technique I use that prevents a lot of problems is that anyone who breaks the build owns it, and *they* get to spend their time figuring out why it doesn’t build or install.

    I think you are assuming a level of knowledge about MSI and installation issues that many developers do not have. Creating a MSI file is often a bastard child of the development process that no one wants to do. It is true that the developer knows better then anyone what needs to be done to add a feature to the setup but that does not always mean that they know best how to add it to the setup. The issues can be more involved then simply adding a file to be copied to the harddrive, especially given the variety of deployment models available to .NET apps. As you state, area experts are needed.

    Perhaps part of the pain we go through is because we currently use the Dev Studio IDE for creating our MSI files. It is not a complete authoring tool and we don’t have direct ORCA-like access to the tables. We use it because it drastically simplifies the authoring process. The side-effect is that if we need additional customization it is somewhat painful – we wind up using transforms to post-process the MSI files.

    About using text files as the only input to the build process….the current state of the art may make that the best for doing versioning, but doesn’t this really argue that more work is needed on the tools that author the setup file? I’d rather have a good GUI that graphically showed the difference between two versions of the setup files rather then an editor that highlighted different lines of text.

  18. Imagine a blog entry where I ramble along after having my wisdom teeth pulled and come up with something of a vision statement.

  19. Imagine a blog entry where I am reminded how important rollback is in setup and take a few minutes remind everyone.

  20. Imagine a blog entry where I talk about my first real day at OSCON 2004 where I saw an amazing keynote, presented WiX, and hung out with a few guys from Ximian while a good time was had by all.

  21. Experiance suggests there are windows of opportunity for breaking the cycle of the centralized setup…

  22. Imagine a blog entry where I talk about my first day at PDC.

  23. Imagine a blog entry where I talk about my second day at PDC 2005.

  24. Kurt says:

    The idea of a distributed setup development came up in our development team as well.

    My question is: Is there a size of the developer team which makes it reasonable to distribute the setup development?

    If the team is just 2 or 12 person I think it is not reasonable to distribute the setup because you may generate more overhead by putting the setup into small pieces, write manuals for the developers (which will not be read anyway) how to use it, instruct them and handle exceptions (I believe not the whole information of a component should be transferred to the distributed setup).

  25. Imagine a blog entry where I try to answer Kurt’s question about how large a development team should be to use distributed setup development.