Single Servicing


What is Single Servicing?

Single Servicing is that when you fix a bug in your component, you drop a single binary with the fix into customer’s machine, everyone using your component automatically get the fix.

Sounds easy. But in reality it is much harder.

1. Applications may statically link to your component. Those applications will have to re-compile to get your fix.

The recent zlib bug is an example.

2. Applications may carry a private copy of your componenent. Those applications will have to re-deploy your new component to their private directory.

The recent GDI+ bug is an example.

You are very screwed.

Well, until you start using managed code. Fortunately, it is very possible to achieve single servicing.

1. Static Linking

You can’t statically link an managed assembly. You simply can’t. The platform does not support it.

Joel Spolsky asked: ” Please Sir May I Have a Linker?”. Jason Zander answered: “No”.  You know, you should be very thankful for that decision.

2. Private Deployment

In Whidbey we made two changes to assembly loading:

Assembly.Load* now apply policy

Assembly.LoadFrom/LoadFile/Load(byte[]) prefers GAC

The two changes together, give you a way to do single servicing in .Net.

If you buy versioning, you change your assembly version number, issue a publish policy, and install your fix to GAC. Everyone picks up the fix.

If you don’t buy versioning, you don’t change your assembly version number, and install your fix to GAC. Everyone picks it up.

Of course, the bottom line is, you have to strongly name your assembly.

In light of the zlib and GDI+ patching problem, aren’t you glad that in .Net you can be assured of single servicing?

Comments (21)

  1. hn says:

    What do you prefer to do distribute the patch to customer machine? msi or write your own exe w/ fusion api?

  2. MSI of course. Why do you want to re-invent the wheel?

  3. David Levine says:

    No one wants to reinvent the wheel but sometimes you need something other then a wheel.

    Single Servicing works great…except when it doesn’t. When patching a component fixes a problem without introducing new problems then all is well and SS is great – it’s when installing a patch fixes one problem but breaks other apps that all the headaches begin. I don’t think we can ever rely on discipline to ensure that this is done right.

    Is there any way to disable to new Whidbey policy of preferring the GAC for assemblies loaded using LoadFrom/LoadFile/Load(byte[])? Perhaps a buy-in/opt-out option for the new policy.

    There are use cases where what the desired result is to load exactly what is asked for regardless of what’s in the GAC. For example, we need to validate entire systems; we need to be sure that the assemblies that we used to validate the system are the ones that are actually used. We would not want a patch to an unrelated application to change the assemblies our app uses – that would require revalidating the entire system.

    Thanks

    PS: I prefer to write my own GAC installer rather then author an MSI file, but that’s just me. For one thing, debugging a faulty MSI install is not the most pleasant task in the world.

  4. David,

    Yes, we understand the danger of potential applications breaking other applications. It is the fact that you *can’t* patch all the applications scares people. zlib and GDI+ patching problem will happen again. And people will point finger.

    You can use ReflectionOnlyLoad/ReflectionOnlyLoadFrom API in your validation scenario. The two changes I mentioned in the main text won’t be applied to ReflectionOnly API.

    Regarding roll your own installer vs MSI, MSI put a lot of effort into reliablity. You probably want to take advantage of that. It is not that easy to do reliable install/uninstall. Another advantage for MSI is management. You can use Group Policy with MSI. Roll your own is fine, as long as you are willing to tackle the problem ahead.

  5. And of course, if you don’t buy single serving, don’t strongly name your assemblies. If you want to use other people’s assemblies, then you have to play other people’s rule.

  6. Well okay I posted to the previous entry about Load preferring the GAC and I guess this was the post that you said would explain the reasoning.

    I think it’s a bad idea. You are basically saying that you will never load a private assembly if there is ever a matching assembly in the GAC. If someone is using a private assembly, it usually means "I want to use THAT version and don’t you dare ever change it."

    I think the problems you will deal with from unexpected assemblies will almost always outweigh the trouble you have to go to patch assemblies in multiple locations.

    I personally rarely install assemblies to the GAC in the first place.

  7. Josh,

    See my comments above.

  8. Peter Torr says:

    I 100% love this (servicing re-disted bits kills both us and the OSS guys) but I don’t see how the suggestion to use ReflectionOnly to solve David’s problem would work… they’ll use ReflectionOnlyLoad, which will return the EXACT assembly they ask for, they’ll verify it’s the right one, then they’ll call Load with the same parameters and fusion will load a different version!

    Is there an API to get the actual location of the assembly that Load would load (without actually loading it, of course 😉 ), so that you can then pass it to ReflectionOnlyLoad to make sure it’s the right one before you load it?

  9. David Levine says:

    Hello Jungfeng,

    Based on your earlier blog’s description about the ReflectionOnlyxxx APIs I don’t believe I can use these as I will be unable to create types from those assemblies. To be honest I am not yet sure how I will use this new capability (but it sure sounds interesting).

    Also, I am unfamiliar with Group Policy – I will investigate this.

    I think there’s a need for both a Single Service model and an explicit "what you asked for is what you get" model.

    There are many applications that want the behavior provided by the SS model; in fact, most will probably want it, so it’s a reasonable default. The issue is with those applications that do not want that particular behavior.

    For example, an app can be installed with a local copy of an strongly-named assembly, and it works quite happily with it. One day a new app comes along, installs a new copy of the same assembly identity into the GAC, but it has a few differences but the same version number (or a binding redirect), and all of a sudden the old app breaks (perhaps in a subtle, hard-to-detect way). In essence, an external application changed the behavior of a different, unrelated application because its private assemblies are no longer used, instead it uses public/shared assemblies from the GAC. This appears to set the stage for a .NET version of assembly h*ll.

    It’s reasonable to state that publishers of assemblies should never change the bits without changing the version, but the reality is that this does happen. It’s also possible that publisher policy will force a binding redirect.

    I believe the problem is actually a bit deeper – there ought to be distinctions between system level components that change infrequently and undergo rigorous testing when they do change, and assemblies that are shared libraries but which do not go through rigorous regression testing to ensure compatibility. There’s a reliability spectrum, with system components at the edge of greatest reliability with decreasing degrees of reliability as it moves towards less trusted code.

    As an application, I "trust" some components more then others – it’s the ones that I don’t trust all that much that I want control over where and how they get loaded into my application. This is especially problematic in a validated environment.

    Do my concerns make sense or am I missing something basic here?

    regards,

    Dave

  10. Ovidiu says:

    Great post, Junfeng. I still have one question: Given that many developers/project managers don’t take the project’s evolution (versioning) into account when making the initial design, is there a Microsoft-provided PAG or some sort of recommendation for versioning in .NET?

    Something defining what "backward compatible" and "forward compatible" means, what a breaking change is and so on. While some of these things are project-dependent, you should take into account that quite a few developers/architects out there know _nothing_ about this. What’s your opinion on this? Is there such a PAG/Is Microsoft considering writing one?

  11. David and Ovidiu,

    You guys are exactly right. Today there is no distinction between system componenets that are considered as strictly backward compatible, and application components that changes quite frequently.

    We are working on this, though not deliver in Whidbey. For starters, you can read Jeff Richter’s paper on the future of assembly versioning:

    http://www.theserverside.net/articles/showarticle.tss?id=AssemblyVersioning

  12. Peter,

    There is no such API. But of course, you can try to simulate fusion behavior. After all, all fusion probing logics are documented, if not in MSDN, then in my blogs/Suzanne’s blog/Alan’s blog.

  13. Rob M. says:

    This seems like it is taking a weakness of LoadFrom/TrickleDownload and just making it worse with no opt-out. I wouldn’t mind this being the default, but I’d really like a way to say "I want this URL to be checked first". Otherwise a customer could install an update to a dll we use and break our product before we have a chance to QA the changes to a third party DLL.

    I like the GAC as a default, but I hate the assumption that the GAC _always_ knows better than the developer.

  14. You *can’t* have both — single servicing and private deployment. It directly contradicts each other.

    If we did provide an option to opt-out this, then what is the point of making this change? So that everyone can opt-out a security fix, like the GDI+ fix?

    Yes, I know developers hate this. But you know what, system administrators always *hate* that developers assume they know better than system administrators.

  15. Rob M. says:

    Not all assembly updates are critical security fixes, or even security fixes at all.

    I think choice _is_ the proper solution. Let the developers flag their products/assemblies as desiring other specific assemblies, let the sys admin choose to respect this or override it. Everyone can be happy. Microsoft just choosing one or the other is going to piss off a lot of people because its not a good solution.

    Guess what, the System Administrators will hate it when some simple update to one application breaks three other apps. Only supporting SingleServicing doesn’t put any extra power in the SysAdmin’s hands it limits everyone.

  16. If versioning is implemented correctly, this is actually possible today.

    Assume you always ship publisher policy with patches, in your application’s config file, you can explicitly opt-out the publiser policy, for specific assembly, or for every assembly. Check out the configuration document, and look for <publishPolicy> under <dependentAssembly>, and <assemblyBinding>.

    The problem today is that application developers decide that. We want to move to a model that administrators decide that, not developers.

  17. Senkwe says:

    It would seem there’s someone out there who’s never deployed (or written) a .Net app in his life and yet assumes he’s smarter than you guys when it comes to this very topic.

    http://www.arstdesign.com/BBS/BulletinBoard.php?qs_id=1925

    Well, good for a chuckle at least methinks 🙂

  18. Ovidiu says:

    I was thinking about your post these days – is there a tool to check two versions of an assembly and say "The second is backward-compatible with the first one"? Given the amount of metadata .NET assemblies have, it shouldn’t be very difficult to check this (when it comes to classes, interfaces, method signatures and so on).

  19. Rob M. says:

    Publisher policy is nice, I wish we could use it. We can’t, we install a simple bootstrapper application _once_ and have LoadFrom (trickle download) do all the updating from there on out. Hence we can not update the app config file once installed. If you don’t cover the corner cases you don’t really have a solution.

  20. Ovidiu,

    You can’t. The two versions of an assembly may have the same metadata, but the behavior could be different. Reflection can’t tell behavior change. Even if it can, the change of behavior could be a breaking change for some apps, but not for others.

    Rob M,

    As a developer, you can think about patching. But patching ultimately is administrators’ job. If admin installs a patch,and it breaks your applications, admin should go and modify the app.config for you.

    Of course, if the admin is not intelligent enough, there is nothing you can do. But in that case, it is even more important to make sure the system is always in safe state.

  21. Rob M. says:

    Unless there is a simple and automatable way for an admin to roll out app.config changes, it won’t happen. Which is really what I am asking for.

    I think there is also a difference in focus. I believe you are very "security patch" focused. That is a good thing, however even from Microsoft I would wager that only 1 out of 10 patches is truly a critical security patch.

    I would prefer the ability to embed in an assemblies metadata a request to not have reference overriden by the GAC, unless the Admin takes specific action to override certain security updated assemblies.

    I believe there can be a compromise where we can have secure systems without breaking applications as a matter of routine. I’m probably worrying that "single servicing" sounds a lot like DLL hell to me.