I’ve just finished a significant push on automating our signing process over at the NuPattern project. And the results are so good that I felt like I need to share them with others, so you can also benefit from this important process step in your software projects.
So what’s the big deal?
I am sure that every .NET developer has some experience with creating strong named assemblies by now. All you need to do is create yourself a *.snk file (sn.exe –k) that contains both a (public and private key by default), and then configure a couple of assembly level configuration in your project and you are done. This is all fine and good if you are permitted to create and access your own private keys. But if you work in an organization and want your software identified to that organization, chances are that your organization wants to control access to those private keys. Because compromising them amounts to a security vulnerability, and you may have to jump though hoops to get your bits signed in the organization. Many organizations wish to have, and have defined strict processes and governance around getting bits signed using their private keys which they keep under lock and key, quite understandably.
The NuPattern project is a member of the The Outercurve Foundation, and as well as attributing the projects ownership to the foundation, we also sign and certify our binary deliverables using certificates provided by the Outercurve Foundation. We DelaySign our code with the public key which we store in the source code tree. But in order for the project to sign the bits for public consumption, we need to sign our bits with a private key, which we don’t have access to.
Like many software projects, we strive to build an identity and trust with our customers, and ensure that the authenticity and the integrity of our deliverables are protected. As part of that, we like to strong name and Authenticode sign our assemblies, we also like to Authenticode sign our deployable packages (i.e VSIXes and MSI’s) so they can be downloaded and trusted. Have you ever downloaded something to be told by your browser that it does not look to be trusted, and you should perhaps not install it? So, signing and certifying deliverables is pretty key to building trust with customers, and is a very common practice in use by many software organizations to ensure that what you download and consume are genuinely from the source, and have not been tampered with.
Back when we first developed and released NuPattern (then called VSPAT) before it was open sourced, it was a proprietary product from Microsoft. At Microsoft, any released binary deliverable must go through a rigorous release process to ensure it meets many compliance standards - as do all software deliverables from Microsoft. There are some very good tools around Microsoft for managing this process as a whole. But one of the critical steps in that process is to get your deliverables signed by the organization. Microsoft, like other large software vendors don’t give their staff access to the private keys of their certificates and asymmetric signing keys. Those private keys are guarded carefully, locked up and protected well behind authorized processes. So how did we get our assemblies strong named signed, and other deliverables Authenticode signed at Microsoft?
Well, at Microsoft they have created their own in-house tools and process flows for getting your deliverables signed for you on behalf of the organization. It is a tightly controlled process, and not that easy to navigate to be fair. The basic idea is that you get time-boxed access to the signing tools in a convoluted approval process. Then you login to a internal corporate web site, you go through an online wizard where you configure the kind of signing you want, then you upload the deliverables to sign, and then at some time later (like days) you get an email to tell you they were signed and can be downloaded from a file share, which requires authorized access as well.
I remember that when I first went thought this process: it took about 4 days to get authorized access from some administrator over email. Then I had to meet strict software configuration on my machine, and dependencies in my browser. Then it forced me to download specific version of some ActiveX control, which was problematic in many browser versions, even Internet Explorer! Then, I got immediately stuck on what option to select for what kind of signing I needed. I just wanted assembly strong name and Authenticode signing at first. But there were many options for all the different kinds of things and keys they have to support. (As you might expect in a company that size, that sign anything from every Microsoft product on the marketplace ever!) After someone (with prior-experience with the tools) gave me the right coordinates, I was away. I would submit my batch job and wait a few days for the results. Overall, end-to-end the process took several days to release the bits – just from signing. It required intense manual configuration in the signing tool, and multiple rounds of signing in our case (see later). Naturally, I was never really very comfortable that I hadn’t forgotten a signing step, or configured each step correctly. The only sense of comfort I could find that things were done right, was intensive, tedious and manual verification that the bits were indeed all signed in the right places, for which there are very rudimentary tools available to help.
When the project was finally assigned to Outercurve and made open source, we faced the same issue of how to get our bits signed with the private keys held by the Outercurve Foundation. At the time, signing bits for other .NET projects at the Outercurve was not in high demand! Which I have to say surprised me, but on reflection appears to be representative of many software organizations. Why? Simply, because providing tools that do such a task in an automated fashion (even partially automated) are HARD to implement correctly and reliably. This is evident from the lack of any online services that can provide such a service to software organizations. [There an idea for an online entrepreneur. Not without its trust challenges I might add].
At the time however the Outercurve strongly desired to support such a key scenario to its projects, and work began in earnest to provide this signing service to its project owners. The results of that push are now realized and fantastic. We have Eric Schultz to thank for all his hard work and dedication for getting us there, and furthermore, for enabling the NuPattern project to continue creating regular releases. But first let me explain how we got there.
At the start, there was nothing except a couple of private keys for both strong naming and Authenticode signing assets, and a will to create a signing process.
Eric began working on providing services for automating the signing of project assets. I actually don’t know the details of what Eric started with or how Eric began the work (perhaps you can ask him yourself), but at some point, shortly after, we were working together intensely on Skype signing our assemblies and packages, by exchanging binaries over a online file share (Skydrive), and some time later (minutes to hours) Eric would pass me back my bits signed.
From the project side, initially we had no automation to help us provide Eric with the right bits at the right time. We started off with an automated build that yielded our built packages (VSIXes in this case, which are renamed ZIP files), and an MSI installer. But our VSIXes contain other VSIXes like a Russian doll they nest inside of each other, and every VSIX contains one or more assemblies that would need strong name signing. We also needed to Authenticode sign every assembly, every VSIX package and the MSI package. The signing process, end-to-end requires several steps, with multiple signing phases, it goes something like this:
- Extract all assemblies from all built VSIXes (we have 8 VSIXes, and about 50 assemblies)
- Sign all 50 assemblies (both Strong-Name sign and Authenticode sign)
- Reconstruct all 8 VSIXes with the signed assemblies.
- Sign the 6 nested VSIXes (Authenticode sign)
- Reconstruct the 2 containing VSIXes with the 6 nested VSIXes
- Sign the 2 containing VSIXes (Authenticode sign)
- Recompile the single MSI installer which packages the 8 signed VSIXes
- Sign the 1 built MSI installer
That’s four signing steps each with different assets, each with different signing keys.
Even though we could automate the building of the code to yield the VSIX packages and MSI package, and automate the extraction of the unsigned assemblies from those VSIXes with some basic batch files, the reconstruction of the VSIXes with signed assemblies had to be performed manually using tools like 7-Zip on the desktop. Over the course of a few hours, we had built a few automated batch files for performing one or two of the steps. None of which automated either the signing or reconstruction steps. You could see those initial batch files in a previous version of the source code (now removed).
The first time around the process, it was basically Eric and myself on Skype uploading the files to a Skydrive, and then Eric downloading them and running his signing routines on them, then uploading back the signed results to the Skydrive. The overall process took about 6 hours to complete. We both had various interruptions with meetings, home life, meals etc., as we were also spanning an 8hrs time zone difference. We finally go it done just before 11pm at night for Eric, and so he signed off for bed. That left me to verify the results and upload the release to our project site. And wouldn't just you know it, it was only at that point that I discovered I had missed a step in the process, and we had signed the wrong bits!! At this point, I had no choice but to wait another half day until Eric and I could get back together again online, and rather frustratingly/embarrassingly (for us both) go over the whole process again. We got lucky that time, at least that’s how it felt given the number of steps and opportunity to miss or muck up a step – hardly a robust and reliable procedure. But it worked that time, thankfully.
Several weeks passed after that first experience, and Eric suddenly announced that the Outercurve signing service had been created and is now available online!!! He even created a GitHub open source project for it! And wouldn’t you know it, just in time for us to create a new release of NuPattern, which would need signing again. It has to be said at this point, that although we strive to create regular maintenance drops of the projects, the signing process and its overhead had caused some resistance to dropping whenever we wanted to. Clearly, not the best position to be in.
After Eric had provided our project with the necessary credentials to use this new signing service, and some examples of how to programmatically use it (in both PowerShell and MSBUILD) we were entirely unblocked from building a totally automated process around signing our bits. One that could be executed anytime! like after a CI build! perhaps not that frequently, but certainly that easily.
Work began in earnest at NuPattern. We had already discovered the process we needed, we had already documented it fully, and had it partially automated it with some rudimentary batch files. All we need to do was convert the batch files into MSBUILD scripts, and weave into that the signing pieces using the new signing service, using the MSBUILD tasks that Outercurve had provided.
The results of this MSBUILD code can now be seen in a single file in the NuPattern project: SignBinaries.targets. We needed to add the MSBUILD tasks/assemblies provided by Outercurve in our source tree, and also include the 7-Zip binaries in our source tree for this to all work. And we have a simple batch file (Make-AllAndSign.bat) that builds the code and executes the signing process in one click.
The outcome is phenomenal! in one click (or one step added to a CI process) we get our bits built, signed, repackaged and re-signed (as described above)! The overall process takes close to 1 hour to complete. (Most of that time taken to upload and download the various assets to and from the signing service).
In terms of productivity, we have gone from tying up 2 people for about 6hrs, to tying up a CPU for 1hr. But the most important improvement has been the reliability of the overall process. No longer do we have to worry about making manual human errors at any step in the process. The process is entirely automated, and as long as we got the automation steps right, we can guarantee the results are 100% reliable. And that cost saving (in verification and rework) is priceless to us.
At anytime, anyone on the project can now sign the bits. and perform a release. The instructions are fully documented as well.
If you are reading this and are interested in how to reap similar benefits in your project, you may want to talk to Eric. The code for this signing service is open sourced, and I believe its based on a set of simple Azure services. So why not build one in your organization? and save heaps of time and money getting your projects bits signed the right way?
Perhaps this signing service could be the first of its kind to lead the way in a new set of services that could be provided for *any* organization? who knows?