Recently, I did an interview about factories for an online magazine (InfoQ) who were asking me to explain what factories were all about. (Here I am thinking everyone knew already!)
To help explain building them I wanted to use a practical analogy. I had hoped to appeal to those of us from a hands-on background using a better understood metaphor. (Everyone is sick of the automobile factory analogy!)
Whilst admittedly, most may have never have owned, fixed or built a motorbike, most boys (and some girls – I am desperately trying to be inclusive here) out there would be familiar with some of the basic terminology used and such – besides which I’ve kept it fairly non-technical to reach a broader audience.
Incidentally, the story actually started off being an analogy about building Airfix models, like the ones we used to build as kids (well I guess, some of us anyhow). Imagine, you can just see the grey plastic wire frames and moulded parts, and smell the enamel paint and feel the glue sticking your fingers together. But the analogy ran a bit dry because, unlike software factories, you don’t really need many custom tools to build airfix models (unless you count the paintbrush and tube of cement of course).
I quickly switched to a better story where you do need a few custom tools, and you get a lot of the parts in raw material form as well as ready made components and detailed instructions – it’s just that the box it comes in is bigger than airfix perhaps. So, the analogy was born around building kit-motorbikes. Of course, it could have just as easily been kit-cars or kit-planes. But custom kits in this sense are pretty close to where we are with factories and software today.
Now personally, I’ve never ridden a ‘cruiser’ type motorbike I describe in the story below. I definitely had a certain friend in mind for building that type of bike. (Myself, I started out riding and fixing Vespas and then graduated to various sizes of Japanese plastic missiles). But for most, cruisers are much sexier than Vespas, but in fact, in retrospect, choppers would have probably been much sexier to use.
Well that friend of mine recently pointed out that he thought the analogy was a perfect vehicle for describing a software factory and explaining the jargon, so I thought I’d re-share just that part of the story here again for you (just in case you missed it).
You can find the complete interview here. What follows is just the relevant middle part of the Q&A piece.
JES: I’ll assume you build custom software solutions today for your customers. Let’s see how we get from these standard parts to building a software factory. I’ll also introduce the factory jargon as we go.
Let’s say that a customer, wants a custom built motorcycle, and they ask you to build it for them. Something a little different from what you can buy off the shelf, not factory-made (excuse the pun, I mean not a standard model), but custom made – but not from scratch – a standard type of bike model like a ‘cruiser’, with a few personal modifications.
Now clearly, the customer is asking you to do this, because (a) they can’t buy this bike off the shelf from a store, because the store only sell standard models, (b) they don’t want to specify a bike from scratch, a specific known classification of bike will suit their needs for now (the ‘cruiser’ type), (c) they want some personal special modifications done to this bike, and finally, (d) clearly they would rather pay someone to deliver it than take the risk of doing it themselves.
Typically, as a bike builder, your simplified options are:
- Build it from scratch – very risky, and expensive and low profit for you
- Pay someone else to make one for you (outsource) – expensive and low profit for you
- Buy a ready-made kit of one (a ‘cruiser bike’ kit), and build it, and customize it yourself – cheaper more profit.
Now, this bike you want to build would be a very specific, known type of motorcycle (a ‘cruiser’). The special kit would include most of the pre-made parts (engine block, carburettors, cylinders, wheels etc), some raw materials (fabric, tubing, sheet metal, plastic etc) and specialised tools to construct the bike with the step by step instructions on what to do and what pieces you need at each step. You still need to build it all yourself, and wield the tools in the right sequence of steps, against the right materials and parts. But most of the really detailed hard parts like carburetion, power transmission, fuel injection etc, are already solved and come in ready-made components. You just have to hook them all together to work on a frame you can sit on.
InfoQ: How is this different from how we build software today?
JES: For a lot of successful solution providers, this is the experience of building software today. But, for others it’s quite a different story when you have to start with only raw materials like plate steel and tubing with screws, spanners, welding torches and milling machines, with a rough sketch or notion of how these things should hang together to create the end result. (A certain TV series comes to mind for where they do that very well for custom bikes). Then actually going ahead and building a highly customized, quality, unique one-off bike for your customer and their particular needs. Not that it can’t be done of course, many will try; a few will even succeed, and some become specialists at it. (As we witness in that TV series).
The cost of doing this custom one-off development though is enormous to the end customer, and few businesses can afford to buy into that market. Few customers today can afford to invest in that type of luxury just to get a nice looking functional bike that gets them from A to B in style. The story of software factories starts with a requirement to want to reach out to larger markets, where the solutions don’t need to be so customized and the customers are more willing to make a ‘reasonable’ investment in a high quality product that is completely functional.
The reason the customer came to you in the first place, was (hopefully) you already possess the capability to build them such a custom solution – you know the domain. You can do it at a reasonable cost, and you’ve probably done it before, and could do it again – you own domain experience and solution domain knowledge. It is likely that you probably already did all the work of creating a custom motorbike solution, so you know what to do. Probably, you did that multiple times before and in that process, understood deeply what all the nuances of building a custom motorbike are. All the tips and tricks (practices, and patterns) and more importantly, how it should fit together properly to satisfy larger requirements like: road safety, performance, fuel efficiency, noise control, comfort etc. (domain specific quality attributes) – in essence you are a domain expert. In the process, as an outcome, you probably developed some of your own custom tools to help building the bike (Assets). You probably also harvested and refined certain parts and architecture you found in common with all your previous attempts at building these types of solutions. No doubt, in the process of building these types of solutions, you would have built several one-off bikes somewhere along the line, certainly some failures, and learned (usually the hard way) how to do it better, more efficiently and with a solution meeting better quality attributes.
In short, you have become a domain expert with domain specific assets for building these types of vehicles.
InfoQ: Sounds familiar. What should happen next?
JES: You may have then realised that, with a little extra work, you could create a whole family of bikes that were essentially similar using the same basic design and materials and tools. Perhaps these bikes could vary by having a custom fender design or a paint job, different length forks, or engine size and tyres. But the heart of all these bikes (the architecture), are all the same. You probably identified a group of other people (end-customers) who would be interested in having such bikes (products) of this particular design (product line). Then you identified a group of people capable of building these products for the end-customer (stakeholders) using a lot less effort than you did, with specific instructions and custom tools, because now you have a demand to make more than just you can handle.
You would have then had to generalise the model of a bike, then defined some of the core parts of it in general terms (work products) and how they fit together (a logical product architecture). Along with that, you would have had to also specify what could be tweaked on each of the logical parts to produce a slightly different variant of it that would suit different end-customers requirements (variability). And then, what was left as common to the particular parts you factored out into reusable assets.
Then you probably found a way to describe these parts in a less technical detailed, simplified custom language that anyone (i.e. bike enthusiasts, hobbyists, customers) could easily understand. This language describes the important aspects of the part, and hides the rest (abstraction). It defines what is needed to be tweaked (variability) to create other variants of the bike in your new product line. This custom language is then used to communicate ideas and requirements (domain specific languages). To communicate the complete description of a bike, you would have needed to provide various separate, but related descriptions of different aspects of the bike (view points), some perhaps spanning all parts of the bike: an electrical wiring diagram, the fuel and lubrication system, the cooling system schematic, the physical structural frame design (all abstractions, and models or different views of a bike).
Finally, you would have to communicate how to assemble each of these parts together to complete the overall architecture of the bike – into a solution. The series of steps to take, to assemble and configure the parts, including what parts need to be fit first and in what order (Activities). With that, you also include which raw materials to use (artefacts), and what tools to use (Assets) and further, what settings the tools should use to assemble those parts together.
As you can see, (if you are following along the hidden analogy to software design), this is not unlike the outcome of what most solution providers (people who solve the same problems over and over again for different customer requirements) get as an outcome from evolving their software development of a particular product from scratch today.
This process is quite natural and logical if you care about saving yourself a lot of work the next time you may have do it. This is what we can a product line mentality.
InfoQ: So how do software factories fit in?
JES: The real crux of the transition to software factories from this – is what happens next? Traditionally, in today’s practices, the domain knowledge and experience is in the people’s heads, and in the solutions that they build. And there is little means to package up their assets (intellectual and physical) into a reusable form for others to use on the next project. That’s even assuming the next project seeks to re-use any of it! There is a human attitude adjustment that comes to play in this too. But more importantly is that these assets are probably not in a form of easy reuse in time for the next project. Typically, the instructions and languages are in detailed written (textual) form and the physical assets are in (low-level) undocumented binary form. Even then, the quality and clarity of those assets can be highly variable. Typically, consuming those assets is often perceived more work than it’s worth, resulting in the fallback position of the ‘not built here syndrome’ we see so often.
Creativity is seriously underrated as a powerful driver for software reuse.
To turn these specifically focused assets into something highly reusable, configurable and customizable, and hence capitalise on the value of investment of learning this domain (and the value of those learning’s and assets) – special, intentional steps need to be taken. In software development today it is often these steps that are typically not accounted for, and therefore receive no resourcing at the end of a software project, in preparation for the next one.
The missing piece that Software Factories brings to the table here is a means to systematically capture that value and those assets and package them into an automated form to be re-used by others. Those others, by the way, are not expected to necessarily possess the detailed expert domain experience, but they are expected to be able to define and assemble the same products quickly and easily using the factory.
The process of building the software factory is taking these assets, define configurable abstractions of them (which describe the various aspects of the bike), and then applying automation to translate those abstractions into physical solutions.
InfoQ: Who uses this software factory, and how does that experience look?
JES: The primary users of a software factory are software professionals building the solutions – these are the factory users that the factory builders design their factories for. Using the example above, the software factory displays various abstractions (models, designers, tool windows etc) to these factory users in familiar languages (to this domain) that describe the various features of the bike, and its configuration from different viewpoints. Each viewpoint may only be of interest to particular factory users in the development team. Maybe your team has different skilled resources. Perhaps wiring specialists, engine specialists, safety specialists or branding specialists, they can focus on one particular aspect of the bike, while others complete the others. The factory provides them with the right tools, and the right view of the bike to do that.
The factory provides instructions to the factory users in a form that guides them through each step of constructing each part, using the relevant abstractions – much like a task list. Continuously presenting the user with tasks that can be worked upon or completed, based on the state of the entire product at this moment in time. With each step the user takes, the factory presents the right tool to them (pre-configured with the right settings) to assemble each part. The user simply ‘turns a few knobs, and switches’, and provides/refines the configuration for the part that suits the particular needs of that instance.
InfoQ: What is the value of the software factory to the users?
JES: The value here, using the example above, is that a software factory built for this kind of bike, would enable a hobbyist or enthusiast biker, (someone with considerably much less technical knowledge of the inner workings of the bikes engine, superstructure, and plumbing) assemble their own custom bike.
They do this by selecting the various parts from the toolkit for the factory provides to them, place them in the right place in one of several diagrams (designers) provided to describe the bike, and tweak their various settings to suit their own custom needs. The software factory then orchestrates the automation of the assembly of the bike (usually in stages), and outputs an almost complete bike, which can then be fine tuned or customized by enthusiast.
The added bonus of course, is that the person using the factory learns from it, during the experience, and the experience encapsulated within the factory from the original builders of the factory starts to rub off on them. In a way, the factory teaches you the domain, without initially committing you to investing in learning it from personal practical experience.
InfoQ: You say ‘outputs an almost complete bike’. Please explain.
JES: We say ‘almost complete bike’ in the previous statement, because the basic expectation, even today, is that very few factories that deliver whole completed software solutions to satisfy, ultimately, all end-user scenarios. This limitation is really a side-effect of the complexity of modelling an entire solution domain to the level of detail that would be needed to provide products to address 100% of all end user scenarios, and therefore a complete set of product variants. Initially, in earlier incarnations of a factory, the models are more simplified (abstractions are higher), and they will be fleshed out over time to accommodate more exception cases of variability to satisfy more refined requirements. Initially, for most factories built today we are seeing a target closer to 60-80% complete product, followed with some manual customization. However, that automation gap is expected to slowly rise over time, as that particular factory is iteratively refined, and the general factory building processes, tools and technologies are refined.
InfoQ: In order to provide a toolkit that allows for “simply” assembling the parts with some additional customizations a Software Factory needs to be based on a fixed architecture, doesn’t it? What if I want to tweak some parts to suit my architectural needs? How flexible or composable is a Software Factory?
JES: Indeed, a software factory defines a product architecture, since it’s this product architecture that provides the model upon which the factory operates. We call this the ‘factory schema’, and it tells us what we are building, with what, and how. It is part of the guidance the factory builder conveys for their model of the solution in this domain. How the factory builder presents the variability of this product architecture, is an attribute of the particular factory. Some factories will want to intentionally fix this architecture; some may allow it to be extended. (Architecture can also be a point of variability).
There is nothing to limit the scope of the product architecture. Typically, when we talk about a factory, we assume that a factory will build a whole solution, but this is not going to be the case for a lot of factories. A factory might build only a part of a solution, and then this part is given to another factory for assembly into a larger part, for assembly by another factory into another part – etc. You might do this with multiple separate factories, or more likely, you will have a grand factory, that orchestrates the component factories (or their products), and manages the assembly for you. Either way, composability is a very important goal of factories. In the latter case, we will be sharing or extending parts of a factory schema to achieve this. As demonstrated by the ‘EFx Factory‘, and now the next version of the ‘Web Service Factory‘ (from patterns &practices), your factory can provide ‘extensibility points’ for other factories to plug-into to provide certain components or aspects of your product or architecture for your factory.
This is also a major requirement to take what we call a ‘horizontal factory’, such as a web service factory (WSSF) (that creates a type of solution – any web service), and tailor it (verticalize it) towards a factory for a particular industry or technology vertical. For example, you could customize the WSSF and build a financial payments web service (FPSSF). This kind of ‘vertical factory’, then fixes or extends certain points of the web service product architecture either to constrain the variability of those parts, or specialise them for a more specialised domain. For example, the financial payments web service factory (FPSSF) could fix or limit the data contracts of the messages you could exchange, with industry standard payment schemas. In the former case, the FPSSF would effectively fix the point of variability of Data Contracts in the WSSF with its own fixed Data Contract, preventing the user configuring a data contract at all. In the later case, it would extend this point of variability, and only allow the user to select an industry standard data contract schema.
InfoQ: What is this “factory schema” in concrete terms?
JES: The heart of a software factory is the factory schema (a specification, itself a domain specific language), which tells us above all the architecture of a solution the factory is creating (the product). It tells us what the various logical parts of product are (the Work Products), how they are related to each other, how to view them (View Points), how each one is constructed (Activities), and with what tools and resources (Assets) it should be constructed with.
The instructions for product assembly – the Activities – are presented to the user as a list of things that can be worked upon at that particular moment in time. These are associated to the things you work on (Work Products) and the tools available to manipulate them (Assets). Activities are conditionally-based on the state of product itself; and driven by what the factory users have already done, and what is left to be done. Using the example from earlier, it does not make sense to mount the engine of the bike to the frame of the bike before the pistons and carburettor are in place yet. But you could start painting the frame of the bike, whilst someone else is building the engine. So these activities appear when it makes sense. In this way, instead of presenting you with a long list of unrelated activities you need to perform from A to Z serially to complete the product, the activities become available to you as you (or others) complete work on the parts they need as pre-conditions for further activities. This makes team development with a factory much easier, and provides contextual guidance to the users on what things need to be done now, and what things can’t be done and why (what other things need to be done).
There are several ways to view a factory product (View Points). Like the electrical wiring diagram, cooling system, and lubrication system from our example earlier. The work products, assets and activities are organised by these view points. Each View (instance of a view point) is typically of interest to only certain people working on the product (Stakeholders), and each view makes available different assets with different activities to work within that view. If you were painting the frame of the bike, you wouldn’t expect to be asked to ‘connect the fuel lines’ as an activity, and you wouldn’t need the torque wrenches handy. Similarly, if you were building the engine, you wouldn’t want the airbrush, and be told you could now ‘prime coat the surfaces of the frame’. The view associates the right activities to work with the right assets to complete the work products you are working on now for your role. Clearly, you don’t need to have mechanic’s skills, and specialised tools just to paint a bike.
now, go build them custom bikes then….