Moving to Software Factories
Jack Greenfield and Keith Short, Architects, Visual Studio Enterprise Tools, Microsoft Corporation
Industrializing Software Development
Total global demand for software will grow by an order of magnitude over the next decade, driven by new forces in the global economy like the growing role of software in social infrastructure, by new application types like business integration and medical informatics, and by new platform technologies like web services, mobile devices and smart appliances. Without comparable increases in productivity, total software development capacity seems destined to fall far short of total demand by the end of the decade. What will change to provide the massive increase in capacity required to meet demand? It is not likely to come from adding developers. Instead, software development methods and practices will have to change dramatically to make developers much more productive.
Other industries multiplied their capacity by moving from craftsmanship, where whole products are created from scratch by individuals or small teams, to manufacturing, where a wide range of product variants is rapidly assembled from reusable components created by multiple suppliers, and where machines automate rote or menial tasks. They standardized processes, designs and packaging, using product lines to facilitate systematic reuse, and supply chains to distribute cost and risk.
What will industrialization look like in the software industry? We cannot know with certainty until it happens, of course, but we can make educated guesses based on the way the software industry has evolved, and on what industrialization looks like in other industries. The key is to leverage experienced developers by encapsulating their knowledge as reusable assets that others can apply. Patterns demonstrate limited but effective knowledge reuse. The next step is to move from documentation to automation, using languages, frameworks and tools to automate more of the software life cycle.
Automating Software Development
Can we automate software development? Of course, we can, and we have already. Widget frameworks and WYSIWYG editors, for example, make it easier to build and maintain graphical user interfaces, providing benefits like device independence and visual assembly. Database design offers similar forms of automation. Looking closely at how this was done, we can see a recurring pattern.
- After developing a number of systems in a given problem domain, we identify a set of reusable abstractions for that domain, and then we document a set of patterns for using those abstractions.
- We then develop a run time, such as a framework or server, to codify the abstractions and patterns. This lets us build systems in the domain by instantiating, adapting, configuring and assembling runtime components.
- We then define a language for the domain and build tools that support the language, such as editors, compilers and debuggers, to automate the assembly process. This helps us respond faster to changing requirements, since part of the implementation is generated, and can be easily changed.
The Role of Models
Raising the level of abstraction for developers using higher level languages, such as modeling or visual assembly languages, is one of the key elements of this pattern. We are using UML for documentation, but we are using models based on small, focused, domain specific languages (DSLs) for automation. DSL based tools can help developers define and assemble components, such as web services, generate their implementations using framework completion, and capture metadata used to automate validation, packaging, deployment, configuration management, test generation, defect tracking and many other aspects of the software life cycle. We are using high fidelity DSL based models as first class software development artifacts.
Is that Model Driven Architecture (MDA)? No, not quite. Like MDA, we are interested in models. However, we are less concerned with portability and platform independence than MDA, and more concerned with productivity. While stereotypes and tags can be used to decorate UML models, experience shows that more precise language features are required to support compilation, debugging, testing and other development tasks. Unlike MDA, we do not propose to use UML where programmatic manipulation of models is a key requirement. We use UML for discussion, sketching diagrams on whiteboards and napkins, using the UML static structure and sequence chart notations that are now almost universally recognized by developers.
Not Just Models
While models play an important role, they are not the whole solution. Scaling up to higher levels of productivity will require the ability to rapidly configure, adapt and assemble independently developed, self-describing, location independent components to produce families of similar but distinct systems. It will require a transition from craftsmanship to manufacturing like the ones we have seen in other industries, and it will eventually produce more advanced earmarks of industrialization, such as supply chains, value chain integration, and mass customization. This vision cannot be achieved with models alone. Domain specific patterns, frameworks, and tools must also be developed and applied systematically, and must be aligned with both the product architecture and the life cycle process.
Also, we must address the processes by which we analyze requirements, develop software and deploy it to our datacenters. While prescriptive methods optimize for complexity not change, modern agile methods optimize for change not complexity. To scale agile methods requires the ready availability of best practices, reusable content and patterns. To ensure prescriptive methods are not overly rigid requires flexibility and variability in their description. We are using a methodology, based on these ideas, called software factories.