FAQ - Why do I need Software Factories?

OK, so first off I want to address the more personal, individual aspect of this question, rather than the more broader industry angle - which can be found in many other references online and offline (books, articles and the like). These other references describe in detail the drivers for software factories, and why we as an industry need them to industrialise software development. But that's not what we are going to explore here.

Instead, I want to tackle the personal view points of the folks perceived to be at the 'pointy end' of this proposition - the developers. It is the developers (the authors, and users of the factories) who ultimately decide the fate of factories, because factories become their new 'tools' to get part of their jobs done. It should be no surprise to anyone, that this won't happen if they (our 'project saviours') don't want to use the tools.

I want to explore the reasons, (during this evolutionary period of software factories), why some of our software professionals, particularly the 'eager-professional' developers, may find factories hard to embrace. I want to explore what lurks behind that, where that cynicism and fear comes from and why.

[I define what I mean by 'eager-professional' developer later on in this post. If anyone has a reference to developer behaviour profiles I can leverage instead, please comment - I'd like to accurately and precisely get to the right term for the behaviour and circumstances I am trying to convey here.]

This has got to be the most controversial question to answer, since:

• We are dealing with individual emotions, opinions, perceptions, experiences of the folk who find themselves in circumstances not favourable for this type of new automation. So, I am expecting that not everyone will see eye-to-eye on this. I merely offer observations from this moment in time, in a new evolving and fluid marketplace.
• We are also dealing with essentially disruptive technologies to replace well-established technologies and tools in an existing marketplace, and therefore this new technology requires (demands) a new approach, with new optimism and newer goals than have been previously defined.

Answer

What we face, with factories, is a departure from established development attitudes, behaviours and approaches towards a vision that has previously failed to deliver - sometimes more than once (in some people's professional careers). We are bound to face resistance and scepticism from the purists, the development empires, and individual kingdoms set up in development establishments, from the persons who have established livelihoods and comfort zones out of developing software the inefficient, unaffordable, one-off, 'artistic' way it's done today. It's a natural reaction to change, and its healthy, since it qualifies exactly what, and whether, factories can actually invoke and achieve industrialisation of software development in the future.

Now - let me scope this back. What I want to focus on is here is how a certain group of developers today feel about this impending change - the 'eager-professionals' . What some of the opinions are of these groups, and why they (quite rightly) are sceptical and downright passionately adverse to what could be perceived as a god-send to others in the profession.

I define 'eager-professionals' as those professional developers who are typically, but not always, less experienced in small companies or more specialised in larger companies. The eager professional feels the need to prove his worth by displaying his prowess at the art of software development. Since the eager professional is driven to prove himself, he tends to compete with other eager professionals in what they perceive is the most important skill - hand writing the best code possible. The eager professional focuses on solving individual problems, and solving them in the most elegant and most efficient possible way, and evolving that in every iteration. That’s a very valuable and necessary set of skills in any development project. However, excelling at these skills puts a lot of self-emphasis (over emphasis perhaps) on their own unique knowledge, their understanding of that knowledge, and the value of that knowledge, as well as the tools they wield - their programming languages. The eager professional considers his freedom to choose his tools and techniques to be absolutely necessary. Otherwise, creativity and innovation will be stifled and the solution will be sub-optimal. For this reason the eager professional strongly resists any perceived threat to his freedom to apply his tools, knowledge and techniques as he sees fit. The 'eager professional' has an ardent distrust for other development professional's contributions brought to bear on, or mix with, their own. It is therefore quite easy to predict that software factories will present a perceived threat and - specifically to this group of developers - evoke strong resistance, stifling widespread adoption - for a number of reasons we will dig into later on.

There are other groups of individuals of interest in this context too that we may touch upon. Such as:

[BTW: I am not attempting to profile all developers here, just the ones I feel will have particular resistance to software factories in this context]

The risk-adverse 'mature-professionals' who have fought and bore the brunt of past failed software revolutions, some they pioneered themselves, who are naturally sceptical. But I believe, it’s the mature-professionals who are ultimately more open and optimistic to change - albeit, super-cautious about it. They are quietly, positively waiting for it to succeed in a big way, so it can vanquish earlier failed attempts, some of which they championed. In fact, would argue that it will be these professionals who in fact make it happen or not, since they see a more maturer picture of the strategic direction of their groups.

Then there are the 'purists' whom are chartered with unraveling and championing the essence of what lies at the core of software development. They are naturally adverse to this kind of change and automation because its perceived to understate the pure art or source of development to which they dedicate their passion and theoretical studies. They will probably, not for a long time, embrace 'such a generic mongrel, and they will scoff at this perceived de-throning of their highly insightful knowledge as being of primary importance to the mass software development audience.Those that follow the purists religiously will also share the same opinions.

Then, there's an interesting group of very ordinary professional developers, I am calling the 'apathetic-practitioners' . In my experience, by far the largest group presently. These are the guys who don't necessarily tout any techno-religions, who are not driven to excel or be creative, who work together to just do their job and return to real-life at the end of the day. Their opinions (and pain) are based on what works and what doesn't - they vote with their feet. They are not paid, nurtured, incented or given latitude to care about the best way to do their jobs, or what the latest techno-trends are, or who the demi-gods of software development are. They don't participate in the raging forums, attend the latest tech-events, read the cool-blogs and techno-sites, or immerse in the best practices. They just want to get the job done because of the circumstances they are in. They find a way to do it, and if it works, they stick with it. As a result, these guys bear the brunt and pain of all past failures and attempts of the other more eager groups and the pioneering attempts of the industry to make life easier for them. They of course, by necessity, will always find the easiest path through the mazes and mine fields presented to them. They couldn't care less if it fails, they would just again raise their eyes to the roof, tut to themselves, and work around it. But if it succeeds, then they are fully onboard. Because it's for these people, collectively in the masses, where it would have the most significant impact on everyday work, and therefore sizable benefits for their businesses, their markets and the industries they work in.

Why the fear?

Now that we have roughly categorised our developers into behaviour profiles based upon their circumstances, we can start to analyse the reasons why software factories will meet initial resistance as they emerge into the existing marketplace.

Most of the objections and resistance we will face (particularly for the 'eager-professionals'), from the introduction and use of software factories, will be from fear of what factories will mean to the individuals themselves, rather than having any technical basis. (and boy are we good in this business at de-bunking ideas based on a marginal technical bases!!). That is not to say that factories are the applicable silver bullet for all developers in all situations, certainly not. Quite the opposite, it is a narrow band of software solution domains that factories apply to, but even in the cases where factories make perfect sense, these threats are perceived as encroaching insidiously on the individuals purpose, usefulness and status.

What I have noticed from my own experiences as a professional developer and from evangelising software factories to developer professionals in the field, is that there are several ways for dealing with fear from these types of perceived 'techo-threats'. For some, denial is a safe and comfortable place to hide. For others it's dis-association. Many others just 'pick the battles' worth fighting. Now, I am no psychoanalyst or professor of human behaviour, but what I have noticed, is that a common way to get over these types of fears - if you decide to confront them, and come out positively on top - is to understand what drives the fears in yourself and then question the basis of that fear within. That all starts with recognising the fear of course, and what threats drive it in you.

I predict that if those who fear software factories were to 'look inside themselves' and identify what threats that fear is based upon, and then rationalise that, they too would come out on top of it. Hopefully then, they would view software factories for what they really are, their value and the benefits they bring many in the circumstances they were designed for, rather than a wholesale threat to their individual status.

This last point is leading to a new discussion about how a future of software factories may change individual status and the mix of developer skills in any given organisation. The required skill sets, the expectations put upon them, the types of creativity and design skills they will need to exercise and the impact that will bring to the end customers. I'll address this in a subsequent post called: 'FAQ - Where are my skills going to be required with factories?' (coming a little later on) but I do touch on some of it below.

What are the threats?

So, finally. Lets have a look at some of the perceived threats to these individuals, primarily from the point of view of our 'eager-professionals' who see themselves as expert professionals. Let's have a look at what's concerns the individuals and how that concern is actually a benefit of factories.

These threats are primarily aimed at those expert professionals who create solutions by hand today. Although most are related, I have tried to tease out the differentiators.

[BTW: This list is obviously not exhaustive. I fully expect others will have identified other threats through their own experiences, which I would encourage you to share in the same format.]

Concern: 'Lowers the bar' required to create software solutions. ('Open the country club to the riff-raff').

Description: With factories, it might appear that novice or undisciplined, even cowboy developers now have the power to wield a 'powertool' such as a factory to do the same job as a 'hard core' expert professional developer would do. This is a similar perception many C/C++ developers had when VB developers became mainstream. This leaves the impression with the experts that the art of software development will be lost (or depreciated) through the abuse and misuse of the 'tools' (in this case - the programming languages). The expert developers are concerned that the markets will appreciate mediocre quality, and cheap solutions rather than precision engineered, aesthetically designed and quality products.

Fear: Depreciation of deep technical understanding, principals, techniques and materials required to coalesce into quality solutions. Fear of technological change.

Benefit: Allows the organisation creating the product (using the factory) to focus upon business requirements implementation and logical design rather than technical implementation details.

Solace: The factory assets will be built by the expert professionals with the deep technical knowledge to encapsulate the principals and techniques for creating a specific solution for a given set of requirements. Therefore, there is little loss of quality or precision engineering of the solution. Unlike the introduction of high-level generic programming languages such as VB, which made creating whole applications much quicker and easier (RAD), software factories are a much more specific tools confined to well defined, managed environments to operate within. The potential for widespread poor use and abuse is greatly reduced since the factory (through the factory schema) defines precisely how the product is to be built, its bounds and the variability of the product and its pieces. 

Concern: Can a tool really do the job of a expert software professional today?

Description: Introducing a tool which can automate the creation of source code for whole or part of a complex solution (rather than just discrete components) appears to an expert developer as means for less skilled developers to do their job - as good, if not better than them. This of course inevitably asks the question "what therefore, is my purpose? ".

Fear: Loss of relevancy, value in the project/organisation- redundancy?

Benefit: Developers with higher level skillset's are empowered to develop the solutions using the factory, more cost effectively. These developers won't be required to possess the highly specialised (and therefore expensive) solution domain expertise to create these solutions in such detail (at such a low level), since that investment has been captured and reused within the factory's assets already created by the expert developers.

Solace: Expert developers skills are still highly valuable, perhaps more so, since the potential scope for re-use is now much greater in a factory rather than in just one-off developments. The pursuit of excellence in low level development skillset's will still be most definitely required. However, the demand for them will no longer be in the bulk of creating well known solution domains that factories address. Instead, expert developers will be required to create the factories themselves and/or the assets within the factories, as well as the customization of the factories and the refinement of their products. Furthermore, there will always be a requirement for customized one-off solutions in the market place, but whilst they are common today, they will be less common in the future.

Concern: Can a tool address a whole solution domain? Shouldn't we be hiring more domain experts? (Generalisation vs. Specialisation)

Description: Solutions today are becoming more and more complex addressing more and more solution domains, each requiring more specialised domain specific expertise. Organisations today therefore require possession of these domain expertise to create even the most basic solutions. This is, for most organisations, a mounting challenge. Until recently, few have conceived the idea that a single tool could address a whole solution domain, and therefore take it upon themselves to master most of all the expertise of these domains. However, this has its practical resource limitations, and poorly or over-engineered inferior solutions are re-crafted or cobbled together to make up for the lack of these expertise. The introduction of software factories brings with it a requirement for specialisation and generalisation, and for most expert developers today, re-use of other's domain expertise will be required to be profitable. For the end customer this means higher quality solutions, at lower costs. It will no longer be acceptable in the marketplace to deliver poor quality standard solutions, which implies for most expert solution developers today that being a generalist in an average delivery capacity will no longer be profitable. Factories will require some expert professionals to generalise solution domains, and some expert professionals to specialise in smaller domains. The generalists will define the factories that assemble the assets created by the specialists.

Fear: Becoming a specialist: become more accountable for a more specialised higher quality reusable asset for use in many solutions. Becoming a generalist: move to a higher level (perhaps less-detailed) understanding of particular technology domains, but more focused on bigger picture scenarios.

Benefit: Better products, higher quality, newer markets, and sustainable businesses.

Concern: Are we to trust this person's implementation (i.e. source code) of our unique problem space? ('Not invented here')

Description: Using factories, to create a products, brings advantages in time-to-market and high-quality which fulfils the customers demands today. These drivers are going to force the software creators to assemble factories from standard assets not necessarily built within their own organisational boundaries. These standard assets are best acquired from the domain specialist organisations that specialise in the best-practices and principals of these domains. For some factories, particularly the first generation factories we see emerging now, this will mean the introduction of assets that create source artefacts - visible, editable source artefacts (as opposed to binary obscured artefacts). Since source artefacts are the focus and domain of most expert professionals today, these assets will be perceived as providing implementation that could have 'just-as-easily been created in-house'.

Fear: That my/our implementation is not good enough, and the other's might be better.

Concern: Is source code really not the most important artefact anymore?

Description: Today, for most expert professionals, it is the creation and management of source code artefacts which defines their purpose and status within an project. Since the primary first-class artefacts of Model Driven Development (MDD) are models, this may be perceived by most expert developers as being no longer any place to realise their value.

Fear: That my/our speciality, skills and mastery of programming languages are no longer more valuable than the application of them.

Benefit: As factories evolve, developers will certainly focus on higher level artefacts as their tools become more graphical and abstract, as the lower level artefacts are generated as outputs of the factory for verification and compilation. The interfaces for configuring and changing the product of the factory may become more important than the organisation or structure of any source code implementation underlying them. (In much the same way it is with C#/VB over the machine generated byte-code today).

Solace: Although source code artefacts and management of them are the focus of most expert professionals today, this may not necessarily be the case or focus for them in the future. The first generation of factories will very much be focused on source code assets to bridge the modelling/automation gap, but in the future it probably won't be the case as we move up to higher level languages and platforms. We may even see in the next generation of factories, models which generate assemblies or other binaries directly, and certainly (even now) factories which generate no source code artefacts at all (no source code, models or otherwise). However, today and for the first generation of factories at least, there will most likely be a requirement for fine tuning or refinement of the factory created products at the source code level. So the requirement for these skills will certainly not dissipate with the advent of software factories.

[Much thanks again to Martin Danner for helping me precisely articulate some profiles and Kimmo Forss for some very good idioms for my concerns.]