Pattern Automation – heard of it?

Not a lot has changed in general with software development the last few years with respect to re-delivering proven software solutions more economically and consistently. Whilst agile and lean processes and XP practices have been monumental in given developers back the freedoms (and responsibility) to build better and more relevant solutions utilizing better feedback loops, increased communication  and more end user involvement. These approaches have not really made enough headway in promoting higher productivity gains and economies with reuse of proven implementations. Although agile methods and reuse practices are indeed orthogonal in nature, they certainly can be well aligned.

For the last few years work has been ongoing in that direction. How to combine agile processes, lean principles and new development practices focused on giving organizations the capability to 1) capture their evolving best practices and 2) build their own custom integrated tooling that gives them repeatability of their proven solutions across multiple projects.

What does that mean concretely? well, lets say your organization builds software solutions, or even deploys IT systems. Over the course of doing this kind of work over and over again doing multiple implementations, the experts doing it learn new and more efficient approaches. Experience makes them experts. As they master the various technologies and techniques of applying them to real-world problems, they will desire to reapply them again to similar projects to achieve mastery of them. To simplify complexity and increase productivity, the experts journal informal ‘patterns of implementation’ (rules, constraints, structures, architectures, processes etc.) for the work that they do, and for the solutions that they implement.

Every time those experts come to develop or deploy a new solution, appropriate patterns are sought after and selected for that kind of solution so that they cover most of the requirements of the solution. These selected patterns are then evaluated against the specific set of needs for that specific implementation, and against a possible set of custom needs, which generally drives another need to tailor the patterns to address the custom needs to. The experience should sound very familiar to most. For every implementation, you try to apply proven patterns (from past experience/awareness) and you select ones that most apply to this specific implementation (pattern matching), expecting that some parts need some level of refinement or customization for each implementation (pattern customization). Over time, and over several similar implementations, as the various patterns are mastered, they congeal and harden. We talk about this as factoring out the commonality and variability of the implementations, where the commonality applies to all implementations and the variability addresses the variance between the implementations.

For many skilled experts in delivering these kinds of solutions, they desire to record or document their successes in some form; so they can remember (or teach to others) how to be successful again with them. Checklists and other various other forms of guidance, instruction manuals, architectural documents, experience reports, frameworks, etc. emerge. A few will be actually templatized for reuse, and even fewer are tooled or automated to some degree.

But who has the extra time to do all that kind of work? And what is that going to cost me or the project? The overhead and cost today to do this kind of extra work is far beyond most individuals or organizations. Nevertheless, despite the desire and need to do extra work, the patterns will need to be reproduced, formally or informally and with varying degrees of accuracy, consistency, and predictability in cost and schedule.

So what else is new? For the most part, you will spend most of the time reproducing what you know to be true and proven (harvested from previous experiences) and then fitting those to something new, and always end up running out of time/money for the last 10%-20% of the more valuable, creative and differentiating customization work. Sound familiar?

Well, maybe not if you are constantly trying to master new technologies, platforms, architectures, and learning the new patterns. Consistency and predictability are a goal of those who have to build the same things over and over again, and have already mastered the basic constraints, arrangements and patterns present in any given technology, platform or architecture. For them, the goal has become how best leverage their investment in mastering those things and re-deploying/re-delivering them quickly and efficiently again, so they can move on to the next best thing, perhaps even improving on them. It’s a necessarily creative process however you look at it. And there is enormous value in being able to do it effectively and efficiently.

So for these folks the challenge remains, how do you capture, define and re-deliver (and support and evolve) these kinds of proven patterns consistently and economically? What tools do you use to do that?, and do you have the skills (and time, and motivation) to use them effectively?

If you could define your own custom automation and tooling that reapplies the majority of the useful patterns for you, then that frees you to focus on what is unique and different for each specific implementation. This would then lead to all those highly desirable outcomes like: reuse, high productivity, consistency and predictably, cost savings, efficiencies, higher integrity and raises quality etc. If you could then add to that custom tooling some instructional guidance that explains how to use the tooling correctly, then you have custom tooling that others can wield just as well as you - as if you were working alongside them. Consistency, accuracy and rapid delivery can be realized. And in this way, your knowledge and experience can be scaled-out consistently and repeatedly across your organization and across multiple implementation projects simultaneously. New capabilities can be predictably capitalized on.

If this tooling can be then be further refined and economically tailored for slightly different applications of the implementation patterns, then whole value-chains of specialized expertise can be mobilized across your organization, capturing your institutionalized expertise, and extending it.

And if building this kind of tooling was easily within the reach and ability of those who actually define the custom solutions and evolve the patterns of implementation to begin with, then efficient and pervasive proliferation of best practices from your experts to implementation can be finally realized.

So that is Pattern Automation. It’s the outcome of being able to economically capture, document, reuse and re-deliver proven and known best practices, into patterns integrated into the development environment using automation. It is achieved by defining a simplified structured orchestration of templates, automation and guidance that together yield an integrated ‘custom toolkit’ for building part of a solution. These ‘toolkits’ are then deployed to projects, and are composed together and configured for each custom implementation to create and configure larger parts of the target solution consistently and predictably.