Software engineering—what’s missing?


 To start the new year, my boss gave an all-hands speech to a large group of developers about being an engineer. He equated being an engineer with taking responsibility for quality and using methods that ensure high quality at checkin (Nailing the nominals). Naturally, a developer in the crowd took issue with calling software developers “engineers.” He claimed, “Engineers assemble parts out of tables and charts of proven components. Developers create something new and unproven every time.” Some disputes never get old.

Is software development engineering or craft? I’ve already argued that it’s pointless to argue. Developers must act like engineers or we all lose our jobs. No one bets his company or health on well-crafted software—it had better be engineered. As I claimed in “A software odyssey” (chapter 5), developers need to know instead of guess. They need to use instrumentation data from their products and engineering systems to guide decision making and drive quality and improvement.

But what about all those handy tables and charts “real” engineers use to assemble parts? Does software have an equivalent? Why do new language paradigms claim to solve the reuse problem, only to later fail? Can we be engineers without repositories and reuse? Hey, control yourself! We do have code warehouses, language paradigms are misguided, and we are engineers. Quit waving your rationalization crutches around, hang up your cowboy hat, and be a professional.

Eric Aside

I had an email conversation with the guy who asked the question. He was interested in setting up a repository, which got me thinking.

If at first you don’t succeed

Since the 1960s, there have been numerous attempts to establish software repositories and reuse. Many of the advances in object-oriented languages were accompanied by sincere claims that they finally solved the reuse problem, and soon software development would take on more of the character of other engineering disciplines. Unfortunately, those claims haven’t been realized.

At Microsoft, there have been many failed attempts to establish code repositories. The most recent internal attempt is CodeBox. It enjoys great success as an internal “open-source-like” collaborative development environment for tools, but it’s not a reusable code repository beyond copy/paste (a separate subject I address later).

Why have these attempts failed? Is there no hope? Oh, please. Folks have been thinking about the problem the wrong way. Imagining parts as class instances failed as an analogy—engineering reuse is quite alive and well in software.

I got what you need

There are two proven instances of code repositories and reuse that work and work well—libraries and design patterns.

Software libraries, including DLLs, interfaces, platforms (like .NET), SDKs, web services (mashups), and even JavaScript libraries like jQuery, have been wildly successful. They have achieved broad adoption, driven tremendous savings in development effort, and resulted in significant quality and security improvements to software systems.

Design patterns have also been highly successful, though not quite as broadly adopted as libraries. They provide proven design approaches for common engineering problems that result in more robust and reliable code that is easier to explain and understand. Design patterns are an old idea used in many other engineering disciplines. They aren’t reusable parts—they are approaches to common engineering challenges.

Something borrowed, something blue

There is a third, broadly utilized example of code reuse—copy/paste. Code “borrowed” from another engineer is prevalent in every significant codebase ever studied. Borrowing code responsibly can save you time and effort by using the best accumulated knowledge available. Borrowing code irresponsibly can cost you your career and your company millions of dollars. The specifics of who owns the code and how it is licensed are critical, but that’s a discussion for a different time.

Microsoft is extremely restrictive about borrowed code. Even internal efforts like CodeSearch are secured so that only people working on the same product can search their product code. The areas with the most code “sharing” are engineering tools and appropriately licensed code samples from SDKs.

Look, I love “sharing” and “borrowing.” My kids were taught these concepts in toddler group. “Borrowing” responsibly is part of being a professional, but it shouldn’t be confused with reusing documented, measured, tested, and published libraries and patterns from an established repository. Those proven repositories provide predictability and repeatability to engineering—the attributes that guy was seeking when he questioned whether or not developers are engineers.

A tremendous sense of isolation

Why do libraries and design patterns succeed where other repositories and reuse fail? I think the key is isolation.

  • Libraries are isolated from the rest of the code. They form a separate logical layer that developers can program against almost as a black box. There’s no need for the vast majority of developers to change the library, and most never even see the library code. This makes for stable libraries and clear separation of responsibilities—quite sustainable.
  • Design patterns are isolated from code by their very nature—they aren’t code. They are abstractions of code structure that defy fixed implementation, which also makes them stable and sustainable in their own way.

Note that while both libraries and design patterns provide abstraction, abstraction isn’t enough—you need isolation. If the developer ever has to mess with the repository for a typical project, then the isolation fails, and the repository becomes unstable and unsustainable for reuse. It’s a bummer, but that’s how things have gone over the past 50 years.

Eric Aside

Note that copy/paste succeeds as a reuse approach for the same reason—isolation. Copying code from another source does not impact the source in any way. This is a blessing and a curse. Should the original source have a serious error, such as a security vulnerability, that error is propagated. However, unlike libraries, a fix to the original code does not propagate to where the code was copied.

What are we doing here?

If software libraries are the only form of proven code repositories (like charts and tables for other engineering disciplines), does software engineering fall short of engineering? No, software development is just like every other form of engineering practice.

If you are an electrical engineer designing circuit boards, each new circuit board is unique. You determine what the circuit board’s inputs and outputs are and what it needs to do. You look through your tables and charts of circuit board components to find the ones that meet your specs. You then use common electrical design patterns, along with your own best judgment and creativity, to uniquely arrange and connect the components on your new board.

If you are a mechanical engineer designing transmissions, each new transmission is unique. You determine what the transmission’s inputs and outputs are and what it needs to do. You look through your tables and charts of gears, shafts, and clutches to find the ones that meet your specs. You then use common mechanical design patterns, along with your own best judgment and creativity, to uniquely arrange and connect the components in your new transmission.

If you are a software engineer designing an application or interface, each new piece of software is unique. You determine what the software’s inputs and outputs are and what it needs to do. You look through your online documentation of software libraries to find the ones that meet your specs. You then use common software design patterns, along with your own best judgment and creativity, to uniquely arrange and connect the software components in your new application or interface.

Living the dream

There are a few lessons to be learned about being a software engineer from analyzing software repositories and reuse.

  • We can encourage reuse by designing well-isolated architectural layers, creating robust libraries and interfaces that reflect how developers use them, and communicating effective design patterns that propagate engineering expertise.
  • Having great software repositories isn’t enough. We must follow other essential engineering practices, such as basing our decisions on data as well as experience, investigating applicable design patterns and libraries we can use, and taking responsibility for quality by having our designs and code peer reviewed, our code analyzed, and our units and components tested.
  • Put those pieces together and there’s nothing missing from being an engineer other than demanding that minimum quality bar from each other and exceeding it ourselves as professionals.

You can be an engineer and you should be an engineer. Stand proud. Be strong. Win the hearts, minds, and well-earned capital of our customers.

Eric Aside

This is my 100th column. I love that it’s about quality and engineering—two things I really care about.

Many thanks to all the readers who’ve spent a little of their valuable time reading my cathartic rants each month. Ranting to you sure beats talking to myself.

0212 Software engineering – what is missing.wma

Comments (6)

  1. Ooh says:

    Thanks Eric for writing this column! It's great to read every single time and I learn a lot from the way you approach and discuss this type of issues. Looking forward to the next 100!

  2. NW says:

    Congrats on 100!

    I really enjoyed this article, but while I do agree that libraries are "the right answer" to reuse, there is a huge problem with them that you don't mention: libraries are almost never "reused", they are simply "used more than once." Snippets (copy/paste sharing), while they do require more responsibility on the part of the borrower, are manifestations of developer experience. They a free side-effect of a project. "Robust libraries and interfaces that reflect how developers use them" are not: they are projects in themselves. Opportunities to build them need to be identified, they need their own requirements, design and development cycles, and users to provide feedback. Isolating code into a good reusable library is *hard* and takes time. Reusable library projects can be successfully scheduled and budgeted into larger projects that use them, but not without an increase in cost that needs to be justified.

    Like any project, truly reusable libraries don't make it out of the gate unless the value they will deliver upon completion is known up-front. Unfortunately, unlike other kinds of projects, the value of a reusable library is hard to see up front, especially when your team could be expending their effort to deliver other projects that external stakeholders are depending on. If a team has a long term vision and a budget that enables them to find opportunities to reduce future work by developing code that can be used more than once, more power to them, but that's definitely not always the case.

  3. The other EricB says:

    Great article Eric. I always enjoy your insights. I've always seen engineering as the process of understanding the array of possible solutions and choosing the best one given the problem to be solved. Traditional engineers do that and good software developers also strive to apply the best solution given the constaints of the problem and the project. There is always a range of choices: which language, framework, libraries, patterns, etc. makes the most sense given the constraints. Your analysis of reusablity is also spot on. Thanks!

  4. Captain PAF says:

    Great article.  Reuse is one of those never ending topics within the industry.  I think that if you really try to solve the "reuse" problem that you will be miserable for the rest of your career.  80% of most reuse is the copy/paste approach and this sort of "organic" reuse is fine just as it is.  The other 20% can be classified as "organized" reuse or platforms/frameworks.  Being intentional about a few key services/capabilities and making sure that you have good controls and process around them can be very beneficial.  If you try to pick one way or ther other across the board you fail.  

  5. Andrew says:

    The primary problem reuse solves is that it reduces the amount of information software engineers have to hold in their head to be productive. However this doesn't mean that those engineers can be ignorant of how reused components are implemented. Every productive engineer I've ever worked with understands to some level how the layer below them is implemented. This is certainly true in the hardware world where datasheets and other documentation describe the internal architecture of chips and devices, sometimes down to the actual circuits. The same has to be true of software libraries as well. Reusing libraries that have poor documentation is a recipe for disaster, doubly so if the source code is unavailable. in my opinion MSDN is a large part of why Windows has been a very successful platform to develop against.

  6. ecommerce reviews says:

    Nice post! No doubt, as the time grow, our need are also grow so we discovers the many idea's ,tools which is useful in our work. As i know that to move the procedural language to object oriented language , we need the reusable code. There are more benefit to adopt  this procedure .The best one is that our software are successfully install and works in the any machine.

    <a href="ecommercesoftwarereviewss.com/">ecommerce reviews</a>

    Thanks for sharing the information.