Reusability Rant


During my lifetime (so far) of programming I’ve worked on a many projects that were built around or including external libraries and components.  These are packaged units of software purchased from a vendor for a specific purpose.  However, no matter how useful the software claimed to be, the specific purposes these packages were designed for never quite matched the specific purpose we were attempting to apply them to. 


Never having been directly involved in these purchasing decisions I can only imagine the reasons that led to them.  I suppose it must have seemed like the easy way out.  Instead of building up expertise in a particular domain, you could just buy that from someone else, plug it into your project, and away you go.  The costs of these libraries were minuscule compared to the cost of the expected effort that would have gone into rolling our own.  And no matter how many times I’m certain this all must have seemed the truth, the reality was far from it.


Because the software never actually did exactly what we wanted it to.  Maybe 70 to 80 percent, sure, but never all that we wanted and so we had to compensate for what it did not do.  Now, this would not have been such a big deal if the package was designed for extensibility from the beginning, but not many were, and even those that were often allowed extensions only in areas we cared not to go.  So, in order to actually get what we wanted we had to work around the flaws ourselves. 


Usually, this lead to a lot of trial and error, building advance frameworks to augment or encapsulate the purchased piece, that often solved many of the shortcomings, but never quite all of them, and certainly not the most critical.  Eventually we would reach the conclusion, over many months, that the only plausible way to achieve our goal was to modify the actual source code, of which we did not have. 


Now you might think that we ought to have abandoned ship at this point, to just take our lumps, learn from the experience and move on.  We would most certainly have gained enough knowledge to understand exactly what we wanted and would have been able to build it ourselves.  That might have even been true in some cases, but not entirely practical.  As the project evolves you only learn about its true cost as you forge onward, only aware of any additional costs after you are so deep in that backing out still seems more costly than plugging ahead.  Of course, you never really appreciate the full cost until much later, and by then it is just too late, you’ve used up a year or two and it is time to ship your product.


Of course, if you flash enough money you can convince any vendor to turn over the source code, provided you are not going to resell it, etc.  And that’s usually what happened.  We’d pay handsomely for the rights to modify the code, and we would do it, fix what we did not like, add what we needed and be done with it.  Our product would ship.  We’d have a party and then relax for a few days.  And then we would hear about the big dreams for the next version, the new features and purchases already made.


I suppose that’s why I sort of chuckle anytime someone talks about reusable software components.  Because, it’s mostly a myth.  It only ever works in misguided wishes and dreams. 


Software is only truly reusable at the source code level.  Read it, learn it, write your own.


 

Comments (10)

  1. Mark Vejvoda says:

    Wow, and I thought I was alone on this one. I cannot begin to express my complete agreement through 10 years of experience on this item. Ususally not being the one involved in the purchase but mandated to implement the product(s). I just wonder what stories you could tell us now about just how stubborn those who made the purchase were to move beyond all logic and include it in the project at all costs! I certainly could! And interesting that the purchasers are soldom those who have to work with it 🙂

  2. Mark Mullin says:

    I’d actually disagree pretty strongly – the issue is not that you can’t make reusable code, it’s that it’s damn hard and there are a _lot_ of costs in doing so.

    OO was heralded as the savior to the creation of general purpose libraries, but the argument was fundamentally flawed – it wasn’t that OOD made it possible, it made what was already possible (the std C libraries) cheaper and more powerful (stl, the .net framework, pick your own poison)

    In designing for reusability, you have to ensure one of two truths is truth

    1) What you are implmenting is a closed set – a list is an example of this – you can define a full suite of operations on the list and implement them – OOD allows for further specialization of the list, but without modification, the list is reusable, because it can perform all the fundamental operations – you could say that mathematically it’s a closed system

    2) As you very insightfully observed and damn near every vendor forgets, if you can’t ensure complete closure, (and you really can’t cause someone can always argue some edge case) then you have to account for extensibility, i.e. you must implement a system specifically engineered for extension. Not only extension by specialization (subclassing) but specialization by delegation, i.e. the ability to change the behavior of the base – consider a list manager that never conceived of right to left ordering present in some languages

    There’s no question it’s real hard to do a truly reusable framework. To many vendors make five toy examples, voila, our frameworks reusable, give us the money. Bull. But to say it’s impossible discounts the efforts of people on your own campus – they have done a reasonably decent job with the .net hierarchy, and earlier, when they adopted stl (missing stl concepts in .net notwithstanding (could we please have a richer collection hierarchy please))

    A friend of mine made an observation once that I thought showed his deep understanding of this issue.

    "Will you shut up already on the reusability!!! It’s too damn expensive. I can only afford usable!"

  3. Matt says:

    I would agree with you that particularly constrained problems can have solutions that are generally reusable. I was not implying that all such frameworks are inherently bad. Certainly, the string class in .Net is fantastic for the most part, but many string libaries in the past were utterly horrible. Still, its not these types of libraries that I’m referring to. I’m thinking of more verticle market style products that try to capture a lot of stuff in a general (and therefore not so useful) sort of way.

    For example, you want to display nice bar graphs in your app. Goobysoft has a bunch of graph drawing controls that you can just pop in. Great, except they don’t support the exact correct style you want, type of axes, units or whatever. Maybe the tick marks point the wrong way and you need to change that, but you can’t because there is no such option.

    There is nothing wrong with the product as is, it just doesn’t match your exact need. The purchaser figured it was close enough, so it was a real bargain, except the effort you put in to get the final few features out of it ends up costing you more than it would have to write the thing yourself.

    If details matter to you, and you really think your best option is to buy from someone else, then you need to consider the cost of the source code up front, because you will most certainly need it.

  4. Anonymous says:

    Lazycoder weblog » MS employees continue to advance towards an open source mentality

  5. John Cavnar-Johnson says:

    Really, are you serious? Do you write your own relational database server? Or do you only use an OS that you have the source code to? Maybe you wrote your own compiler, too? I guess you don’t use any of the classes built in to the .NET framework, either. I’m sure your forms engine is better than WinForms.

    This is one of the silliest pieces I’ve ever read. You clearly don’t appreciate the giants on whose shoulders you stand.

  6. matt says:

    I’m totally serious.

    If the precise behavior of a database were critical to the functioning of my product, and no such database product offered that behavior, I would seriously consider building a database product or buying rights to another product’s source. I’ve never actually had the need, but it could happen.

    The same with the OS. Luckily, the OS is often extensible enough as is to solve any discrepancies.

    As for forms packages, yes I’ve built my own. I’ve also worked on projects that started off using standard forms packages such as MFC and later switched to rewriting those libraries based on need. I’ve even worked on projects that had purchased rights to VC runtime and shipped our own. Many projects I’ve worked on have in fact had their own compiler, when a feature of the product was to expose programmable capabilities.

    One interesting fact to point out, is that prior to .Net, most Microsoft products were built without using standard libraries offered by other Microsoft products. And the Windows team do have their own copy of the C compiler.

    Eventually, yes, a problem domain becomes so well understood and the compilation of functions in a library become so complete that there is seldom need to stray from what is available. At this point, you can indeed stand on the backs of those giants.

    However, this is not the case for most products sold in the market for resuable components or libraries.

  7. John Cavnar-Johnson says:

    I still don’t understand the point your post. You say you worked on a lot of projects where you had to use third party components. They never do exactly what you need (Well, duh, anybody who’s been in software development for more than 5 minutes knows that). Are you arguing that it’s always wrong to use other people’s code or that you’re just some much smarter than the rest of us that you should not be forced to use third-party software? Or, is it, as your replies to the comments indicate, that you’re more than willing to use other people’s code when it’s what you need? And if that’s the case, should you be directing your anger at the people who chose those components for you rather than the component vendors? Because I don’t recall ever seeing any advertising that said "This component does exactly what Matt Warren wants!"

  8. Jonno says:

    One of the very few sensible things I ever read by ESR was The Art Of Unix Programming – http://www.catb.org/~esr/writings/taoup/html/

    Somewhere in there he talks about how it’s the fact that unix tools all read text from stdin and write to stdout that allowed for infinite recomposition and reuse.

    I think the biggest problem with most 3rd party components is not the lack of source, it’s the fact their outputs cannot be easily piped in to some other component’s input.

    Consider you’re graphing component example – even if I had the source I don’t really want to go poking around it. It would probably be easier to integrate if the output of the component was something more easily hackable – SVG or something that can betransformed using standard tools.

  9. Matt says:

    All I said was that the purchased libraries did not satisfy some need we were trying to apply them to, not that they were inappropriate for what they were designed.

    My point of the whole post is to not underestimate the cost of buying a prepackaged component, as you will likely spend as much in effort to work around its shortcomings (relative to what you actually need) than it wouild have cost to build it yourself.

    In cases were that is not true, where the cost of building it yourself is still unreasonable, it is prudent to go ahead and buy a license for the source code right up front, because you’ll need and it will save you a lot of time and anguish.

    Still there may be those libraries that are exactly perfect for what you apply them to and there is no need to do any of this. However, this has not been my experience.

  10. nb5 says:

    I have been throwing around ideas for writing my own operating system for a while (OK, so call me wildly ambitious — I am). I have two goals in particular for this OS that I was happy to see an application for in this post: it would be open-source (no cost to see and modify the code) and even the GUI would be built on a greatly expanded pipeline model, a little like PowerShell’s, but much richer (allowing piping applications together).