Motley says: "Ugh, not again..."

  • Article

Summary

 

Motley:  Developers keep making the same mistakes over and over. Can't they learn?!? Some kind of punishment is in order, like no more free soda!

 

Maven: Don't just address the symptom - get to the root cause of a problem. Leverage the "Five Why's" to find root cause and come up with solutions that are realistic, cost-effective, and within your control.

______________________________

 

[Context: After taking part in a few structured code reviews, Motley is getting frustrated by the patterns of errors he is seeing]

 

Motley: You know, I'm almost sorry we started this code inspection thing. As the senior developer, I have been participating in quite a few of them. It has really come to light that (a) human beings are creatures of habit, and (b) we don't learn from our mistakes! Every review that we have finds the same kinds of errors. Can't people learn?!?!? Perhaps we should revoke the free soda as punishment.

 

Maven: I feel for you. The real question is, what can you do about it? Taking away soda is not the answer.

 

Motley: I can think of lots of other things too - beating on people, making them wear dunce caps, cutting their pay, -

 

Maven: Whoa! Let's try to be more constructive. How can we proactively cut down on the problems that occur?  If you keep getting flat tires on your car, you could first consider fixing the symptom. Patch the tire.  Buy run-flat tires. Both of these solutions won't solve the real problem, and you'll still keep getting flat tires.

 

Motley: Well, duh! You want to figure out why you are getting the flat tires. This is just common sense. Is it a fender rub? Do I have a cheap brand of tires? Is there something protruding from the wheel? Am I driving in an area that is prone to junk all over the road causing flats? Could be a multitude of reasons.

 

Maven: Exactly! Why not apply this same principle to software bugs?  If we can figure out what the root cause for some of the bugs we find, perhaps we can prevent the problem from occurring at the source and not make the same mistakes again.

 

Motley: Well, um, of course. There's nothing to say, however, that the developers will actually address the root cause and not make the same mistakes. And with software, it is usually quite difficult to actually determine the real reason for a bug.

 

Maven: Well, once you find the root cause, there are some creative solutions you can usually apply to help developers do the right thing. Before we address that, let's talk about how to do root-cause analysis. One simple way is to leverage "The Five Why's". The idea is that if you ask "Why?" five times you'll ultimately get to the root cause of an issue.

 

Motley: Why five? Oh, wait. I know. If someone keeps asking me "Why?" after about 5 times I'll get frustrated and provide a knuckle sandwich for their effort. Five is about all I can take.

 

Maven: Actually, I don't know why it's five. I treat it as a guideline anyway, and not a hard and fast rule. Want to try an example?

 

Motley: I'll humor you, just please don't annoy me.

 

Maven: The foreman of an assembly line shuts down the line because Mark got injured. The manager could just start up the line again or he could pursue the cause of the shut-down. You play the manager.

 

Motley: Ugh. Fine...

 

Maven: Well, ask the question!

 

Motley: Ok, why did the line shut down?

 

Maven: Very good - you are a good actor! The line shut down because Mark slipped in a puddle of water and hurt himself.

 

Motley: Why is there a puddle of water on the floor?

 

Maven: Very good! There is a leak in the roof and it rained last night.

 

Motley: Jeez. Why is there a leak in the roof?

 

Maven: The rain storm last night also exhibited high winds and some tree branches were blown into the roof.

 

Motley: Why was it windy?

 

Maven: Ha. Ha. Nice. The key is to keep the "why" questions realistic.  If you were to ask me "Why did the tree branches poke a hole in the roof?" I might answer because we have some trees that are too close to the building. At this point, we're probably done. Our root cause for the line being shut-down and Mark getting hurt was because trees are too close to the building. These two seem unrelated at first glance, but once you delve into the details, it becomes clear how to fix the root issue.

 

Motley: That's a pretty ridiculous example!

 

Maven: Well, it's fictitious, but the same technique can be used with bugs. Keep asking "Why?" until you get to the root issue for the bugs.

 

Motley: Okay, back to my previous question. Once we find the root cause we can tell developers how to fix it, but there is no guarantee that they will.

 

Maven: Yes, sometimes that is true. However, there are certain types of solutions we can put into place to minimize the odds of the bugs creeping in again. Here are some examples:

  • Run tools proactively: Before we check-in code, we can run static analysis tools like FxCop to proactively find bugs on our behalf. FxCop is great at finding security and other quality issues. Even before check-in we should run this kind of tool prior to code review to nail the obvious ones before we use up the time of the team.
  • Update tools to find our common bugs: Most tools like FxCop have an extensibility mechanism. If we consistently make the same error and the tool currently does not look for it, consider creating an add-in that finds the issue in code.
  • Update coding standards: The developers should all be very familiar with our team coding standards. Make it mandatory reading. Add patterns of errors to the standards document to ensure new developers in particular do not make the same mistakes.
  • Update team checklists: Checklists are important for reviews. If a mistake keeps happening, add it to the checklist that developers use during code reviews.
  • Get training: If mistakes are happening in an area that team isn't very familiar with, consider books or training to fill the gap. If this is our first foray into multithreading, for example, take a course on how to develop concurrent applications.
  • Change development processes: Perhaps the steps that are required before code check-in need updating. For example, consider updating the process to require a mandatory personal checklist-based review before a team review before using the time of others.

 

Commonalities between all of these solutions are that:

  • They are realistic. Doing a rain dance most likely is not going to solve the assembly line issue.
  • They are cost-effective. Building a dome around the factory is not feasible.
  • They are within your control. Controlling the weather is not within your control.

 

Motley: We obviously cannot afford the time to do root cause analysis on every bug!

 

Maven: No one says you need to do this for every bug. Concentrate on using this technique for patterns of bugs. One place to make use of bug root-cause analysis is for a few minutes at the end of an inspection. Have a quick chat that covers the following:

  • What went well with the inspection?
  • What could we improve the next time we do an inspection?
  • What is the root cause of some of the major issues we found?
  • What can we do to prevent those issues from re-occurring in the future?

 

Motley: I am always up for preventing boneheaded mistakes from appearing over and over again. We'll try it at the end of the next inspection to start, and then perhaps think about some of the other patterns of bugs we've seen over the past couple of weeks and address them.

 

Maven: Awesome! I think the team will be better for it moving forward...

______________________________

 

Maven's Pointer:  Root cause analysis is used in many different industries from accident investigation to automotive assembly lines to healthcare. Addressing the symptoms of a problem are usually temporary fixes. Individuals and businesses that believe in continuous improvement must dig deeper beneath the surface to ensure the real problem is being addressed. Fishbone diagrams (a.k.a cause-effect diagrams or Ishikawa diagrams) are a tool to help you visualize and delve deeper into problems to discover root causes. I prefer a classic mind map, but the principles are the same.

 

Maven's Resources: