- Unit tests. We have talked about unit testing at length in other discussions. This is the number one quality-driven developer technique. You must have a set of unit tests in place that exercise all of your new code and you can check-in when all the tests pass. This items should be bolded on a checklist.
- Code coverage. Hand-in-hand with unit testing is code coverage. Aim for 70%+ code coverage for all of your code. More important than the coverage number is the feedback you receive from the coverage tool regarding source lines that were missed in testing so that you can add new tests.
- Debug and retail builds. A debug build is generally different than a retail build, particularly when you make use of debug-time primitives such as assertions. You need to build both flavors to ensure the build succeeds either way. Broken builds are not tolerable as they block the entire team. In fact, do a clean build (vs. incremental build) to be safe.
- Static analysis. A static analysis tool like FxCop finds bugs for you. Per the cost of change discussion, why not fix them now instead of later when it is more expensive? Why risk security defects escaping your defenses and being released into the wild where hackers and customers will find them?
- Reliability tools. Running a tool like AppVerifier is more relevant to C++ applications. Running this tool is a must prior to check-in as it will help identify heap corruptions, improper use of handles, and misuse of critical sections. Additionally, for native code it will point out memory leaks.
- Warning level. Crank up the compiler to generate warnings at level 4 (or whatever is highest for your compiler) to ensure that you have addressed all possible compiler-identified issues. Get into the habit of perpetually building at the highest warning level. Get clean and stay clean.
- Other tools. Run any other organization-wide tools such as a copyright header checker, XML documentation checker, or coding style checker (e.g. Source Analysis) prior to check-in. You don’t want to be the grunt fixing a ton of these issues later. Trust me on this one.
- Code reviews. All code should be at least peer-reviewed, with larger changes (e.g. new features) being put through a code inspection. Having at least one other pair of eyes look at your code is likely to find many issues before even the test team gets their hands on the code.
- Change list description. Include complete descriptions for check-in change lists, including how you tested the code, why the change was made, how you built, possible regressions, and other risks.
- Buddy build. To ensure you did not forget to add a file to your change list and to protect against other bad assumptions, package up your change and pass it to a buddy for her to build. Most source control systems allow you to package up a change list for distribution. If you have a continuous integration system that builds changes before check-in, this is less necessary.
- TED (http://www.ted.com): A great web site of some of the world’s best speakers. There are talks on a multitude of topics. You are guaranteed to find something of interest.
- AppVerifier (http://msdn.microsoft.com/en-us/library/ms220948.aspx): A mandatory tool for C++ developers on the Microsoft Windows platform.
- Source Analysis (http://code.msdn.microsoft.com/sourceanalysis): A customizable tool that validates coding standards and is a good complement to FxCop.
Motley: There is no harm shortcutting a few steps of development to meet a deadline. Some up-front check-in steps just take too much time.
Maven: Slow down to go fast. Obey check-in checklists or you will pay for it later in the development cycle. Pre-check-in tasks such as code reviews, unit testing, static analysis, and buddy builds of various flavors must not be avoided.
[Context: The code check-in deadline is approaching. Motley has code that is in progress and he is hurrying to get his change in prior to the build snap]
Maven: Hey, Mot. How’s it-
Motley: Quiet! I have to get this code checked-in by the end of the day to meet the deadline. I don’t have time to chat and need to rush to get this code in.
Maven: Whoa – maybe we should chat. Please just give me a few minutes here – it won’t affect anything in the bigger scheme of things. What do you mean you have to “rush”?
Motley: The daily build starts at 6pm and it’s 5:35pm. I only have 25 minutes to get the code checked-in so I have to take a few shortcuts.
Maven: I cannot stand the word “shortcut”. Care to clarify?
Motley: It doesn’t really matter what you can and cannot stand. But since you asked, I am going to skip writing unit tests for this last class, I am going to forgo code review because my code is always perfect anyway, and I am going to skip running FxCop just this once.
Maven: Have you ever heard the phrase, “Pay me now, or pay me more later”? What you are doing is akin to racking up credit card debt. On the outside it looks like a $5000 purchase, but in the long run it is going to cost you a lot more in interest if you do not pay off the debt as soon as the bill comes due.
Motley: Oh, no. More Maven analogies. I can do without those. And by the way, Boyz II Men called – they want their shirt back.
Maven: Always insulting my clothes! This blue and pink sweater combo isn’t that bad, is it? Anyway, taking shortcuts now when the code is cheap to fix pays off in the long run and leads to quicker ship cycles and shorter bug-fix cycles later in the development cycle.
Motley: How so? Bugs are bugs, I need to fix them anyway.
Maven: It’s much cheaper to fix bugs now than later. Fixing them later usually involves someone else creating and maintaining bugs in a bug database, fixing them when the code is no longer in context and fresh in your memory, and the tendency to bolt-on fixes and shortcut fixes. Additionally, if the test team does not find an issue, the cost of fixing it post-release is large due to the overhead in working with the customer, the bug getting back to the product group, deploying the fix, the negative press, and other factors. If you find an issue that modifies the design later, you may have to change a lot of code possibly destabilizing the product.
Motley: Yeah, yeah – the cost of change curve. I know it.
Maven: Then you know that you shouldn’t rush your development. Would you rather fix bugs later when they are more expensive or now when they are cheap without the overhead of a post-check-in bug? Leaving bugs until later forces long “stabilization” periods on the team, which, in my opinion, are evil and can be avoided by a diligent team.
It is well worth pushing back on the deadline to ensure that you can adhere to the following code check-in checklist:
Motley: You cannot possibly be serious! If I do all that now there is no way I would ever make the deadline, let alone make a check-in.
Maven: All of the above is necessary, particularly in a large, quality-focused organization. We want to exterminate bugs before they are even checked-in. The checklist above helps with that.
Motley: Can you say “Sloooooooooooooowwwwww”? I knew you could.
Maven: Get in touch with your inner tortoise. I heard that in a TED talk on Slowing Down in a World Built for Speed by Carl Honore. Going a little slower up front will save time in the overall development cycle, and I would claim that you ship sooner as a result. Of course, that was not the specific theme of the talk, but the lesson can be applied here. Also, be pragmatic. Perhaps you avoid a step or two for small bug fixes.
Motley: One thing I do not quite buy is the lack of long stabilization periods at the end of the development cycle. We still need to put all the pieces together!
Maven: Ultimately we want to avoid long stabilization cycles and instead be stable when you check-in. A tail of the development cycle needs to be dedicated to integration and overall acceptance test validation, but is not the same thing as “stabilization”. We talked about this before in our discussion on developer testing.
Motley: So you are telling me that I need to pay my boss a visit as soon as possible to tell him that I am going to miss the deadline? He is going to kill me.
Maven: Well, seeing as you only have 5 minutes to the build snap-
Motley: <POW>. You deserved that shot in the nose for wasting all my time and missing the build snap!
Maven: Ouch! Okay, I deserved that. But seriously, your boss is a reasonable person and will understand. Firstly, you are communicating bad news as soon as possible (well, not really, but let’s pretend), which all managers appreciate. Secondly, present it such that you are focused on quality and want to avoid taking shortcuts, and that you feel this course of action is best for the product. I will bet you lunch that he understands.
Motley: There are some days where I wish I stayed in bed…
James’ Pointer: The checklist items I mention above are from a real team in at Microsoft in Windows Mobile. Quality is our #1 value as a team. We want solid check-ins that adhere to the checklist to ensure we don’t have a long tail of bugs later. Yes, it is a lot of work but ultimately saves time in the long run. Slow down. Don’t rush. Nail the checklist on every check-in. You will thank yourself later. Okay, maybe not, but you should.
James’ Double Pointer Indirection: A good developer knows how much work goes into each and every check-in and factors these things into their estimates. These things take time.
James’ Triple Pointer Indirection: “Slow down to go fast” is real advice in car racing circles. Intuitively it does not make a lot of sense. However, if I go into the corner too fast I have to brake to avoid the wall and then take a lot of time and gas to come back up to speed exiting the corner. If I slow down on entry, roll through the corner and smoothly apply the accelerator on exit, I actually go faster even though I slowed down entering the corner.