The key step? We have to change something that will actually help us. Here, “test every line of code” made a huge difference in this software organization. This is the first of several articles on basic changes that made a huge difference and why they worked.

Test Every Change You Make

It all started after I had spent an afternoon one day digging through the project archives, reading file after file on software defects repaired. While I was looking for project management information in general, I noticed a very consistent pattern. The pattern was the simple notion that just about every defect I reviewed would have been prevented if the software programmer had simply executed the line of code that caused the defect. That was it. Get the software to run its logic through that line of code in the program, and the defect would have exposed itself.

It helped that during my undergraduate computer science work at the University of Southern California, I had studied under Per Brinch Hansen (http://en.wikipedia.org/wiki/Per_Brinch_Hansen). In his classes he often mentioned how his teams, when building Concurrent Pascal compilers, would use a “test every line of code” approach and how it resulted in rock solid compilers. While I did not learn the specifics of his technique at that time, I recognized that I was now looking at a situation that could benefit from this type of approach.

I implemented the technique by requiring my software programmers to have a test plan that caused every line of code they added or changed to be executed by the system at least once. This annoyed most of my programmers. They felt their functional testing was more than adequate. Sure, the test team and the customers were finding errors in their code, but that was normal. Software changes always have some errors. It just worked that way was their thinking.

I had to send a few programmers back to redo their testing when, upon declaring their software changes complete, they admitted their test plans didn’t specifically test every line of code. I asked them to verify whether their plan tested every line, and if it did not, to add to their tests the additional steps that would result in all lines being tested. Being good troops (these were military software programmers from five nations) they gamely went back and reviewed their test plans.

They dutifully checked their test plans. I soon noticed they were also very quietly making changes to their code. I assumed testing every line of code had uncovered some problems. Success! It was making a difference. Well, almost.

Adding steps to test all lines of code did in fact find some issues. However, it was as often the case that just looking at the code, one more time, to figure out how to add such tests found the issue before the test was ever executed. The mental discipline needed to construct the test plan resulted in a code review that then caught many of the errors.

Changes Do Not Have To Have Errors

This one elementary improvement, to test every line of code the programmer changed or added, resulted in a huge reduction in defects being reported by our independent testing department (see the somewhat humorous story on the test teams reaction to this increase in quality). I do not claim that this improvement would always have the same effect on other software organizations. But for this development team, my afternoon deep dive into the records of past defects found that this approach was a good idea for handling the problems in quality we had observed. We made a change that directly addressed a problem that we could objectively measure.

I did make one big mistake. In my hubris of success, I joked to the test department manager that we had left a few defects in the code for them to try and find. I had a hard time convincing him that it was just a joke, and he would continually ask me to tell him what the defects were. His test team could find few significant defects in our software releases. He, like my programmers, assumed that software just had to have lots of defects. I’m sure we had more than the few we found. They were just no longer the kind our customers would readily find in their normal operational use.

We can use the latest and greatest techniques to improve our project management: Six Sigma, Lean, TPS, Agile, Scrum, CMMI, ISO9000, etc. However, they only work well when they address real opportunities to improve. Finding that opportunity makes the difference. What formal technique we use to find it is almost irrelevant.

What examples do you have of a single change that made a huge difference where other changes didn’t seem to accomplish much?

Next “It Should Work The First Time.”