A busy test organization can mean your products are having problems.  In this case however, the fact they were very busy helped the test organization to stay objective about reporting dramatically improved test results without missing a beat.  Other test teams described in this series of articles reacted in inefficient ways when product quality improved dramatically.  This is the last in a three part series on test organizations and how they responded to dramatic improvements in product quality.

We had delivered the premier company product on time and with quality.  It was the first time in recent corporate history.  A few months leading up to this release, I had talked with the Director of Testing about how his department might need to change if software quality improved significantly.  It was a somewhat humorous discussion as he listened to me actually saying these kind of words in a company that was known for its late and buggy products (some of which sold very, very well).  However, after he heard some of my examples, I found him listening very closely.  I am not sure he really believed it could happen, but I am sure I got his attention.

While I had been happy that we delivered on time, I had not seen the improvement in quality that I expected.  In this case, I had eight months before predicted that the count of final defects left in the product would be significantly lower than previous products.  Unfortunately, this just didn’t seem the case.  We seemed to have a normal amount of defects.  I was a little disappointed.  In the past, when finally getting the schedule right and delivering on time, software quality had not only improved, it had improved dramatically.  While the customer acceptance reports suggested a higher quality,  it just did not seem convincing that the quality was that much better.

Our next product release in this line was to be what we called a refresh product.  It would lower some costs and add new capabilities.  In fact, as it often happens, it was to add a lot of new capabilities.  We had so many new capabilities to be added that we were running out of memory in the product.  Normally, we would have expected to go a year with the product, adding new capabilities, before running short of memory.  Instead, it was happening within six months.  So this refresh release was not typical.  It was a much larger release than in the past.

Development had challenges fitting everything into the product.  We were still doing well with the schedule, but I was concerned about quality.  The development organization had fought hard against fixing all the existing defects.  They felt that putting in so many would threaten the stability of the product as it had in the past with other products.  They wanted to treat the product, still very early in its lifecycle, as if it was going to ship next week, and only fix those things that we thought the customer would find unacceptable.  I pushed hard for them to fix as many defects as possible.  I directed that we would not start to restrict what defects would be fixed and put into the product until closer to when the product was to be released.

So we had a ton of new features and a ton of defect fixes going into this product.  It was not looking like a refresh release, it was looking like a new product launch in size.  This was going to be a challenge.  The test team was going to have a lot to do,  or so I thought.

Our test organization was spread thin over many products.  Often, getting them to initially test a product was a challenge as they inevitably had another dozen products to test.  Those other dozen products followed the pattern of being late and buggy.  Therefore, the test teams were always in a critical mode because the product needed to be finished by “next week.”  So when we were ready to test and we were still months from our release date, it was tough to get much attention.  However, we finally got the test team to start running their suite of product tests.

The first week of testing went by without seeing much in the way of test results.  In fact, they had entered only one defect report when we would normally have hundreds of defects at this point.  We had a meeting with the test manager and asked him how things were going.  We were expecting him to explain why they were not testing.  He reported that they had finished their first week of testing.  They only reported one defect.  Pressed hard, the test manager began to get agitated. Finally, his comment was “the product is fine, everything worked, what is your problem?”  He was busy and needed to get back to those other products to manage their testing.  Those other products had multiple new development releases with hundreds of defect being fixed and were in need of additional testing.

I reported only one defect in the first week of testing to my boss. His response was for me to go pressure the test manager to get started testing. I explained they had completed their first week of test.  He said “something is wrong then” and to go and find out why the test team wasn’t reporting defects.

It turned out that this was to be the pattern for the whole release.  Very few defects.  This allowed us to run on time.  Almost.  It turned out, as it happens, that we shipped a week late.  This was a reminder that while we had few errors, it only takes one error to block a shipment.  We had that one error and missed our deadline.  From the customers perspective we were fine.  We still had a reputation of shipping months late, so a miss of a week was on time for our customers who were prepared to not get a working product for months.

This test department wasn’t threatened by good quality. They were too busy.  They did the tests, noted that there were not many errors, and were happy to report that and move on.

We know we are improving our quality when our test or quality organization starts to get stressed when finding fewer issues.   If we’ve implemented the latest and greatest new methodology (e.g., Agile, Scrum, Lean, Six Sigma, TPS, etc.) but our test or quality organization is happily doing what they’ve always done in the past, we may want to consider if our new methodology has truly provided the benefits we expected.  For senior managers, keep in mind when assessing results that often the test department has a vested interest in having quality problems.

Previously, an Almost A Great Test Organization, and the Best and Worst Test Organizations.