Getting the right project management answer, a useful answer, means asking the right questions of our people, metrics, analytics, and data mining efforts. Too many questions have no useful meaning and too many answers have no corresponding question that needed to be answered. Here is a real world example and a few key principles that got us the answers we needed and vaulted our projects on to success.

First, Ask A Question That Needs Answering

I did something that everyone said was useless and made no sense. I answered the question of how long was it taking to fix defects in our product. I just added up how long it took from when the defects were reported to when the defects were put in a product build and then divided by the total number of defects reported. I got an average. One well respected engineering manager retorted “That makes no sense! You are mixing large and small defects and defects from different parts of the product. You are averaging out apples and oranges and getting useless fruit cocktail!”

There was only one hitch with this manager’s theory (see theorizing vs managing in projects) — this average was very good at predicting when our defects got fixed. It was much better than the estimates, plans and projections found in all the PowerPoint slides from equally well respected engineers and managers. The numbers further showed that we would not be done for months (see defect reports can be our best friend), when all the “happy” slides said we were sure to be done by next week or maybe in two weeks. “Oh, no!” they said. “It won’t take months. We are the experts and have been managing these efforts longer than you have.”

What happened?

We finished up about four months later, when we finally got all the defects identified and fixed.

Make Sure Our Answer Is Really For The Question Asked

As it happened, the Quality department decided that they liked the idea of calculating how long it was taking to fix defects. So they computed their own number. It was huge. Much larger than what I had computed. Suddenly, the development teams liked my numbers! I liked what Quality was trying to do, so I had a few meetings with them to understand what we were doing differently.

Simply put, Quality was asking and answering the wrong question. For the next ten years — yes, ten years — Quality would continually get it wrong and the numbers they produced would never get into any real use nor help projects deliver on time. Instead they were relegated to being part of the “standard charts” produced on a Quality dashboard. These charts were never predictive of when defects would be fixed nor when the the project was to be completed.

What happened?

They never understood the concept of asking the right question.

What was the right question in this case?

It was very simple.

It was a question we asked every day. It was asked constantly. It was the source of great and often heated debates.

The question was “How long will it take to fix those defects that were just reported today?”

Simple yes? No, not really.

Quality would never answer this question. Even after discussing it with them, for years, they insisted that the question they asked and answered was the right one.

What did they answer? “How long has it taken to fix those defects we just fixed today?”

That is it.

Isn’t it the same question?

No. Not even close for our purposes.

Just Because We Can Compute A Number Doesn’t Mean It Is Useful

The first reality check is that no one, no customer, no senior manager, no product manager, and no project manager ever asked that question Quality was answering.

When these same folks were asked, as an experiment, what question they wanted me to answer (“how long will it take to fix today’s reported defects vs how long did it take to fix the now fixed defects in the product”) — I often got a “are you nuts” look and a repeat of wanting to know when the current defects we just identified today would get fixed.

The differences in calculations were simple. In my case, I’d take the defects that were reported found in any one day, and compute the average on how long it took to fix them.

Quality did it a bit differently. They took the defects that were reported fixed on any given day, and computed the average of how long it had been since they were reported.

There primary argument for taking this approach was that on any given day that we fixed defects (installed them in the product), we knew all the information we needed and could compute a number.

The problem is, they said, with my approach was that to determine how long it took to fix the defects reported today, we had to wait until they were all fixed to know a number. I agreed. (OK, so how would you do it? It is a bit tricky.  Share your ideas in the comments.)

Quality was computing something they found easy to compute, rather than answering the question being asked.

Truly Answering A Key Question Can Have A Huge Impact

Asking the right question is not as easy as it sounds. Just because a question can be asked or an answer can be computed, doesn’t mean that it is useful. Pay attention to the questions that are asked in our project, and then try to see what information we have that allows us to answer that question. When we simply did that, we suddenly knew how long things were really taking, and we started meeting our promises, building customer confidence, and hitting our deadlines.

Are you producing any metrics or numbers that don’t seem to answer a question people ask everyday?