A simple project management tool, such as computing an average, can transform a project from chaotic and out of control to suddenly predictable and manageable. Here are some simple averages that made a huge difference in our management of large and complex projects.
What is an average?
Yes, we know to take a list of numbers, add them up, and then divide by the total count of the numbers in the list.
But what does it mean to our software project (or any project for that matter), especially when our project management tools do it for us?
Let’s say that it takes us ten days on average to fix a software defect (or any defect).
Today, software testing (or quality control, see the pattern?) reported twenty-five defects.
If “on average” it takes ten days to fix a defect, how many defects will probably be fixed ten days from now?
A. All of them because 10 days have passed and the typical time to fix a defect is 10 days
B. Half of them because 10 days have passed and an average is generally just a mid point
C. We can’t know because every defect is different as some are hard and some are easy
D. You are wasting my time with guessing games when I have 25 defects to fix
E. It doesn’t matter because they all need to be fixed by Friday since that is what we told the customer
F. Impossible to know as you didn’t tell me the standard deviation nor tested that the fix rate distribution is normal
I’ve heard all these answers during real projects, so let’s look at each of them.
A. All of them because 10 days have passed and the typical time to fix a defect is 10 days. I’m not kidding, an awful lot of people will say this. It is usually in the form of saying something like “it takes us 3 days to fix a defect, so we should have them all completed by Friday.” If we then ask the person what an average means, we see the cogs spin in their head and then they say “well, ok, it is an average, but we still think everything will be done by then ….”
B. Half of them because 10 days have passed and an average is generally just a mid point. This is a pretty good answer. The key is to think “half” when talking about an average. However, what we are really asking is “when will ALL these defects be fixed?” This is the question we and more senior management will ask all the time. This is the question we will work hard trying to answer. This is why we will collect data on how long things take to fix. I’ll address how to answer the “all fixed” question below.
C. We can’t know because every defect is different as some are hard and some are easy. This is a good answer when someone asks us how long it will take to fix a particular defect which was just reported. However, if we have a dozen or more defects and someone is asking us how long for all of them to be fixed then we can indeed have a good answer. More on this below.
D. You are wasting my time with guessing games when I have 25 defects to fix! Too many managers resort to this reply for what is really an important question – one that we should be able to answer. Yes, even when we don’t know anything yet about those 25 new defects, we should be able to give a rock solid answer to this question – an answer that will be accurate the vast majority of the time. No, it does not mean making a very large guesstimate. We want to provide an estimate that will most probably be the case. Yes, we can provide that estimate. We’ll talk about that below.
E. It doesn’t matter because they all need to be fixed by Friday since that is what we told the customer! Here the problem is we as project managers have not educated our account managers about how long things take and what they should be telling the customer. This is also one of the biggest challenges, to educate our customer facing team. The chances are that they’ve been telling their customer roughly the same thing for a long time (often years) on how fast we can respond. Inevitably, when we educate the account team on how long things take, it very well might be twice what they have been normally telling the customer. So instead of “by the end of the week” it becomes standard to say “by the end of next week.” Once we start to deliver on those promises, our account team will come to trust us. Hopefully, the customer will come to trust the account team and may – just may – no longer demand everything be fixed by tomorrow (see more on being brutally honest as a project management tool).
F. Impossible to know as you didn’t tell me the standard deviation nor tested that the fix rate distribution is normal. Too many Six Sigma Black Belts wannabes reply with this kind of answer. The key for us as project managers is to track and understand how well our teams perform critical tasks. By “living” with the numbers every day, we will understand them better than anyone trained in any level of statistics who does not have this experience. This gut level feel for the numbers is what we are ultimately trying to achieve by knowing our performance. If we are not collecting and tracking this kind of performance data, we will probably not know these numbers by virtue of just working with the process everyday. In one company, our best development managers would routinely say something like “it takes us about three days to resolve a defect,” but when the data was compiled the average was closer to ten days. Many defects were indeed resolved in just three days (the defect time-to-repair curve was a skew shape), but when looking for a median (the middle point) or the average, the results were two to three times longer. To really know, we have to routinely measure and track over the life of our projects.
If we have a critical process that happens regularly, we want to measure that process. I recommend you do it yourself to start, so you understand the numbers (we can delegate or automate the task later). Some examples:
1. If we are regularly fixing defects, keep track of how long it takes to fix them. Track them from when they get reported to when they get fixed in a product build (i.e., when they are truly done in your environment). As the project manager, we want to know the end-to-end time it is taking. The development team might only want to hear how long it is taking them to fix the defect once they finally receive it or once they finally understand the problem. As the project manager, we care about the full time.
2. If our test team is running test cases, keep track of how long it takes to run the entire test suite. Once again, the test team might want to exclude test cases that didn’t work right or times when the testing environment had problems. We, as the project managers, want to keep track of how long it is currently taking, including all the typical things that go wrong. We always want to be able to answer the question: how long will this process take on average?
Anything that is repeated over and over is a candidate for this kind of tracking and measuring. Focus on the key critical processes that happen regularly. Documentation updates may not be on the critical path for example. Be able to answer the question “how long is this going to take?” for critical processes.
Humorously, I’ve come to realize that I know we now have good estimates when half the folks ask “why is it now taking so long?” and the other half say “it is about time we started using realistic estimates!”
OK, but often this question “how long will it take” is more often phrased as “how long will it take to fix all the problems?”
Let’s take a real world example.
At one company, the average time to resolve a defect was about seven days.
What we want to know, as I mentioned above, is generally how long is it going to take to fix the current bumper crop of defects that just got reported.
It turns out that the standard deviation for these defects was also seven days. So the average of 7 plus 2 times the standard deviation of 7 gives 7+2*7= 21 days. (For the statistically minded, yes there are some problems here, but the results are close enough, much closer than what had been used, and this is quick and easy to calculate in Excel, Google Docs, OpenOffice Calc, etc.).
So when the question comes up of “when will all these defects get fixed?” — a good answer is 21 days. Using our 25 defects that got reported, this works out to a statement that we expect all but one or two of the hardest defects to be fixed by the end of three weeks. Yes, one or two will probably be left over. This is because adding in two standard deviation to the average (the mean) tells us when 95% of the defects will get resolved (.95 * 25 = 23.75 defects resolved). (For applying this tool to schedules, see getting the project management tool schedule right. For more on managing defects with this tool, see project management tool defect reports are your best friend.)
Once we know the average we have a project management tool that puts us in control and allows us to manage in an objective manner. The average allows us to distinguish between what is truly chaos and what is really just normal. We will know how our project is doing overall and how things will turn out in the near term. We will be much more accurate than those folks who are still estimating without the benefit of knowing their average.