Knowing Your Project Management Average Is Powerful!
A simple project management tool, such as computing an average, can transform a project from chaotic and out of control to suddenly predictable and manageable. Here are some simple averages that made a huge difference in our management of large and complex projects.
What is an average?
Yes, we know to take a list of numbers, add them up, and then divide by the total count of the numbers in the list.
But what does it mean to our software project (or any project for that matter), especially when our project management tools do it for us?
Let’s say that it takes us ten days on average to fix a software defect (or any defect).
Today, software testing (or quality control, see the pattern?) reported twenty-five defects.
If “on average” it takes ten days to fix a defect, how many defects will probably be fixed ten days from now?
A. All of them because 10 days have passed and the typical time to fix a defect is 10 days
B. Half of them because 10 days have passed and an average is generally just a mid point
C. We can’t know because every defect is different as some are hard and some are easy
D. You are wasting my time with guessing games when I have 25 defects to fix
E. It doesn’t matter because they all need to be fixed by Friday since that is what we told the customer
F. Impossible to know as you didn’t tell me the standard deviation nor tested that the fix rate distribution is normal
I’ve heard all these answers during real projects, so let’s look at each of them.
A. All of them because 10 days have passed and the typical time to fix a defect is 10 days. I’m not kidding, an awful lot of people will say this. It is usually in the form of saying something like “it takes us 3 days to fix a defect, so we should have them all completed by Friday.” If we then ask the person what an average means, we see the cogs spin in their head and then they say “well, ok, it is an average, but we still think everything will be done by then ….”
B. Half of them because 10 days have passed and an average is generally just a mid point. This is a pretty good answer. The key is to think “half” when talking about an average. However, what we are really asking is “when will ALL these defects be fixed?” This is the question we and more senior management will ask all the time. This is the question we will work hard trying to answer. This is why we will collect data on how long things take to fix. I’ll address how to answer the “all fixed” question below.
C. We can’t know because every defect is different as some are hard and some are easy. This is a good answer when someone asks us how long it will take to fix a particular defect which was just reported. However, if we have a dozen or more defects and someone is asking us how long for all of them to be fixed then we can indeed have a good answer. More on this below.
D. You are wasting my time with guessing games when I have 25 defects to fix! Too many managers resort to this reply for what is really an important question – one that we should be able to answer. Yes, even when we don’t know anything yet about those 25 new defects, we should be able to give a rock solid answer to this question – an answer that will be accurate the vast majority of the time. No, it does not mean making a very large guesstimate. We want to provide an estimate that will most probably be the case. Yes, we can provide that estimate. We’ll talk about that below.
E. It doesn’t matter because they all need to be fixed by Friday since that is what we told the customer! Here the problem is we as project managers have not educated our account managers about how long things take and what they should be telling the customer. This is also one of the biggest challenges, to educate our customer facing team. The chances are that they’ve been telling their customer roughly the same thing for a long time (often years) on how fast we can respond. Inevitably, when we educate the account team on how long things take, it very well might be twice what they have been normally telling the customer. So instead of “by the end of the week” it becomes standard to say “by the end of next week.” Once we start to deliver on those promises, our account team will come to trust us. Hopefully, the customer will come to trust the account team and may – just may – no longer demand everything be fixed by tomorrow (see more on being brutally honest as a project management tool).
F. Impossible to know as you didn’t tell me the standard deviation nor tested that the fix rate distribution is normal. Too many Six Sigma Black Belts wannabes reply with this kind of answer. The key for us as project managers is to track and understand how well our teams perform critical tasks. By “living” with the numbers every day, we will understand them better than anyone trained in any level of statistics who does not have this experience. This gut level feel for the numbers is what we are ultimately trying to achieve by knowing our performance. If we are not collecting and tracking this kind of performance data, we will probably not know these numbers by virtue of just working with the process everyday. In one company, our best development managers would routinely say something like “it takes us about three days to resolve a defect,” but when the data was compiled the average was closer to ten days. Many defects were indeed resolved in just three days (the defect time-to-repair curve was a skew shape), but when looking for a median (the middle point) or the average, the results were two to three times longer. To really know, we have to routinely measure and track over the life of our projects.
So, what do we do?
If we have a critical process that happens regularly, we want to measure that process. I recommend you do it yourself to start, so you understand the numbers (we can delegate or automate the task later). Some examples:
1. If we are regularly fixing defects, keep track of how long it takes to fix them. Track them from when they get reported to when they get fixed in a product build (i.e., when they are truly done in your environment). As the project manager, we want to know the end-to-end time it is taking. The development team might only want to hear how long it is taking them to fix the defect once they finally receive it or once they finally understand the problem. As the project manager, we care about the full time.
2. If our test team is running test cases, keep track of how long it takes to run the entire test suite. Once again, the test team might want to exclude test cases that didn’t work right or times when the testing environment had problems. We, as the project managers, want to keep track of how long it is currently taking, including all the typical things that go wrong. We always want to be able to answer the question: how long will this process take on average?
Anything that is repeated over and over is a candidate for this kind of tracking and measuring. Focus on the key critical processes that happen regularly. Documentation updates may not be on the critical path for example. Be able to answer the question “how long is this going to take?” for critical processes.
Humorously, I’ve come to realize that I know we now have good estimates when half the folks ask “why is it now taking so long?” and the other half say “it is about time we started using realistic estimates!”
OK, but often this question “how long will it take” is more often phrased as “how long will it take to fix all the problems?”
Let’s take a real world example.
At one company, the average time to resolve a defect was about seven days.
What we want to know, as I mentioned above, is generally how long is it going to take to fix the current bumper crop of defects that just got reported.
It turns out that the standard deviation for these defects was also seven days. So the average of 7 plus 2 times the standard deviation of 7 gives 7+2*7= 21 days. (For the statistically minded, yes there are some problems here, but the results are close enough, much closer than what had been used, and this is quick and easy to calculate in Excel, Google Docs, OpenOffice Calc, etc.).
So when the question comes up of “when will all these defects get fixed?” — a good answer is 21 days. Using our 25 defects that got reported, this works out to a statement that we expect all but one or two of the hardest defects to be fixed by the end of three weeks. Yes, one or two will probably be left over. This is because adding in two standard deviation to the average (the mean) tells us when 95% of the defects will get resolved (.95 * 25 = 23.75 defects resolved). (For applying this tool to schedules, see getting the project management tool schedule right. For more on managing defects with this tool, see project management tool defect reports are your best friend.)
Once we know the average we have a project management tool that puts us in control and allows us to manage in an objective manner. The average allows us to distinguish between what is truly chaos and what is really just normal. We will know how our project is doing overall and how things will turn out in the near term. We will be much more accurate than those folks who are still estimating without the benefit of knowing their average.
24 thoughts on “Knowing Your Project Management Average Is Powerful!”
Comments are closed.
What I find interesting here (by the way this is a very interesting discussion) is that I haven’t seen one person weighting some of these errors into PRIORITY categories. When faced with these kinds of problems in real world situations you are working through these in a priority order because of resource constraint. After all, if all of the errors were in the area of “look and feel” and not in wrong calculations or “death screens” appearing at a point in time, the pressure and use of resources would change would it not? And, would that not adversely effect the time from discovery to fix of ‘simple’ problems? And, it would follow that the numbers for each type would skew?
Jim,
Yes, a priority ordering is normal. The priority is often set by how “bad” the problem is, such as “death screens” being first, and “cosmetic” issues in the GUI being the lowest.
While one would think that this would change how fast a defect is fixed, it is not quiet as obvious as it would seem. What we’ve found in large projects is:
1. If it is readily reproducible and it is a “death” defect, it gets fixed real fast. This one is pretty obvious.
2. If is is readily reproducible and it is a failure of major functionality, again it will be fixed very fast.
3. If it is hard to reproduce (such as an intermittent issue) and is a “death” defect, it will probably not get fixed as quickly as #1 or even #2.
4. If it is hard to reproduce (such as an intermittent issue) and is a failure of major functionality, it will probably not get fixed as quickly as #1/#2
5. If it is readily reproducible and is any other kind of failure, it will often be fixed faster than #3/#4 above ….
You can see the pattern. It becomes more the ease of reproducibility that drives the overall speed and not the severity/badness of the issue. This again is pretty obvious – but to many people this is a source of frustration. What we found is programmers will generally fix things – lower priority issues – that can readily be fixed, at a faster rate than the harder issues that are difficult to isolate or reproduce. The response to this is usually to increase the priority or visibility of the hard issues (reviews, priority lists, meetings, etc.) to make sure everyone is working on the highest priority issues first, not those that are easiest to fix.
This seems logical, but often does not work as well as expected. I’ve seen a lot of “forcing” techniques used, but in general the “hard” issues got fixed at their *own* average/normal rate regardless of how hard we pushed or how high we prioritized them (or how many people we assigned to them). Telling programmers that they could not fix anything else until they fixed the “hard” issues, for example, did not result in the hard issues getting fixed any faster and only resulted in significantly fewer easier issues getting fixed in the same period (this also suggests that Kanban techniques may not be as efficient as expected). This drives many folks absolutely crazy.
To many it just does not seem logical or reasonable and there must be something wrong they think, when lower priority issues get fixed faster than higher priority issues. I’ve watched teams lose significant productivity (and morale) because someone was trying to “fix” this “problem.”
Thanks for the comment!
Bruce
Another approach to estimating:
http://www.galorath.com/wp/the-estimate-maturity-model-can-improve-project-success.php
I like the notion of a maturity model for estimating. I’m not quite convinced this is it, but it is a good reminder that we just don’t get good at something, we incrementally improve and get better with time.
Okay so this is the source of misunderstanding.
You are talking about many people working in parallel, and I’m talking about one team where defect fixing is one-at-a-time.
If we have a set of defects that take 30 days to finish, the question of how many are fixed after 10 days or 25 is a function of how we order them. If we do the hard defects first, everything else moves back, and your measure would suggest that the mean time to completion is 20 days. But if we do the easy ones first (and there are more of those), mean time to completion is 10 days.
If one defect came today, it would take 1 day to fix (on average). If 2 came today, one would be fixed today, and one tomorrow. If 10 came today, half would be fixed at the end of the week. But if there are more difficult problems, one tough bug might push back the entire schedule.
So it seems to me your projections simply don’t apply to my context.
Alan,
Yes, this technique especially helps to manage efforts where many engineers are repairing defects in parallel.
However, I’ve used it successfully in a small team of less than 10 engineers. We were still repairing defects in parallel and then integrating them together. Typically a release would have around 20 fixes in it.
I’ve never seen anyone not finally go “wow” after collecting and seeing how long it was taking to repair. This by folks who insisted they really did know how long things take or who said they couldn’t see how this could be useful information. The second biggest reaction is “no way, it can’t be taking that long!” This second reaction comes from teams that usually have a history of delivering software late.
Good luck and thanks.
Bruce
Alan,
“I have 1000 defects” is, I believe, the sticking point.
Did you get all of those today? Probably not. Are they all currently being worked on and are at various stages of being repaired? Hopefully!
At a Fortune 50 company where I used this technique, it was not unusual at peak testing to get 2000 defects reported over the span of a *month*. Two thirds of those would ultimately be known errors (duplicates) or configuration/test issues. That would leave about 660+ as real issues that needed code changes.
Time to resolve an issue (i.e., have a working solution) during these peak times would average 7 days. It would take 7 more days on average to get this fix integrated, tested and in an official product build. So this was 14 days on average to a complete repair (which is what I think you are referring to).
Ninety-five percent of the defects achieved completed repair within 28 days. This is taken from the actual repair history. This is *not* an estimate or probability. It is a statement of the characteristic of the total population of defects repaired for the period.
The fact we will use this number to anticipate the near term repair rate makes it an estimate but only for the defects that have not yet achieved repair completion. From my experience it can be a *very* good estimate.
This is 28 days from the *day* the defects *arrived*. If 60 came in today, then 20 would end up being things we needed to fix. Four weeks from today about 19 would have complete fixes and about one would still be outstanding.
So “1000 defects”, in an ongoing development environment, means 1000 defects reported over time and in various states of being repaired: from new today to completing being fixed today. We never got in 1000 defects in a single day or even a week.
What you want to do is chart the numbers for your environment and see what is your rate. What someone else can do or is doing is not too relevant except as an example (we had big products containing up to a million lines of code, and upwards of 450 software engineers working per month).
I’m thinking of doing an e-book/white paper on the technique with complete examples that folks could download from this site.
Bruce
I don’t have a spreadsheet of times, but your site is inspiring me to build one. If you want to illustrate, maybe you can give me your data as an example.
Meanwhile, I think your missing my point (or my misunderstanding). You seem to be saying that if I have 1000 defects, I can complete 950 in 21 days and only have 50 left to repair. This is absurd, so you must be saying something else.
I understand the math, but the conclusion makes no sense to me. If the average (mean) time to complete one task is 7 days, and the standard deviation says 95% of tasks can be completed in 21 days, then my conclusions differ from yours:
1) One task can be completed 95% probability in 21 days.
2) 25 tasks can be completed very high probability in 25*7=175 days.
3) If you have 5 teams working in parallel, 25 tasks can be completed on average in 5*7=35 days. The standard deviation suggests there’s a good possibility it might take 50 days or whatever, but I don’t know how to calculate this.
How do you figure 21?
Alan,
Possibly the best way to see this is to graph your defects into a how-many-days-to-repair vertical bar chart: (number of defects y-axis, days to fix x-axis):
|
| *
| ***
| *** **
| ******* *
|*********** * *
————————-
You will generally see a somewhat skewed to the left rise, peak, and fall of defect repair times. So the fastest defects will be fixed in a day, the majority of the defects will show up being repaired in 5 to 7 days (as I said, a bit skewed to the left in the data we experienced) not as many will be repaired up through 14 days, tailing off through 21 days.
What this says is actual history shows that 95% of all defects are being repaired in 21 days or under. There are few, about 5% that will take longer (see note below in the PS on computing these numbers).
So you want to use your real data. Chart it and see what percentage of defects is falling in what interval of days. Chart it and count it. Don’t worry about the math or the notion of probability. These often only serve to get in the way of seeing what is really happening (I talk about this in the paragraph on “diet dilemma” – http://pmtoolsthatwork.com/get-schedule-right/).
Send me a pointer to a spreadsheet of your defect repair times and I’ll show you what I mean. All I need is a pair of dates for each defect: date reported and date fixed. Dates are best because repair capacity and performance will often vary over time (which is also useful to see). Often lumping in all defects found and fixed over, for example, six months will not be as useful as showing the defect repair rates for each month over the last six months.
Bruce
P.S. I use mean + 2 X standard deviation to find the point where about 95% of all defects are finally fixed. My Six Sigma Black Belt buddies cringe at this and say “but defect repair times are skewed and not normally distributed!” True, and so I chart out the times and count the actual number of days that account for 95% of all the defects being repaired. Somewhat humorously and to the chagrin of the BBs it turns out in our case that the mean plus 2 sigma (std dev) falls nicely around 90-95%. This was true for our data for the almost 10 years we did this. So often instead of saying 95% I’ll say something like mean + 2 X std dev tells us when “just about all” or the “vast majority” of defects will be repaired.