“Systems development is primarily a knowledge discovery activity. Even in coding, the thing that takes the most time is figuring out what we don’t know. … But sometimes there are things about the system we don’t know we don’t know, which means we have Second Order Ignorance (2OI) … So asking the question “… how complete are you?” or its corollary “… how much more work is there left to do?” is the equivalent of asking “… how much do you not yet know?” There is no easy way for someone to answer that question. Most people extrapolate from their past experience and what they have learned on the task so far to figure how much they might have left to learn. But they don’t actually know.” How We Build Things … and why things are 90% complete.” Phillip G. Armour, Communications of the ACM, January 2013.

“But they don’t actually know.”

Yes, actually we probably do know. Let me explain.

Phillip gets close to it when he says “Most people extrapolate from their past experience ….”

Let me add that, in my experience, most people work in an environment where we do similar things over and over again. Typically in the software world, we regularly put out new releases of existing software or develop all new software in major projects.

Yeah, I know, each of those projects are different. They have different requirements, they are building on different versions of the existing system or they are an all-new system.

However, if we look at our past history at our organization we will often find consistencies in project after project after project. It may be hard to get the data on those projects, yet most organizations keep project archives, lessons learned, meeting minutes, project plans (Microsoft Project, etc.), project status slides, etc. For most organizations I’ve worked with over my career this data pretty much tells us what we need to know. In fact, I can say it always told us what we needed to know with the only exceptions when we really did something we never had done before … and that was in reality very rare (and still solvable).

At one Fortune 50 company I worked at, I would regularly look at the project plans and archives of every project I could get my hands on  I wanted to know how long it took them, what they produced, how “big” were their requirements, how many defects did they find, how long did it take them to develop the original requirements and how long did it take them to test and achieve shippable quality. How many other people did I encounter in my career who did the same thing? None.

For more, see how to get the data we need

Sometimes, I’d find a special effort or team that dug into corporate past performance data and produced some interesting findings. In general, however, this digging into past performance was an exception, something to be done when pressed, but not a normal everyday activity. Certainly, the typical project manager didn’t have time to dig through old data on past projects. At best they would call meetings and ask experienced people for their inputs on what they needed to know to plan and manage and track a project.

So, how has that worked for us lately? Usually, not well. Especially in the middle of the project when we are thinking we should be towards the end:

“Then a certain amount of optimism, a modicum of professional pride, and perhaps a dash of hubris and the calculation yields the ubiquitous “90%.”

I had a manager tell me his part of the project would take six weeks. I told him he needed more time than that. He was floored. Usually, someone in my position was telling him to deliver things faster. I was telling him that it was not enough time. Humorously, he asked me how much time I would give him. I told him to go back, look through his team’s past performance over the last few projects, and come back and tell me how much time it has been taking his team. He was frustrated because he knew I had a number but I would not share it with him. I wanted him to be telling me his team’s performance and not me knowing more than he did about his team.

It turned out that it was more typically 13 weeks and they had never delivered in six weeks but had always promised six weeks because that is what they had been badgered into saying for many years. He also had a catalog of excuses why it was not their fault for not making it in six weeks but the humorous part was he was right, it was not their fault, but none of their excuses ever included that six weeks was way too short.

Where did I get this information to know more than the manager? I simply looked at the projects we had completed over the last few years. The pattern was clear. Note, that I didn’t look at the requirements or the details of the projects. I simply looked at our past performance for this major task and noted that we consistently said six weeks, never delivered in six weeks — generally spent another six weeks at “90%” complete, and we consistently delivered in an average of 13 weeks. So the odds were in my favor that telling the manager he needed more time was probably more accurate than what he had given me. Since I wanted to deliver the whole project on time and needed accurate estimates this use of past performance actually reduced the risk of missing a major milestone. Also note that in this organization our projects were always late, with another catalog of well-used excuses, so I also knew that in general we were underestimating our tasks and again this was because I reviewed (dug deep) into our past projects.

For more on why this is, see In Project Management 3 + 9 Is Not Equal to 12 and Software Development Is A Repeatable Process

I love the notion of Second Order Ignorance. I also love the insight that we have, in many — if not most — practical situations, a solution — a way to manage the risk — for Second Order Ignorance. The notion that we can be successful at something without first knowing everything we need to know in advance still just strikes many people as just not right. However, knowing and using our past performance is a great project management tool that works for mastering the problems of Second Order Ignorance.

How do you handle the risk of Second Order Ignorance on your project?