My job was to evaluate how well our government projects were doing.  The projects were expected to be using the prescribed latests and greatest project management tools and development technologies to help ensure they were successful.   I was generally only brought in on projects that were not going well.  I was to help senior management assess how things were really going and to recommend ways forward. Here is how we finally achieved an “A” in our project management.

The Approach

I would sit in on meetings and briefings, interview managers and practitioners, evaluate what was provided and supply detailed feedback on what I saw was going on. I used a fairly simple three step approach: Listen, Look, Report.

1. Listen For Understanding

I would listen to what the project team was saying.  Were they using the prescribed concepts and management techniques?  For example, when they talked about estimating and using Function points (used in software project management tools), did they understand the technique, or were they just going through the motions?  The same for CMMI, object oriented development, metrics, quality management, agile and other hot methodologies at the time.

I could quickly tell when a project was doing whatever they had been doing in the past but then just layering onto it terminology borrowed from the latest edict to try and say they were using the techniques.  Yes, we can probably all see some problems here.  I was evaluating efforts, but these efforts by definition where probably doomed because they were chasing silver bullets to help us suddenly get better at what we were doing.

Some projects were really good at talking the talk.  They would say the rights things, handle questions with reasoned nuances, and so sounded pretty good.  The problems always were that the project was behind schedule, over cost and the quality of what they had could not be demonstrated (e.g., lots of PowerPoint slides, nothing to show for it).  So once they were saying the right words, I then had to find evidence that they were actually following those words.

2. Look For Evidence Of Actual Use

I would sit down with the individual practitioners and see how they were going through their tasks.  This is often where we would see a huge disconnect.  This disconnect was frequently that the foot soldiers were just working away, as they had always done, and few if any real new techniques were being employed.  Often there was an additional cadre of folks who would take their work and try and retrofit it or spruce it up so that it looked like technique X was being used.  This not only resulted in not a very good implementation of the techniques, but also increased cost, which we bore as the government, due to the extra folks brought in to “implement” the new government initiatives.

I would write up a detailed description of what I had found, the concepts being used, and the implementation being accomplished.  Often, as the projects were not doing well, we would find they didn’t understand the concepts or having brought in an “expert” could talk up a smoke screen, but were not implementing the concepts in any effective manner.  Again, I point out the the environment here was not real good anyway, as we were using silver bullet after silver bullet as our primary method of improving project performance.

3. Report What Is Found As Factually As Possible

I briefed and passed these dense reports onto the project management team, the contractors and more senior management.  They read them, or claimed to read them.  They were too overly technical, dealing with nuances of what was being done or not done and what should be done differently — in page after page after page.  Most of them read like primers on the management or technical method that were being used.  Needless to say, these detailed findings had no impact whatsoever.  I was just another source of “noise” in the process.  The projects continued on their inevitable path to a failed or at best, late and buggy completion.  However, everyone was seen as doing their jobs.  Software intensive projects were difficult to do — we all knew that — and we were satisfied that we had done our best.

Well, I wasn’t.  Something just was not right.  These projects were failing — consistently. We were providing what we felt was good guidance on how to get the projects back on track.  Many of the practitioners, including many managers, agreed with the actions and insights we published, in large part because these folks were the source of the recommendations. Many of them were in fact excited about using a new technique or methodology, but admitted the lack of effective training (e.g., a mass two week classes in technique “X”) insufficient schedule allocated (always overly aggressive) and random direction given (daily crisis management) didn’t allow them to really make effective use of the new methodologies.  How could we be doing all this and nothing really change? What could I do to make a difference?

Assign A Letter Grade?

My boss’s boss, an individual who had been a senior VP at a fortune 50 company, was talking about some things he had done at that company before he came here.  In passing he had mentioned how a team of his had once used school grades (e.g., A, B, C, D, F) to report on how something was going.  He indicated that it had been effective in getting attention. I was intrigued.  He had not told me to use it.  It was just a passing comment along with a lot of other comments.  It just struck me as something that might be useful.

I thought about it for awhile and, looking over many of my reports, I assigned a school grade to each effort reviewed in the report. I finally came down to something like the following scale:

Grade	Effectiveness of management/development tool or technique implementation
A	One of the best implementations and approaches we’ve seen.  It is effective and shows deep insight into how to do this kind of work.
B	Good implementation of the technique.  Knows what they are doing and are effective in what they do.
C	OK implementation of the technique.  There are some issues and we are struggling in places with the ideas, but can in most cases get the job done using the technique, but we won’t yet see the full benefits.
D	Not a good implementation.  We have real problems and we don’t understand the technique well enough to get much benefit out of it.
F	Fatally flawed. It is simply not being done or there are significant conceptual disconnects that show we just have no idea what the technique is intended to accomplish.

In my next project evaluation I did my analysis as I had done in the past.  The difference now was I put grades on each area evaluated as well as an overall grade on the summary page along with the grading scale.  Needless to say, since this was a struggling project, most of the grades were of the C, D, and F variety. There was a sprinkling of As and Bs, where someone or a small team was doing great things, usually with enthusiasm.

Survive The Firestorm of Protest!

The impact on publishing my report was electric.  The whole organization was in an uproar.  I had graded a major government project an “F”.  Yes, we knew it was behind schedule, over cost and had yet to generate anything that worked.  The contractor and government project management team had, as usual, already implemented a series of actions that would cause the effort, miraculously it always seemed, to be back on track and on schedule.  It wasn’t working, might have been the 3rd or 4th “redo” of the plan, and they had brought me in to provide recommendations.  Yet, the notion that I would put in writing that the current effort was an “F”, fatally flawed, was just not something that was done!  It was OK to list all the problems and recommendations, which few senior managers could understand, in a dense report that was hard to read.  But clearly stating something was an “F” now that just could not be right!

Suddenly, everyone had a clear statement — at least based upon my report — of where the project stood.  Most people, even those who requested I come evaluate the effort, were uncomfortable with this clarity.  It was too clear.  Too simple.  Too “unprofessional” I was told by one manager, to give an official project by the government and highly paid contractors a letter grade!  This was not how it was done!

No one was disagreeing with any of the content, only with the grade.  This grade was something everyone could finally understand, as opposed to arcane project management and software development techniques.  They could argue with this and “negotiate” what the letter grade should really be.  Unfortunately, being young at that time and politically un-savvy and overly idealistic, I was not interested in negotiating new grades for each effort.  I was interested in helping the project!

The highlight of the initial maelstrom was when I spotted a young manager for the contractor literally skipping down the hallway. She was singsonging “I got a ‘B’, I got a ‘B'”  and I knew I had hit a chord.  I just needed to survive the firestorm and make some progress with it.

Not surprisingly, the project eventually was closed down, like so many before it, after years of effort and millions of dollars.  This current collection of management and development tools just didn’t help the project management and software development to become effective within the given costs and schedule. The project was cancelled earlier than similar failed projects in part because of the “F” assigned to it.  I was told, privately, by a manager how great it was that someone had the courage to grade the effort as a fatally flawed, before it had completely failed.

Sometimes we just need to use simple techniques to help clarify and communicate our findings.  Using school letter grades can be a successful project management tool at communicating the essence of the findings when the impact of the technical details is hard to understand by sponsors and management.

What techniques have you seen or used that got the attention needed to help a project?