Project Management Tools

« go back — your next project management tip »

Are We Measuring Project Quality Or Just Activity?

Things must have not been going too well on our project.  The VP of Quality and the VP of Test both showed up at one of our project meetings and wanted to know if all of our defects had been prioritized correctly.

I told them that the review process was ongoing, as usual.

They didn’t like our process nor my answer.

The problem they saw was I was taking too much time and not immediately reprioritizing the defects.

They were sure of this because they were not seeing the spike in defects that occurred as the quality review team reviewed and increased the defect priorities.   The periodic spikes were taken as indicators of Quality doing its job — identifying quality problems.  The Test VP also liked it because higher priority defects showed that his team was finding important issues that needed fixing.

They were mad.

The Quality Team Reviewed Our Defect Prioritization

They told me my job was to immediately implement the new priorities.

I responded that in the previous product (which shipped on time with good quality) we always first reviewed the recommended changes. This they clearly took as rank insubordination  (see initiative or insubordination for more on this notion).

Humorously, seeing that I was going to lose this one, during this conversation I messaged my team and told them to just apply all the new priorities without a review.  Simultaneously the VP of Quality was messaging my Product VP. At about the same time my staff confirmed all the priorities were updated, I got a message from my VP telling me to “just do it.”

I informed the VPs that all defects were now prioritized per their recommendations.

We had a real time defect tracking system, so they immediately pulled up the trends to see the impact.

There was no change in the priority defect trend.

The expected spike did not stand out in the normal ups and downs of the overall trend.

So what happened?

The Quality Reviews Had No Measurable Impact

What happened was the normal process disposed of the defects, at the original priority, faster than the reviewing of the priorities.  This highlighted a simple truth.  The  priority changes that were recommended made no difference as to how fast defects were disposed of nor of which ones got worked on or not.  The only thing it did was to introduce a periodic spike in the trend, based upon the batches of new priorities the review team would set.

One review team member admitted in confidence that the team had agreed to always increase some defect priorities at each daily review.  This was to show management that they were doing the job they were given.

I had at one time, again humorously, suggested that we should just write a script that randomly upgraded priorities and this would save hundreds of staff hours.  I don’t think anyone was amused.

The Quality Team Just Believed That Quality Was Not Good

Why didn’t the VPs of Quality and Test see the data as we did?  This is always hard to analyze, but in a nutshell those organizations had a belief that the development and product teams were “tweaking” the reported defects to claim quality was better than what quality and test found.  They saw the periodic defect spikes after a review as evidence of this.

The fact that the data never showed any increase in how fast or how many defects were fixed, even when their priorities were updated fast enough to cause a noticeable spike, was never addressed.  As long as the review caused periodic spikes in the trend, Quality felt this showed they were contributing.

Quality Had No Effective Measure Of Quality

By the way, this product shipped on time and even got recognition from our field test team (another test organization – see the best test team) for the best quality product the team had ever seen.

Quality is not always self evident, especially on large projects or products.  We need to have clear and objective methods of measuring quality.  Too often what passes for quality checks can be driven more by emotion and habit than by an objective measure.  Knowing our project or product quality requires hard data, not just beliefs.

Are the quality measures on your project objective and do they truly measure quality?

Filed Under Quality | 6 Comments

Tagged With Defects, Measurement, Metrics, Tracking

« go back — your next project management tip »

Solutions (note the scroll buttons < > )

• Subscribe To These Articles Using RSS

• Tools, Techniques, Hints & Tips

  • Can Your IT Project Manager Sell Better Than Sales?
  • Are We Measuring Project Quality Or Just Activity?
  • Why We Can Never Make Just One Change As A Project Manager
  • Sometimes When They Say They Want The Project Fast It Is Just Talk
  • Why Project Managers Should Not Set Up Metrics To Measure Improvement Efforts
  • How To Ask The Right Project Management Question And Get The Right Answer
  • Project Managers Get The Vision Right!
  • We Have To Do Something – Is Often Bad Project Management
  • Project Managers Get The Motivation Right!
  • Project Management Tools And Why Being Highly Customizable Can Be A Bad Thing
  • Project Managers – Now Is The Time!
  • Project Manager Don’t Ring That Bell For The Wrong Reason
  • Successful Project Management Is As Easy As Safely Flying A Plane
  • How to Gracefully Degrade Your Project Management in a Crisis
  • Project Manager Is It Really A Crisis?
  • Project Managers Are We Managing Or Are We Theorizing?
  • Why Project Management Dilbert Style Works
  • Project Managers Are You Managing Or Are You Experimenting?
  • Project Management Improvement Is As Easy As Brewing Coffee
  • Use Bad Situations To Make Good Project Management Changes
  • The One Great But Often Overlooked Project Management Improvement Technique
  • Project Managers, Maybe We Should Ignore Our Boss
  • Project Managers, How To Stop Making The Same Old Mistakes
  • Project Managers Follow The Crowd As Your Last Resort
  • Do We Manage Projects By Intuition or by Methodology?
  • Two Steps to Getting Truly Useful Project Management Metrics
  • Project Manager Are We Really Managing Both Risk And Change?
  • On Time Successful Projects? This Year We Can
  • Project Managers Beware of Oversimplifying Your Problems
  • Project Managers – Check your Assumptions!
  • Senior Manager It Is OK Not To Be An Expert Which Is Why You Have A Project Manager
  • Rooting Out And Removing The Causes Of Project Management Bad Habits
  • Why We Need Project Management Islands of Excellence
  • Project Manager If The Talk Is All Bad Then There Are Great Opportunities
  • Being A Manager Is Hard But Not For The Reasons I Expected
  • Beware The Moneyball Myth In Project Management
  • Key Metrics Often Tell Us More Than Classic Project Management Tools
  • Project Managers Finding A Noble Purpose For That Project Motivates Success
  • Project Managers Teach Your Daughters War Games!
  • Project Managers There Is A Better Use For Those Long Work Hours
  • Are We Project Managing Or Sending In The Clowns?
  • Project Manager Who Is Really In Charge Of Your Project?
  • Project Managers Use That Leadership Vacuum To Take The Initiative
  • Project Manager This Is Your Number One Priority
  • 50% Of Our Weekly Project Tasks Are Not Completed But How To Succeed Anyway
  • Too Many Rules Mean Something Is Wrong With Our Project Management
  • Do You Spend More Project Management Time With Your Eagles or With Your Turkeys?
  • What Project Management Can Learn From Medical Weight Loss Management
  • Project Managers Quit Talking Jargon To Our Customers
  • Five Secrets For Creating Successful Project Management Metrics
  • A Real Project Management Plan May Confuse Many People
  • Being Chaotic Is Often Smart In Project Managing Innovation
  • Project Manager It Is Ok To Charge A Customer For Our Services!
  • How to get An “A” On Our Project Management
  • Project Managers Get The Communications Right!
  • How To Keep Your Project Management On The Cutting Edge
  • We Know Everything We Need To Know To Be Successful Project Managers
  • The Right Project Management But The Wrong Project?
  • Project Management And Innovation Can Work Against Each Other
  • How Not To Kill Your Project Management Data With Kindness and Best Intentions
  • Why Change Management Oriented Projects Are Hard and How To Succeed Anyway
  • Why Project Management Success Can Make People Unhappy
  • Hiding The Problem As A Project Management Tool
  • STOP DOING THAT As A Project Management Tool
  • Why Waterfall Project Management Threatens To Return
  • How To Avoid Optional Chaos In Your Project
  • Project Management Choose Quality or Speed?
  • Project Management Choose Consensus or Speed?
  • How To Project Manage People Who Are Smarter Than You
  • Project Managing Other People’s Efforts Is Risky
  • If Quality Assurance Is Project Managing Then We Are In Big Trouble
  • Get The Project Budget Right!
  • In Project Management The “A” Student Can Be A Problem
  • Project Management Perseverance Is Still One Way to Finally Get It Done
  • Are Projects Really In Chaos?
  • The Most Underused Project Management Tool – Asking Your Staff’s Opinion
  • Is Your Project Management Pragmatic or Zealous?
  • Project Management By Taking a Break
  • In Project Management We Are Only Good At What We Spend Our Time On
  • Project Management Success by Failing Often
  • Why Your Project Does Not Need More People, More Time And More Money
  • Is it Our Project Management Or Our Strategy That Is Not Working?
  • Project Management Helps Us In Everyday Life Too
  • How Mentoring Can Be An Effective Project Management Tool
  • Is It Really OK For Your Project To Fail?
  • Manage Your Projects By Example To Improve Performance
  • Boycotting Meetings As A Project Management Tool
  • Do You Manage Your Project Team By Data Or By Direction?
  • 5 Reasons Managing Projects Objectively Is Hard And How To Succeed Anyway
  • Is Your Project Schedule Really Driven By Your Requirements?
  • How Not To Lose Our Way When Projects Go Bad
  • Project Managers – Double Check Your Temperature!
  • How To Get The Project Management Data We Need
  • Project Management By Hostage Taking?
  • Yes, You Can Quickly And Accurately Estimate And Commit Your Project
  • Is Project Management Quality Just A Belief?
  • Project Management Quality Needs Data Not Beliefs
  • Project Management By Living With Our Data
  • Four Ways The Project Manager Helps Innovations To Succeed
  • In Project Management — Patience Is Not A Virtue
  • Are 67 Percent Of Your Projects Delivered On Time?
  • The Project Management Secret Of Innovation
  • Project Management By Suits
  • Get the Project Management Cost Right!
  • It Must Be Something Other Than The Project Management Schedule
  • Its The Project Management Schedule, Stupid
  • Project Management Metrics Madness? Don’t Do It!
  • Project Management is Simple But Not Obvious
  • Are Project Management Costs Really The Problem?
  • Self-Managing Teams In Project Management? Is That Possible?
  • Compressed Project Management Schedules Discourage Innovation and Improvement
  • Three Ways Project Management Estimates Fail And How To Avoid Them
  • The Most Effective Part of the Day In Project Management?
  • Project Managers Repeat After Me: Software Development Is A Repeatable Process
  • Improving Project Management Productivity Does Not Save Time
  • One Reason To Always Support The Other Guy’s Project
  • Don’t Squash Your Project Management Pockets of Excellence
  • Don’t Kill The Golden Goose of Project Management – The Crisis
  • Project Management Crisis? Hurry, But Do Nothing
  • Project Management Initiative or Insubordination?
  • Your Project Needs Business Rules, Not Meetings
  • Project Management Or Anesthetized By Analytics?
  • Project Management or Death by Detail?
  • One Key Secret for Improving Project Management – Just Do It!
  • In Project Management 9+3 Is Not Equal To 12
  • The One Perfect Project Management Methodology
  • Three Reasons Not To Finish A Project Early
  • One Way To Make Your Project Risk Free?
  • 2 Ways To Make Project Management Problems No Big Deal By Managing Risk
  • Three Secrets For Successfully Putting Your Project Management Team At Risk
  • Don’t Get Everyone Project Management Certified … For The Wrong Reason
  • One Great Way Of Using Your Project Management Staff Hours
  • Four Insights Into The Best Use Of Your Project Management Experience
  • Fixing The Project Management Plan: What Should We Be Doing?
  • Visibility – The Fifth Project Planning Secret
  • The Official Plan: Four Project Management Planning Secrets
  • Yes Virginia, There Is A Project Management Plan
  • Get Out Of The Rut Of Solutions That Don’t Work!
  • We Really Can Know If Our Project Estimates Are Accurate
  • Your Project Team Really Can Do A Great Job
  • One More Way To Fight E-mail Infoglut: Talk!
  • Don’t Treat Your E-mails As Dead Skeletal Husks!
  • Five Techniques For Overcoming E-Mail Infoglut
  • Using Hard Facts To Reduce Honesty Buffers
  • Nine Ways To Eliminate Honesty Buffers
  • Eliminate Your Project Management Honesty Buffers
  • Focusing On High Quality Gives You High Productivity At No Extra Cost
  • Project Managing Climate Change And Your Kids On Blog Action Day
  • It Should Work The First Time
  • Getting The Schedule Right Is Hard! But Not For The Obvious Reasons.
  • The One Key Step For Successful Improvements
  • The Busiest Project Test Organization Was Also The Best
  • Almost a Great Project Test Organization
  • Want To Stress Your Project Test Team? Give Them Good Quality!
  • Is Your Project Management On Steroids? Maybe You Should Kick The Habit!
  • Disrupt The Project To Make The Big Improvements
  • Project Management Needs Business Intelligence!
  • Global Project Management? Further Away Can Be Better!
  • Are You Managing the Critical Project Thread?
  • Want Project Management Quality? Use the “B” Team
  • Successful Projects Are Boring!
  • A Successful Manager But Never A Successful Project?
  • Project Managers, Are You Being Heard?
  • Does Anyone Really Care If Our Project Is On Time?
  • Is Your Project On Time?
  • The Ultimate Project Management Tool
  • Project Management Meeting Madness? Don’t Do it!
  • Seven Ways To Make That “Silver Bullet” Work
  • Avoid This Project Management Schedule vs Requirements Trade-Off Trap!
  • Knowing Your Project Management Average Is Powerful!
  • Being Slightly Out Of Control Can Be A Good Thing
  • The Leap From Mediocrity To The Exceptional Is Shorter Than We Think
  • Don’t Project Manage Change – Provide A Service!
  • Defect Reports Are Your Best Friend!
  • Get the Project Management Schedule Right!
  • If Nothing Else, Honesty Is Just More Efficient

• Browse by Category

  • Change Management
  • Communication
  • Cost
  • Leadership
  • Management
  • Metrics
  • Personal Practices
  • Planning
  • Process Improvement
  • Quality
  • Reporting
  • Requirements Management
  • Risk Management
  • Schedule
  • Team Management
  • Testing
  • Time Management
  • Training

Tags

Agile Change Management Communication Defects Education Estimation Honesty Initiative Innovation Measurement Metrics On-Time Planning Productivity Project Management Tools Quality Reporting Requirements Risk Schedule Software Software Defects Successful Projects Tracking training

• Recommended Resources

  • Blog Catalog
  • Blogged
  • Business Blog Directory
  • Cloud Computing
  • Customer Service Helper
  • Financial Project Management
  • Search Open Directory
  • Seer by Galorath
  • Software Engineering Institute
  • Softwarebuzz – Tech Blog
  • Tom Peters