“As my internist friend likes to say, ‘Nobody’s healthy — they just haven’t been tested enough,’” says Nigel Paneth, a professor of epidemiology and biostatistics at Michigan State University. Hacking the Human OS, IEEE Spectrum, June 2015.

I love data. Once I started to collect and pay attention to data coming out of our software development projects I found profound insights into how our organization ticked. From there we were able to predict our future performance with much greater accuracy than our old traditional standardized bottom-up detailed planning techniques. We went from late and buggy software releases to on-time with continually improving software quality.

For more see How To Quickly Give Your Boss An Accurate Estimate

The catch was always that anyone could come up with data and with enough data, one could show just about anything they wanted. Oftentimes we were awash with data analysis results because everyone wanted to jump on the bandwagon and find the next great insight. We came to trust data so much that anything that could be plotted on a chart could become persuasive.

So as much as I love data and the current big push into analyzing “big data” the quote above reminded me that we can go awfully wrong and do considerable harm when we don’t do data analysis well. 

Intensely monitoring healthy individuals could make them less well, not more. Take today’s screenings for prostate cancer, which often flag benign lesions and slow-growing tumors that will never become life threatening. Worried that patients were undergoing unnecessary surgeries, medical groups recently scaled back screening recommendations. The more measurements you take, the more likely you are to find something wrong. Hacking the Human OS, IEEE Spectrum, June 2015.

I always hated the idea of stopping the collection of data. I’ve had to fight many a battle to get access to data or to prevent historical data from being thrown out (yeah, they really did). Instead of stopping the data collection, what worked for me was often to just back off and watch the data for a while and keep asking myself “What is really going on here? Is this really showing me something that is an enduring characteristic of the organization? What is this telling me that is useful, that I can make decisions based upon?” We would spend some time looking at the data and not trying to act on it until we truly understood what it was telling us.

With time and the progress of projects, the insights would ultimately come and I could see how the data was telling us how we were doing. I also would always dig into data from past projects and compare that to what I was seeing in our current projects. This helped tremendously to understand what was normal and what might be a significant difference we would need to respond to.

Data can be a tricky thing to work with. Just having lots of data doesn’t guarantee anything, except maybe lots of time spent looking at it. We just want to make sure an over-zealousness in looking at data doesn’t result in doing more harm than good.

See related Managing Projects Objectively Is Hard And How To Succeed Anyway

Is your data analysis helping your project to succeed or is it just making more work for your team without adding any real benefits?