These spacecraft also underscore the value of data preservation.  In the early days of the Pioneer missions, scientists and engineers often viewed the medium as more valuable than the data it contained.  Many considered raw data to be worthless once “useful” scientific and technical information had been extracted.  Nowadays data storage may be cheap, but we’re still in danger of suffering from shortsightedness when it comes to data custodianship.  … It may very well be the only way we’ll resolve the next confounding mystery. Find The Source Of The Pioneer Anomaly, IEEE Spectrum, December 2012.

The engineering manager had rearranged his software project.  He had organized his team differently and so the database that tracked the team’s features and defects no longer matched the team’s organization.  His solution?  Just throw out all the past data and start with a new database that reflected his organizational structure.   Of course, no one could now tell how the team performed in the past in delivering features and fixing defects. This manager didn’t see why that would matter and besides he reorganized his team to fix its many problems and so that past history was irrelevant anyway. I almost cried.

Luckily, I had regularly captured snapshots of our performance data and kept them in an archive.  I had all the past data, just no one else did.  Guess what, his team didn’t perform any differently than they had in the past.  I could use this past data, with the new data, and could predict when they would, for example, finish removing defects from their current software.

I would regularly use the performance of past projects to explain to people how our current project would most likely unfold.  Too often I would get people, including the company’s VPs, telling me I needed to forget the past — it had too many mistakes — and just focus on the current project and it’s current data.  Too many people just wanted to forget the challenges of the past (i.e., the late and low quality project deliveries) with the belief that doing so would free them to do better stuff in the future.

For more see Key Metrics That Often Tell Us More Than Classic Project Metrics

It never happened that way.  In particular, because I didn’t ignore the past, I could readily see how we consistently repeated the same problems as from past projects. They just couldn’t see it because most projects were so large and so complex that few engineers or managers, no matter how intellectually gifted they were, could remember enough details to realize that we were seeing and experiencing  the same things we had in the past, both good and bad.  Specifically, the details were often different, but the pattern was the same and resulted in the same results, which were late and buggy projects.

Before big data, which is often focused on our customers, we’ve always had medium data right here in our company that reflected how we were doing.  While we’ve come to understand the need to superanalyze our customers, we still often seem to forget to seek the same insights on our own performance in our own organization.  Anytime we throw away “old data” we are often times effectively forgetting our past and so will inevitably be doomed to repeat it.

See more at: We Probably Have All The Data We Need For Success

Are you throwing away data that could help you be very successful on your next project?