Threats of federal intervention aside, it will be difficult to fix the problem of high murder rates without first addressing clearance rates. So it’s fortuitous, perhaps, that we are living in an age in which the analysis of data is supposed to help us decipher, detect, and predict everything from the results of presidential elections to the performance of baseball players. … Law enforcement would seem to be a fertile area for data to be helpful: In the 1990s the New York Police Department famously used data to more shrewdly deploy its officers to where the crimes were, and its CompStat system became the standard for other departments around the country. What Hargrove has managed to do goes a few orders of magnitude beyond that. His innovation was to teach a computer to spot trends in unsolved murders, using publicly available information that no one, including anyone in law enforcement, had used before. Bloomberg Businessweek, Murder He Calculated, Feb 13, 2017.

Exploiting existing data is a no-brainer these days, but it still takes someone to first notice and then to dig in and see what insights are available. Getting the first drops of useful information out of a data source that was often constructed for a different purpose can be a challenge.

I dug into our “feature” database, which was built to track our new requirements, to try and determine how long it was taking us to actually develop a feature. I wanted to know from when a feature was proposed to when it was approved and then to when it was ready to go into a product. I annoyed the keepers of the database when I asked if they could put timestamps on each of these events. They saw no reason for such information but luckily the database system they used had its own internal timestamps as well as a transaction journal of when certain things were changed in the database.

In no time flat I had insight into how long it was taking features to progress through their lifecycles. There was even a movement to try and block me accessing the information because, not surprisingly, the data showed it was taking significantly longer to deliver than we were promising. The observation that the data gleaned from the database matched well with our actual experience only further annoyed folks who insisted we could deliver new products quickly and had already promised such things to our customers. Luckily, our customers ultimately thanked us for finally giving them realistic estimates that we then actually met.

Getting useful data and then getting it into actual use can be, and often is, a challenge. The good news is that good data is often at our fingertips. It just takes a bit of curiosity and courage to dig in and see what is lurking there.

What are you doing to find the hidden gems of insight in your project data?