Investing in big data is all the rage right now. But as with many topics that become buzz worthy, its core values often get lost in the hype. In customer conversations it has become increasingly clear that many companies are investing in the wrong places when it comes to data science, predictive analytics, or big data. They think they’re taking shortcuts to execute quickly, but are generally then left with systems that fall far short of their business needs. Here are the three major mistakes I see in choosing big data solutions and how to overcome them:  Mistake #1: Thinking the Business Problem Will Be Solved on the First Try I see this all the time; companies want a recommendation algorithm or some sort of machine learning capability. When broadly defined, their business problem seems straightforward enough, so they believe they should be able to simply buy a black-box solution or hire a person or firm to “build a model” and solve the problem.  As we see with companies like Netflix, which employs 300 people to maintain and improve its content recommendations, customer data is a continuously changing environment. That’s why the company also spends $150 million recommending movies and TV shows to its members every year to keep up with the cyclical state of machine learning.  Symptoms: Deliverables are delayed, everything requires manual effort, or systems don’t seem to return the business value promised.  Solution: No matter how simple the problem may seem at first, exceptions, edge cases, data hygiene challenges and integration issues will inevitably slow the execution and the results. The first application should be seen as an initial attempt to solve a complex problem, which will undoubtedly require constant iteration to be successful. Don’t think there’s a silver bullet or that you’ll be able to get there without a lot of effort.  Mistake #2: Leaving Your Business Problem to Be Solved by Data Scientists There is no doubt that having experts in statistics, machine learning and other areas of data science is incredibly important to the success of an initiative. However, left to their own devices, data scientists will rarely be able to achieve the business results an organization needs. Data scientists typically build new models and solve intricate equations, leaving a business problem, however obvious, not a priority. Data scientists are only one part of the complex, cross-functional team required to create business value.  Symptoms: The deliverables never seem to precisely address the business challenges, processes require more work to maintain and execute, or bugs seem to constantly present themselves.
Big data is all the rage in businesses large and small -- yet all too often, the actual data goes unused and misunderstood.