Approaching big data analytics is no different than learning how to surf.
Ever wonder why surfers have broad shoulders? When surfers ride a wave, the effort does not seem to rely on their shoulders, or their legs. The true reason for a surfer’s broad upper body physique is that it is used to prepare for the next wave: paddling out to sea, paddling towards the peak and paddling to catch the ultimate wave. Along with upper body strength, surfers rely on balance, agility and the ability to read an incoming swell.
Big data analytics is no different. Business users know big data analytics for its “cool” visualizations and “cool” findings. However, we forget all the paddling that need to happen to get to that point.
Getting to the visualizations and findings require data integration, data collection, data cleansing, and data organization. The barriers can be daunting. How are we going to collect billions of rows? How are we going to find our way across the tens of thousands of tables and fields where our data is located? Even after data collection, how will we connect the dots between the billions of data points to expose them in a business sense?
These tasks, which often represents about 80 percent of a big data project, are difficult to tackle as they require the perfect mix of processing fire power, data sources, knowledge and business expertise to weave data in a meaningful way.
In terms of effort and time, data integration is usually where the bulk of the effort is when starting big data analytics. One of the key game changers that application-based big data solution brings is that it encapsulate that entire process: they readily offer data collectors that can take on huge data volumes while embedding business expertise created from decades of business research. Yes they could give up some flexibility, in terms of breadth of field tapped into, but they guarantee performance and data quality for fields they are designed for.
The benefits are doubled: it leaves all the heavy lifting to the machine that can execute data acquisition flawlessly, sustainably and at unmatched speed. It also creates a foundation, a clean foundation at that, to run statistical predictive analytics from.
For new surfers, one of the first things learned is that one can never ride a wave if the effort of catching it has taken too much energy. The same principle can be applied to business analytics: if the core data collection is a breeze, users can focus on the added value analytics.
I would venture one step further: big data applications also carries all the math and statistics payload that is already connected with acquired data, allowing users to focus on interactions and investigations.
Have you ever tried to apply statistical models to a data set? The problem does not lie in understanding the formula, it’s the application–how to apply it the right way, to the right data, without creating false conclusions or erroneous predictions. Predictive analytics, with the capability to deliver robust results or statistics, are way more complex than our favorite linear regression in excel: they touch on concepts that are, technically speaking, within the domain of mathematicians, statisticians, and PhD’s—not finance people.
I tried to believe the contrary, but working with Stanford University and Heidelberg University statisticians humbled me not to chase that vain race of understanding that level of statistics (and neither should we spend our time with data plumbing). It’s for the best because our value, as business people, is not to spend our time on playing sorcerer’s apprentice with formulas we could never understand. Our value is to take analytics results and use them to find answers for the most complex business questions.
In surfing, I dream of a solution that can take me back to the top of a wave, freshly rested—and launch me back on an amazing ride. In business, this dream is achieved. New big data applications offer a realm of endless possibilities for business users. The right application can take the user to the top of their data: by offering readily accessible statistical results, simulation capabilities and predictions each time with no effort. The right application can give business users confidence, allowing them to focus solely on their analytics ride.
And surfer’s rules dictate, the first person standing at the peak has right of way. Let’s be that guy in big data.