Managing Big Data
Just about everyone these days suffers from the information overload explosion from our computers, smartphones, media, colleagues, and customers. Information is essential to making intelligent decisions, but more often than not, it simply overwhelms us. It’s like trying to drink from a fire hose. How do you find the truly essential nuggets of information and use them with confidence?
The amount of data and information we consume has grown exponentially. Big Data refers to the large datasets of information being amassed as a result of our social, mobile, and digital world. It is as McKinsey defines “datasets whose size is beyond the ability of typical database software to capture, store, manage, and analyze.” Every day, we create 2.5 quintillion bytes of data — so much that 90 percent of the data in the world today has been created in the last two years alone. Not only has the volume of data increased, but also the variety of data, including structured and unstructured data (text, sensor, A/V), and the velocity of data including the doubling of business data every 1.2 years.
The trend towards larger data sets is the result of our ability to gather data from multiple sources. Yet, our ability to process the information is slow to follow. Although data production in 2020 is expected to be 44 times that of 2009, IT spending is expected to increase only five percent in the same period.
Christopher Frank, Vice President of American Express and Paul Magnone, co-authors of Drinking from the Fire Hose, believe the big data pendulum will rebound giving rise to ‘essential information’. They urge organizations to understand their readiness for big data—Are you ready to capture it? Do you have the right people to analyze it? Do you have an organization that is willing to listen? And, is the data addressing a business need?
Frank and Magnone predict five shifts that will require organizations to rethink everything from go-to-market and service strategies to hiring, skill building and decision-making. Shifts such as the rise of easily accessible analytical tools, control to collaboration, and self-serve to social service will be prevalent in the near future. An evolution from ‘ask to listen’ type market research will lead to a new method of questioning to discover essential information.