The amount of data being created and consumed in the world has been exploding, and analyzing large data sets – so-called big data – will become a huge challenge and has prompted use of new technologies to search, extract and analyze this data. The increasing volume and detail of information captured by manufacturing applications, financial trading applications, web analytics, operational analytics, enterprises and the rise of multimedia, social media, and the Internet will fuel exponential growth in data for the foreseeable future.
As the various IT vendors have analyzed in the past, in the data deluge faced by various businesses, there is an increasing need to store and analyze vast amounts of unstructured data including data from sensors, devices, manufacturing applications, trading applications, social media, bots and crawlers and this volume is predicted to grow exponentially over the next few years. Consumers have been requesting the IT vendors to create new technologies to help store, manage, and analyze their big data.
For discussing the challenges of Big Data, let’s pick the special report proposed by Gartner which examines leverage Pattern-Based strategy to gain value in Big Data. Gartner characterizes Big Data as challenges in three distinct areas, characterized by the 3Vs:
– Volume: The volume dimension describes the challenges an organization faces because of the large and increasing amounts of data that need to be stored or analyzed.
– Velocity: The velocity dimension captures the speed at which the data needs to be processed and analyzed so that results are available in a timely manner for an organization to act on the data.
– Variety: The variety dimension finally looks at the different kinds of data that need to be processed and analyzed, ranging from tabular data in relational databases to multimedia content like text, audio or video.
To discuss the challenges faced by Big Data, the first two dimensions, i.e., volume and velocity, are the most interesting dimensions. To cover these dimensions in Big Data solution, an intuitive approach will lead you to a picture like the one illustrated in Figure 1. It shows the dimensions volume and velocity and overlays it with technologies such as MapReduce or Complex Event Processing.