Juan Antonio Ruiz Project Manager and Senior CRM Consultant with more than 12 years of experience in multicultural and international IT projects. Siebel specialist in several Siebel modules, EIM, ETL tools, Business Intelligence, OBIEE, batch integration and massive data migration. Degree in Mathematics from the University of Granada and Master in Strategic Project Management from IE (Instituto de Empresa), he combines his responsibilities as a partner in the company and project manager in the clients
A very fashionable concept lately and that according to some analysts is one of the 5 trends of the next decade: Mobile, Social, Cloud, Consumerization and Big Data
Big Data is, as the name suggests, "Big Data", that is, analysis and processing of large volumes of data. So large that current tools will not be able to process.
Big Data: Set of techniques and technologies that allow storing and processing huge amounts of data in an effective and efficient way.
According to IBM, more than 2.5 quintillion bytes are generated per day, to the point that 90% of the world's data has been created during the last two years.
We are therefore facing a great challenge in the coming years.
Not all the information that is generated can be considered Big Data. There are four main characteristics that identify what type of data can be considered Big Data:
- When we talk about volume, we mean from petabytes up.
- Speed: Near real-time
- Variety of information sources
- Added value, being able to discover patterns and predict future behaviors
To put ourselves in context:
- Kilobyte = 103 bytes
- Megabyte = 106 bytes
- Gigabyte = 109 bytes
- Terabyte = 1012 bytes
- Petabyte = 1015 bytes (1000 terraces)
- Exabyte = 1018 bytes (1 million Teras)
- Zettabyte = 1021 bytes (Billion Teras)
- Yottabyte = 1024 bytes (1 trillion teras)
According to Cisco's Visual Networking Index, by the end of 2016 the world will have crossed the threshold of the zettabyte of data transferred over the Internet for one year. Truly these numbers give vertigo (http://technografy.blogspot.com.es/2013/04/so-how-much-is-zettabyte-infographic.html (opens in a new tab))
Traditionally, data warehouse systems analyze past behaviors. The great challenge of Big Data is to predict the future.
For the usual data warehouse you need large infrastructures, large servers, large spaces conditioned to measure, it is very expensive, it goes out of fashion quickly.
Big Data requires a new technological infrastructure.
In the opensource project of apache they have created Hadoop (http://hadoop.apache.org/ (opens in a new tab)), which allows you to use hardware in "commodity" mode, that is, cheap hardware that instead of growing "vertically" (+ processors + memory + disk), grows horizontally (cluster of nodes), that is, adding nodes as you need more storage and processing capacity.
Large manufacturers are also betting on big data and creating new distributions of hadoop oriented to the business world. Other providers such as Amazon allow you to mount your cluster of nodes in the cloud with low-cost hardware.
Hadoop is an open source framework that allows you to develop and run parallel processing applications for large amounts of data, both structured and unstructured.
It is designed to work from a few servers to hundreds, where each takes care of both processing and storage.
Some haddop solutions:
- Hadoop Common: Accessing Hadoop Modules
- Hadoop Distributed File System (HDFSTM): Distributed file system for data access
- Hadoop MapReduce (YARN): parallel processing of large volumes of data
- Other components for task management and available resources
Finally, some possible applications of BigData:
- Customer Churn: Predict the likelihood that any customer can unsubscribe based on their behavior
- Improve campaign management: Forecast demand and improve campaign performance
- Risk modelling: managing exposure and delivering better products to clients.
- Improving Sales and Customer Segmentation: Increasing Revenue and Maintaining Financial Strength
