There has been a lot of hype around Hadoop for a long time. This hype was expected because Hadoop is perceived an extremely efficient big data processing tool. But time has come to look at some cold, hard facts. It is the time when the hype slowly dies down and businesses start to look at the Return on investment (ROI). A survey, conducted by Gartner, seems to show that a lot of companies are not planning to invest in Hadoop because they do not have the skills to use it or it is still deemed a user-unfriendly tool. There are other reasons as well. However, there is another group of people who are bullish about the prospects of Hadoop. If the situation seems confusing, it is because the attitude towards Hadoop is entering the phase of disillusionment from that of hype and that is natural. This is the time when businesses start to get realistic. This is the time when companies will objectively evaluate and use Hadoop.
Contrasting opinions on Hadoop adoption
As stated above, businesses are divided into two groups when it comes to Hadoop adoption: one group comprises enterprises that are reluctant, hesitant or circumspect when it comes to Hadoop adoption and the second group comprises enterprises that believe Hadoop is going to give good ROI. The attitude of the former group is reflected in the Gartner survey. Given below are the salient findings from the survey. Note that the survey findings were released in May, 2015. So, the results are pretty updated. The survey target audience comprised small and medium sized companies and C-level executives.
- 54% of the respondents do not plan to invest in Hadoop in the future.
- Just 18% of the respondents have plans to invest in Hadoop in the next two years.
- 26% of the respondents are deploying or experimenting with Hadoop.
- Companies that were reluctant or hesitant with Hadoop adoption cited skills shortage and user-unfriendliness as reasons for not thinking about Hadoop.
According to Merv Adrian (vice president at Gartner), “With such large incidence of organizations with no plans or already on their Hadoop journey, future demand for Hadoop looks fairly anemic over at least the next 24 mesi". Moreover, the lack of near-term plans for Hadoop adoption suggests that, despite continuing enthusiasm for the big data phenomenon, demand for Hadoop specifically is not accelerating”. The main reasons for such a negative response to Hadoop are given below.
- Lack of Hadoop skills is an important constraint. Enterprises claim that their staff is not capable of using Hadoop. Hadoop, in its original form, has been largely confined to an exclusive group who could use it productively. Though a number of third-party tools are coming up to facilitate the use of Hadoop, even they are not easy to use. The main complaint against Hadoop and the third-party tools are they require new skills to be learnt which means additional investment. Existing skills cannot be used. Though training is available for these tools, experts believe that it will take another 2 to 3 years for these programs to gain credence.
- For many enterprises, Hadoop is not a priority. They think that Hadoop is overkill for the business problems it is supposed to solve. It is like deploying a missile to kill a group of flies. Also, the cost of Hadoop adoption is more than the benefit derived by solving the business problems enterprises are facing.
Il secondo gruppo è ottimista e fiducioso circa l'adozione Hadoop. Ci sono aziende che hanno iniziato a utilizzare Hadoop per il loro business mainstream e stanno raccogliendo i benefici. La caratteristica principale che viene utilizzato è l'elaborazione dei dati in tempo reale. For example, le aziende possono prevenire le frodi analizzando i dati su una base in tempo reale. Le aziende sono in grado di fornire prodotti migliori attraverso l'analisi su una valutazione dei dati base in tempo reale dai loro clienti. Essi stanno ricevendo i dati dal sito-uso, video, Banca online, social media e varie altre fonti. In un sondaggio condotto da TechValidate, 96% degli intervistati sono in esecuzione diversi casi d'uso su un singolo cluster Hadoop e tra loro, 20% sono la distribuzione di quasi 50 casi d'uso su un singolo cluster Hadoop. Il sondaggio ha rivelato che 73% degli intervistati sono la distribuzione dei loro prodotti e servizi e 59% are benefiting from reduced costs. Most of the above companies are MapR customers. According to Bryon Dover, a big data engineer with the Rubicon Project, “MapR gives me the reliability to process 3 trillion transactions a month with 99.999% uptime.”
What to make of the above findings?
The two extreme attitudes towards Hadoop adoption can be generally confusing but in the context of business cycle, this stage represents just another phase: that of enterprises leaving the hype stage and entering the evaluation stage. When there is hype, everything seems rosy but when there is evaluation, the disadvantages also come out. So, businesses are finding out how to best use Hadoop to solve their business problems or whether, Hadoop is at all required.
Hadoop ha bisogno di superare la sua esclusività di sicuro perché è considerato uno strumento difficile da usare, disponibile solo per le persone che sono specializzati. Non vi è alcun buon front-end che lo rende facile per le persone per elaborare e analizzare i dati. Also, vi è la necessità di imparare Hadoop e che ha bisogno di ulteriori investimenti. strumenti di terze parti che pretendono di rendere Hadoop facile da usare non sono esattamente vivere fino alle loro richieste. So, tutta l'offerta ha bisogno di modifiche e sta andando a prendere tempo. Basically, Hadoop deve dimostrare che è facile da usare.
Le imprese devono rendersi conto che uno dei migliori usi di Hadoop è quando si elaborano i dati in tempo reale. È qui che il secondo gruppo di clienti è benefici raccogliendo. L'elaborazione batch non è dove dovrebbe essere incentrato. Real time processing can enable you prevent frauds and offer customized products and services to your customers. Hadoop is not meant for static data. The image below shows that real-time usage is the biggest consideration for Hadoop.
In this context, it is pertinent to mention Apache Spark which has been doing a stellar job analyzing big data real time. It gives you a unified and comprehensive framework that helps you to manage huge data sets from variety of sources in real-time basis. The biggest advantage with Spark is that you can rapidly write applications in Python, Java or Scala and has more than 80 high-level operators. It also supports SQL queries, machine learning, streaming data and processing of graph data, other than Map and Reduce operations. In a nutshell, it can prove to be an effective real-time processing application.
Real time usage of Hadoop
Image1: Real time usage of Hadoop
Any adoption of a new technology takes time. Hype and adoption are different things. It is quite possible that a percentage of the 57% of the respondents of the Gartner survey who did not plan to invest in Hadoop may do so after some time as Hadoop enters the mainstream production stage of many companies and its benefits start showing. This is especially after new technologies of products start to make Hadoop more usable. The SQL on Hadoop, per esempio, may be just the starting point of making Hadoop more accessible to a wider community.
Summary
The indifference towards Hadoop does not make it an unproductive tool. It is only that businesses are still unfamiliar with its ways. As MapR customers will corroborate, you need to identify how to best use Hadoop for solving your business problems. Using it for real-time data processing in the mainstream production appears to be the way to go. Similarly, there are other benefits too that still seem undiscovered. Of course, much around Hadoop and its ecosystem needs to change. It needs to be more accessible to anyone who wants to use it. The slow adoption rate of Hadoop could turn out into an acceptable proposition after 2 to 3 years.
