Apache Hadoop is an open source big data processing platform. It has its own eco-system products to support various needs. Different big data products/platforms can integrate Hadoop and NoSQL into one platform so it provides better performance and a single source of truth. Let us have a look at how NoSQL and Hadoop can work together for big data challenges.
Sometime people often get confused that Hadoop is a database as it has a storage system associated. But let us clearly understand that Apache Hadoop is not at all a database.
Apache Hadoop is an open source big data platform consists of the following main components.
- HDFS: A file system known as Hadoop Distributed File System (HDFS)
- MapReduce: A distributed programming framework known as MapReduce
- Hadoop Common: It contains the libraries and utilities to support associated Hadoop modules.
- Hadoop YARN: This is called ‘Yet Another Resource Negotiator’. It is basically the resource management platform for managing computing resources and scheduling tasks.
Hadoop also has other host of software packages to support the eco-system components. The framework supports the processing of data intensive distributed applications. It enables applications to work in a distributed environment consists of thousands of nodes and petabytes of data. The nodes are independent computers, also known as low cost commodity hardware. Hadoop cluster means a group of computational units (basically machines) running in a general environment with Hadoop distributed file system (HDFS) to support scaling.
The fundamental design goal of Hadoop was to overcome the hardware failure. Because hardware failures are very common and the framework should be able to overcome it automatically. The Hadoop eco-system achieves this goal by all the modules.
The key features of Hadoop platform are a distributed storage and a distributed processing framework. The distributed storage (HDFS) splits large files into small blocks (default is 64MB) and distribute it across the clustered nodes. The distributed processing framework also known as ‘MapReduce’ supports very efficient parallel processing. The key feature of MapReduce is that, it ships the code (which will do the processing) to the node where the data resides. This is also called ‘data locality’, where the data remains in its original location and the code comes to it for doing the processing. This is indeed a revolution in parallel processing domain.
The main components of Hadoop (HDFS and MapReduce) are derived from Google’s File System (GFS) and Google’s MapReduce. Apart from the above components, Hadoop consists of a number of related projects like Apache Hive, Apache HBase, and Apache Pig etc.
On the other hand, NoSQL (interpreted as ‘Not only SQL’) is a non-relational database management system. It is identified by the non-adherence to the relational database model. NoSQL databases are not primarily based on tables.
NoSQL database technology provides efficient mechanism for storage and retrieval of data, but it is not similar as relational model. The main design goals of NoSQL databases are simple design, horizontal scaling and better availability. The name interprets as ‘Not only SQL’, so it supports some SQL like query languages like HQL etc. NoSQL databases are mostly used in big data and analytic applications.
So, in short we can define Hadoop and NoSQL as follows
- Hadoop: Distributed computing framework.
- NoSQL: Non-relational database.
How Hadoop and NoSQL can work together?
From the above discussion, it is clear that Hadoop and NoSQL is not the same thing, but they are both related to data intensive calculation. Hadoop framework is mostly used for processing huge amount of data (also known as big data) and NoSQL is designed for efficient storing and retrieval of large volume of data. So there is always a chance to have NoSQL as a part of Hadoop implementation. In most of the cases, the processed data from Hadoop system is stored in a NoSQL database. But they can always have independent use cases which may not need a support of both the platforms. For example, if we need only the parallel processing of big data and storing it in HDFS, then may be Hadoop alone is sufficient. Similarly, only for storing and retrieval of unstructured data, any NoSQL database and its associated query language can meet the requirement.
So the integration of NoSQL with Hadoop is always a preferred environment for large scale parallel processing and real time data access. Different Hadoop based products provide the integration of Hadoop and NoSQL in one platform. And this ‘in-Hadoop’ NoSQL database provides real time, operational analytics capabilities. Hadoop products including Apache Hadoop is the best fit for business critical production deployments. These products do not require any additional administrative tasks for the NoSQL data. The integrated platform (NoSQL and Apache Hadoop) supports high performance, extreme scalability, high availability, snapshots, disaster recovery, integrated security and many more, suitable for any production ready operational analytics.
So we can conclude that Apache Hadoop and NoSQL are not the same technology platform, but they are always recommended as an integrated environment suitable for big data solutions.
