Hadoop 2.0 and Advantages of Hadoop 2.0 over 1.0

This post explains the benefits of Hadoop two.0 and is in continuation to our previous diary post saying the arrival of stable unharness of Hadoop two.0 for production deployments.

Since then Apache has free 2 a lot of releases of Hadoop two. the foremost recent unharness two.4.0 of Hadoop two currently supports Automatic Failover of the YARN ResourceManager. as a result of several such enterprise prepared options, Hadoop is creating news and positive predictions.

This post explains the new options well and clarifies several rife doubts regarding Hadoop two.0. If you're unaccustomed Hadoop, review our previous diary posts on HDFS and MapReduce and HDFS design. if you have any queries contact Hadoop administration online training 

Following area unit the four main enhancements in Hadoop two.0 over Hadoop one.x:

HDFS Federation – horizontal quantifiability of NameNode

NameNode High availableness – NameNode is not any longer one purpose of Failure

YARN – the ability to method Terabytes and Petabytes of information obtainable in HDFS exploitation Non-MapReduce applications like MPI, GIRAPH

Resource Manager – splits up {the 2|the 2} major functionalities of loaded down JobTracker (resource management and job scheduling/monitoring) into two separate daemons: a worldwide Resource Manager and per-application ApplicationMaster

There area unit further options like capability hardware (Enable Multi-tenancy support in Hadoop), information exposure, Support for Windows, NFS access, sanctionative exaggerated Hadoop adoption within the business to resolve huge information issues.

HDFS Federation

Even though a Hadoop Cluster will rescale to many DataNodes, the NameNode keeps all its data in memory (RAM). This ends up in the limitation on most variety of files a Hadoop Cluster will store (typically 50-100M files). As your information size and cluster size grow this becomes a bottleneck as size of your cluster is restricted by the NameNode memory.

Hadoop 2.0 feature HDFS Federation permits horizontal scaling for Hadoop distributed filing system (HDFS). this can be one in every of the numerous asked for options by enterprise category Hadoop users like Amazon and eBay. HDFS Federation supports multiple NameNodes and namespaces.

In order to scale the name service horizontally, federation uses multiple freelance Namenodes and Namespaces. The Namenodes area unit federates, that is, the Namenodes area unit freelance and don’t need coordination with one another. The DataNodes area unit used as common storage for blocks by all the Namenodes. every DataNode registers with all the NameNodes within the cluster. DataNodes send periodic heartbeats and block reports and handle commands from the NameNodes.

NameNode High availableness

In Hadoop one.x, NameNode was the single purpose of failure. NameNode failure makes the Hadoop Cluster inaccessible. Usually, this can be a rare prevalence as a result of business-critical hardware with RAS options used for NameNode servers.

In the case of NameNode failure, Hadoop directors ought to manually recover the NameNode exploitation Secondary NameNode.

Hadoop 2.0 design supports multiple NameNodes to get rid of this bottleneck. Hadoop 2.0, NameNode High availableness feature comes with support for a Passive Standby NameNode. These Active-Passive NameNodes area unit designed for automatic failover.

All namespace edits area unit logged to a shared NFS storage and there's solely one author (with fencing configuration) to the present shared storage at any purpose of your time. The passive NodeNode reads from this storage ANd keeps updated data for the cluster. just in case of Active NameNode failure, the passive NameNode becomes the Active NameNode and starts writing to the shared storage. The fencing mechanism ensures that there's just one write to the shared storage at any purpose of your time.

With Hadoop unharness two.4.0, High availableness support for Resource Manager is additionally obtainable. more details visit Hadoop administration online course  

YARN – yet one more Resource communicator

A large amount of information from multiple stores is keeping in HDFS however you'll solely run MapReduce framework jobs on to method and analyze constant (with Pig and Hive). To method with alternative framework applications like Graph or Streaming, you would like to require this information out of HDFS, as an example, into Cassandra or HBase.

Hadoop 2.0 provides YARN API‘s to write down alternative frameworks to run on prime of HDFS. this allows running Non-MapReduce huge information Applications on Hadoop. Spark, MPI, Giraph, and HAMA area unit few of the applications written or ported to run at intervals YARN.

YARN provides the daemons and Apis necessary to develop generic distributed applications of any kind, handles and schedules resource requests (such as memory and CPU) from such applications, and supervises their execution.

YARN – Resource Manager

In Hadoop, JobTracker is that the master daemon for each Job resource management and scheduling/monitor of Jobs. In giant Hadoop Cluster with thousands of Map and scale back tasks running with task trackers on DataNodes, this ends up in the central processing unit and Network bottlenecks.

It takes care of the complete life cycle of employment from programming to prosperous completion –Scheduling and observance. It conjointly needs to maintain resource data on every one of the nodes like the variety of map and scale back slots obtainable on DataNodes – Resource management.

The Next Generation MapReduce framework (MRv2) is AN application framework that runs at intervals YARN. The new MRv2 framework divides the 2 major functions of the JobTracker, resource management, and job scheduling/monitoring, into separate elements.

The new ResourceManager manages the worldwide assignment of reckoning resources to applications and therefore the per-application ApplicationMaster manages the application’s programming and coordination.

YARN provides higher resource management in Hadoop, leading to improved cluster potency and application performance. This feature not solely improves the MapReduce processing however conjointly permits Hadoop usage in alternative processing applications.

YARN’s execution model is a lot of generic than the sooner MapReduce implementation in Hadoop one.0. YARN will run applications that don't follow the MapReduce model, not like the first Apache Hadoop MapReduce (also referred to as MRv1).

It is necessary to grasp that YARN and MRv2 area unit 2 totally different ideas and may be used interchangeably. YARN is that the resource management framework that has infrastructure and Apis to facilitate the request for, allocation of, and programming of cluster resources. As explained earlier, MRv2 is AN application framework that runs at intervals YARN.any queries contact Hadoop admin online course 

Capacity hardware – Multi-tenancy Support

In Hadoop one.0 all DataNodes area unit dedicated to Map and scale back tasks and can't be used for the alternative process. In Hadoop one.0, the cluster’s capability is measured in MapReduce slots. every node within the cluster contains a pre-defined set of slots, and therefore the hardware ensures that a share of these slots area unit obtainable to a collection of users and teams. therefore if you're not running MapReduce jobs, you're wasting DataNode resources.

With capability hardware support in Hadoop two.0, DataNode resources are often used for alternative Applications too. The capability hardware (CS) ensures that teams of users and applications can get a warranted share of the cluster, whereas increasing overall utilization of the cluster. Through AN elastic resource allocation, if the cluster has obtainable resources then users and applications will take up a lot of-of the cluster than their warranted minimum share.

In Hadoop two.0 with YARN and MapReduce v2, the cluster capability is measured because the physical resource (RAM currently, and central processing unit similarly within the future) that's obtainable across the complete cluster.

The ResourceManager supports hierarchic application queues and people queues have often warranted a share of the cluster resources. It performs no observance or pursuit of standing for the appliance and works as a pure hardware.

The ResourceManager performs its programming operate supported the resource necessities of the applications. every application has multiple resource request like memory, CPU, disk, network etc. it's a big amendment from this model of fixed-type slots in Hadoop MapReduce, that ends up in important negative impact on cluster utilization.

You can consider our post on HDFS and MapReduce, HDFS design, five Reasons to find out Hadoop and conjointly however essential is Hadoop admin online Training 

Share: