Multi-Node Cluster Install

Setting up a multi-node Hadoop cluster involves configuring several nodes (master and worker nodes) to work together. In this guide, we'll focus on a three-node cluster, where one node will act as the NameNode (master) and the other nodes will act as DataNodes (workers). We'll use a Linux-based OS (Ubuntu/CentOS) for this installation.

Pre-requisites

  1. Java 8 or higher installed on all nodes.

  2. SSH passwordless setup between all nodes.

  3. All nodes should have static IP addresses.

  4. Network configuration:

    • Ensure that all nodes can communicate with each other via their IP addresses.

    • All nodes should be able to resolve hostnames (hostname should resolve to an IP).

Step 1: Configure SSH on All Nodes

  1. Generate SSH key on the master node:

    ssh-keygen -t rsa

  2. Copy the public key to all nodes (master and workers):

    ssh-copy-id user@worker-node-ip
ssh-copy-id user@master-node-ip

    This will enable passwordless SSH access between the nodes.

  3. Test SSH between the nodes:

    ssh user@worker-node-ip
ssh user@master-node-ip

  4. Set up a hostname for each node in /etc/hosts (on all nodes):

    Open /etc/hosts and add entries like:

    192.168.1.1    master-node
192.168.1.2    worker-node-1
192.168.1.3    worker-node-2

    This allows each node to resolve the names of the others.

Step 2: Install Java on All Nodes

  1. Install Java 8 or higher (if not already installed) on all nodes:

    On Ubuntu:

    sudo apt update
sudo apt install openjdk-8-jdk

    On CentOS:

    sudo yum install java-1.8.0-openjdk-devel

  2. Set JAVA_HOME on all nodes:

    • Add the following to ~/.bashrc or ~/.profile:

      export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PATH=$PATH:$JAVA_HOME/bin

    • Reload the profile:

      source ~/.bashrc

Step 3: Install Hadoop on All Nodes

  1. Download Hadoop and extract it on all nodes:

    • On the master node:

      wget https://downloads.apache.org/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz
tar -xzvf hadoop-3.3.0.tar.gz
sudo mv hadoop-3.3.0 /usr/local/hadoop

    • On each worker node: Download Hadoop and extract it in the same way as on the master node:

      wget https://downloads.apache.org/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz
tar -xzvf hadoop-3.3.0.tar.gz
sudo mv hadoop-3.3.0 /usr/local/hadoop

  2. Set Hadoop environment variables on all nodes: Add the following to the end of ~/.bashrc:

    export HADOOP_HOME=/usr/local/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

  3. Reload ~/.bashrc:

    source ~/.bashrc

Step 4: Configure Hadoop on All Nodes

1. Configure hadoop-env.sh

  1. Edit hadoop-env.sh on all nodes (/usr/local/hadoop/etc/hadoop/hadoop-env.sh):

    nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh

    Add the following line to set JAVA_HOME:

    export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

2. Configure core-site.xml

  1. Edit core-site.xml on all nodes (/usr/local/hadoop/etc/hadoop/core-site.xml):

    nano /usr/local/hadoop/etc/hadoop/core-site.xml

    Add the following configuration:

    <configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://master-node:9000</value>
  </property>
</configuration>

3. Configure hdfs-site.xml

  1. Edit hdfs-site.xml on all nodes (/usr/local/hadoop/etc/hadoop/hdfs-site.xml):

    nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml

    Add the following configuration:

    <configuration>
  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///usr/local/hadoop/hadoop_data/hdfs/namenode</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///usr/local/hadoop/hadoop_data/hdfs/datanode</value>
  </property>
</configuration>

4. Configure mapred-site.xml

  1. Edit mapred-site.xml on all nodes (/usr/local/hadoop/etc/hadoop/mapred-site.xml):

    cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml
nano /usr/local/hadoop/etc/hadoop/mapred-site.xml

    Add the following configuration:

    <configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

5. Configure yarn-site.xml

  1. Edit yarn-site.xml on all nodes (/usr/local/hadoop/etc/hadoop/yarn-site.xml):

    nano /usr/local/hadoop/etc/hadoop/yarn-site.xml

    Add the following configuration:

    <configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>master-node:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>master-node:8030</value>
  </property>
</configuration>

Step 5: Set Up Directories for HDFS

  1. On the master node: Create the directories for HDFS and give appropriate permissions:

    sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/namenode
sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/datanode
sudo chown -R user:user /usr/local/hadoop/hadoop_data

  2. On each worker node: Create the directories for HDFS:

    sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/datanode
sudo chown -R user:user /usr/local/hadoop/hadoop_data

Step 6: Format the Hadoop Filesystem (Only on Master Node)

Run the following command on the master node to format the NameNode:

hdfs namenode -format

Step 7: Start the Hadoop Cluster

  1. Start HDFS: On the master node:

    start-dfs.sh

  2. Start YARN: On the master node:

    start-yarn.sh

  3. Check the status of your Hadoop cluster:

    • NameNode: http://master-node:9870

    • ResourceManager: http://master-node:8088

Step 8: Stop the Hadoop Cluster

To stop Hadoop, use the following commands:

  1. Stop HDFS:

    stop-dfs.sh

  2. Stop YARN:

    stop-yarn.sh

Troubleshooting Tips

  • SSH issues: Make sure passwordless SSH works correctly between nodes.

  • Permissions: Ensure that the Hadoop user has appropriate permissions for directories.

  • Firewall settings: Check firewall settings if the nodes cannot communicate.

  • Logs: Look into Hadoop log files (located in $HADOOP_HOME/logs) for detailed error messages if something goes wrong.

This multi-node setup allows Hadoop to operate efficiently across multiple machines, enabling scalable and distributed data processing.

PreviousSingle Node InstallNextCluster Install from Ambari

Last updated