Ansible Playbook To Configure Hadoop And Start Cluster Service

For configuring hadoop cluster we need a name node and a data node. Data node will share its storage to name node to store data For donig this we will use ansible playbook.

Configuration of hadoop cluster consist of several steps:

Firstly we have to configure Name Node:

  1. We have to send hadoop and jdk(java) software to the target node using ansible for this these two software or rpm we should have in our system.
tasks:    
- name: "copying hadoop software"
copy:
src: "/root/hadoop-1.2.1-1.x86_64.rpm"
dest: "/root/"
- name: "copying jdk"
copy:
src: "/root/jdk-8u171-linux-x64.rpm"
dest: "/root/"

2. We will install the software in target node

- name: “installing software” 
shell: “rpm -ivh hadoop-1.2.1–1.x86_64.rpm — force”
— name: “installing jdk”
shell: “rpm -ivh jdk-8u171-linux-x64.rpm”
#Instead of shell command here we can use "command" module also.

3. Now we will configure Hdfs file and core file

file: 
state: directory
path: “{{namenode_dir}}”
— name: “hdfs file configuration”
blockinfile:
path: “/etc/hadoop/hdfs-site.xml”
insertafter: “<configuration>”
block:
<property>
<name>dfs.name.dir</name>
<value>{{namenode_dir}}</value>
</property>
— name: “core file configuration”
blockinfile:
path: “/etc/hadoop/core-site.xml”
insertafter: “<configuration>”
block:
<property>
<name>fs.default.name</name>
<value>hdfs://0.0.0.0:9001</value>
</property>

4.Formating Name node

- name: “formating namenode”
shell: "echo Y | hadoop namenode -format"

5.Starting services

- name: “starting namenode service”   s
shell: "hadoop-daemon.sh start namenode"

Same kind of setup we have to do for Data node

Name Node

When we run These ansible playbook our hadoop setup will configured

VEDIO LINK

https://youtu.be/o7BDwNi53r0

ENJOY LEARNING…..

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Our experience at SOUTHWORKS

Meteor: good practices with API & Data

Graph Data Structure

2022 Is For Building Lean Web Apps

Partial Updates (PATCH) in Spring Boot

Creating High Availability Architecture with AWS CLI

Single Responsibility Principle

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ashish Dwivedi

Ashish Dwivedi

More from Medium

How to use secrets without relying on .config files

Use Ansible to customize AWS EMR through bootstrap actions

Run A Small/Medium Sized Wikipedia Clone on Microsoft Azure

Data Masking & Encryption in Azure