hadoop_setup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# hadoop setup\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### wget http://mirrors.estointernet.in/apache/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz\n",
    "\n",
    "extract in /opt/hadoop "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# configure sitexmls\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### cd hadoop-2.7.3/etc/hadoop/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### core-site.xml\n",
    "core-site.xml informs Hadoop daemon where NameNode runs in the cluster. \n",
    "It contains configuration settings of Hadoop core such as I/O settings that are common to HDFS & MapReduce."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
    "<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>\n",
    "<configuration>\n",
    "<property>\n",
    "<name>fs.default.name</name>\n",
    "<value>hdfs://localhost:9000</value>\n",
    "</property>\n",
    "</configuration>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### hdfs-site.xml\n",
    "hdfs-site.xml contains configuration settings of HDFS daemons (i.e. NameNode, DataNode, Secondary NameNode). \n",
    "It also includes the replication factor and block size of HDFS."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
    "<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>\n",
    "<configuration>\n",
    "<property>\n",
    "<name>dfs.replication</name>\n",
    "<value>1</value>\n",
    "</property>\n",
    "<property>\n",
    "<name>dfs.permission</name>\n",
    "<value>false</value>\n",
    "</property>\n",
    "</configuration>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### mapred-site.xml\n",
    "mapred-site.xml contains configuration settings of MapReduce application like number of \n",
    "JVM that can run in parallel, the size of the mapper and the reducer process,  \n",
    "CPU cores available for a process, etc."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
    "<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>\n",
    "<configuration>\n",
    "<property>\n",
    "<name>mapreduce.framework.name</name>\n",
    "<value>yarn</value>\n",
    "</property>\n",
    "</configuration>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### yarn-site.xml\n",
    "yarn-site.xml contains configuration settings of ResourceManager and NodeManager \n",
    "like application memory management size, \n",
    "the operation needed on program & algorithm, etc."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "<?xml version=\"1.0\">\n",
    "<configuration>\n",
    "<property>\n",
    "<name>yarn.nodemanager.aux-services</name>\n",
    "<value>mapreduce_shuffle</value>\n",
    "</property>\n",
    "<property>\n",
    "<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>\n",
    "<value>org.apache.hadoop.mapred.ShuffleHandler</value>\n",
    "</property>\n",
    "</configuration>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ==>open hadoop-env.sh and update  JAVA_HOME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### ==> open ~/.bashrc"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "update below details \n",
    "\n",
    "export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/\n",
    "export HADOOP_HOME=/opt/hadoop/hadoop-3.1.2/\n",
    "export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop\n",
    "export HADOOP_MAPRED_HOME=$HADOOP_HOME\n",
    "export HADOOP_COMMON_HOME=$HADOOP_HOME\n",
    "export HADOOP_HDFS_HOME=$HADOOP_HOME\n",
    "export YARN_HOME=$HADOOP_HOME\n",
    "export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_HOME/bin\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### ==> format via name node for the first time \n",
    "bin/hadoop namenode -format\n",
    "\n",
    "\n",
    "initializing the variable dfs.name.dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ==> change directory to hadoop-version/sbin\n",
    "\n",
    "cd /opt/hadoop/hadoop-3.1.2/sbin"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "1. sudo apt-get install openssh-server\n",
    "2. sudo /etc/init.d/ssh restart\n",
    "3. ssh-keygen\n",
    "4. cat /home/<user>/.ssh/id_rsa.pub >> .ssh/authorized_keys\n",
    "5. ssh localhost\n",
    "6. ./start-all.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### shawak@shawak:/opt/hadoop/hadoop-3.1.2/sbin$ \n",
    "\n",
    "./start-all.sh\n",
    "\n",
    "WARNING: Attempting to start all Apache Hadoop daemons as shawak in 10 seconds.\n",
    "\n",
    "WARNING: This is not a recommended production deployment configuration.\n",
    "\n",
    "WARNING: Use CTRL-C to abort.\n",
    "\n",
    "Starting namenodes on [localhost]\n",
    "\n",
    "Starting datanodes\n",
    "\n",
    "Starting secondary namenodes [shawak]\n",
    "\n",
    "Starting resourcemanager\n",
    "\n",
    "Starting nodemanagers\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "shawak@shawak:/opt/hadoop/hadoop-3.1.2/sbin$ \n",
    "    \n",
    "jps\n",
    "\n",
    "19890 Jps\n",
    "18727 NameNode\n",
    "19177 SecondaryNameNode\n",
    "18909 DataNode\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### hadoop server is up and running\n",
    "\n",
    "\n",
    "open http://localhost:9870/ in browser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}