Hadoop 伪分布模式安装

In |

1121621048902_.pic_hd.jpg

环境

  • CentOS 6.8 64位 1核 2GB

  • JDK 1.7.0_55 64 位

  • Hadoop 1.1.2

用户及目录配置

  • 创建 yohann 用户

使用 root 用户创建了一个 yohann 用户，并授权 sudo 命令的使用。

  • 创建 /app 目录
1
$ mkdir /app

在系统根目录下创建 /app 目录，用于存放 hadoop 等组件运行包。

  • 修改 /app 目录拥有者为 yohann
1
$ chown yohann:yohann  –R /app

防火墙配置

  • 查看防火墙状态
1
2
3
4
5
6
7
8
9
10
$ sudo service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

如下所示表示 iptables 已经开启。

  • 关闭防火墙
1
$ sudo service iptables stop

SElinux配置

  • 查看 SElinux 状态
1
2
$ getenforce
Disabled

如上所示表示 SElinux 已经关闭。

  • 修改 /etc/selinux/config 文件

将SELINUX=enforcing改为SELINUX=disabled。

Host配置

  • 设置机器名
1
2
3
$ sudo vi /etc/sysconfig/network
# HOSTNAME=VM-2-14-centos
HOSTNAME=hadoop

使用 yohann 用户登录，修改服务器机器名为hadoop。

  • 查看服务器内网IP
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ ifconfig
eth0      Link encap:Ethernet  HWaddr 52:54:00:7C:CC:3F
          inet addr:10.0.2.14  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe7c:cc3f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:768548 errors:0 dropped:0 overruns:0 frame:0
          TX packets:735773 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:97032640 (92.5 MiB)  TX bytes:122594225 (116.9 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

如上面返回，10.0.2.14 是当前服务器的内网IP。

  • 设置 host 映射文件
1
2
$ sudo vi /etc/hosts
10.0.2.14 VM-2-14-centos hadoop

配置主机名对应的IP地址。

  • 执行 ping 验证是否设置成功
1
2
3
4
$ ping hadoop
PING VM-2-14-centos (10.0.2.14) 56(84) bytes of data.
64 bytes from VM-2-14-centos (10.0.2.14): icmp_seq=1 ttl=64 time=0.017 ms
64 bytes from VM-2-14-centos (10.0.2.14): icmp_seq=2 ttl=64 time=0.021 ms

JDK安装及配置

  • 下载 JDK 64bit 安装包

访问 https://www.oracle.com/java/technologies/javase/javase7-archive-downloads.html ，找到 jdk-7u55-linux-x64.gz 进行下载。

  • 创建 /app/lib 目录
1
$ mkdir /app/lib
  • 把下载的安装包解压并移动到 /app/lib 目录下
1
$ tar -zxf jdk-7u55-linux-x64.gz && mv jdk1.7.0_55/ /app/lib
  • 修改 /etc/profile 文件，设置 JDK 路径
1
2
3
export JAVA_HOME=/app/lib/jdk1.7.0_55
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
  • 编译并验证
1
2
3
4
5
6
7
8
9
$ source /etc/profile
$ java --version
Unrecognized option: --version
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
[yohann@VM-2-14-centos lib]$ java -version
java version "1.7.0_55"
Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)
  • 更新 OpenSSL
1
$ sudo yum update openssl

SSH免密配置

  • 修改 /etc/ssh/sshd_config 文件
1
2
3
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

修改以上三项配置。

  • 重启 sshd
1
2
3
$ sudo service sshd restart
Stopping sshd:                                             [  OK  ]
Starting sshd:                                             [  OK  ]
  • 生成私钥和公钥
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/yohann/.ssh/id_rsa):
Created directory '/home/yohann/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/yohann/.ssh/id_rsa.
Your public key has been saved in /home/yohann/.ssh/id_rsa.pub.
The key fingerprint is:
e0:9b:20:a4:cf:27:74:c6:0b:ef:99:41:73:89:01:78 yohann@VM-2-14-centos
The key's randomart image is:
+--[ RSA 2048]----+
| ..              |
|. E.             |
| .. . .          |
| o . + o         |
|. + O + S        |
| + B = o         |
|  + = o          |
|   + +           |
|    +            |
+-----------------+
  • 进入 /home/yohann/.ssh 目录
1
$ cd /home/yohann/.ssh
  • 复制公钥 id_rsa.pub 命名为 authorized_keys
1
$ cp id_rsa.pub authorized_keys
  • 设置 authorized_keys 读写权限
1
$ chmod 400 authorized_keys
  • 测试 ssh 免密是否生效
1
$ ssh hadoop

Hadoop配置

  • 下载 hadoop 安装包到 /app/lib 目录
1
$ wget -P /app/lib https://archive.apache.org/dist/hadoop/core/hadoop-1.1.2/hadoop-1.1.2-bin.tar.gz
  • 进入 /app/lib 目录
1
$ cd /app/lib
  • 解压 hadoop 安装包
1
$ tar -xzf hadoop-1.1.2-bin.tar.gz
  • 移动 hadoop-1.1.2 到上一级目录
1
$ mv hadoop-1.1.2 ../
  • 在 hadoop-1.1.2 目录下创建子目录
1
2
$ cd /app/hadoop-1.1.2
$ mkdir -p tmp hdfs hdfs/name hdfs/data
  • 修改 hdfs/data 目录的权限
1
$ chmod -R 755 hdfs/data
  • 修改 /app/hadoop-1.1.2/conf/hadoop-env.sh
1
2
export JAVA_HOME=/app/lib/jdk1.7.0_55
export PATH=$PATH:/app/hadoop-1.1.2/bin

设置 hadoop 中 jdk 和 hadoop/bin 路径。

  • 编译并验证
1
2
3
4
5
6
$ source /app/hadoop-1.1.2/conf/hadoop-env.sh
$ hadoop version
Hadoop 1.1.2
Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782
Compiled by hortonfo on Thu Jan 31 02:03:24 UTC 2013
From source with checksum c720ddcf4b926991de7467d253a79b8b
  • 进入 /app/hadoop-1.1.2/conf/ 目录
1
$ cd /app/hadoop-1.1.2/conf/
  • 修改 core-site.xml 文件
1
2
3
4
5
6
7
8
9
10
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://hadoop:9000</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/app/hadoop-1.1.2/tmp</value>
  </property>
</configuration>
  • 修改 hdfs-site.xml 文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.name.dir</name>
    <value>/app/hadoop-1.1.2/hdfs/name</value>
  </property>
  <property>
    <name>dfs.data.dir</name>
    <value>/app/hadoop-1.1.2/hdfs/data</value>
  </property>
</configuration>
  • 修改 mapred-site.xml 文件
1
2
3
4
5
6
<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>hadoop:9001</value>
  </property>
</configuration>
  • 配置主从节点
1
2
$ echo 'hadoop' > masters
$ echo 'hadoop' > slaves
  • 格式化 namenode
1
2
$ cd /app/hadoop-1.1.2/bin
$ ./hadoop namenode -format

启动Hadoop

  • 在 /app/hadoop-1.1.2/bin 目录下启动
1
2
3
4
5
6
$ ./start-all.sh
starting namenode, logging to /app/hadoop-1.1.2/libexec/../logs/hadoop-yohann-namenode-VM-2-14-centos.out
hadoop: starting datanode, logging to /app/hadoop-1.1.2/libexec/../logs/hadoop-yohann-datanode-VM-2-14-centos.out
hadoop: starting secondarynamenode, logging to /app/hadoop-1.1.2/libexec/../logs/hadoop-yohann-secondarynamenode-VM-2-14-centos.out
starting jobtracker, logging to /app/hadoop-1.1.2/libexec/../logs/hadoop-yohann-jobtracker-VM-2-14-centos.out
hadoop: starting tasktracker, logging to /app/hadoop-1.1.2/libexec/../logs/hadoop-yohann-tasktracker-VM-2-14-centos.out
  • 查看启动进程
1
2
3
4
5
6
7
$ jps
27841 TaskTracker
27897 Jps
27416 NameNode
27522 DataNode
27735 JobTracker
27650 SecondaryNameNode

确保以上进程都存在。

-------------本文结束感谢您的阅读-------------
0%