Hadoop 搭建Sqoop环境

环境

• CentOS 6.8 64位 1核 2GB

• JDK 1.7.0_75 64 位

• Hadoop 1.1.2

• Sqoop 1.4.5

安装Sqoop

• 下载 sqoop 安装包

$ wget http://archive.apache.org/dist/sqoop/1.4.5/sqoop-1.4.5.bin__hadoop-1.0.0.tar.gz

• 解压 sqoop 安装包

$ tar -xzf sqoop-1.4.5.bin__hadoop-1.0.0.tar.gz
$ mv sqoop-1.4.5.bin__hadoop-1.0.0 /app/sqoop-1.4.5

解压后，将 sqoop-1.4.5.bin__hadoop-1.0.0 移动到 /app 目录下。

• 修改 /etc/profile 文件

$ sudo vi /etc/profile

• 添加 sqoop 环境变量

export SQOOP_HOME=/app/sqoop-1.4.5
export PATH=$PATH:$SQOOP_HOME/bin

• 保存生效

$ source /etc/profile

• 验证安装成功

$ sqoop version
Warning: $HADOOP_HOME is deprecated.

21/07/03 21:01:56 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5
Sqoop 1.4.5
git commit id 5b34accaca7de251fc91161733f906af2eddbe83
Compiled by abe on Fri Aug  1 11:15:29 PDT 2014

配置Sqoop

• 将 mysql-connector-java-5.1.22-bin.jar 复制到 /app/sqoop-1.4.5/lib 目录下。

$ cp /home/yohann/mysql-connector-java-5.1.22-bin.jar /app/sqoop-1.4.5/lib/

• 进入 /app/sqoop-1.4.5/bin 目录

$ cd /app/sqoop-1.4.5/bin

• 修改 configure-sqoop 文件

# if [ ! -d "${HBASE_HOME}" ]; then
#   echo "Warning: $HBASE_HOME does not exist! HBase imports will fail."
#   echo 'Please set $HBASE_HOME to the root of your HBase installation.'
# fi

##  Moved to be a runtime check in sqoop.
# if [ ! -d "${HCAT_HOME}" ]; then
#   echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
#   echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
# fi

# if [ ! -d "${ACCUMULO_HOME}" ]; then
#   echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
#   echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
# fi
# if [ ! -d "${ZOOKEEPER_HOME}" ]; then
#   echo "Warning: $ZOOKEEPER_HOME does not exist! Accumulo imports will fail."
#   echo 'Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.'
# fi

注释掉 HBase 和 Zookeeper 等检查。

• 进入 /app/sqoop-1.4.5/conf 目录

$ cd /app/sqoop-1.4.5/conf

• 复制 sqoop-env-template.sh 命名为 sqoop-env.sh

$ cp sqoop-env-template.sh sqoop-env.sh

• 修改 sqoop-env.sh 文件

export HADOOP_COMMON_HOME=/app/hadoop-1.1.2
export HADOOP_MAPRED_HOME=/app/hadoop-1.1.2

设置 hadoop 运行程序所在路径。

• 保存生效

$ source /etc/profile

• 验证配置完成

sqoop help