环境下载
首先将下载的 sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz放到 /usr/hadoop/目录下(该目录可以自定义,一般为Hadoop集群安装目录),然后对安装包解压、修改文件名和修改用户权限。
[hadoop@centpy /]$ cd /usr/hadoop/
[hadoop@centpy hadoop]$ ls
flume hadoop-2.6.
[hadoop@centpy hadoop]$ rz //上传安装包 [hadoop@centpy hadoop]$ ls
flume hadoop-2.6. sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
[hadoop@centpy hadoop]$ tar -zxf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz //解压
[hadoop@centpy hadoop]$ mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha sqoop //修改安装文件目录
[hadoop@centpy hadoop]$ ls
flume hadoop-2.6. sqoop sqoop-1.4..bin__hadoop-2.0.-alpha.tar.gz
[hadoop@centpy hadoop]$ rm -f sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz //删除安装包
[hadoop@centpy hadoop]$ ls
flume hadoop-2.6. sqoop
环境配置
1.配置hadoop版本号
切换到/sqoop/conf 目录下,执行以下命令修改对应Hadoop版本号为自己所安装的集群的版本号。
[hadoop@centpy hadoop]$ cd sqoop/
[hadoop@centpy sqoop]$ ls
bin COMPILING.txt ivy LICENSE.txt README.txt sqoop-test-1.4..jar
build.xml conf ivy.xml NOTICE.txt sqoop-1.4..jar src
CHANGELOG.txt docs lib pom-old.xml sqoop-patch-review.py testdata
[hadoop@centpy sqoop]$ vi build.xml
修改为:
2.配置sqoop-env.sh文件
执行以下命令生成sqoop-env.sh配置文件。
[hadoop@centpy sqoop]$ cd conf/
[hadoop@centpy conf]$ ls
oraoop-site-template.xml sqoop-env-template.sh sqoop-site.xml
sqoop-env-template.cmd sqoop-site-template.xml
[hadoop@centpy conf]$ mv sqoop-env-template.sh sqoop-env.sh
然后使用 vi sqoop-env.sh 命令,打开文件添加如下内容。
如果数据读取不涉及hbase和hive,那么相关hbase和hive的配置可以不加;如果集群有独立的zookeeper集群,那么配置zookeeper,反之,不用配置。
3.配置环境变量
[hadoop@centpy conf]$ su
Password:
[root@centpy conf]# vim /etc/profile //追加以下内容
#sqoop
export SQOOP_HOME=/usr/hadoop/sqoop
export PATH=$PATH:$SQOOP_HOME/bin
[root@centpy conf]# source /etc/profile
4.上传jar包
将相关的驱动 jar 包拷贝到 sqoop/lib 目录下。数据库驱动 jar 包需要导入,这里我们使用的是 mysql 数据库,所以需要导入mysql-connector-java-5.1.32.jar包。
5.确定用户组
需要确定sqoop所属用户组是否为hadoop
[hadoop@centpy hadoop]$ ll
total
drwxrwxr-x. hadoop hadoop Jun : flume
drwxr-xr-x. hadoop hadoop Jun : hadoop-2.6.
drwxr-xr-x. hadoop hadoop Jun : sqoop
如果不是请通过以下语句修改:
chown –R hadoop:hadoop sqoop
测试运行
//首先启动hadoop集群
[hadoop@centpy sqoop]$ cd ../hadoop-2.6.0/
[hadoop@centpy hadoop-2.6.]$ sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [centpy]
centpy: starting namenode, logging to /usr/hadoop/hadoop-2.6./logs/hadoop-hadoop-namenode-centpy.out
centpy: starting datanode, logging to /usr/hadoop/hadoop-2.6./logs/hadoop-hadoop-datanode-centpy.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/hadoop/hadoop-2.6./logs/hadoop-hadoop-secondarynamenode-centpy.out
starting yarn daemons
starting resourcemanager, logging to /usr/hadoop/hadoop-2.6./logs/yarn-hadoop-resourcemanager-centpy.out
centpy: starting nodemanager, logging to /usr/hadoop/hadoop-2.6./logs/yarn-hadoop-nodemanager-centpy.out
[hadoop@centpy hadoop-2.6.]$ jps
SecondaryNameNode
Jps
ResourceManager
NameNode
NodeManager
DataNode
//测试sqoop
[hadoop@centpy hadoop-2.6.]$ cd ../sqoop/
[hadoop@centpy sqoop]$ sqoop list-databases --connect jdbc:mysql://202.193.60.117/dataweb --username root --password 20134997
Warning: /usr/hadoop/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/hadoop/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/hadoop/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/hadoop/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.
// :: WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
// :: INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
dataweb
mysql
performance_schema
test
sqoop 命令执行成功,代表安装成功。
sqoop list-databases --connect jdbc:mysql://202.193.60.117/dataweb --username root --password 20134997
该语句中使用我们上传的mysql-connector-java-5.1.32.jar包链接本地数据库,其中ip地址可以通过命令行中输入ipconfig进行查看。
执行该命令之前要先启动数据库服务,若出现2003错误时因为mysql服务没有成功启动,解决方案如下:
选中后右键开启服务即可。
到此,sqoop的安装和配置就结束了!
以上就是博主为大家介绍的这一板块的主要内容,这都是博主自己的学习过程,希望能给大家带来一定的指导作用,有用的还望大家点个支持,如果对你没用也望包涵,有错误烦请指出。如有期待可关注博主以第一时间获取更新哦,谢谢!