Datax 源码打包报错与解决方法

2021-10-06 13:25:23

打包Datax源码遇到的所有问题与解决方法

问题一

一、问题描述

下载Datax源码后，根据
官方文档：
https://github.com/alibaba/DataX/blob/master/userGuid.md

提供的打包命令：

mvn -U clean package assembly:assembly -Dmaven.test.skip=true

执行打包命令报错：Failed to execute goal on project hdfsreader: Could not resolve dependencies for project com.alibaba.datax:hdfsreader:jar:0.0.1-SNAPSHOT: Failed to collect dependencies

maven的setting.xml中mirror配置如下

		<mirror>
            <id>alimaven</id>
            <mirrorOf>central</mirrorOf>
            <name>aliyun maven</name>
            <url>http://maven.aliyun.com/nexus/content/repositories/central/</url>
        </mirror>
        <mirror>
            <id>repo1</id>
            <mirrorOf>central</mirrorOf>
            <name>Human Readable Name for this Mirror.</name>
            <url>http://repo1.maven.org/maven2/</url>
        </mirror>
        <mirror>
            <id>repo2</id>
            <mirrorOf>central</mirrorOf>
            <name>Human Readable Name for this Mirror.</name>
            <url>http://repo2.maven.org/maven2/</url>
        </mirror>
		<mirror>
            <id>spring-milestones</id>
            <name>Spring Milestones</name>
            <url>https://repo.spring.io/libs-milestone</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </mirror>
		<mirror>
			<id>aliyunmaven</id>
			<mirrorOf>*</mirrorOf>
			<name>阿里云spring插件仓库</name>
			<url>https://maven.aliyun.com/repository/spring-plugin</url>
		</mirror>
		<mirror>
            <id>repo3</id>
            <mirrorOf>central</mirrorOf>
            <name>Human Readable Name for this Mirror.</name>
            <url>https://maven.aliyun.com/repository/spring-plugin</url>
        </mirror>
		<mirror>
            <id>repo4</id>
			<mirrorOf>central</mirrorOf>
            <name>Spring Milestones</name>
            <url>https://repo.spring.io/plugins-release/</url>
        </mirror>

报错日志如下：

[INFO] Reactor Summary:
[INFO]
[INFO] datax-all 0.0.1-SNAPSHOT ........................... SUCCESS [  0.060 s]
[INFO] datax-common 0.0.1-SNAPSHOT ........................ SUCCESS [  1.253 s]
[INFO] datax-transformer 0.0.1-SNAPSHOT ................... SUCCESS [  2.051 s]
[INFO] datax-core 0.0.1-SNAPSHOT .......................... SUCCESS [  5.294 s]
[INFO] plugin-rdbms-util 0.0.1-SNAPSHOT ................... SUCCESS [  0.731 s]
[INFO] mysqlreader 0.0.1-SNAPSHOT ......................... SUCCESS [  2.573 s]
[INFO] drdsreader 0.0.1-SNAPSHOT .......................... SUCCESS [  2.576 s]
[INFO] sqlserverreader 0.0.1-SNAPSHOT ..................... SUCCESS [  2.478 s]
[INFO] postgresqlreader 0.0.1-SNAPSHOT .................... SUCCESS [  2.682 s]
[INFO] kingbaseesreader 0.0.1-SNAPSHOT .................... SUCCESS [  2.461 s]
[INFO] oraclereader 0.0.1-SNAPSHOT ........................ SUCCESS [  2.522 s]
[INFO] odpsreader 0.0.1-SNAPSHOT .......................... SUCCESS [  4.557 s]
[INFO] otsreader 0.0.1-SNAPSHOT ........................... SUCCESS [  4.408 s]
[INFO] otsstreamreader 0.0.1 .............................. SUCCESS [  4.277 s]
[INFO] plugin-unstructured-storage-util 0.0.1-SNAPSHOT .... SUCCESS [  0.636 s]
[INFO] txtfilereader 0.0.1-SNAPSHOT ....................... SUCCESS [ 11.641 s]
[INFO] hdfsreader 0.0.1-SNAPSHOT .......................... FAILURE [ 25.316 s]
[INFO] streamreader 0.0.1-SNAPSHOT ........................ SKIPPED
[INFO] ossreader 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] ftpreader 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] mongodbreader 0.0.1-SNAPSHOT ....................... SKIPPED
[INFO] rdbmsreader 0.0.1-SNAPSHOT ......................... SKIPPED
[INFO] hbase11xreader 0.0.1-SNAPSHOT ...................... SKIPPED
[INFO] hbase094xreader 0.0.1-SNAPSHOT ..................... SKIPPED
[INFO] tsdbreader 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] opentsdbreader 0.0.1-SNAPSHOT ...................... SKIPPED
[INFO] cassandrareader 0.0.1-SNAPSHOT ..................... SKIPPED
[INFO] gdbreader 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] oceanbasev10reader 0.0.1-SNAPSHOT .................. SKIPPED
[INFO] mysqlwriter 0.0.1-SNAPSHOT ......................... SKIPPED
[INFO] drdswriter 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] odpswriter 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] txtfilewriter 0.0.1-SNAPSHOT ....................... SKIPPED
[INFO] ftpwriter 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] hdfswriter 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] streamwriter 0.0.1-SNAPSHOT ........................ SKIPPED
[INFO] otswriter 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] oraclewriter 0.0.1-SNAPSHOT ........................ SKIPPED
[INFO] sqlserverwriter 0.0.1-SNAPSHOT ..................... SKIPPED
[INFO] postgresqlwriter 0.0.1-SNAPSHOT .................... SKIPPED
[INFO] kingbaseeswriter 0.0.1-SNAPSHOT .................... SKIPPED
[INFO] osswriter 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] mongodbwriter 0.0.1-SNAPSHOT ....................... SKIPPED
[INFO] adswriter 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] ocswriter 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] rdbmswriter 0.0.1-SNAPSHOT ......................... SKIPPED
[INFO] hbase11xwriter 0.0.1-SNAPSHOT ...................... SKIPPED
[INFO] hbase094xwriter 0.0.1-SNAPSHOT ..................... SKIPPED
[INFO] hbase11xsqlwriter 0.0.1-SNAPSHOT ................... SKIPPED
[INFO] hbase11xsqlreader 0.0.1-SNAPSHOT ................... SKIPPED
[INFO] elasticsearchwriter 0.0.1-SNAPSHOT ................. SKIPPED
[INFO] tsdbwriter 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] adbpgwriter 0.0.1-SNAPSHOT ......................... SKIPPED
[INFO] gdbwriter 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] cassandrawriter 0.0.1-SNAPSHOT ..................... SKIPPED
[INFO] clickhousewriter 0.0.1-SNAPSHOT .................... SKIPPED
[INFO] oscarwriter 0.0.1-SNAPSHOT ......................... SKIPPED
[INFO] oceanbasev10writer 0.0.1-SNAPSHOT .................. SKIPPED
[INFO] hbase20xsqlreader 0.0.1-SNAPSHOT ................... SKIPPED
[INFO] hbase20xsqlwriter 0.0.1-SNAPSHOT ................... SKIPPED
[INFO] kuduwriter 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:16 min
[INFO] Finished at: 2021-06-16T15:40:42+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project hdfsreader: Could not resolve dependencies for project com.alibaba.datax:hdfsreader:jar:0.0.1-SNAPSHOT: Failed to collect dependencies at org.apache.hive:hive-exec:jar:1.1.1 -> org.apache.calcite:calcite-core:jar:1.0.0-incubating -> org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde: Failed to read artifact descriptor for org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde: Could not transfer artifact org.
pentaho:pentaho-aggdesigner-algorithm:pom:5.1.5-jhyde from/to repo1 (http://repo1.maven.org/maven2/): Transfer failed for http://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algori
thm-5.1.5-jhyde.pom 501 HTTPS Required -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hdfsreader

二、解决

1 google搜索报错

找到一个相关Issue，并没有解决方案
https://github.com/alibaba/DataX/issues/704

找到一个教程，让在setting.xml里加几个mirror
https://www.programmersought.com/article/25773844741/
照着加上，还是找不到jar包

2 检查本地仓库

看到底是哪个jar包没下载下来
依次检查，发现只有pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar这个jar包没下载下来
google搜索这个jar包，发现maven仓库有，手动下载提示输入用户名密码，找了一下注册登录（下载jar还需要登录真是无fuck说），没找到。

2.1 下载jar包

经过不懈的寻找，终于在另外一个提供jar包下载的网站找到这个jar。
https://javalibs.com/artifact/org.pentaho/pentaho-aggdesigner-algorithm

2.2 将下载好的jar包放到本地仓库jar包对应目录下

比如我的本地仓库是

// 本地仓库路径
D:\data\mvn_repository
// 放jar包的路径
D:\data\mvn_repository\org\pentaho\pentaho-aggdesigner-algorithm\5.1.5-jhyde

2.3 将jar包目录下除jar包之外的其他文件删除

2.4 新建pom文件

继续打开上面的url

点击Download可以看到下图所示的内容
Datax 源码打包报错与解决方法

点击pom，会出现如下内容

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.pentaho</groupId>
<artifactId>pentaho-aggdesigner-algorithm</artifactId>
<version>5.1.5-jhyde</version>
<description>POM was created by Sonatype Nexus</description>
</project>
  1. 新建一个名为pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom的文件
  2. 用文本编辑器打开文件
  3. 复制上面的pom内容，粘贴到文件里，保存

现在jar包所在目录应该像这样
Datax 源码打包报错与解决方法

继续执行打包命令
发现hdfsreader已经可以打包成功

执行到打包oscarwrite时又报了第二个错

问题二

一、问题描述

执行打包命令报错：
Failed to execute goal on project oscarwriter: Could not resolve dependencies for project com.alibaba.datax:oscarwriter:jar:0.0.1-SNAPSHOT: Could not find artifact com.oscar:oscar:jar:7.0.8

日志

[INFO] Reactor Summary:
[INFO]
[INFO] datax-all 0.0.1-SNAPSHOT ........................... SUCCESS [  0.060 s]
[INFO] datax-common 0.0.1-SNAPSHOT ........................ SUCCESS [  1.264 s]
[INFO] datax-transformer 0.0.1-SNAPSHOT ................... SUCCESS [  2.045 s]
[INFO] datax-core 0.0.1-SNAPSHOT .......................... SUCCESS [  5.495 s]
[INFO] plugin-rdbms-util 0.0.1-SNAPSHOT ................... SUCCESS [  0.720 s]
[INFO] mysqlreader 0.0.1-SNAPSHOT ......................... SUCCESS [  2.584 s]
[INFO] drdsreader 0.0.1-SNAPSHOT .......................... SUCCESS [  2.617 s]
[INFO] sqlserverreader 0.0.1-SNAPSHOT ..................... SUCCESS [  2.509 s]
[INFO] postgresqlreader 0.0.1-SNAPSHOT .................... SUCCESS [  2.730 s]
[INFO] kingbaseesreader 0.0.1-SNAPSHOT .................... SUCCESS [  2.486 s]
[INFO] oraclereader 0.0.1-SNAPSHOT ........................ SUCCESS [  2.531 s]
[INFO] odpsreader 0.0.1-SNAPSHOT .......................... SUCCESS [  4.637 s]
[INFO] otsreader 0.0.1-SNAPSHOT ........................... SUCCESS [  4.472 s]
[INFO] otsstreamreader 0.0.1 .............................. SUCCESS [  4.335 s]
[INFO] plugin-unstructured-storage-util 0.0.1-SNAPSHOT .... SUCCESS [  0.658 s]
[INFO] txtfilereader 0.0.1-SNAPSHOT ....................... SUCCESS [ 11.636 s]
[INFO] hdfsreader 0.0.1-SNAPSHOT .......................... SUCCESS [ 31.336 s]
[INFO] streamreader 0.0.1-SNAPSHOT ........................ SUCCESS [  2.017 s]
[INFO] ossreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 12.163 s]
[INFO] ftpreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 11.482 s]
[INFO] mongodbreader 0.0.1-SNAPSHOT ....................... SUCCESS [ 11.661 s]
[INFO] rdbmsreader 0.0.1-SNAPSHOT ......................... SUCCESS [  2.555 s]
[INFO] hbase11xreader 0.0.1-SNAPSHOT ...................... SUCCESS [ 12.942 s]
[INFO] hbase094xreader 0.0.1-SNAPSHOT ..................... SUCCESS [ 11.717 s]
[INFO] tsdbreader 0.0.1-SNAPSHOT .......................... SUCCESS [  2.798 s]
[INFO] opentsdbreader 0.0.1-SNAPSHOT ...................... SUCCESS [  6.044 s]
[INFO] cassandrareader 0.0.1-SNAPSHOT ..................... SUCCESS [  5.355 s]
[INFO] gdbreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 10.133 s]
[INFO] oceanbasev10reader 0.0.1-SNAPSHOT .................. SUCCESS [  2.919 s]
[INFO] mysqlwriter 0.0.1-SNAPSHOT ......................... SUCCESS [  2.545 s]
[INFO] drdswriter 0.0.1-SNAPSHOT .......................... SUCCESS [  2.533 s]
[INFO] odpswriter 0.0.1-SNAPSHOT .......................... SUCCESS [  4.515 s]
[INFO] txtfilewriter 0.0.1-SNAPSHOT ....................... SUCCESS [ 11.471 s]
[INFO] ftpwriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 11.544 s]
[INFO] hdfswriter 0.0.1-SNAPSHOT .......................... SUCCESS [ 23.885 s]
[INFO] streamwriter 0.0.1-SNAPSHOT ........................ SUCCESS [  1.850 s]
[INFO] otswriter 0.0.1-SNAPSHOT ........................... SUCCESS [  4.381 s]
[INFO] oraclewriter 0.0.1-SNAPSHOT ........................ SUCCESS [  2.531 s]
[INFO] sqlserverwriter 0.0.1-SNAPSHOT ..................... SUCCESS [  2.547 s]
[INFO] postgresqlwriter 0.0.1-SNAPSHOT .................... SUCCESS [  2.651 s]
[INFO] kingbaseeswriter 0.0.1-SNAPSHOT .................... SUCCESS [  2.500 s]
[INFO] osswriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 12.018 s]
[INFO] mongodbwriter 0.0.1-SNAPSHOT ....................... SUCCESS [ 12.132 s]
[INFO] adswriter 0.0.1-SNAPSHOT ........................... SUCCESS [  8.309 s]
[INFO] ocswriter 0.0.1-SNAPSHOT ........................... SUCCESS [  4.925 s]
[INFO] rdbmswriter 0.0.1-SNAPSHOT ......................... SUCCESS [  2.664 s]
[INFO] hbase11xwriter 0.0.1-SNAPSHOT ...................... SUCCESS [ 13.108 s]
[INFO] hbase094xwriter 0.0.1-SNAPSHOT ..................... SUCCESS [ 11.841 s]
[INFO] hbase11xsqlwriter 0.0.1-SNAPSHOT ................... SUCCESS [ 23.918 s]
[INFO] hbase11xsqlreader 0.0.1-SNAPSHOT ................... SUCCESS [ 24.070 s]
[INFO] elasticsearchwriter 0.0.1-SNAPSHOT ................. SUCCESS [  3.759 s]
[INFO] tsdbwriter 0.0.1-SNAPSHOT .......................... SUCCESS [  2.525 s]
[INFO] adbpgwriter 0.0.1-SNAPSHOT ......................... SUCCESS [  5.888 s]
[INFO] gdbwriter 0.0.1-SNAPSHOT ........................... SUCCESS [  5.877 s]
[INFO] cassandrawriter 0.0.1-SNAPSHOT ..................... SUCCESS [  5.032 s]
[INFO] clickhousewriter 0.0.1-SNAPSHOT .................... SUCCESS [  5.627 s]
[INFO] oscarwriter 0.0.1-SNAPSHOT ......................... FAILURE [  0.007 s]
[INFO] oceanbasev10writer 0.0.1-SNAPSHOT .................. SKIPPED
[INFO] hbase20xsqlreader 0.0.1-SNAPSHOT ................... SKIPPED
[INFO] hbase20xsqlwriter 0.0.1-SNAPSHOT ................... SKIPPED
[INFO] kuduwriter 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  06:29 min
[INFO] Finished at: 2021-06-16T16:44:38+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project oscarwriter: Could not resolve dependencies for project com.alibaba.datax:oscarwriter:jar:0.0.1-SNAPSHOT: Could not find artifact com.oscar:oscar:jar:7.0.8 at specified path D:\os_ws\DataX\o
scarwriter\src\main\lib\oscarJDBC.jar -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :oscarwriter

二、解决

1 google搜了一下这个jar包名字

发现一个datax的issue，下面有兄弟说如果没用到这个可以注释掉，就不会打这个包了
https://github.com/alibaba/DataX/issues/962

1.1 找到项目根目录下的pom.xml

用文本编辑器打开文件，搜索oscar，将oscar有关的注释掉

如图所示
Datax 源码打包报错与解决方法

1.2 找到项目根目录下的package.xml

用文本编辑器打开文件，搜索oscar，将oscar有关的注释掉

如图所示
Datax 源码打包报错与解决方法
重新执行打包命令
终于打包成功
Datax 源码打包报错与解决方法

上一篇：Saltstack 介绍、安装、配置语法（一）


下一篇：glusterfs 快照