ICode9
精准搜索请尝试： 精确搜索
首页
编程语言>
数据库
系统相关
互联网
其他分享
Python
Java
JavaScript
android
PHP
首页 > 编程语言> 文章详细

Datax 源码打包报错与解决方法

2021-06-16 18:04:06  阅读：2197  来源： 互联网

标签：INFO SKIPPED SUCCESS 0.0 jar 源码 SNAPSHOT Datax 报错


打包Datax源码遇到的所有问题与解决方法

问题一

一、问题描述

下载Datax源码后，根据
官方文档：
https://github.com/alibaba/DataX/blob/master/userGuid.md

提供的打包命令：

mvn -U clean package assembly:assembly -Dmaven.test.skip=true

执行打包命令报错：Failed to execute goal on project hdfsreader: Could not resolve dependencies for project com.alibaba.datax:hdfsreader:jar:0.0.1-SNAPSHOT: Failed to collect dependencies

maven的setting.xml中mirror配置如下

		<mirror>
            <id>alimaven</id>
            <mirrorOf>central</mirrorOf>
            <name>aliyun maven</name>
            <url>http://maven.aliyun.com/nexus/content/repositories/central/</url>
        </mirror>
        <mirror>
            <id>repo1</id>
            <mirrorOf>central</mirrorOf>
            <name>Human Readable Name for this Mirror.</name>
            <url>http://repo1.maven.org/maven2/</url>
        </mirror>
        <mirror>
            <id>repo2</id>
            <mirrorOf>central</mirrorOf>
            <name>Human Readable Name for this Mirror.</name>
            <url>http://repo2.maven.org/maven2/</url>
        </mirror>
		<mirror>
            <id>spring-milestones</id>
            <name>Spring Milestones</name>
            <url>https://repo.spring.io/libs-milestone</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </mirror>
		<mirror>
			<id>aliyunmaven</id>
			<mirrorOf>*</mirrorOf>
			<name>阿里云spring插件仓库</name>
			<url>https://maven.aliyun.com/repository/spring-plugin</url>
		</mirror>
		<mirror>
            <id>repo3</id>
            <mirrorOf>central</mirrorOf>
            <name>Human Readable Name for this Mirror.</name>
            <url>https://maven.aliyun.com/repository/spring-plugin</url>
        </mirror>
		<mirror>
            <id>repo4</id>
			<mirrorOf>central</mirrorOf>
            <name>Spring Milestones</name>
            <url>https://repo.spring.io/plugins-release/</url>
        </mirror>

报错日志如下：

[INFO] Reactor Summary:
[INFO]
[INFO] datax-all 0.0.1-SNAPSHOT ........................... SUCCESS [  0.060 s]
[INFO] datax-common 0.0.1-SNAPSHOT ........................ SUCCESS [  1.253 s]
[INFO] datax-transformer 0.0.1-SNAPSHOT ................... SUCCESS [  2.051 s]
[INFO] datax-core 0.0.1-SNAPSHOT .......................... SUCCESS [  5.294 s]
[INFO] plugin-rdbms-util 0.0.1-SNAPSHOT ................... SUCCESS [  0.731 s]
[INFO] mysqlreader 0.0.1-SNAPSHOT ......................... SUCCESS [  2.573 s]
[INFO] drdsreader 0.0.1-SNAPSHOT .......................... SUCCESS [  2.576 s]
[INFO] sqlserverreader 0.0.1-SNAPSHOT ..................... SUCCESS [  2.478 s]
[INFO] postgresqlreader 0.0.1-SNAPSHOT .................... SUCCESS [  2.682 s]
[INFO] kingbaseesreader 0.0.1-SNAPSHOT .................... SUCCESS [  2.461 s]
[INFO] oraclereader 0.0.1-SNAPSHOT ........................ SUCCESS [  2.522 s]
[INFO] odpsreader 0.0.1-SNAPSHOT .......................... SUCCESS [  4.557 s]
[INFO] otsreader 0.0.1-SNAPSHOT ........................... SUCCESS [  4.408 s]
[INFO] otsstreamreader 0.0.1 .............................. SUCCESS [  4.277 s]
[INFO] plugin-unstructured-storage-util 0.0.1-SNAPSHOT .... SUCCESS [  0.636 s]
[INFO] txtfilereader 0.0.1-SNAPSHOT ....................... SUCCESS [ 11.641 s]
[INFO] hdfsreader 0.0.1-SNAPSHOT .......................... FAILURE [ 25.316 s]
[INFO] streamreader 0.0.1-SNAPSHOT ........................ SKIPPED
[INFO] ossreader 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] ftpreader 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] mongodbreader 0.0.1-SNAPSHOT ....................... SKIPPED
[INFO] rdbmsreader 0.0.1-SNAPSHOT ......................... SKIPPED
[INFO] hbase11xreader 0.0.1-SNAPSHOT ...................... SKIPPED
[INFO] hbase094xreader 0.0.1-SNAPSHOT ..................... SKIPPED
[INFO] tsdbreader 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] opentsdbreader 0.0.1-SNAPSHOT ...................... SKIPPED
[INFO] cassandrareader 0.0.1-SNAPSHOT ..................... SKIPPED
[INFO] gdbreader 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] oceanbasev10reader 0.0.1-SNAPSHOT .................. SKIPPED
[INFO] mysqlwriter 0.0.1-SNAPSHOT ......................... SKIPPED
[INFO] drdswriter 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] odpswriter 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] txtfilewriter 0.0.1-SNAPSHOT ....................... SKIPPED
[INFO] ftpwriter 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] hdfswriter 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] streamwriter 0.0.1-SNAPSHOT ........................ SKIPPED
[INFO] otswriter 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] oraclewriter 0.0.1-SNAPSHOT ........................ SKIPPED
[INFO] sqlserverwriter 0.0.1-SNAPSHOT ..................... SKIPPED
[INFO] postgresqlwriter 0.0.1-SNAPSHOT .................... SKIPPED
[INFO] kingbaseeswriter 0.0.1-SNAPSHOT .................... SKIPPED
[INFO] osswriter 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] mongodbwriter 0.0.1-SNAPSHOT ....................... SKIPPED
[INFO] adswriter 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] ocswriter 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] rdbmswriter 0.0.1-SNAPSHOT ......................... SKIPPED
[INFO] hbase11xwriter 0.0.1-SNAPSHOT ...................... SKIPPED
[INFO] hbase094xwriter 0.0.1-SNAPSHOT ..................... SKIPPED
[INFO] hbase11xsqlwriter 0.0.1-SNAPSHOT ................... SKIPPED
[INFO] hbase11xsqlreader 0.0.1-SNAPSHOT ................... SKIPPED
[INFO] elasticsearchwriter 0.0.1-SNAPSHOT ................. SKIPPED
[INFO] tsdbwriter 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] adbpgwriter 0.0.1-SNAPSHOT ......................... SKIPPED
[INFO] gdbwriter 0.0.1-SNAPSHOT ........................... SKIPPED
[INFO] cassandrawriter 0.0.1-SNAPSHOT ..................... SKIPPED
[INFO] clickhousewriter 0.0.1-SNAPSHOT .................... SKIPPED
[INFO] oscarwriter 0.0.1-SNAPSHOT ......................... SKIPPED
[INFO] oceanbasev10writer 0.0.1-SNAPSHOT .................. SKIPPED
[INFO] hbase20xsqlreader 0.0.1-SNAPSHOT ................... SKIPPED
[INFO] hbase20xsqlwriter 0.0.1-SNAPSHOT ................... SKIPPED
[INFO] kuduwriter 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:16 min
[INFO] Finished at: 2021-06-16T15:40:42+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project hdfsreader: Could not resolve dependencies for project com.alibaba.datax:hdfsreader:jar:0.0.1-SNAPSHOT: Failed to collect dependencies at org.apache.hive:hive-exec:jar:1.1.1 -> org.apache.calcite:calcite-core:jar:1.0.0-incubating -> org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde: Failed to read artifact descriptor for org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde: Could not transfer artifact org.
pentaho:pentaho-aggdesigner-algorithm:pom:5.1.5-jhyde from/to repo1 (http://repo1.maven.org/maven2/): Transfer failed for http://repo1.maven.org/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algori
thm-5.1.5-jhyde.pom 501 HTTPS Required -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hdfsreader

二、解决

1 google搜索报错

找到一个相关Issue，并没有解决方案
https://github.com/alibaba/DataX/issues/704

找到一个教程，让在setting.xml里加几个mirror
https://www.programmersought.com/article/25773844741/
照着加上，还是找不到jar包

2 检查本地仓库

看到底是哪个jar包没下载下来
依次检查，发现只有pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar这个jar包没下载下来
google搜索这个jar包，发现maven仓库有，手动下载提示输入用户名密码，找了一下注册登录（下载jar还需要登录真是无fuck说），没找到。

2.1 下载jar包

经过不懈的寻找，终于在另外一个提供jar包下载的网站找到这个jar。
https://javalibs.com/artifact/org.pentaho/pentaho-aggdesigner-algorithm

2.2 将下载好的jar包放到本地仓库jar包对应目录下

比如我的本地仓库是

// 本地仓库路径
D:\data\mvn_repository
// 放jar包的路径
D:\data\mvn_repository\org\pentaho\pentaho-aggdesigner-algorithm\5.1.5-jhyde

2.3 将jar包目录下除jar包之外的其他文件删除

2.4 新建pom文件

继续打开上面的url

点击Download可以看到下图所示的内容
在这里插入图片描述

点击pom，会出现如下内容

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.pentaho</groupId>
<artifactId>pentaho-aggdesigner-algorithm</artifactId>
<version>5.1.5-jhyde</version>
<description>POM was created by Sonatype Nexus</description>
</project>
  1. 新建一个名为pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom的文件
  2. 用文本编辑器打开文件
  3. 复制上面的pom内容，粘贴到文件里，保存

现在jar包所在目录应该像这样
在这里插入图片描述

继续执行打包命令
发现hdfsreader已经可以打包成功

执行到打包oscarwrite时又报了第二个错

问题二

一、问题描述

执行打包命令报错：
Failed to execute goal on project oscarwriter: Could not resolve dependencies for project com.alibaba.datax:oscarwriter:jar:0.0.1-SNAPSHOT: Could not find artifact com.oscar:oscar:jar:7.0.8

日志

[INFO] Reactor Summary:
[INFO]
[INFO] datax-all 0.0.1-SNAPSHOT ........................... SUCCESS [  0.060 s]
[INFO] datax-common 0.0.1-SNAPSHOT ........................ SUCCESS [  1.264 s]
[INFO] datax-transformer 0.0.1-SNAPSHOT ................... SUCCESS [  2.045 s]
[INFO] datax-core 0.0.1-SNAPSHOT .......................... SUCCESS [  5.495 s]
[INFO] plugin-rdbms-util 0.0.1-SNAPSHOT ................... SUCCESS [  0.720 s]
[INFO] mysqlreader 0.0.1-SNAPSHOT ......................... SUCCESS [  2.584 s]
[INFO] drdsreader 0.0.1-SNAPSHOT .......................... SUCCESS [  2.617 s]
[INFO] sqlserverreader 0.0.1-SNAPSHOT ..................... SUCCESS [  2.509 s]
[INFO] postgresqlreader 0.0.1-SNAPSHOT .................... SUCCESS [  2.730 s]
[INFO] kingbaseesreader 0.0.1-SNAPSHOT .................... SUCCESS [  2.486 s]
[INFO] oraclereader 0.0.1-SNAPSHOT ........................ SUCCESS [  2.531 s]
[INFO] odpsreader 0.0.1-SNAPSHOT .......................... SUCCESS [  4.637 s]
[INFO] otsreader 0.0.1-SNAPSHOT ........................... SUCCESS [  4.472 s]
[INFO] otsstreamreader 0.0.1 .............................. SUCCESS [  4.335 s]
[INFO] plugin-unstructured-storage-util 0.0.1-SNAPSHOT .... SUCCESS [  0.658 s]
[INFO] txtfilereader 0.0.1-SNAPSHOT ....................... SUCCESS [ 11.636 s]
[INFO] hdfsreader 0.0.1-SNAPSHOT .......................... SUCCESS [ 31.336 s]
[INFO] streamreader 0.0.1-SNAPSHOT ........................ SUCCESS [  2.017 s]
[INFO] ossreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 12.163 s]
[INFO] ftpreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 11.482 s]
[INFO] mongodbreader 0.0.1-SNAPSHOT ....................... SUCCESS [ 11.661 s]
[INFO] rdbmsreader 0.0.1-SNAPSHOT ......................... SUCCESS [  2.555 s]
[INFO] hbase11xreader 0.0.1-SNAPSHOT ...................... SUCCESS [ 12.942 s]
[INFO] hbase094xreader 0.0.1-SNAPSHOT ..................... SUCCESS [ 11.717 s]
[INFO] tsdbreader 0.0.1-SNAPSHOT .......................... SUCCESS [  2.798 s]
[INFO] opentsdbreader 0.0.1-SNAPSHOT ...................... SUCCESS [  6.044 s]
[INFO] cassandrareader 0.0.1-SNAPSHOT ..................... SUCCESS [  5.355 s]
[INFO] gdbreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 10.133 s]
[INFO] oceanbasev10reader 0.0.1-SNAPSHOT .................. SUCCESS [  2.919 s]
[INFO] mysqlwriter 0.0.1-SNAPSHOT ......................... SUCCESS [  2.545 s]
[INFO] drdswriter 0.0.1-SNAPSHOT .......................... SUCCESS [  2.533 s]
[INFO] odpswriter 0.0.1-SNAPSHOT .......................... SUCCESS [  4.515 s]
[INFO] txtfilewriter 0.0.1-SNAPSHOT ....................... SUCCESS [ 11.471 s]
[INFO] ftpwriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 11.544 s]
[INFO] hdfswriter 0.0.1-SNAPSHOT .......................... SUCCESS [ 23.885 s]
[INFO] streamwriter 0.0.1-SNAPSHOT ........................ SUCCESS [  1.850 s]
[INFO] otswriter 0.0.1-SNAPSHOT ........................... SUCCESS [  4.381 s]
[INFO] oraclewriter 0.0.1-SNAPSHOT ........................ SUCCESS [  2.531 s]
[INFO] sqlserverwriter 0.0.1-SNAPSHOT ..................... SUCCESS [  2.547 s]
[INFO] postgresqlwriter 0.0.1-SNAPSHOT .................... SUCCESS [  2.651 s]
[INFO] kingbaseeswriter 0.0.1-SNAPSHOT .................... SUCCESS [  2.500 s]
[INFO] osswriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 12.018 s]
[INFO] mongodbwriter 0.0.1-SNAPSHOT ....................... SUCCESS [ 12.132 s]
[INFO] adswriter 0.0.1-SNAPSHOT ........................... SUCCESS [  8.309 s]
[INFO] ocswriter 0.0.1-SNAPSHOT ........................... SUCCESS [  4.925 s]
[INFO] rdbmswriter 0.0.1-SNAPSHOT ......................... SUCCESS [  2.664 s]
[INFO] hbase11xwriter 0.0.1-SNAPSHOT ...................... SUCCESS [ 13.108 s]
[INFO] hbase094xwriter 0.0.1-SNAPSHOT ..................... SUCCESS [ 11.841 s]
[INFO] hbase11xsqlwriter 0.0.1-SNAPSHOT ................... SUCCESS [ 23.918 s]
[INFO] hbase11xsqlreader 0.0.1-SNAPSHOT ................... SUCCESS [ 24.070 s]
[INFO] elasticsearchwriter 0.0.1-SNAPSHOT ................. SUCCESS [  3.759 s]
[INFO] tsdbwriter 0.0.1-SNAPSHOT .......................... SUCCESS [  2.525 s]
[INFO] adbpgwriter 0.0.1-SNAPSHOT ......................... SUCCESS [  5.888 s]
[INFO] gdbwriter 0.0.1-SNAPSHOT ........................... SUCCESS [  5.877 s]
[INFO] cassandrawriter 0.0.1-SNAPSHOT ..................... SUCCESS [  5.032 s]
[INFO] clickhousewriter 0.0.1-SNAPSHOT .................... SUCCESS [  5.627 s]
[INFO] oscarwriter 0.0.1-SNAPSHOT ......................... FAILURE [  0.007 s]
[INFO] oceanbasev10writer 0.0.1-SNAPSHOT .................. SKIPPED
[INFO] hbase20xsqlreader 0.0.1-SNAPSHOT ................... SKIPPED
[INFO] hbase20xsqlwriter 0.0.1-SNAPSHOT ................... SKIPPED
[INFO] kuduwriter 0.0.1-SNAPSHOT .......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  06:29 min
[INFO] Finished at: 2021-06-16T16:44:38+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project oscarwriter: Could not resolve dependencies for project com.alibaba.datax:oscarwriter:jar:0.0.1-SNAPSHOT: Could not find artifact com.oscar:oscar:jar:7.0.8 at specified path D:\os_ws\DataX\o
scarwriter\src\main\lib\oscarJDBC.jar -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :oscarwriter

二、解决

1 google搜了一下这个jar包名字

发现一个datax的issue，下面有兄弟说如果没用到这个可以注释掉，就不会打这个包了
https://github.com/alibaba/DataX/issues/962

1.1 找到项目根目录下的pom.xml

用文本编辑器打开文件，搜索oscar，将oscar有关的注释掉

如图所示
在这里插入图片描述

1.2 找到项目根目录下的package.xml

用文本编辑器打开文件，搜索oscar，将oscar有关的注释掉

如图所示
在这里插入图片描述
重新执行打包命令
终于打包成功
在这里插入图片描述

标签：INFO,SKIPPED,SUCCESS,0.0,jar,源码,SNAPSHOT,Datax,报错
来源： https://blog.csdn.net/qq_39945938/article/details/117959051

本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享；
2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关；
3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关；
4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除；
5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。

关于我们 | 联系我们 | 留言反馈

专注分享技术，共同学习，共同进步。侵权联系[81616952@qq.com]

Copyright (C)ICode9.com, All Rights Reserved.

ICode9版权所有