ICode9
精准搜索请尝试： 精确搜索
首页
编程语言>
数据库
系统相关
互联网
其他分享
Python
Java
JavaScript
android
PHP
首页 > 其他分享> 文章详细

DataX的ftpreader读插件和ftpwriter写插件的中文目录乱码或无法正常同步数据问题

2022-06-28 16:06:03  阅读：158  来源： 互联网

标签：core 插件 java ftpwriter springframework 乱码 apache org ApplicationFilterChain


DataX的ftpreader读插件和ftpwriter写插件的path为中文目录时，可能会出现乱码或无法正常同步数据问题，如报以下错误：

 

1、ftpreader读插件报错：

com.alibaba.datax.common.exception.DataXException: Code:[FtpReader-04], Description:[您配置的目录文件路径不存在或者没有权限读取.]. - 请确认您的配置项path:[/新建文件夹/datasource_202206271639.csv]存在，且配置的用户有权限读取
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
at com.shsnc.dbtdemo.data.reader.ftpreader.StandardFtpHelper.getListFiles(StandardFtpHelper.java:175)
at com.shsnc.dbtdemo.data.reader.ftpreader.FtpHelper.getAllFiles(FtpHelper.java:101)
at com.shsnc.dbtdemo.data.reader.ftpreader.FtpReader$Job.prepare(FtpReader.java:108)
at com.shsnc.dbtdemo.data.core.job.JobContainer.prepareJobReader(JobContainer.java:746)
at com.shsnc.dbtdemo.data.core.job.JobContainer.prepare(JobContainer.java:310)
at com.shsnc.dbtdemo.data.core.job.JobContainer.start(JobContainer.java:116)
at com.shsnc.dbtdemo.data.core.Engine.start(Engine.java:92)
at com.shsnc.dbtdemo.data.core.Engine.entry(Engine.java:171)
at com.shsnc.dbtdemo.service.impl.ITransferServiceImpl.lambda$executeJob$0(ITransferServiceImpl.java:58)
at java.util.ArrayList.forEach(ArrayList.java:1249)
at com.shsnc.dbtdemo.service.impl.ITransferServiceImpl.executeJob(ITransferServiceImpl.java:54)
at com.shsnc.dbtdemo.controller.TransferController.executeDatax(TransferController.java:473)
at com.shsnc.dbtdemo.controller.TransferController.transferFtpOrSftpToSame(TransferController.java:404)
at com.shsnc.dbtdemo.controller.TransferController$$FastClassBySpringCGLIB$$1294915c.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:371)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:134)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at com.shsnc.dbtdemo.controller.TransferController$$EnhancerBySpringCGLIB$$a4c0126.transferFtpOrSftpToSame(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:893)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:807)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1061)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:961)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

 

解决办法：

找到ftpreader读插件的源码的StandardFtpHelper类，找到以下方法：

isDirExist，
isFileExist，
isSymbolicLink，
getListFiles，
getInputStream

将方法里面的这句代码new String(filePath.getBytes(),FTP.DEFAULT_CONTROL_ENCODING)，

加入“GBK”，即new String(filePath.getBytes("GBK"),FTP.DEFAULT_CONTROL_ENCODING)，测试通过。

 

2、ftpwriter写插件报错：

com.alibaba.datax.common.exception.DataXException: Code:[FtpWriter-14], Description:[与ftp服务器连接异常.]. - 创建目录:/新建文件夹/新建文件夹/时发生异常,请确认与ftp服务器的连接正常,拥有目录创建权限
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
at com.shsnc.dbtdemo.data.writer.ftpwriter.util.StandardFtpHelperImpl.mkDirRecursive(StandardFtpHelperImpl.java:151)
at com.shsnc.dbtdemo.data.writer.ftpwriter.FtpWriter$Job.prepare(FtpWriter.java:111)
at com.shsnc.dbtdemo.data.core.job.JobContainer.prepareJobWriter(JobContainer.java:755)
at com.shsnc.dbtdemo.data.core.job.JobContainer.prepare(JobContainer.java:311)
at com.shsnc.dbtdemo.data.core.job.JobContainer.start(JobContainer.java:116)
at com.shsnc.dbtdemo.data.core.Engine.start(Engine.java:92)
at com.shsnc.dbtdemo.data.core.Engine.entry(Engine.java:171)
at com.shsnc.dbtdemo.service.impl.ITransferServiceImpl.lambda$executeJob$0(ITransferServiceImpl.java:58)
at java.util.ArrayList.forEach(ArrayList.java:1249)
at com.shsnc.dbtdemo.service.impl.ITransferServiceImpl.executeJob(ITransferServiceImpl.java:54)
at com.shsnc.dbtdemo.controller.TransferController.executeDatax(TransferController.java:473)
at com.shsnc.dbtdemo.controller.TransferController.transferFtpOrSftpToSame(TransferController.java:404)
at com.shsnc.dbtdemo.controller.TransferController$$FastClassBySpringCGLIB$$1294915c.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:371)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:134)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at com.shsnc.dbtdemo.controller.TransferController$$EnhancerBySpringCGLIB$$32ef374.transferFtpOrSftpToSame(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:893)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:807)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1061)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:961)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

 

解决方法：

 

1、首先创建设置编码方法

/**
 * datax的ftp写插件的目录中文乱码问题，或无法同步中文目录数据问题，获取utf-8的编码格式处理 中文路径乱码问题
 * 写插件的StandardFtpHelperImpl的mkDirSingleHierarchy和getOutputStream方法调用***
 *
 * @param filePath
 * @param ftpClient
 * @return
 * @throws IOException
 */
public static String getDataxFtpWriterUtf8(String filePath, FTPClient ftpClient) {
    String LOCAL_CHARSET = null;
    try {
        // 开启服务器对UTF-8的支持，如果服务器支持就用UTF-8编码，否则就使用本地编码（GBK）.
        if (FTPReply.isPositiveCompletion(ftpClient.sendCommand("OPTS UTF8", "ON"))) {
            LOCAL_CHARSET = "UTF-8";
        }
        ftpClient.setControlEncoding(LOCAL_CHARSET);
        ftpClient.setFileType(FTPClient.BINARY_FILE_TYPE);
        ftpClient.enterLocalPassiveMode();
        String newFtpPath = new String(filePath.getBytes(LOCAL_CHARSET), "UTF-8");
        return newFtpPath;
    } catch (Exception e) {
        e.printStackTrace();
    }
    return null;
}


2、找到ftpwriter写插件的源码的StandardFtpHelperImpl类

（1）、找到mkDirSingleHierarchy方法，在方法里面的第一行调用上面创建的方法，即加入这句代码：
 directoryPath = FtpUtil.getDataxFtpWriterUtf8(directoryPath, ftpClient);//中文乱码问题，或无法同步中文目录数据问题处理 klp 20220628


（2）、找到getOutputStream方法，在方法里面的第一行调用上面创建的方法，即加入这句代码：



filePath = FtpUtil.getDataxFtpWriterUtf8(filePath, ftpClient); //中文乱码问题，或无法同步中文目录数据问题处理 klp 20220628


以上，通过测试均能正常同步数据了。




 


 
 


标签：core,插件,java,ftpwriter,springframework,乱码,apache,org,ApplicationFilterChain	

来源： https://www.cnblogs.com/haidaogege/p/16419769.html 




本站声明：
1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享；

2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关；

3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关；

4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除；

5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。


	

     

      

      

      

      

     

     

    

     
    
      
  • 
       
    


    • 
      
  •  
    • 
      
  • 
       
    


    • 
      
  • 
       
    
 
       
    • 
     

     
    
      
  • 
       
    
       
    • 
      
  • 
       
    
       
    • 
     

    

   

  

  

     

    

     关于我们
     |
     联系我们
     |
     留言反馈
    

    

     
专注分享技术，共同学习，共同进步。侵权联系[81616952@qq.com]

     
Copyright (C)ICode9.com, All Rights Reserved.

     
ICode9版权所有