标签：25 Flink java 部署 flink yarn apache org
目录
1. 基于docker部署cdh
基于Docker搭建CDH单机单节点集群_litlit023的专栏-CSDN博客
2. 遇到的异常
2.1 flink下缺少hadoop相关依赖
基于docker 部署cdn后，启动flink on yarn模式，报了以下错误
./bin/yarn-session.sh -n 2 -jm 1024 -tm 1024
报了以下错误
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:650)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:632)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 7 more解决方法：
提示flink启动缺少Hadoop yarn的相关jar包，从官网下载Flink依赖Hadoop的相关jar包；
进入官网，下载对应的相关依赖jar包，放到flink lib目录下即可
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException - RICH-ATONE - 博客园Apache Flink: Downloads
根据官网部署Flink on YARN过程中遇到的问题及解决方法 - 简书
2.2 jdk7造成的错误
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1643103905134_0001
at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1219)
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:607)
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:419)
... 7 more
2022-01-25 17:49:17,827 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cancelling deployment from Deployment Failure Hook
2022-01-25 17:49:17,840 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
2022-01-25 17:49:17,845 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Killing YARN application
2022-01-25 17:49:17,931 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Killed application application_1643103905134_0001
2022-01-25 17:49:18,033 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deleting files in hdfs://quickstart.cloudera:8020/user/root/.flink/application_1643103905134_0001.
[root@vm01 flink-1.14.3]# yarn logs -applicationId application_1643103905134_0001
22/01/25 17:49:32 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
Container: container_1643103905134_0001_01_000001 on quickstart.cloudera_8041
===============================================================================
LogType:jobmanager.err
Log Upload Time:星期二 一月 25 17:49:19 +0800 2022
LogLength:160
Log Contents:
Unrecognized VM option 'MaxMetaspaceSize=268435456'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
LogType:jobmanager.out
Log Upload Time:星期二 一月 25 17:49:19 +0800 2022
LogLength:0
Log Contents:Unrecognized VM option 'MaxMetaspaceSize=268435456'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
需要将cdh组件所用的jdk改为jdk8以上，我的环境时基于docker来部署的cdh 5.x，因此进入到容器内，修改容器内的jdk为jdk8即可
修改jdk为自己的jdk8
进入到容器docker exec -it mycdh3 bash
docker cp /work/jdk/jdk1.8.0_271 02155f0f3679:/root/
cd /usr/java/jdk1.7.0_67-cloudera
rm -rf ./*
cp -r /root/jdk1.8.0_271/* /usr/java/jdk1.7.0_67-cloudera/修改jdk版本：
java - Unrecognized option: -MaxMetaspaceSize=256m - Stack Overflow
How to configure java home on flink job on yarn - JiKe DevOps Community
java - Unrecognized option: -MaxMetaspaceSize=256m - Stack Overflow
Exposing a port on a live Docker container - Stack Overflow
flink on yarn 集群模式启动报错及解决方案汇总_大数据技术博客-CSDN博客_flink on yarn 报错
flink on yarn遇到的yarn资源不足问题_无名子七叶莲的博客-程序员宝宝 - 程序员宝宝
3. 启动flink on yarn
模式
[root@vm01 flink-1.14.3]# ./bin/yarn-session.sh -jm 1024 -tm 1024
2022-01-25 20:38:55,885 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, localhost
2022-01-25 20:38:55,894 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123
2022-01-25 20:38:55,896 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m
2022-01-25 20:38:55,897 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m
2022-01-25 20:38:55,897 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2022-01-25 20:38:55,898 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1
2022-01-25 20:38:55,898 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2022-01-25 20:38:57,062 INFO org.apache.flink.runtime.security.modules.HadoopModule [] - Hadoop user set to root (auth:SIMPLE)
2022-01-25 20:38:57,095 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-3655855445473519286.conf.
2022-01-25 20:38:57,390 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/work/flink/streamx_dir/flink-1.14.3/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2022-01-25 20:38:57,951 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
2022-01-25 20:38:58,732 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2022-01-25 20:38:58,736 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2022-01-25 20:38:58,737 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction network memory (57.600mb (60397978 bytes)) is less than its min value 64.000mb (67108864 bytes), min value will be used instead
2022-01-25 20:38:59,287 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1024, slotsPerTaskManager=1}
2022-01-25 20:39:00,148 WARN org.apache.hadoop.util.NativeCodeLoader [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2022-01-25 20:39:05,590 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2022-01-25 20:39:05,671 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting application master application_1643114118293_0002
2022-01-25 20:39:05,740 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted application application_1643114118293_0002
2022-01-25 20:39:05,740 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for the cluster to be allocated
2022-01-25 20:39:05,745 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deploying cluster, current state ACCEPTED
2022-01-25 20:39:24,071 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - YARN application has been deployed successfully.
2022-01-25 20:39:24,072 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface quickstart.cloudera:36283 of application 'application_1643114118293_0002'.
JobManager Web Interface: http://quickstart.cloudera:36283
标签：25,Flink,java,部署,flink,yarn,apache,org 来源： https://blog.csdn.net/litlit023/article/details/122761791
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。