标签：zookeeper standalone 部署 cfg flink echo zoo data

最近上面想要通过flink-cdc来实现mysql数据实时同步至Elasticsearch,由于可以通过sql来实现数据同步，难度和投入都相对较小。于是自己研究了下flink,由于flink-cdc 现在最高支持flink1.13的版本，所有本文使用1.13.5的版本演示部署flink集群。

flink大数据框架，官方推荐了三种集群模式：

1.standalone

2.kubernetes

3.YARN

本文通过standlone+zookeeper构建flink HA Service

项目环境：

 操作系统均为ubuntu20.1

master	192.168.1.101
worker1	192.168.1.102
worker2	192.168.1.103

软件版本：

flink	flink-1.13.5
zookeeper	apache-zookeeper-3.5.9-bin.tar.gz

官方推荐jdk是8或者11

jdk	openjdk-11.0.2_linux-x64_bin.tar.gz

1.所有的主机安装好jdk

tar xf openjdk-11.0.2_linux-x64_bin.tar.gz -C /usr/local/
cd /usr/local
mv jdk-11.0.2 jdk
echo "export JAVA_HOME=/usr/local/jdk">>/etc/profile
echo 'export PATH=$PATH:$JAVA_HOME/bin'>>/etc/profile
source /etc/profile	

java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

2.做免密登录（重要的步骤）

由于大部分人都会做免密所以不在阐述详细的步骤。

3.安装zookeeper

3.1master主机上安装zookeeper

mkdir -p /data/zookeeper
echo 1 > /data/zookeeper/myid
tar xf apache-zookeeper-3.5.9-bin.tar.gz -C /data/
cd /data/apache-zookeeper-3.5.9
cp zoo_sample.cfg zoo.cfg
sed -i "s/tmp/data/" zoo.cfg
echo "server.1=192.168.1.101:2888:3888" >> zoo.cfg
echo "server.2=192.168.1.102:2888:3888" >> zoo.cfg
echo "server.3=192.168.1.103:2888:3888" >> zoo.cfg

3.2worker1上安装zookeeper

mkdir -p /data/zookeeper
echo 2 > /data/zookeeper/myid
tar xf apache-zookeeper-3.5.9-bin.tar.gz -C /data/
cd /data/apache-zookeeper-3.5.9
cp zoo_sample.cfg zoo.cfg
sed -i "s/tmp/data/" zoo.cfg
echo "server.1=192.168.1.101:2888:3888" >> zoo.cfg
echo "server.2=192.168.1.102:2888:3888" >> zoo.cfg
echo "server.3=192.168.1.103:2888:3888" >> zoo.cfg

3.3worker2上安装zookeeper

mkdir -p /data/zookeeper
echo 3 > /data/zookeeper/myid
tar xf apache-zookeeper-3.5.9-bin.tar.gz -C /data/
cd /data/apache-zookeeper-3.5.9
cp zoo_sample.cfg zoo.cfg
sed -i "s/tmp/data/" zoo.cfg
echo "server.1=192.168.1.101:2888:3888" >> zoo.cfg
echo "server.2=192.168.1.102:2888:3888" >> zoo.cfg
echo "server.3=192.168.1.103:2888:3888" >> zoo.cfg

zookeeper的配置文件如下：

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper
clientPort=2181
server.1=192.168.1.101:2888:3888
server.2=192.168.1.102:2888:3888
server.3=192.168.1.103:2888:3888

 

3.4依次启动三个节点上的zookeeper

cd /data/apache-zookeeper-3.5.9-bin
./bin/zkServer.sh --config conf start

4.安装flink(切记三个主机上的flink的安装路劲需要一致，推荐全部安装在/opt这个目录，如果是云服务器推荐安装在data目录)

4.1三个节点上面依次执行下面命令。

tar xf flink-1.13.5-bin-scala_2.12.tgz -C /data/
echo "192.168.1.101:8081" > /data/flink-1.13.5/conf/master
echo -e "192.168.1.102\n192.168.1.103" > /data/flink-1.13.5/conf/worker
 

 4.2 请保证三台服务器的配置文件如下。

jobmanager.rpc.address: 192.168.1.101
jobmanager.rpc.port: 6123
jobmanager.memory.process.size: 8600m
taskmanager.memory.process.size: 8728m
taskmanager.numberOfTaskSlots: 32
parallelism.default: 1
high-availability: zookeeper
high-availability.storageDir: file:///flink_store/recovery
high-availability.zookeeper.quorum: 192.168.1.101:2181,192.168.10.102:2181,192.168.1.103:2181
state.backend: filesystem
state.checkpoints.dir: file:///flink-checkpoints
state.savepoints.dir: file:///flink-savepoints
state.backend.incremental: true
execution.checkpointing.interval: 100
jobmanager.execution.failover-strategy: region
taskmanager.memory.network.min: 64mb
taskmanager.memory.network.max: 1gb

请注意/flink_store/recovery, /flink-checkpoints,/flink-savepoints都是nfs挂载目录。

在master上执行：

cd /data/flink-1.13.5
./bin/start-cluster.sh

为了简化操作，提供一个简易功能不完全的shell脚本

#！/bin/bash

cur_dir=`dirname $0`
root_dir=`cd $cur_dir|pwd`
echo $root_dir
source /etc/profile
flink_tar=flink-1.13.5-bin-scala_2.12.tgz
flink_dir=flink-1.13.5
zookeeper_tar=apache-zookeeper-3.5.9-bin.tar.gz
zookeeper_dir=apache-zookeeper-3.5.9-bin
nfs_host= #你的nfs服务器
master=$1
host2=$2
host3=$3
id=$4


function jdk_install(){
   java -version > /dev/null
   if [ $? -ne 0 ];then
	cd $root_dir
        tar xf openjdk-11.0.2_linux-x64_bin.tar.gz -C /usr/local/
        cd /usr/local
        mv jdk-11.0.2 jdk
        echo "export JAVA_HOME=/usr/local/jdk">>/etc/profile
        echo 'export PATH=$PATH:$JAVA_HOME/bin'>>/etc/profile	
   fi
}

function flink_install(){
   cd $root_dir
   tar xf $flink_tar -C /data/
   cp flink-sql-connector-elasticsearch6_2.11-1.13.5.jar /data/$flink_dir/lib/
   cp flink-sql-connector-mysql-cdc-2.1.1.jar /data/$flink_dir/lib/
   sed -i -e "s/{master}/${master}/g" -e "s/{host2}/${host2}/" -e "s/{host3}/${host3}/" flink-conf.yaml
   cp -f flink-conf.yaml /data/$flink_dir/conf/
   echo "${master}:8081" >> /data/$flink_dir/conf/master
   echo "${host2}:8081" >> /data/$flink_dir/conf/master
   echo "${host2}" >> /data/$flink_dir/conf/worker
   echo "${host3}" >> /data/$flink_dir/conf/worker
}

function zookeeper_install(){
   cd $root_dir
   tar xf $zookeeper_tar -C /data/
   cd /data/$zookeeper_dir/conf
   mkdir /data/zookeeper
   echo $id > /data/zookeeper/myid
   cp zoo_sample.cfg zoo.cfg
   echo "server.1=${master}:2888:3888" >> zoo.cfg
   echo "server.2=${host2}:2888:3888" >> zoo.cfg
   echo "server.3=${host3}:2888:3888" >> zoo.cfg
   sed -i "s/tmp/data/" zoo.cfg
}

function mount_nfs(){
	apt install -y nfs-common
	mkdir /flink_job
	mkdir /flink_store
	mkdir /flink-checkpoints
	mkdir /flink-savepoints
	echo "${nfs_host}:/flink/flink_job         /flink_job nfs4 rw  0 0" >> /etc/fstab
	echo "${nfs_host}:/flink/flink-checkpoints /flink-checkpoints nfs4 rw 0 0" >> /etc/fstab
	echo "${nfs_host}:/flink/flink-savepoints /flink-savepoints nfs4 rw 0 0" >> /etc/fstab
	echo "${nfs_host}:/flink/flink_store /flink_store/  nfs4 rw 0 0" >> /etc/fstab
	mount -a
}

if [ -z $master ];then
   exit
fi
if [ -z $host2 ];then
   exit
fi
if [ -z $host3 ];then
   exit
fi
if [ -z $4 ];then
   exit
fi
jdk_install
flink_install
zookeeper_install
mount_nfs

了解更多关于flink和flink-cdc：

flink: Docs | Apache Flink

flink-cdc: Welcome to Flink CDC — Flink CDC documentation

标签：zookeeper,standalone,部署,cfg,flink,echo,zoo,data
来源： https://blog.csdn.net/wchao111111/article/details/122849645

本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享；
2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关；
3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关；
4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除；
5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。