ICode9

精准搜索请尝试: 精确搜索
首页 > 其他分享> 文章详细

docker跨网段搭建gp测试环境

2021-04-12 21:32:52  阅读:210  来源: 互联网

标签:24 10 网段 gp gpadmin 172.19 docker mdw


(1)制作docker镜像:

#docker run -i -d --name sdw1 -h sdw1 centos  #拉取一个基础镜像,然后在此基础上修改

 

容器里面操作:

#yum install vim wget compat-openssl10 langpacks-en glibc-all-langpacks readline-devel bzip2 krb5-devel perl rsync libevent apr apr-util openssl passwd libnsl net-tools which openssh-clients openssh-server less zip unzip iproute -y
#ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ""
#ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ""
#ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N "" #/usr/sbin/sshd

vi /etc/security/limits.conf
# End of file
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
vi /etc/sysctl.conf
#等一些简单设置以后,上传镜像到harbor仓库: 

然后拉取镜像开始测试:

申请2台ec2,ip为:

a机器  10-10-70-186     docker0 : 172.19.1.1

b机器 10-10-77-42       docker0 : 172.19.2.1
2台ec2的vpc网络放通,避免不必要的麻烦

(3)修改2台ec2默认docker0网卡ip地址(个人习惯)

修改/etc/docker/daemon.json文件,没有就编辑,修改后如下:

root@ip-10-10-70-186:~# cat /etc/docker/daemon.json
{"bip":"172.19.1.1/24"}
root@ip-10-10-77-42:~# cat /etc/docker/daemon.json
{"bip":"172.19.2.1/24"}

systemctl restart docker.service

(4)创建docker,写docker-compse.yaml配置文件:

root@ip-10-10-70-186:~# cat docker-compose.yml
version: '3'
services:
sdw3:
image: xxxx/ops/gpdb:v1.2
hostname: sdw3
container_name: db_sdw3
ports:
- "1225:22"
- "15435:5432"
stdin_open: true
tty: true
networks:
mynet1:
ipv4_address: 172.19.3.3
sdw4:
image: xxxx/ops/gpdb:v1.2
hostname: sdw4
container_name: db_sdw4
ports:
- "1226:22"
- "15436:5432"
stdin_open: true
tty: true
networks:
mynet1:
ipv4_address: 172.19.3.4
sdw5:
image: xxxx/ops/gpdb:v1.2
hostname: sdw5
container_name: db_sdw5
ports:
- "1227:22"
- "15437:5432"
stdin_open: true
tty: true
networks:
mynet1:
ipv4_address: 172.19.3.5
networks:
mynet1:
ipam:
config:
- subnet: 172.19.3.0/24



root@ip-10-10-77-42:~# cat docker-compose.yml
version: '3'
services:
mdw:
image: xxxx/ops/gpdb:v1.2
hostname: mdw
container_name: db_mdw
ports:
- "1222:22"
- "15432:5432"
stdin_open: true
tty: true
networks:
mynet1:
ipv4_address: 172.19.4.2
sdw1:
image: xxxx/ops/gpdb:v1.2
hostname: sdw1
container_name: db_sdw1
ports:
- "1223:22"
- "15433:5432"
stdin_open: true
tty: true
networks:
mynet1:
ipv4_address: 172.19.4.3
sdw2:
image: xxxx/ops/gpdb:v1.2
hostname: sdw2
container_name: db_sdw2
ports:
- "1224:22"
- "15434:5432"
stdin_open: true
tty: true
networks:
mynet1:
ipv4_address: 172.19.4.4
networks:
mynet1:
ipam:
config:
- subnet: 172.19.4.0/24

2台ec2启动docker-compose正常启动docker:


然后配置互信,进入db_mdw容器:

root@10-10-77-42:~# docker exec -ti db_mdw /bin/bash
[root@mdw /]# su - gpadmin
Last login: Thu Apr  8 14:52:59 UTC 2021 on pts/2
[gpadmin@mdw ~]$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/gpadmin/.ssh/id_rsa): 

...

配置mdw主机host文件:

[gpadmin@mdw ~]$ cat /etc/hosts
127.0.0.1       localhost

172.19.4.2 mdw
172.19.4.3 sdw1
172.19.4.4 sdw2
172.19.3.3 sdw3
172.19.3.4 sdw4
172.19.3.5 sdw5

然后再2台宿主机添加路由使2个宿主机内部docker能互访:



实现: 路由表方式:
a机器添加路由:route add -net 172.19.2.0/24 gw 10.10.77.42
b机器添加路由:route add -net 172.19.1.0/24 gw 10.10.70.186


a机器添加转发:iptables -t nat -A POSTROUTING -s 172.19.1.0/24 -d 172.19.2.0/16 -j MASQUERADE
iptables -t nat -I PREROUTING -s 172.19.3.0/24 -d 172.19.4.0/24 -j DNAT --to 172.19.1.1

b机器添加转发:iptables -t nat -A POSTROUTING -s 172.19.2.0/24 -d 172.19.1.0/16 -j MASQUERADE
iptables -t nat -I PREROUTING -s 172.19.4.0/24 -d 172.19.3.0/24 -j DNAT --to 172.19.2.1

操作以后,6个docker能互访,当时a机器内部docker与b机器的docker互信始终不行,折腾了许久没搞定,于是换了一种方法达到这6个docker互访没有问题。

使用https://github.com/weaveworks/weave这个工具充当路由器的功能。实现6个docker互访。

2台ec2分别安装weave工具:
curl -L git.io/weave -o /usr/local/bin/weave

root@10-10-70-186:~#

weave connect 10.10.77.42 to ## 连接另外一台ec2
weave attach 172.19.44.5/24 cd9754fd68b0 #db_sdw3
weave attach 172.19.44.6/24 ff2bddf34119 #db_sdw4
weave attach 172.19.44.6/24 ff2bddf34119 #db_sdw5

root@10-10-77-42:~#

weave connect to 10.10.70.186 ## 连接另外一台ec2
weave attach 172.19.44.2/24 ac3077ebfd0a #db_mdw
weave attach 172.19.44.3/24 b6349ed1305d #db_sdw1
weave attach 172.19.44.4/24 d51d1cbd7658 #db_sdw2

这样6个docker就实现了互访。

(5) 开始安装greenplum数据库:

root@10-10-77-42:~# docker exec -ti db_mdw /bin/bash
[root@mdw /]# su - gpadmin
##ssh-keygen设置ssh配置
##配置host文件
[gpadmin@mdw ~]$ cat /etc/hosts
127.0.0.1 localhost
172.19.44.2 mdw
172.19.44.3 sdw1
172.19.44.4 sdw2
172.19.44.5 sdw3
172.19.44.6 sdw4
172.19.44.7 sdw5

因为已经用weave改变了6个docker的ip,才能互访,所以用修改后的ip地址,之前docker-compse.yaml里面的ip可保留也可以丢弃。

[gpadmin@mdw ~]$ cat seg_hosts # 我暂时用4个docker搭建机器,留2个用来添加删除segment 

然后开始安装gp数据库

确保6个docker互信没有问题才可以安装数据库。

gp初始化参数如下:

[gpadmin@mdw ~]$ cat gpinitsystem_config
ARRAY_NAME="Greenplum"
MACHINE_LIST_FILE=/home/gpadmin/seg_hosts
# Segment 的名称前缀
SEG_PREFIX=gpseg
# Primary Segment 起始的端口号
PORT_BASE=33000
# 指定 Primary Segment 的数据目录
declare -a DATA_DIRECTORY=(/home/gpadmin/gpdata/gpdatap1 /home/gpadmin/gpdata/gpdatap2)
# Master 所在机器的 Hostname
MASTER_HOSTNAME=mdw
# 指定 Master 的数据目录
MASTER_DIRECTORY=/home/gpadmin/gpdata/gpmaster
# Master 的端口
MASTER_PORT=5432
# 指定Bash的版本
TRUSTED_SHELL=/usr/bin/ssh
# Mirror Segment起始的端口号
MIRROR_PORT_BASE=43000
# Primary Segment 主备同步的起始端口号
REPLICATION_PORT_BASE=34000
# Mirror Segment 主备同步的起始端口号
MIRROR_REPLICATION_PORT_BASE=44000
# Mirror Segment 的数据目录
declare -a MIRROR_DATA_DIRECTORY=(/home/gpadmin/gpdata/gpdatam1 /home/gpadmin/gpdata/gpdatam2)

[gpadmin@mdw ~]$ gpssh -f seg_hosts

=>

mkdir -p /home/gpadmin/gpdata/gpmaster/gpseg-1
mkdir -p /home/gpadmin/gpdata/gpdatap1
mkdir -p /home/gpadmin/gpdata/gpdatam1
mkdir -p /home/gpadmin/gpdata/gpdatam2
mkdir -p /home/gpadmin/gpdata/gpdatap2

chmod 777 -R /home/gpadmin/gpdata
chown gpadmin.gpadmin -R /home/gpadmin/gpdata

echo 'export MASTER_DATA_DIRECTORY=/home/gpadmin/gpdata/gpmaster/gpseg-1' >> /home/gpadmin/.bashrc

这样在4个docker均已创建好相应目录已经设置好权限。

for i in $(docker ps -a -q);do docker exec -ti $i ip a|grep inet|awk '{print $2}'|grep -vE '^fe80|^::|^127'|awk -F '/' '{print $1}';done  # 2台ec2宿主机执行,检查docker ip地址

[gpadmin@mdw ~]$ gpinitsystem -c gpinitsystem_config

。。。。

最后完成。



最大的坑,安装的时候报错,排查安装日志报错提示:

/usr/local/greenplum-db-6.15.0/bin/psql: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory


解决方法:
ls -la /usr/lib64 |grep libreadline
[gpadmin@660f628171a7 gpAdminLogs]$ ldd /usr/local/greenplum-db-6.15.0/bin/psql
linux-vdso.so.1 (0x00007ffde69ca000)
libpq.so.5 => /usr/local/greenplum-db-6.15.0/lib/libpq.so.5 (0x00007fab0fbb4000)
libssl.so.10 => /lib64/libssl.so.10 (0x00007fab0f945000)
libreadline.so.6 => not found
libc.so.6 => /lib64/libc.so.6 (0x00007fab0f582000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fab0f11f000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fab0eeca000)
libldap_r-2.4.so.2 => /lib64/libldap_r-2.4.so.2 (0x00007fab0ec73000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fab0ea53000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fab0e84f000)
libz.so.1 => /lib64/libz.so.1 (0x00007fab0e638000)
/lib64/ld-linux-x86-64.so.2 (0x00007fab0fdea000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fab0e34f000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fab0e138000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fab0df34000)

ln -s /usr/lib64/libreadline.so.7.0 /usr/lib64/libreadline.so.6

  

标签:24,10,网段,gp,gpadmin,172.19,docker,mdw
来源: https://www.cnblogs.com/5sdba-notes/p/14650394.html

本站声明: 1. iCode9 技术分享网(下文简称本站)提供的所有内容,仅供技术学习、探讨和分享;
2. 关于本站的所有留言、评论、转载及引用,纯属内容发起人的个人观点,与本站观点和立场无关;
3. 关于本站的所有言论和文字,纯属内容发起人的个人观点,与本站观点和立场无关;
4. 本站文章均是网友提供,不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属;如您发现该文章侵犯了您的权益,可联系我们第一时间进行删除;
5. 本站为非盈利性的个人网站,所有内容不会用来进行牟利,也不会利用任何形式的广告来间接获益,纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。

专注分享技术,共同学习,共同进步。侵权联系[81616952@qq.com]

Copyright (C)ICode9.com, All Rights Reserved.

ICode9版权所有