0.2.kubeadm搭建kubernetes高可用集群

2020-05-17 17:07:51 阅读：328 来源： 互联网

标签：kube kubernetes 0.2 192.168 etc kubeadm k8s flannel

kubeadmin安装Kubernetes

kubeadmin安装Kubernetes

kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具；使用staticPod(容器)运行的管理组件,镜像使用gcr.io域名仓库里的。

0.部署环境&需求

部署版本 kubernetes1.16

IP	Hostname	内核	CPU&Memory	系统版本	备注
192.168.121.81	k8s-master01	4.4.206-1.el7.elrepo.x86_64	4C&8M	7.6.1810	control plane
192.168.121.82	k8s-master02	4.4.206-1.el7.elrepo.x86_64	4C&8M	7.6.1810	control plane
192.168.121.83	k8s-master03	4.4.206-1.el7.elrepo.x86_64	4C&8M	7.6.1810	control plane
192.168.121.84	k8s-node01	4.4.206-1.el7.elrepo.x86_64	4C&8M	7.6.1810	worker nodes
192.168.121.85	k8s-node02	4.4.206-1.el7.elrepo.x86_64	4C&8M	7.6.1810	worker nodes
192.168.121.86	k8s-node03	4.4.206-1.el7.elrepo.x86_64	4C&8M	7.6.1810	worker nodes
192.168.121.88	keepalived			v1.3.5	vip

在开始之前，部署Kubernetes集群机器需要满足以下几个条件：

一台或多台机器，操作系统 CentOS7.x-86_x64
硬件配置：2GB或更多RAM，2个CPU或更多CPU，硬盘30GB或更多
集群中所有机器之间网络互通
可以访问外网，需要拉取镜像
禁止swap分区

1.系统初始化

1.1.主机名

设置永久主机名称，然后重新登录:

]# hostnamectl set-hostname k8s-master01 # 将  k8s-master01 替换为当前主机名

设置的主机名保存在 /etc/hostname 文件中；

如果 DNS 不支持解析主机名称，则需要修改每台机器的 /etc/hosts 文件，添加主机名和 IP 的对应关系：

]# cat >> /etc/hosts <<EOF
192.168.121.81 k8s-master01
192.168.121.82 k8s-master02
192.168.121.83 k8s-master03
192.168.121.84 k8s-node01
192.168.121.85 k8s-node02
192.168.121.86 k8s-node03
EOF

1.2.添加 docker 账户

在每台机器上添加 docker 账户：

]# useradd -m docker

1.3.无密码 ssh 登录其它节点

如果没有特殊指明，本文档的所有操作均在 k8s-master01 节点上执行，然后远程分发文件和执行命令，所以需要添加该节点到其它节点的 ssh 信任关系。

设置 k8s-master 的 root 账户可以无密码登录所有节点：

]# ssh-keygen -t rsa
]# ssh-copy-id root@k8s-master01
]# ssh-copy-id root@k8s-master02
]# ssh-copy-id root@k8s-master03

1.4.更新 PATH 变量

将可执行文件目录添加到 PATH 环境变量中：

]# echo 'PATH=/opt/k8s/bin:$PATH' >>/root/.bashrc
]# source /root/.bashrc

1.5.安装依赖包

在每台机器上安装依赖包：

]# yum install -y epel-release
]# yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget

1.6.关闭防火墙

在每台机器上关闭防火墙，清理防火墙规则，设置默认转发策略：

]# systemctl stop firewalld
]# systemctl disable firewalld
]# iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat
]# iptables -P FORWARD ACCEPT

1.7.关闭 swap 分区

如果开启了 swap 分区，kubelet 会启动失败(可以通过将参数 --fail-swap-on 设置为 false 来忽略 swap on)，故需要在每台机器上关闭 swap 分区。同时注释 /etc/fstab 中相应的条目，防止开机自动挂载 swap 分区：

]# swapoff -a
]# sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

1.8.关闭 SELinux

关闭 SELinux，否则后续 K8S 挂载目录时可能报错 Permission denied：

]# setenforce 0
]# sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

1.9.关闭 dnsmasq（可选）

linux 系统开启了 dnsmasq 后(如 GUI 环境)，将系统 DNS Server 设置为 127.0.0.1，这会导致 docker 容器无法解析域名，需要关闭它：

]# systemctl stop dnsmasq
]# systemctl disable dnsmasq

1.10.加载内核模块

]# modprobe ip_vs_rr
]# modprobe br_netfilter

1.11.优化内核参数

]# cat > kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0 # 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它
vm.overcommit_memory=1 # 不检查物理内存是否够用
vm.panic_on_oom=0 # 开启 OOM
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF
]# cp kubernetes.conf  /etc/sysctl.d/kubernetes.conf
]# sysctl -p /etc/sysctl.d/kubernetes.conf

必须关闭 tcp_tw_recycle，否则和 NAT 冲突，会导致服务不通；
关闭 IPV6，防止触发 docker BUG；

1.12.设置系统时区

# 调整系统 TimeZone
]# timedatectl set-timezone Asia/Shanghai

# 将当前的 UTC 时间写入硬件时钟
]# timedatectl set-local-rtc 0

### 重启依赖于系统时间的服务
]# systemctl restart rsyslog 
]# systemctl restart crond

1.13.关闭无关的服务

]# systemctl stop postfix && systemctl disable postfix

1.14.升级内核

CentOS 7.x 系统自带的 3.10.x 内核存在一些 Bugs，导致运行的 Docker、Kubernetes 不稳定，例如：

高版本的 docker(1.13 以后) 启用了 3.10 kernel 实验支持的 kernel memory account 功能(无法关闭)，当节点压力大如频繁启动和停止容器时会导致 cgroup memory leak；
网络设备引用计数泄漏，会导致类似于报错："kernel:unregister_netdevice: waiting for eth0 to become free. Usage count = 1";

解决方案如下：

升级内核到 4.4.X 以上；
或者，手动编译内核，disable CONFIG_MEMCG_KMEM 特性；
或者，安装修复了该问题的 Docker 18.09.1 及以上的版本。但由于 kubelet 也会设置 kmem（它 vendor 了 runc），所以需要重新编译 kubelet 并指定 GOFLAGS="-tags=nokmem"；
这里采用升级内核的解决办法：

]# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
# 安装完成后检查 /boot/grub2/grub.cfg 中对应内核 menuentry 中是否包含 initrd16 配置，如果没有，再安装一次！
]# yum --enablerepo=elrepo-kernel install -y kernel-lt
# 设置开机从新内核启动
]# grub2-set-default 0

安装内核源文件（可选，在升级完内核并重启机器后执行）:

]#  yum erase kernel-headers
]# yum --enablerepo=elrepo-kernel install kernel-lt-devel-$(uname -r) kernel-lt-headers-$(uname -r)

1.15.关闭 NUMA

]# cp /etc/default/grub{,.bak}
]# vim /etc/default/grub # 在 GRUB_CMDLINE_LINUX 一行添加 `numa=off` 参数，如下所示：
]# diff /etc/default/grub.bak /etc/default/grub
6c6
< GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rhgb quiet"
---
> GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rhgb quiet numa=off"

重新生成 grub2 配置文件：

]# cp /boot/grub2/grub.cfg{,.bak}
]# grub2-mkconfig -o /boot/grub2/grub.cfg

1.16.文件最大数

]# cat>/etc/security/limits.d/kubernetes.conf<<EOF
*       soft    nproc   131072
*       hard    nproc   131072
*       soft    nofile  131072
*       hard    nofile  131072
root    soft    nproc   131072
root    hard    nproc   131072
root    soft    nofile  131072
root    hard    nofile  131072
EOF

2.开始部署

2.1.部署Docker

每台机器上安装Docker，建议使用18.09版本。

]#  wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
]#  yum list docker-ce --showduplicates | sort -r
]#  yum install docker-ce-18.09.9 docker-ce-cli-18.09.9 containerd.io -y

镜像下载加速：

]# curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io

设置cgroup驱动，推荐systemd：

]#  mkdir -p /etc/docker
]# cat > /etc/docker/daemon.json <<EOF
{
"registry-mirrors": ["https://docker.mirrors.ustc.edu.cn","http://f1361db2.m.daocloud.io"],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {"max-size": "100m"},
"storage-driver": "overlay2"
}
EOF
]#  systemctl enable docker
]#  systemctl start docker

修改cgroupdriver是为了消除告警：
[WARNING IsDockerSystemdCheck]: detected “cgroupfs” as the Docker cgroup driver. The recommended driver is “systemd”. Please follow the guide at https://kubernetes.io/docs/setup/cri/

启动&验证

]# systemctl enable --now docker
]# docker --version
Docker version 18.09.9, build 039a7df9ba

2.2.部署keepalived

k8s-master节点都执行本部分操作。
安装keepalived

]# yum -y install keepalived

keepalived配置

master01上keepalived配置：

[root@master01 ~]# more /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
   router_id master01
}
vrrp_instance VI_1 {
    state MASTER 
    interface eth0
    virtual_router_id 50
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.121.88
    }
}

master02上keepalived配置：

[root@master02 ~]# more /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
   router_id master02
}
vrrp_instance VI_1 {
    state BACKUP 
    interface eth0
    virtual_router_id 50
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.121.88
    }
}

master03上keepalived配置：

[root@master03 ~]# more /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
   router_id master03
}
vrrp_instance VI_1 {
    state BACKUP 
    interface eth0
    virtual_router_id 50
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.121.88
    }

启动keepalived

所有k8s-mster节点启动keepalived服务并设置开机启动

]# service keepalived start
]# systemctl enable keepalived

VIP查看

]# ip a

确保vip在master01上；否则在初始化安装会无法连接api-server

2.3.部署管理工具kubeadm

在所有节点操作

]# cat <<EOF >/etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
EOF

maser节点安装

]# yum install -y \
    kubeadm-1.16.4 \
    kubectl-1.16.4 \
    kubelet-1.16.4 
]# systemctl enable kubelet

node节点安装

]# yum install -y \
    kubeadm-1.16.4 \
    kubelet-1.16.4 
]# systemctl enable kubelet

命令补全

安装bash-completion

]# yum -y install bash-completion

加载bash-completion

]# source /etc/profile.d/bash_completion.sh

启动kubelet并设置开机启动

]# systemctl enable kubelet && systemctl start kubelet

kubectl命令补全

]# echo "source <(kubectl completion bash)" >> ~/.bash_profile
]# source .bash_profile

2.4.下载镜像

镜像下载的脚本
Kubernetes几乎所有的安装组件和Docker镜像都放在goolge自己的网站上,直接访问可能会有网络问题，这里的解决办法是从阿里云镜像仓库下载镜像，拉取到本地以后改回默认的镜像tag。本文通过运行image.sh脚本方式拉取镜像。

]# more image.sh 
#!/bin/bash
url=registry.aliyuncs.com/google_containers
version=v1.16.4
images=(`kubeadm config images list --kubernetes-version=$version|awk -F '/' '{print $2}'`)
for imagename in ${images[@]} ; do
  docker pull $url/$imagename
  docker tag $url/$imagename k8s.gcr.io/$imagename
  docker rmi -f $url/$imagename
done

url为阿里云镜像仓库地址，version为安装的kubernetes版本。

下载镜像

运行脚本image.sh，下载指定版本的镜像

]# chmod +x image.sh
]# ./image.sh
]# docker images

3.初始化Master&Node

master01节点执行本部分操作。

3.1.配置初始化文件

]# more kubeadm_config.yaml 
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.4
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
apiServer:
  extraArgs:
    authorization-mode: "Node,RBAC"
    runtime-config: api/all,settings.k8s.io/v1alpha1=true
    storage-backend: etcd3
    etcd-servers: https://192.168.121.81:2379,https://192.168.121.82:2379,https://192.168.121.83:2379
  certSANs:    #填写所有kube-apiserver节点的hostname、IP、VIP
  - k8s-master01
  - k8s-master02
  - k8s-master03
  - k8s-node01
  - k8s-node02
  - k8s-node03
  - 192.168.121.81
  - 192.168.121.82
  - 192.168.121.83
  - 192.168.121.84
  - 192.168.121.85
  - 192.168.121.86
  - 192.168.121.87
  - 192.168.121.88
  extraVolumes:
  - hostPath: /etc/localtime
    mountPath: /etc/localtime
    name: localtime
    readOnly: true
etcd:
  local:
    dataDir: /var/lib/etcd
    serverCertSANs:
    - master
    - 192.168.121.81
    - 192.168.121.82
    - 192.168.121.83
    - k8s-master01
    - k8s-master02
    - k8s-master03
    peerCertSANs:
    - master
    - 192.168.121.81
    - 192.168.121.82
    - 192.168.121.83
    - k8s-master01
    - k8s-master02
    - k8s-master03
    extraArgs:
      auto-compaction-retention: "1h"
      max-request-bytes: "33554432"
      quota-backend-bytes: "8589934592"
      enable-v2: "false"
controlPlaneEndpoint: "192.168.121.88:6443"
networking:
  podSubnet: "10.244.0.0/16"
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs #启动ipvs mode；默认iptables mode
ipvs:
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: "rr"
  strictARP: false
  syncPeriod: 15s
iptables:
  masqueradeAll: true
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
failSwapOn: true

kubeadm_config.yaml为初始化的配置文件

3.2.master初始化

检查文件是否错误，忽略warning，错误的话会抛出error

]# kubeadm init --config /root/initconfig.yaml --dry-run

进行初始化

]# kubeadm init --config=kubeadm_config.yaml
......
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities 
and service account keys on each node and then running the following as root:

  kubeadm join 192.168.121.88:6443 --token kkpntz.yc5ow033k2mnp0z4 \
    --discovery-token-ca-cert-hash sha256:bbc3c89aa94f90c457fc6a61f66c39b8bfccb12e84cd8df31ce226e09f0ea839 \
    --control-plane 	  

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.121.88:6443 --token kkpntz.yc5ow033k2mnp0z4 \
    --discovery-token-ca-cert-hash sha256:bbc3c89aa94f90c457fc6a61f66c39b8bfccb12e84cd8df31ce226e09f0ea839

复制kubectl的kubeconfig，kubectl的kubeconfig路径默认是~/.kube/config

]# mkdir -p $HOME/.kube
]# sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

init的yaml信息实际上会存在集群的configmap里，我们可以随时查看，该yaml在其他node和master join的时候会使用到

kubectl -n kube-system get cm kubeadm-config -o yaml

加载环境变量

]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
]# source .bash_profile

配置其他master的k8s管理组件

for node in 192.168.33.102 192.168.33.103;do
    ssh $node 'mkdir -p /etc/kubernetes/pki/etcd'
    scp -r /root/kubeadm_config.yaml $node:/root/kubeadm_config.yaml
    scp -r /root/image.sh $node:/root/image.sh
    scp -r /etc/kubernetes/pki/ca.* $node:/etc/kubernetes/pki/
    scp -r /etc/kubernetes/pki/sa.* $node:/etc/kubernetes/pki/
    scp -r /etc/kubernetes/pki/front-proxy-ca.* $node:/etc/kubernetes/pki/
    scp -r /etc/kubernetes/pki/etcd/ca.* $node:/etc/kubernetes/pki/etcd/
done

配置其余k8s-master

]# kubeadm join 192.168.121.88:6443 --token kkpntz.yc5ow033k2mnp0z4 \
    --discovery-token-ca-cert-hash sha256:bbc3c89aa94f90c457fc6a61f66c39b8bfccb12e84cd8df31ce226e09f0ea839 \
    --control-plane

所有master配置kubectl,准备kubectl的kubeconfig

]# mkdir -p $HOME/.kube
]# sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

设置kubectl的补全脚本

]# kubectl completion bash > /etc/bash_completion.d/kubectl
]# source  /etc/bash_completion.d/kubectl

3.3.node初始化

在node节点执行

和master的join一样，提前准备好环境和docker，然后join的时候不需要带--control-plane

]# kubeadm join 192.168.121.88:6443 --token kkpntz.yc5ow033k2mnp0z4 \
    --discovery-token-ca-cert-hash sha256:bbc3c89aa94f90c457fc6a61f66c39b8bfccb12e84cd8df31ce226e09f0ea839

3.4.初始化失败

如果初始化失败，可执行kubeadm reset后重新初始化

]# kubeadm reset
]# rm -rf $HOME/.kube/config

4.配置etcdctl

4.1.复制出容器里的etcdctl

]# docker cp `docker ps -a | awk '/k8s_etcd/{print $1}'`:/usr/local/bin/etcdctl /usr/local/bin/etcdctl
]# cat >/etc/profile.d/etcd.sh<<'EOF'
ETCD_CERET_DIR=/etc/kubernetes/pki/etcd/
ETCD_CA_FILE=ca.crt
ETCD_KEY_FILE=healthcheck-client.key
ETCD_CERT_FILE=healthcheck-client.crt
ETCD_EP=https://192.168.121.81:2379,https://192.168.121.82:2379,https://192.168.121.83:2379

alias etcd_v3="ETCDCTL_API=3 \
    etcdctl   \
   --cert ${ETCD_CERET_DIR}/${ETCD_CERT_FILE} \
   --key ${ETCD_CERET_DIR}/${ETCD_KEY_FILE} \
   --cacert ${ETCD_CERET_DIR}/${ETCD_CA_FILE} \
   --endpoints $ETCD_EP"
EOF

source  /etc/profile.d/etcd.sh 

etcd_v3 endpoint status --write-out=table
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://192.168.121.81:2379 | d572bef816d9cb7e |  3.3.15 |  3.2 MB |     false |        52 |      64002 |
| https://192.168.121.82:2379 | a4b42aafeef9bab3 |  3.3.15 |  3.2 MB |      true |        52 |      64002 |
| https://192.168.121.83:2379 | c5d52def73f50ceb |  3.3.15 |  3.2 MB |     false |        52 |      64002 |
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+


etcd_v3 endpoint health --write-out=table
+-----------------------------+--------+-------------+-------+
|          ENDPOINT           | HEALTH |    TOOK     | ERROR |
+-----------------------------+--------+-------------+-------+
| https://192.168.121.82:2379 |   true | 28.336842ms |       |
| https://192.168.121.81:2379 |   true | 29.132682ms |       |
| https://192.168.121.83:2379 |   true | 47.635089ms |       |
+-----------------------------+--------+-------------+-------+

4.2.配置etcd备份脚本

mkdir -p /opt/etcd
cat>/opt/etcd/etcd_cron.sh<<'EOF'
#!/bin/bash
set -e

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

:  ${bak_dir:=/root/} #缺省备份目录,可以修改成存在的目录
:  ${cert_dir:=/etc/kubernetes/pki/etcd/}
:  ${endpoints:=https://192.168.121.81:2379,https://192.168.121.82:2379,https://192.168.121.83:2379}

bak_prefix='etcd-'
cmd_suffix='date +%Y-%m-%d-%H:%M'
bak_suffix='.db'

#将规范化后的命令行参数分配至位置参数（$1,$2,...)
temp=`getopt -n $0 -o c:d: -u -- "$@"`

[ $? != 0 ] && {
    echo '
Examples:
  # just save once
  bash $0 /tmp/etcd.db
  # save in contab and  keep 5
  bash $0 -c 5
    '
    exit 1
    }
set -- $temp


# -c 备份保留副本数量
# -d 指定备份存放目录

while true;do
    case "$1" in
        -c)
            [ -z "$bak_count" ] && bak_count=$2
            printf -v null %d "$bak_count" &>/dev/null || \
                { echo 'the value of the -c must be number';exit 1; }
            shift 2
            ;;
        -d)
            [ ! -d "$2" ] && mkdir -p $2
            bak_dir=$2
            shift 2
            ;;
         *)
            [[ -z "$1" || "$1" == '--' ]] && { shift;break; }
            echo "Internal error!"
            exit 1
            ;;
    esac
done



function etcd_v3(){

    ETCDCTL_API=3 etcdctl   \
       --cert $cert_dir/healthcheck-client.crt \
       --key  $cert_dir/healthcheck-client.key \
       --cacert $cert_dir/ca.crt \
       --endpoints $endpoints $@
}

etcd::cron::save(){
    cd $bak_dir/
    etcd_v3 snapshot save  $bak_prefix$($cmd_suffix)$bak_suffix
    rm_files=`ls -t $bak_prefix*$bak_suffix | tail -n +$[bak_count+1]`
    if [ -n "$rm_files" ];then
        rm -f $rm_files
    fi
}

main(){
    [ -n "$bak_count" ] && etcd::cron::save || etcd_v3 snapshot save $@
}

main $@
EOF

crontab -e添加下面内容自动保留四个备份副本

]# `crontab -e 0 0 * * * bash /opt/etcd/etcd_cron.sh  -c 4 -d /opt/etcd/ &>/dev/null

5.部署插件

5.1.网络查件

5.1.1部署flannel网络

新建flannel网络

]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

由于网络原因，可能会安装失败，可以使用如下yaml，然后再执行apply

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
    - configMap
    - secret
    - emptyDir
    - hostPath
  allowedHostPaths:
    - pathPrefix: "/etc/cni/net.d"
    - pathPrefix: "/etc/kube-flannel"
    - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups: ['extensions']
    resources: ['podsecuritypolicies']
    verbs: ['use']
    resourceNames: ['psp.flannel.unprivileged']
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - amd64
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.12.0-amd64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.12.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

5.1.2部署Calico网络(可选)

]# curl https://docs.projectcalico.org/v3.9/manifests/calico-etcd.yaml -o calico.yaml

修改yaml文件、并部署

]# sed -i -e "s@192.168.0.0/16@10.244.0.0/16@g" calico.yaml
]# kubectl apply -f calico.yaml

5.2.Web UI（Dashboard搭建）

5.2.1进行配置

下载yaml**

]# swget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

如果连接超时，可以多试几次

配置yaml

修改镜像地址

]#  sed -i 's/registry.aliyuncs.com\/google_containers/g' recommended.yaml

由于默认的镜像仓库网络访问不通，故改成阿里镜像

外网访问

]# sed -i '/targetPort: 8443/a\ \ \ \ \ \ nodePort: 30001\n\ \ type: NodePort' recommended.yaml

配置NodePort，外部通过https://NodeIp:NodePort 访问Dashboard，此时端口为30001

新增管理员帐号

]# cat >> recommended.yaml << EOF
---
# ------------------- dashboard-admin ------------------- #
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard-admin
  namespace: kubernetes-dashboard

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: dashboard-admin
subjects:
- kind: ServiceAccount
  name: dashboard-admin
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin

创建超级管理员的账号用于登录Dashboard

5.2.2部署访问

部署Dashboard

]# kubectl apply -f recommended.yaml

状态查看

]# kubectl get all -n kubernetes-dashboard

令牌查看

创建service account并绑定默认cluster-admin管理员集群角色：

]# kubectl describe secrets -n kubernetes-dashboard $(kubectl -n kubernetes-dashboard get secret | awk '/dashboard-admin/{print $1}')

使用输出的token登录Dashboard，使用火狐浏览。

标签：kube,kubernetes,0.2,192.168,etc,kubeadm,k8s,flannel
来源： https://www.cnblogs.com/Gmiaomiao/p/12905795.html

本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享；
2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关；
3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关；
4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除；
5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。

ICode9