标签：hdss7 opt 10.4 二进制 host 集群 etcd k8s com
准备环境
-
准备1台虚拟机，要求centos7系统，用VMvware Workstation安装
-
用安装好的虚拟机，克隆5台虚拟
创建完成的效果图：
-
更改虚拟机 VMnet1 和 VMnet8 的设置。点击WMware Workstation: 编辑->虚拟网络编辑器，在弹出的对话框中修改
-
五台虚机的ip规划为 10.4.7.11、10.4.7.12、10.4.7.21、10.4.7.22、10.4.7.200，对应的名称分别为：hdss7-11.host.com、hdss7-12.host.com、hdss7-21.host.com、hdss7-22.host.com、hdss7-200.host.com
-
修改虚拟机ip
vim /etc/sysconfig/network-scripts/ifcfg-ens33将文件 ifcfg-ens33内容，改为如下
TYPE=Ethernet BOOTPROTO=none NAME=ens33 ONBOOT=yes IPADDR=10.4.7.11 NETMASK=255.255.255.0 GATEWAY=10.4.7.254 DNS1=10.4.7.254修改虚拟机名称：
hostnamectl set-hostname hdss7-11.host.com -
关闭firewalld服务
setenforce 0 systemctl diable firewalld systemctl stop firewalld -
安装epel源
yum install -y epel-release -
安装必要工具
yum install wget net-tools telnet tree nmap sysstat lrzsz dos2unix bind-utils -y
在hdss7-11上安装bind
-
安装bind
yum install bind -y -
修改bind主配置文件 vi /etc/named.conf
listen-on port 53 { 10.4.7.11; }; allow-query { any; }; forwarders { 10.4.7.254; }; dnssec-enable no; dnssec-validation no; -
检查配置文件是否报错
named-checkconf -
在区域配置文件配置 /etc/named.rfc1912.zones中，添加
zone "host.com" IN { type master; file "host.com.zone"; allow-update { 10.4.7.11; }; }; zone "od.com" IN { type master; file "od.com.zone"; allow-update { 10.4.7.11; }; }; -
配置 host.com.zone 配置文件 /var/named/host.com.zone
$ORIGIN host.com. $TTL 600 ; 10 minutes @ IN SOA dns.host.com. dnsadmin.host.com. ( 2021082201 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.host.com. $TTL 60 ; 1 minute dns A 10.4.7.11 HDSS7-11 A 10.4.7.11 HDSS7-12 A 10.4.7.12 HDSS7-21 A 10.4.7.21 HDSS7-22 A 10.4.7.22 HDSS7-200 A 10.4.7.200 -
配置 od.com.zone 配置文件 /var/named/od.com.zone
$ORIGIN od.com. $TTL 600 ; 10 minutes @ IN SOA dns.od.com. dnsadmin.od.com. ( 2021082201 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.od.com. $TTL 60 ; 1 minute dns A 10.4.7.11配置完成后，检查配置是否有错
named-checkconf -
启动
systemctl start named # 查看53端口 netstat -luntp|grep 53 # 查看域名解析 dig -t A hdss7-21.host.com @10.4.7.11 +short -
修改 五台虚拟机的 /etc/sysconfig/network-scripts/ifcfg-ens33 配置文件，将 DNS 改为 10.4.7.11
DNS1=10.4.7.11 -
编辑五台虚拟机 /etc/resolv.conf 配置文件，增加
search host.com -
配置VMnet8 的dns 为10.4.7.11
-
在window宿主机中验证是否成功p
ping hdss7-200.host.com
准备签发证书环境（运维主机 hdss7-200.host.com 上）
-
安装CFSSL
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/bin/cfssl wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/bin/cfssl-json wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/bin/cfssl-certinfo -
创建/opt/cents目录并创建一个/opt/certs/ca-csr.json文件
{ "CN": "FengYun", "hosts": [ ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "beijing", "L": "beijing", "O": "od", "OU": "ops" } ], "ca": { "expiry": "175200h" } } -
自签证书
cfssl gencert -initca ca-csr.json | cfssl-json -bare ca
部署docker 环境（hdss7-200.host.com、hdss7-21.host.com、hdss7-22.host.com）上
- 在三台虚拟机都安装docker
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun - 在三台虚拟机上都执行创建目录命令
mkdir -p /data/docker /etc/docker - 在 hdss7-21.host.com 创建 /etc/docker/daemon.json 文件，加入如下配置
同理 hdss7-200.host.com、hdss7-22.host.com 只需要将配置文件中的"bip": “172.7.21.1/24” 变为 “bip”: “172.7.200.1/24” 和 “bip”: “172.7.22.1/24”{ "graph": "/data/docker", "storage-driver": "overlay2", "insecure-registries": ["registry.access.redhat.com", "quay.io","harbor.od.com"], "registry-mirrors": ["https://k7sj660n.mirror.aliyuncs.com"], "bip": "172.7.21.1/24", "exec-opts": ["native.cgroupdriver=systemd"], "live-restore": true }
部署 docker 镜像私有仓库 harbor （hdss7-200.host.com）
- 下载离线二进制安装包
- 解压二进制包
tar -xvf harbor-offline-installer-v1.10.7.tgz -C /opt/ - 创建软连接，方便升级
cd /opt/ mv harbor/ harbor-v1.10.7 ln -s /opt/harbor-v1.10.7/ /opt/harbor - 进入 /opt/harbor目录，修改harbor.yml配置文件
hostname: harbor.od.com http: port: 180 # https related config #https: # # https port for harbor, default is 443 # port: 443 # # The path of cert and key files for nginx # certificate: /your/certificate/path # private_key: /your/private/key/path harbor_admin_password: Harbor12345 data_volume: /data/docker log: locatioin: /data/harbor/logs - 安装docker-compose
yum install docker-compose -y - 执行 /opt/harbor/install.sh 脚本
- 在 harbor目录向下，执行如下命令，验证安装结果。
docker-compose ps
安装 nginx （hdss7-200.host.com）
- 安装命令
yum install nginx -y - 在 nginx 中配置harbor，编辑/etc/nginx/conf.d/harbor.od.com.conf，加入如下配置：
报错退出后，执行如下命令：server { listen 80; server_name harbor.od.com; client_max_body_size 1000m; location / { proxy_pass http://127.0.0.1:180 } }nginx -t systemctl start nginx systemctl enable nginx - 执行 curl 命令：
如果报如下错误curl harbor.od.comcurl: (6) Could not resolve host: harbor.od.com; Unknown error - 由于有错，回到
（hdss7-11.host.com）主机上，修改如下：
原序列化 2021082201 变为 2021082202，增加$ORIGIN od.com. $TTL 600 ; 10 minutes @ IN SOA dns.od.com. dnsadmin.od.com. ( 2021082202 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.od.com. $TTL 60 ; 1 minute dns A 10.4.7.11 harbor A 10.4.7.200harbor A 10.4.7.200 - 重启dns 并测试解析结果
（hdss7-11.host.com）systemctl restart named dig -t A harbor.od.com +short - 重新回到
（hdss7-200.host.com），执行curl命令，确保成功。curl harbor.od.com - 测试自己私有harbor仓库。登录 http://harbor.od.com
- 创建一个项目
- 从docker hub中拉取nginx镜像，并且打tag 推送到自己创建harbor仓库中
docker pull nginx:1.7.9 docker tag 84581e99d807 harbor.od.com/public/nginx:v1.7.9 # 84581e99d807 这个是镜像id docker login harbor.od.com # 登录自己的 harbor docker push harbor.od.com/public/nginx:v1.7.9 # 将镜像推送到自己 harbor - 成功将镜像推送到自己的私有仓库
安装etcd集群 （hdss7-12.host.com、hdss7-21.host.com、hdss7-22.host.com）
etcd集群规划
| 主机名 | 角色 | ip |
|---|---|---|
| hdss7-12.host.com | etcd lead | 10.4.7.12 |
| hdss7-21.host.com | etcd follow | 10.4.7.21 |
| hdss7-22.host.com | etcd follow | 10.4.7.22 |
创建基于根证书的config配置文件（hdss7-200.host.com）
/opt/certs/ca-config.json
{
"signing": {
"default": {
"expiry": "175200h"
},
"profiles": {
"server": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
创建 etcd 的证书 （hdss7-200.host.com）
- 创建生成自签证书签名请求文件 /opt/certs/etcd-peer-csr.json
{ "CN": "k8s-etcd", "hosts": [ "10.4.7.11", "10.4.7.12", "10.4.7.21", "10.4.7.22" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "beijing", "L": "beijing", "O": "od", "OU": "ops" } ] } - 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json |cfssl-json -bare etcd-peer
安装 etcd（先安装 hdss7-12.host.com ）
-
创建用户
useradd -s /sbin/nologin -M etcd id etcd -
下载 etcd 并解压
wget https://github.com/etcd-io/etcd/releases/download/v3.1.20/etcd-v3.1.20-linux-amd64.tar.gz tar -xf etcd-v3.1.20-linux-amd64.tar.gz -C /opt/ cd /opt/ mv etcd-v3.1.20-linux-amd64 etcd-v3.1.20 ln -s /opt/etcd-v3.1.20/ /opt/etcd -
创建目录，拷贝证书，私钥
mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server将运维主机 hdss7-200.host.com 生成的 ca.pem、etcd-peer-key.pem、etcd-peer.pem拷贝到 /opt/etcd/certs 目录中，注意私钥文件权限600s
cd /opt/etcd/certs scp hdss7-200:/opt/certs/ca.pem . scp hdss7-200:/opt/certs/etcd-peer.pem . scp hdss7-200:/opt/certs/etcd-peer-key.pem . -
创建etcd服务启动脚本 /opt/etcd/etcd-server-startup.sh（注意，改如果以下脚本在hdss7-21、hdss7-22上执行时，修改相应的地方）
#!/bin/sh ./etcd --name etcd-server-7-12 \ --data-dir /data/etcd/etcd-server \ --listen-peer-urls https://10.4.7.12:2380 \ --listen-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \ --quota-backend-bytes 8000000000 \ --initial-advertise-peer-urls https://10.4.7.12:2380 \ --advertise-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \ --initial-cluster etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \ --ca-file ./certs/ca.pem \ --cert-file ./certs/etcd-peer.pem \ --key-file ./certs/etcd-peer-key.pem \ --client-cert-auth \ --trusted-ca-file ./certs/ca.pem \ --peer-ca-file ./certs/ca.pem \ --peer-cert-file ./certs/etcd-peer.pem \ --peer-key-file ./certs/etcd-peer-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file ./certs/ca.pem \ --log-output stdout -
增加执行权限，并且改属主
chmod +x etcd-server-startup.sh chown -R etcd:etcd /opt/etcd-v3.1.20/ chown -R etcd:etcd /data/etcd/ chown -R etcd:etcd /data/logs/etcd-server/ -
安装 supervisor（管理后台进程的软件）
yum install supervisor -y systemctl start supervisord systemctl enable supervisord -
创建 etcd-server 的启动配置 /etc/supervisord.d/etcd-server.ini
[program:etcd-server-7-12] command=/opt/etcd/etcd-server-startup.sh numprocs=1 directory=/opt/etcd autostart=true autorestart=true startsecs=30 startretries=3 exitcodes=0,2 stopsignal=QUIT stopwaitsecs=10cd user=etcd redirect_stderr=true stdout_logfile=/data/logs/etcd-server/etcd.stdout.log stdout_logfile_maxbytes=64MB stdout_logfile_backups=4 stdout_capture_maxbytes=1MB stdout_events_enabled=false -
启动etcd
supervisorctl update supervisorctl status tail -fn 200 /data/logs/etcd-server/etcd.stdout.log #查看启动日志 netstat -luntp |grep etcd -
同理，在hdss7-21.host.com、hdss7-22.host.com上，按照1~8步骤，安装etcd
-
用 etcdctl 检测集群健康状态
./etcdctl cluster-health看到如下信息，说明集群是健康的
member 988139385f78284 is healthy: got healthy result from http://127.0.0.1:2379
member 5a0ef2a004fc4349 is healthy: got healthy result from http://127.0.0.1:2379
member f4a0cb0a765574a8 is healthy: got healthy result from http://127.0.0.1:2379
cluster is healthy./etcdctl member list988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.22:2379 isLeader=false
5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.21:2379 isLeader=false
f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=true
部署 kube-apiserver 集群
集群规划
| 主机名 | 角色 | ip |
|---|---|---|
| hdss7-21.host.com | kube-apiserver | 10.4.7.21 |
| hdss7-22.host.com | kube-apiserver | 10.4.7.22 |
| hdss7-11.host.com | 4层负载均衡 | 10.4.7.11 |
| hdss7-12.host.com | 4层负载均衡 | 10.4.7.12 |
这里的10.4.7.11 和 10.4.7.12使用nginx做4层负载均衡器，用 keepalived 跑 vip：10.4.7.10，代理两个kube-apiserver，实现高可用。
下载k8s安装报，解压，做软链接
- 下载二进制安装包https://dl.k8s.io/v1.15.2/kubernetes-server-linux-amd64.tar.gz
- 解压二进制包，并创建软连接
tar -xf kubernetes-server-linux-amd64.tar.gz -C /opt/ cd /opt/ mv kubernetes kubernetes-v1.15.2 ln -s /opt/kubernetes-v1.15.2/ /opt/kubernetes
签发client证书（apiserver与etcd集群通信使用）
此通信过程中，apiserver是client端，而etcd是server端。
- 创建生成证书签名请求（csr）的JSON配置文件 (此过程在hdss7-200.host.com上执行）
/opt/certs/client-csr.json
{ "CN": "k8s-node", "hosts": [ ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "beijing", "L": "beijing", "O": "od", "OU": "ops" } ] } - 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json |cfssl-json -bare client
签发 kube-apiserver 证书（由于其它组件访问api-server时候的证书）
- 创建生成证书签名请求（csr）的JSON配置文件 (此过程在hdss7-200.host.com上执行）
/opt/certs/apiserver-csr.json
{ "CN": "k8s-apiserver", "hosts": [ "127.0.0.1", "192.168.0.1", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local", "10.4.7.10", "10.4.7.21", "10.4.7.22", "10.4.7.23" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "beijing", "L": "beijing", "O": "od", "OU": "ops" } ] } - 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json | cfssl-json -bare apiserver
拷贝证书到 apiserver（hdss7-21.host.com,hdss7-22.host.com）
- 进入 /opt/kubernetes/server/bin下，创建 cert目录
cd /opt/kubernetes/server/bin mkdir cert - 将证书拷贝到 /opt/kubernetes/server/bin/cert目录下
scp hdss7-200:/opt/certs/apiserver-key.pem . scp hdss7-200:/opt/certs/apiserver.pem . scp hdss7-200:/opt/certs/ca-key.pem . scp hdss7-200:/opt/certs/ca.pem . scp hdss7-200:/opt/certs/client.pem . scp hdss7-200:/opt/certs/client-key.pem .
创建k8s集群api-server启动的配置文件（hdss7-21.host.com,hdss7-22.host.com）
- 创建conf目录 /opt/kubernetes/server/bin/conf
- 创建 /opt/kubernetes/server/bin/conf/audit.yaml 启动文件
apiVersion: audit.k8s.io/v1beta1 # This is required. kind: Policy # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "RequestReceived" rules: # Log pod changes at RequestResponse level - level: RequestResponse resources: - group: "" # Resource "pods" doesn't match requests to any subresource of pods, # which is consistent with the RBAC policy. resources: ["pods"] # Log "pods/log", "pods/status" at Metadata level - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] # Don't log requests to a configmap called "controller-leader" - level: None resources: - group: "" resources: ["configmaps"] resourceNames: ["controller-leader"] # Don't log watch requests by the "system:kube-proxy" on endpoints or services - level: None users: ["system:kube-proxy"] verbs: ["watch"] resources: - group: "" # core API group resources: ["endpoints", "services"] # Don't log authenticated requests to certain non-resource URL paths. - level: None userGroups: ["system:authenticated"] nonResourceURLs: - "/api*" # Wildcard matching. - "/version" # Log the request body of configmap changes in kube-system. - level: Request resources: - group: "" # core API group resources: ["configmaps"] # This rule only applies to resources in the "kube-system" namespace. # The empty string "" can be used to select non-namespaced resources. namespaces: ["kube-system"] # Log configmap and secret changes in all other namespaces at the Metadata level. - level: Metadata resources: - group: "" # core API group resources: ["secrets", "configmaps"] # Log all other resources in core and extensions at the Request level. - level: Request resources: - group: "" # core API group - group: "extensions" # Version of group should NOT be included. # A catch-all rule to log all other requests at the Metadata level. - level: Metadata # Long-running requests like watches that fall under this rule will not # generate an audit event in RequestReceived. omitStages: - "RequestReceived" - 创建api-server启动脚本 /opt/kubernetes/server/bin/kube-apiserver.sh
#!/bin/bash ./kube-apiserver \ --apiserver-count 2 \ --audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log \ --audit-policy-file ./conf/audit.yaml \ --authorization-mode RBAC \ --client-ca-file ./cert/ca.pem \ --requestheader-client-ca-file ./cert/ca.pem \ --enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \ --etcd-cafile ./cert/ca.pem \ --etcd-certfile ./cert/client.pem \ --etcd-keyfile ./cert/client-key.pem \ --etcd-servers https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \ --service-account-key-file ./cert/ca-key.pem \ --service-cluster-ip-range 192.168.0.0/16 \ --service-node-port-range 3000-29999 \ --target-ram-mb=1024 \ --kubelet-client-certificate ./cert/client.pem \ --kubelet-client-key ./cert/client-key.pem \ --log-dir /data/logs/kubernetes/kube-apiserver \ --tls-cert-file ./cert/apiserver.pem \ --tls-private-key-file ./cert/apiserver-key.pem \ --v 2 - 调整权限，创建目录
chmod +x /opt/kubernetes/server/bin/kube-apiserver.sh mkdir -p /data/logs/kubernetes/kube-apiserver - 创建supervisor 配置 /etc/supervisord.d/kube-apiserver.ini
[program:kube-apiserver-7-21] command=/opt/kubernetes/server/bin/kube-apiserver.sh numprocs=1 directory=/opt/kubernetes/server/bin autostart=true autorestart=true startsecs=30 startretries=3 exitcodes=0,2 stopsignal=QUIT stopwaitsecs=10 user=root redirect_stderr=true stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log stdout_logfile_maxbytes=64MB stdout_logfile_backups=4 stdout_capture_maxbytes=1MB stdout_events_enabled=false - 启动api-servers
api-server启动成功后，我们可用看到了apiserver监听了8080端口和6443端口supervisorctl update netstat -luntp|grep kube-api
安装部署主控节点L4反向代理服务
- 在 hdss7-11、hdss7-12安装nginx
yum install nginx -y - 配置反向代理，在文件中添加 /etc/nginx/nginx.conf
stream { upstream kube-apiserver { server 10.4.7.21:6443 max_fails=3 fail_timeout=30s; server 10.4.7.22:6443 max_fails=3 fail_timeout=30s; } server { listen 7443; proxy_connect_timeout 2s; proxy_timeout 900s; proxy_pass kube-apiserver; } } - 检查配置并启动nginx
nginx -t systemctl start nginx systemctl enable nginx
安装keepalived(hdss7-11、hdss7-12)
-
安装 keepalived
yum install keepalived -y -
创建监听端口的脚本 /etc/keepalived/check_port.sh，主要是为了监听7443端口
#!/bin/bash CHK_PORT=$1 if [ -n "$CHK_PORT" ];then PORT_PROCESS=`ss -lnt|grep $CHK_PORT|wc -l` if [ $PORT_PROCESS -eq 0 ];then echo "Port $CHK_PORT Is Not Used,End." exit 1 fi else echo "Check Port Cant Be Empty!" fi -
给与执行权限
chmod +x /etc/keepalived/check_port.sh -
主 keepalived.conf 配置(hdss7-11)
! Configuration File for keepalived global_defs { router_id 10.4.7.11 } vrrp_script chk_nginx { script "/etc/keepalived/check_port.sh 7443" interval 2 weight -20 } vrrp_instance VI_1 { state MASTER interface ens33 virtual_router_id 251 priority 100 advert_int 1 mcast_src_ip 10.4.7.11 nopreempt authentication { auth_type PASS auth_pass 11111111 } track_script { chk_nginx } virtual_ipaddress { 10.4.7.10 } } -
从 keepalive.conf 配置 ( hdss7-12 )
! Configuration File for keepalived global_defs { router_id 10.4.7.12 script_user root enable_script_security } vrrp_script chk_nginx { script "/etc/keepalived/check_port.sh 7443" interval 2 weight -20 } vrrp_instance VI_1 { state BACKUP interface ens33 virtual_router_id 251 mcast_src_ip 10.4.7.12 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 11111111 } track_script { chk_nginx } virtual_ipaddress { 10.4.7.10 } } -
启动keepalived
systemctl start keepalived systemctl enable keepalived -
验证virtual-ip 是否能够漂移
nginx -s stop # 在hdss7-11上执行 ip addr # 在hdss7-11上执行 ip addr # 在hdss7-12上执行此时，发现virtual-ip漂移到 hdss7-12上了
标签：hdss7,opt,10.4,二进制,host,集群,etcd,k8s,com 来源： https://blog.csdn.net/weixin_32196893/article/details/119849734
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。