ICode9

精准搜索请尝试: 精确搜索
首页 > 其他分享> 文章详细

第十一章 kubernetes 核心技术-Controller控制器

2021-11-23 18:04:47  阅读:139  来源: 互联网

标签:web kubernetes 第十一章 nginx Controller master deployment root


一、Controller控制器概述

1.含义

在集群上管理和运行容器的对象。

2.Controller和Pod之间的关系

Pod是通过Controller实现应用的运维比如伸缩,滚动升级等等

Pod和Controller之间通过label标签建立关系

二、Replication Controller

Replication Controller(RC)是 Kubernetes 系统中核心概念之一,当我们定义了一个 RC 并提交到 Kubernetes 集群中以后,Master 节点上的 Controller Manager 组件就得到通知, 定期检查系统中存活的 Pod,并确保目标 Pod 实例的数量刚好等于 RC 的预期值,如果有过多或过少的 Pod 运行,系统就会停掉或创建一些 Pod.此外我们也可以通过修改 RC 的副本数量,来实现 Pod 的动态缩放功能。

kubectl scale rc nginx --replicas=5

由于 Replication Controller 与 Kubernetes 代码中的模块 Replication Controller 同名,所以在 Kubernetes v1.2 时, 它就升级成了另外一个新的概念Replica Sets,官方解释为下一代的 RC,它与 RC 区别是:Replica Sets 支援基于集合的 Label selector,而 RC 只支持基于等式的 Label Selector。我们很少单独使用 Replica Set,它主要被 Deployment 这个更高层面的资源对象所使用,从而形成一整套 Pod 创建、删除、更新的编排机制。最好不要越过 RC 直接创建 Pod, 因为 Replication Controller 会通过 RC 管理 Pod 副本,实现自动创建、补足、替换、删除 Pod 副本,这样就能提高应用的容灾能力,减少由于节点崩溃等意外状况造成的损失。即使应用程序只有一个 Pod 副本,也强烈建议使用 RC 来 定义 Pod.

三、Replica Set

ReplicaSet 跟 ReplicationController 没有本质的不同,只是名字不一样,并且ReplicaSet 支持集合式的 selector(ReplicationController 仅支持等式)。
Kubernetes 官方强烈建议避免直接使用 ReplicaSet,而应该通过 Deployment 来创建 RS 和Pod。由于 ReplicaSet 是 ReplicationController 的代替物,因此用法基本相同,唯一的区别在于 ReplicaSet 支持集合式的 selector。

四、Deployment控制器

1.含义

Deployment 是 Kubenetes v1.2 引入的新概念,引入的目的是为Pod提供声明式更新。Deployment 内部使用了 Replica Set 来实现。Deployment 的定义与 Replica Set 的定义很类似,除了 API 声明与 Kind 类型有所区别.

在Deployment对象中描述所需的状态,然后Deployment控制器将实际状态以受控的速率更改为所需的状态。您可以定义部署以创建新的副本集,或删除现有部署并在新部署中采用其所有资源。一句话:Deployment主要功能是保证有足够的Pod正常对外提供服务。

2.Deployment应用场景

#1.部署无状态应用

#2.管理Pod和Replica Set部署,滚动升级等功能

#3.应用场景:web服务,微服务

3.Deployment实例

1)手写Deployment

[root@kubernetes-master-001 ~]# vim demo.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80

2)导出Deployment模板

[root@kubernetes-master-001 ~]# kubectl create deployment web --image=nginx --dry-run=client -o yaml > web.yaml

[root@kubernetes-master-001 ~]# ll
total 20
-rw-r--r--  1 root root  376 Nov  8 15:31 web.yaml

3)应用部署

将kubectl标志设置--record为true允许您将当前命令记录在正在创建或更新的资源的注释中。这对于将来的检查很有用。
[root@kubernetes-master-001 ~]# kubectl apply -f web.yaml --record
deployment.apps/web created

[root@kubernetes-master-001 ~]# kubectl  get deployments.apps 
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
web     1/1     1            1           39s

[root@kubernetes-master-001 ~]# kubectl rollout history deployment web
deployment.apps/web 
REVISION  CHANGE-CAUSE
1         <none>

4)对外发布(暴露端口号)

[root@kubernetes-master-001 ~]# kubectl  expose  deployment web  --port=80 --type=NodePort --target-port=80 --name=web1 -o yaml > web1.yaml

[root@kubernetes-master-001 ~]# kubectl apply -f web1.yaml
service/web1 configured

[root@kubernetes-master-001 ~]# kubectl  get  pod,svc
NAME                      READY   STATUS    RESTARTS   AGE
pod/web-96d5df5c8-kgw6v   1/1     Running   0          3m58s

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        3h17m
service/web1         NodePort    10.98.97.60      <none>        80:31413/TCP   82s

5)验证访问

[root@kubernetes-master-001 ~]# curl 192.168.13.100:31413
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

4.查看部署状态

[root@kubernetes-master-001 ~]# kubectl  get  pods
NAME                  READY   STATUS    RESTARTS   AGE
web-96d5df5c8-kgw6v   1/1     Running   0          23m

5.查看部署详情

[root@kubernetes-master-001 ~]# kubectl describe deployments.apps web
Name:                   web
Namespace:              default
CreationTimestamp:      Mon, 08 Nov 2021 17:22:44 +0800
Labels:                 app=web
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=web
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=web
  Containers:
   nginx:
    Image:        nginx
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   web-96d5df5c8 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  25m   deployment-controller  Scaled up replica set web-96d5df5c8 to 1

五、Deployment应用升级回滚

一般对应用程序升级或者版本迭代时,会通过Deployment对Pod进行滚动更新。

1.修改配置文件

[root@kubernetes-master-001 ~]# vim web.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: web
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: web
    spec:
      containers:
      - image: nginx:1.14
        name: nginx
        resources: {}
status: {}

2.应用部署

#1.查看应用部署状态
[root@kubernetes-master-001 ~]# kubectl  get pods
NAME                   READY   STATUS    RESTARTS   AGE
web-6c57bdf5f4-cwr8r   1/1     Running   0          4s
web-6c57bdf5f4-r57s9   1/1     Running   0          73s

#2.查看部署详情
[root@kubernetes-master-001 ~]# kubectl  describe  deployments.apps  web
Name:                   web
Namespace:              default
CreationTimestamp:      Mon, 08 Nov 2021 17:55:40 +0800
Labels:                 app=web
Annotations:            deployment.kubernetes.io/revision: 2
Selector:               app=web
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=web
  Containers:
   nginx:
    Image:        nginx:1.14
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   web-5bb6fd4c98 (2/2 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  5m51s  deployment-controller  Scaled up replica set web-6c57bdf5f4 to 1
  Normal  ScalingReplicaSet  4m42s  deployment-controller  Scaled up replica set web-6c57bdf5f4 to 2
  Normal  ScalingReplicaSet  3m9s   deployment-controller  Scaled up replica set web-5bb6fd4c98 to 1
  Normal  ScalingReplicaSet  2m41s  deployment-controller  Scaled down replica set web-6c57bdf5f4 to 1
  Normal  ScalingReplicaSet  2m41s  deployment-controller  Scaled up replica set web-5bb6fd4c98 to 2
  Normal  ScalingReplicaSet  2m38s  deployment-controller  Scaled down replica set web-6c57bdf5f4 to 0

3.应用升级

#1.修改nginx镜像1.14版本为1.18
[root@kubernetes-master-001 ~]# kubectl  set image deployment web nginx=nginx:1.18
deployment.apps/web image updated

#2.直接修改部署应用镜像
[root@kubernetes-master-001 ~]# kubectl  edit deployments.apps web

#2.查看升级状态
[root@kubernetes-master-001 ~]# kubectl  rollout  status  deployment  web 
deployment "web" successfully rolled out

4.应用回滚

当新版本不稳定时,可以对其进行回滚操作,默认情况下,所有Deployment的rollout历史都保留在系统中,可以随时回滚。

#1.查看构建历史
[root@kubernetes-master-001 ~]# kubectl  rollout  history  deployment web 
deployment.apps/web 
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

5.回滚到上一版本

#1.开始回滚
[root@kubernetes-master-001 ~]# kubectl  rollout undo  deployment  web 
deployment.apps/web rolled back

#2.查看构建状态
[root@kubernetes-master-001 ~]# kubectl  rollout  status  deployment  web 
Waiting for deployment "web" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "web" rollout to finish: 1 old replicas are pending termination...
deployment "web" successfully rolled out

#2.查看新的构建历史, 发现历史2被删除,新增了历史3
[root@kubernetes-master-001 ~]# kubectl  rollout history deployment  web 
deployment.apps/web 
REVISION  CHANGE-CAUSE
1        <none>
3         <none>

6.回滚到指定版本

#1.查看当前历史版本
[root@kubernetes-master-001 ~]# kubectl  rollout  history deployment web 
deployment.apps/web 
REVISION  CHANGE-CAUSE
3         <none>
4         <none>
=

#2.回滚至指定版本
[root@kubernetes-master-001 ~]# kubectl  rollout  undo  deployment  web --to-revision=4
deployment.apps/web skipped rollback (current template already matches revision 4)

#3.查看回滚状态
[root@kubernetes-master-001 ~]# kubectl  describe  deployments.apps  web 
Name:                   web
Namespace:              default
CreationTimestamp:      Mon, 08 Nov 2021 17:55:40 +0800
Labels:                 app=web
Annotations:            deployment.kubernetes.io/revision: 4
Selector:               app=web
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=web
  Containers:
   nginx:
    Image:        nginx:1.14
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   web-5bb6fd4c98 (2/2 replicas created)
Events:
  Type    Reason             Age                From                   Message
  ----    ------             ----               ----                   -------
  Normal  ScalingReplicaSet  26m (x2 over 34m)  deployment-controller  Scaled up replica set web-6c57bdf5f4 to 1
  Normal  ScalingReplicaSet  26m (x2 over 33m)  deployment-controller  Scaled up replica set web-6c57bdf5f4 to 2
  Normal  ScalingReplicaSet  26m                deployment-controller  Scaled down replica set web-5bb6fd4c98 to 1
  Normal  ScalingReplicaSet  26m                deployment-controller  Scaled down replica set web-5bb6fd4c98 to 0
  Normal  ScalingReplicaSet  14m (x2 over 31m)  deployment-controller  Scaled up replica set web-5bb6fd4c98 to 1
  Normal  ScalingReplicaSet  14m (x2 over 31m)  deployment-controller  Scaled down replica set web-6c57bdf5f4 to 1
  Normal  ScalingReplicaSet  14m (x2 over 31m)  deployment-controller  Scaled up replica set web-5bb6fd4c98 to 2
  Normal  ScalingReplicaSet  14m (x2 over 31m)  deployment-controller  Scaled down replica set web-6c57bdf5f4 to 0

六、Deployment弹性伸缩和扩容

当业务的用户越来越多,目前的后端服务已经无法满足业务要求当前的业务要求,传统的解决办法是将其横向增加服务器,从而满足我们的业务要求。K8S中也是支持横向扩容的方法的。

1.扩容

#1.第一种方式:scale
[root@kubernetes-master-001 ~]# kubectl  scale  deployment  web  --replicas=5
deployment.apps/web scaled

[root@kubernetes-master-001 ~]# kubectl  get pods
NAME                   READY   STATUS    RESTARTS   AGE
web-5bb6fd4c98-7cgjs   1/1     Running   0          18s
web-5bb6fd4c98-8mn4k   1/1     Running   0          18s
web-5bb6fd4c98-9ssxq   1/1     Running   0          16h
web-5bb6fd4c98-wrd7d   1/1     Running   0          18s
web-5bb6fd4c98-zgvhz   1/1     Running   0          16h

[root@kubernetes-master-001 ~]# kubectl  get  deployments.apps  web 
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    5/5     5            5           16h

#2.第二种方式:patch
[root@kubernetes-master-001 ~]# kubectl  patch deployments.apps  web  -p '{"spec": {"replicas": 5}}'
deployment.apps/web patched

[root@kubernetes-master-001 ~]# kubectl  get pods
NAME                   READY   STATUS    RESTARTS   AGE
web-5bb6fd4c98-9ssxq   1/1     Running   0          16h
web-5bb6fd4c98-bqcmt   1/1     Running   0          7s
web-5bb6fd4c98-pvpc8   1/1     Running   0          7s
web-5bb6fd4c98-tqnzx   1/1     Running   0          7s
web-5bb6fd4c98-zgvhz   1/1     Running   0          16h

[root@kubernetes-master-001 ~]# kubectl  get  deployments.apps  web 
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    5/5     5            5           16h

2.暂停部署

#1.暂停部署
[root@kubernetes-master-01 test]# kubectl rollout pause deployment nginx-deployment
error: deployments.apps "nginx-deployment" is already paused

#2.设置镜像升级
[root@kubernetes-master-001 ~]# kubectl  set image deployment web nginx=nginx:1.20
deployment.apps/web image updated

#3.查看部署状态,发现没有开始部署
[root@kubernetes-master-001 ~]# kubectl  get pods
NAME                   READY   STATUS    RESTARTS   AGE
web-5bb6fd4c98-9ssxq   1/1     Running   0          16h
web-5bb6fd4c98-bqcmt   1/1     Running   0          6m19s
web-5bb6fd4c98-pvpc8   1/1     Running   0          6m19s
web-5bb6fd4c98-tqnzx   1/1     Running   0          6m19s
web-5bb6fd4c98-zgvhz   1/1     Running   0          16h

#4.取消暂停,开始部署
[root@kubernetes-master-001 ~]# kubectl  rollout  resume  deployment web 
deployment.apps/web resumed

#5.查看状态,发现开始部署
[root@kubernetes-master-001 ~]# kubectl  get pods
NAME                   READY   STATUS              RESTARTS   AGE
web-5bb6fd4c98-9ssxq   1/1     Running             0          16h
web-5bb6fd4c98-bqcmt   1/1     Running             0          6m55s
web-5bb6fd4c98-pvpc8   1/1     Running             0          6m55s
web-5bb6fd4c98-tqnzx   0/1     Terminating         0          6m55s
web-5bb6fd4c98-zgvhz   1/1     Running             0          16h
web-7f5844f89c-94w65   0/1     ContainerCreating   0          2s
web-7f5844f89c-ggqz5   0/1     ContainerCreating   0          2s
web-7f5844f89c-tw8rd   0/1     ContainerCreating   0          2s

#6.查看应用镜像
[root@kubernetes-master-001 ~]# kubectl  describe deployments.apps web  |grep Image
    Image:        nginx:1.20

七、Horizontal Pod Autoscaler

在生产环境中,总会有一些意想不到的事情发生,比如公司网站流量突然升高,此时之前创建的Pod已不足以撑住所有的访问,而运维人员也不可能24小时守着业务服务,这时就可以通过配置HPA,实现负载过高的情况下自动扩容Pod副本数以分摊高并发的流量,当流量恢复正常后,HPA会自动缩减Pod的数量。HPA是根据CPU的使用率、内存使用率自动扩展Pod数量的,所以要使用HPA就必须定义Requests参数。

Horizontal Pod Autoscal(Pod 横向扩容 简称 HPA)与 RC、Deployment 一样,也属于一种 Kubernetes 资源对象。通过追踪分析 RC 控制的所有目标 Pod 的负载变化情况,来确定是否需要针对性地调整目标 Pod 的副本数,这是 HPA 的 实现原理。

Kubernetes 对 Pod 扩容与缩容提供了手动和自动两种模式,手动模式通过 kubectl scale 命令对一个 Deployment/RC 进行 Pod 副本数量的设置。自动模式则需要用户根据某个性能指标或者自定义业务指标,并指定 Pod 副本数量的范围,系统将自动在这个范围内根据性能指标的变化进行调整。

1.手动扩容和缩容

[root@kubernetes-master-001 ~]# kubectl  scale  deployment  web  --replicas=1
deployment.apps/web scaled

[root@kubernetes-master-001 ~]# kubectl  get pods
NAME                   READY   STATUS        RESTARTS   AGE
web-7f5844f89c-94w65   0/1     Terminating   0          15m
web-7f5844f89c-ggqz5   0/1     Terminating   0          15m
web-7f5844f89c-lc6r5   0/1     Terminating   0          14m
web-7f5844f89c-tw8rd   1/1     Running       0          15m
web-7f5844f89c-wmrsq   0/1     Terminating   0          14m

[root@kubernetes-master-001 ~]# kubectl  get pods
NAME                   READY   STATUS    RESTARTS   AGE
web-7f5844f89c-tw8rd   1/1     Running   0          15m

2.自动扩容和缩容

HPA 控制器基本 Master 的 kube-controller-manager 服务启动参数 --horizontal-pod- autoscaler-sync-period 定义的时长(默认值为 30s),周期性地监测 Pod 的 CPU 使用率, 并在满足条件时对 RC 或 Deployment 中的 Pod 副 本数量进行调整,以符合用户定义的平均Pod CPU 使用率。

#1.创建HPA
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
          resources:
            limits:
              cpu: 10m
              memory: 50Mi
            requests:
              cpu: 10m
              memory: 50Mi
---
kind: Service
apiVersion: v1
metadata:
  name: svc
  namespace: default
spec:
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80
---
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: docs
  namespace: default
spec:
  # HPA的最小pod数量和最大pod数量
  maxReplicas: 10
  minReplicas: 1
  # HPA的伸缩对象描述,HPA会动态修改该对象的pod数量
  scaleTargetRef:
    kind: Deployment
    name: nginx
    apiVersion: apps/v1
  # 监控的指标数组,支持多种类型的指标共存
  metrics:
    - type: Resource
      #  核心指标,包含cpu和内存两种(被弹性伸缩的pod对象中容器的requests和limits中定义的指标。)
      resource:
        name: cpu
        # CPU 阈值
        # 计算公式:所有目标pod的metric的使用率(百分比)的平均值,
        # 例如limit.cpu=1000m,实际使用500m,则utilization=50%
        # 例如deployment.replica=3, limit.cpu=1000m,则pod1实际使用cpu=500m, pod2=300m, pod=600m
        ## 则averageUtilization=(500/1000+300/1000+600/1000)/3 = (500 + 300 + 600)/(3*1000))
        targetAverageUtilization: 5
[root@kubernetes-master-001 ~]# kubectl run test --rm -it --image=busybox --command -- sh
If you don't see a command prompt, try pressing enter.
/ # while true; do wget -O- -q 10.96.62.55; done

[root@kubernetes-master-001 ~]# kubectl get pods -o wide -w
NAME                     READY   STATUS    RESTARTS   AGE    IP             NODE                 NOMINATED NODE   READINESS GATES
hook-demo1               1/1     Running   0          20h    10.242.112.3   kubernetes-node-02   <none>           <none>
nginx-7bf65b57b6-6sfp5   1/1     Running   0          5m6s   10.240.120.3   kubernetes-node-03   <none>           <none>
tcp-pod                  1/1     Running   0          36m    10.242.0.3     kubernetes-node-05   <none>           <none>
test                     1/1     Running   0          2m3s   10.240.120.4   kubernetes-node-03   <none>           <none>
nginx-7bf65b57b6-2qwn9   0/1     Pending   0          0s     <none>         <none>               <none>           <none>
nginx-7bf65b57b6-2qwn9   0/1     Pending   0          0s     <none>         kubernetes-node-01   <none>           <none>
nginx-7bf65b57b6-2qwn9   0/1     ContainerCreating   0          0s     <none>         kubernetes-node-01   <none>           <none>

[root@kubernetes-master-001 ~]# kubectl top pods
NAME                     CPU(cores)   MEMORY(bytes)
nginx-7bf65b57b6-2qwn9   10m          4Mi
nginx-7bf65b57b6-6sfp5   10m          4Mi
nginx-7bf65b57b6-h8gp2   0m           0Mi
nginx-7bf65b57b6-mb2rz   0m           0Mi

标签:web,kubernetes,第十一章,nginx,Controller,master,deployment,root
来源: https://www.cnblogs.com/jhno1/p/15594428.html

本站声明: 1. iCode9 技术分享网(下文简称本站)提供的所有内容,仅供技术学习、探讨和分享;
2. 关于本站的所有留言、评论、转载及引用,纯属内容发起人的个人观点,与本站观点和立场无关;
3. 关于本站的所有言论和文字,纯属内容发起人的个人观点,与本站观点和立场无关;
4. 本站文章均是网友提供,不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属;如您发现该文章侵犯了您的权益,可联系我们第一时间进行删除;
5. 本站为非盈利性的个人网站,所有内容不会用来进行牟利,也不会利用任何形式的广告来间接获益,纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。

专注分享技术,共同学习,共同进步。侵权联系[81616952@qq.com]

Copyright (C)ICode9.com, All Rights Reserved.

ICode9版权所有