ICode9

精准搜索请尝试: 精确搜索
首页 > 其他分享> 文章详细

Envoy:离群点检测 outlier detection

2021-10-24 21:04:31  阅读:236  来源: 互联网

标签:10 13 06 outlier 09 Envoy detection 2020 GIN


outlier detection

在异常检测领域中,常常需要决定新观察的点是否属于与现有观察点相同的分布(则它称为inlier),或者被认为是不同的(称为outlier)。离群是异常的数据,但是不一定是错误的数据点。

在Envoy中,离群点检测是动态确定上游集群中是否有某些主机表现不正常,然后将它们从正常的负载均衡集群中删除的过程。outlier detection可以与healthy check同时/独立启用,并构成整个上游运行状况检查解决方案的基础。

此处概念不做过多的说明,具体可以参考官方文档与自行google

监测类型

  • 连续的5xx
  • 连续的网关错误
  • 连续的本地来源错误

更多介绍参考官方文档 outlier detection

离群检测测试

说明,此处只能在单机环境测试更多还的参考与实际环境

环境准备

docker-compose 模拟后端5个节点

version: '3'
services:
  envoy:
    image: envoyproxy/envoy-alpine:v1.15-latest
    environment: 
    - ENVOY_UID=0
    ports:
    - 80:80
    - 443:443
    - 82:9901
    volumes:
    - ./envoy.yaml:/etc/envoy/envoy.yaml
    networks:
      envoymesh:
        aliases:
        - envoy
    depends_on:
    - webserver1
    - webserver2
  
  webserver1:
    image: sealloong/envoy-end:latest
    networks:
      envoymesh:
        aliases:
        - myservice
        - webservice
    expose:
    - 90
  webserver2:
    image: sealloong/envoy-end:latest
    networks:
      envoymesh:
        aliases:
        - myservice
        - webservice
    expose:
    - 90
  webserver3:
    image: sealloong/envoy-end:latest
    networks:
      envoymesh:
        aliases:
        - myservice
        - webservice
    expose:
    - 90
  webserver4:
    image: sealloong/envoy-end:latest
    networks:
      envoymesh:
        aliases:
        - myservice
        - webservice
    expose:
    - 90
  webserver5:
    image: sealloong/envoy-end:latest
    networks:
      envoymesh:
        aliases:
        - myservice
        - webservice
    expose:
    - 90
networks:
  envoymesh: {}

envoy 配置文件

admin:
  access_log_path: /dev/null
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 0.0.0.0, port_value: 80 }
    filter_chains:
    - filters:
      - name: envoy_http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: [ "*" ]
              routes:
              - match: { prefix: "/" }
                route: { cluster: local_service }
          http_filters:
          - name: envoy.filters.http.router

  clusters:
  - name: local_service
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: local_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: webservice, port_value: 90 }
    health_checks:
      timeout: 3s
      interval: 90s
      unhealthy_threshold: 5
      healthy_threshold: 5
      no_traffic_interval: 240s
      http_health_check:
        path: "/ping"
        expected_statuses:
          start: 200
          end: 201
    outlier_detection:
      consecutive_5xx: 2
      base_ejection_time: 30s
      max_ejection_percent: 40
      interval: 20s
      success_rate_minimum_hosts: 5
      success_rate_request_volume: 10

配置说明

    outlier_detection:
      consecutive_5xx: 2 # 连续的5xx错误数量
      base_ejection_time: 30s # 弹出主机的基准时间。实际时间等于基本时间乘以主机弹出的次数
      max_ejection_percent: 40 # 可弹出主机集群的最大比例,默认值为10% ,此处为40% 即集群中5个节点的2个节点
      interval: 20s # 间隔时间
      success_rate_minimum_hosts: 5 # 集群中最小主机数量
      success_rate_request_volume: 10 # 在一个时间间隔内中收集请求检测的最小数量

此处为了效果,将主动检测状态时间增加,主机弹出时间增加

路由

/502bad 模拟一个502的错误

运行结果

模拟一些5xx请求和200请求

 workers
envoy_1       | [2020-09-13 06:10:01.093][1][warning][main] [source/server/server.cc:537] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
webserver2_1  | [GIN] 2020/09/13 - 06:10:08 | 200 |      63.272?s |      172.22.0.7 | GET      "/"
webserver5_1  | [GIN] 2020/09/13 - 06:10:10 | 200 |      46.732?s |      172.22.0.7 | GET      "/"
webserver1_1  | [GIN] 2020/09/13 - 06:10:11 | 200 |       45.43?s |      172.22.0.7 | GET      "/"
webserver3_1  | [GIN] 2020/09/13 - 06:10:13 | 502 |      43.858?s |      172.22.0.7 | GET      "/502bad"
webserver4_1  | [GIN] 2020/09/13 - 06:10:14 | 502 |      47.486?s |      172.22.0.7 | GET      "/502bad"
webserver2_1  | [GIN] 2020/09/13 - 06:10:15 | 200 |      15.691?s |      172.22.0.7 | GET      "/"
webserver5_1  | [GIN] 2020/09/13 - 06:10:16 | 200 |      14.719?s |      172.22.0.7 | GET      "/"
webserver1_1  | [GIN] 2020/09/13 - 06:10:16 | 200 |      15.758?s |      172.22.0.7 | GET      "/"
webserver3_1  | [GIN] 2020/09/13 - 06:10:17 | 502 |      15.697?s |      172.22.0.7 | GET      "/502bad"
webserver2_1  | [GIN] 2020/09/13 - 06:10:17 | 502 |      14.002?s |      172.22.0.7 | GET      "/502bad"
webserver5_1  | [GIN] 2020/09/13 - 06:10:17 | 502 |      14.913?s |      172.22.0.7 | GET      "/502bad"
webserver1_1  | [GIN] 2020/09/13 - 06:10:18 | 502 |      14.911?s |      172.22.0.7 | GET      "/502bad"
webserver4_1  | [GIN] 2020/09/13 - 06:10:18 | 502 |      30.429?s |      172.22.0.7 | GET      "/502bad"
webserver5_1  | [GIN] 2020/09/13 - 06:10:19 | 200 |      14.377?s |      172.22.0.7 | GET      "/"
webserver1_1  | [GIN] 2020/09/13 - 06:10:19 | 200 |      14.861?s |      172.22.0.7 | GET      "/"
webserver2_1  | [GIN] 2020/09/13 - 06:10:19 | 200 |      18.924?s |      172.22.0.7 | GET      "/"
webserver5_1  | [GIN] 2020/09/13 - 06:10:19 | 200 |      15.899?s |      172.22.0.7 | GET      "/"
webserver1_1  | [GIN] 2020/09/13 - 06:10:19 | 200 |      24.849?s |      172.22.0.7 | GET      "/"

集群已弹出 20%的节点,健康检查结果为 failed_outlier_check

请求已分配到其余三台节点

30秒后,弹出主机已回复正常

再次模拟请求

30秒后,如在时间间隔内,无新增请求,节点依旧为 failed_outlier_check,有新增请求时恢复。

标签:10,13,06,outlier,09,Envoy,detection,2020,GIN
来源: https://blog.csdn.net/sinat_24092079/article/details/120940760

本站声明: 1. iCode9 技术分享网(下文简称本站)提供的所有内容,仅供技术学习、探讨和分享;
2. 关于本站的所有留言、评论、转载及引用,纯属内容发起人的个人观点,与本站观点和立场无关;
3. 关于本站的所有言论和文字,纯属内容发起人的个人观点,与本站观点和立场无关;
4. 本站文章均是网友提供,不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属;如您发现该文章侵犯了您的权益,可联系我们第一时间进行删除;
5. 本站为非盈利性的个人网站,所有内容不会用来进行牟利,也不会利用任何形式的广告来间接获益,纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。

专注分享技术,共同学习,共同进步。侵权联系[81616952@qq.com]

Copyright (C)ICode9.com, All Rights Reserved.

ICode9版权所有