基于 consul 注册中心自动发现

consul 是一款基于 golang 开发的开源工具，主要面向分布式，服务化的系统提供服务注册、服务发现和配置管理的服务，提供服务注册/发现、健康检查、Key/Value 存储、多数据中心和分布式一致性保证等功能。

服务部署

networks:
  monitoring:
    driver: bridge

services:
  consul:
    image: consul:1.15.4
    volumes:
      - ./consul_configs:/consul/config
      - ./consul_data:/consul/data/
    networks:
      - monitoring
    ports:
      - 8500:8500
    command: ["consul","agent","-dev","-bootstrap","-config-dir","/consul/config","-data-dir","/consul/data","-ui","-log-level","INFO","-bind","127.0.0.1","-client","0.0.0.0"]

  consul-exporter:
    image: prom/consul-exporter:v0.8.0
    networks:
      - monitoring
    ports:
      - 9107:9107
    command:
      - "--consul.server=consul:8500"
    depends_on:
      - consul

编辑 Prometheus.yml

需要注意，使用consul自动发现时，需要在job中通过标签来匹配对应的target，例如：

使用 consul 自动发现时，需要在 job 中通过标签来匹配对应的 target

  - job_name: consul
    consul_sd_configs:
       - server: '10.10.0.2:8500'
    relabel_configs:
    - source_labels: ['__meta_consul_service_metadata_project_code']
      target_label: 'project_code'
    - source_labels: ['__meta_consul_service_metadata_env_code']
      target_label: 'env_code'
    - source_labels: ['__meta_consul_service_metadata_role_code']
      target_label: 'role_code'
    - source_labels: ['__meta_consul_service_metadata_server_code']
      target_label: 'server_code'
    - source_labels: ['__meta_consul_service_metadata_server_name']
      target_label: 'server_name'
    - source_labels: ['__meta_consul_service_metadata_remote_ip']
      target_label: 'remote_ip'
    - source_labels: ['__meta_consul_service_metadata_monitor_type']
      target_label: 'monitor_type'
    - source_labels: ['__meta_consul_service_metadata_monitor_port']
      target_label: 'monitor_port'
    - source_labels: [__meta_consul_tags]
      regex: '(.*port.*|.*ping.*)'
      action: drop

服务注册到 consul

1. api 式注册演示

1. 准备 json 文件 node-exporter.json

{
  "ID": "node-exporter-10.10.0.2",
  "Name": "node-exporter-10.10.0.2",
  "Tags": ["node-exporter", "v1"],    # 包含的标签
  "Address": "10.10.0.1",
  "Port": 9100,
  "Meta": {
    "node-exporter_version": "9"    # 元数据，可自定义
  },
  "EnableTagOverride": false,
  "Check": {    # 检查健康状态的方法
    "http": "http://10.10.0.2:9100/metrics",  #     "Args": ["/usr/local/bin/check_redis.py"], 也可以是 脚本
    "interval": "5s",
    "Timeout": "5s"
  },
  "Weights": {
    "Passing": 1,
    "Warning": 1
  }
}

2. 注册服务

curl -XPUT --data @node-exporter.json http://10.10.0.2:8500/v1/agent/service/register

通过 consul 自动发现的 target 会有很多 __meta_consul 开头的标签，我们可以通过 relabel 来重新利用这些标签。 常用的 api 指令：

# 查看当前所有注册的service
curl http://10.10.0.2:8500/v1/agent/services

# 查看tomcat service的健康状态
curl http://10.10.0.2:8500/v1/agent/health/service/name/tomcat

# 注册服务，需提前准备好json文件
curl -XPUT --data @grafana.json http://10.10.0.2:8500/v1/agent/service/register

# 注销服务
curl -XPUT http://10.10.0.2:8500/v1/agent/service/deregister/grafana

consul 命令方式注册演示

准备 @.json 文件，同一类型的 target 可以写到一个 json 文件中，便于编辑注册。

{
  "services": [
    {
      "ID": "node-exporter-10.10.0.1",
      "Name": "node-exporter-10.10.0.1",
      "Tags": ["node-exporter", "v1"],    
      "Address": "10.10.0.1",
      "Port": 9100,
      "Meta": {
        "node-exporter_version": "9"
      },
      "Check": {
        "http": "http://10.10.0.1:9100/metrics",
        "interval": "5s",
        "Timeout": "5s"
      },
      "Weights": {
        "Passing": 1,
        "Warning": 1
      }
    },
    {
      "ID": "node-exporter-10.10.0.2",
      "Name": "node-exporter-10.10.0.2",
      "Tags": ["node-exporter", "v1"],    
      "Address": "10.10.0.2",
      "Port": 9100,
      "Meta": {
        "node-exporter_version": "9"    
      },
      "Check": {
        "http": "http://10.10.0.2:9100/metrics",
        "interval": "5s",
        "Timeout": "5s"
      },
      "Weights": {
        "Passing": 1,
        "Warning": 1
      }
    },
    {
      "ID": "node-exporter-10.10.0.4",
      "Name": "node-exporter-10.10.0.4",
      "Tags": ["node-exporter", "v1"],
      "Address": "10.10.0.4",
      "Port": 9100,
      "Meta": {
        "node-exporter_version": "9"
      },
      "Check": {
        "http": "http://10.10.0.4:9100/metrics",
        "interval": "5s",
        "Timeout": "5s"
      },
      "Weights": {
        "Passing": 1,
        "Warning": 1
      }
    }
  ]
}

执行 config 重新加载

docker compose exec -it consul consul reload

将 node-exporter 使用 consul 自动采集

1. 修改 prometheus 配置

1. vim prometheus.yml

  - job_name: 'node-exporter'
    consul_sd_configs:
    - server: "10.10.0.2:8500"
      tags:
      - "node-exporter"    # 匹配在 consul 注册的服务中带有 node-exporter 标签的 service
      refresh_interval: 2m
    scrape_interval: 15s

2. 添加 consul 中 json 配置文件

consul_configs/node-exporter.json

vim consul_configs/node-exporter.json

{
  "services": [
    {
      "ID": "node-exporter-10.10.0.1",
      "Name": "node-exporter-10.10.0.1",
      "Tags": ["node-exporter", "v1"],
      "Address": "10.10.0.1",
      "Port": 9100,
      "Meta": {
        "node-exporter_version": "9"
      },
      "Check": {
        "http": "http://10.10.0.1:9100/metrics",
        "interval": "5s",
        "Timeout": "5s"
      },
      "Weights": {
        "Passing": 1,
        "Warning": 1
      }
    },
    {
      "ID": "node-exporter-10.10.0.2",
      "Name": "node-exporter-10.10.0.2",
      "Tags": ["node-exporter", "v1"],
      "Address": "10.10.0.2",
      "Port": 9100,
      "Meta": {
        "node-exporter_version": "9"
      },
      "Check": {
        "http": "http://10.10.0.2:9100/metrics",
        "interval": "5s",
        "Timeout": "5s"
      },
      "Weights": {
        "Passing": 1,
        "Warning": 1
      }
    },
    {
      "ID": "node-exporter-10.10.0.4",
      "Name": "node-exporter-10.10.0.4",
      "Tags": ["node-exporter", "v1"],
      "Address": "10.10.0.4",
      "Port": 9100,
      "Meta": {
        "node-exporter_version": "9"
      },
      "Check": {
        "http": "http://10.10.0.4:9100/metrics",
        "interval": "5s",
        "Timeout": "5s"
      },
      "Weights": {
        "Passing": 1,
        "Warning": 1
      }
    }
  ]
}

3. 重启服务

systemctl restart prometheus.service
docker exec -it consul consul reload

将 cAdvisor 使用 consul 自动采集

1. 修改 prometheus 配置

1. vim prometheus.yml

  - job_name: "cAdvisor"
    consul_sd_configs:
    - server: "10.10.0.2:8500"
      tags:
      - "cAdvisor"    # 匹配在consul注册的服务中带有nodes标签的service
      refresh_interval: 2m
    scrape_interval: 15s

2. 添加 consul 中 json 配置文件

consul_configs/cAdvisor.json

vim consul_configs/cAdvisor.json

{
  "services": [
    {
      "ID": "cAdvisor-10.10.0.1",
      "Name": "cAdvisor-10.10.0.1",
      "Tags": ["cAdvisor", "v1"],
      "Address": "10.10.0.1",
      "Port": 8080,
      "Meta": {
        "cAdvisor_version": "9"
      },
      "Check": {
        "http": "http://10.10.0.1:8080/metrics",
        "interval": "5s",
        "Timeout": "5s"
      },
      "Weights": {
        "Passing": 1,
        "Warning": 1
      }
    },
    {
      "ID": "cAdvisor-10.10.0.2",
      "Name": "cAdvisor-10.10.0.2",
      "Tags": ["cAdvisor", "v1"],
      "Address": "10.10.0.2",
      "Port": 8080,
      "Meta": {
        "cAdvisor_version": "9"
      },
      "Check": {
        "http": "http://10.10.0.2:8080/metrics",
        "interval": "5s",
        "Timeout": "5s"
      },
      "Weights": {
        "Passing": 1,
        "Warning": 1
      }
    }
  ]
}

3. 重启服务

systemctl restart prometheus.service
docker exec -it consul consul reload

查看效果

1. 查看 consul 自动注册效果图

2. 查看 prometheus 自动注册效果图

ansible 批量管理主机

ansible 批量部署 node_exporter

node_exporter.service 文件

node_exporter.service

[Unit]
Description=node_exporter for linux server
Documentation=https://prometheus.io/
After=network-online.target
[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure
[Install]
WantedBy=multi-user.target

hosts 文件

[root@ansible ansible]# vi hosts
[webservers]
10.10.0.2 name=web1

[dbservers]
10.10.0.1 name=db1

[networkservers]
10.10.0.4 name=network1

consul-register.sh 文件

consul-register.sh

#!/bin/bash
service_name=$1
instance_id=${service_name}
ip=$2
port=$3
tags=$4

curl -X PUT -d '{"id": "'"$instance_id"'","name": "'"$service_name"'","address": "'"$ip"'","port": '"$port"',"tags": ["'"$tags"'"],"checks": [{"http": "http://'"$ip"':'"$port"'","interval": "5s"}]}' http://10.10.0.2:8500/v1/agent/service/register

playbook.yaml

playbook.yaml

- hosts: webservers
  gather_facts: no
  vars:
    exporter_port: 9100
  tasks:
  - name: 推送二进制文件
    unarchive: src=node_exporter-1.8.2.linux-amd64.tar.gz dest=/usr/local
  - name: 重命名
    shell: |
         cd /usr/local
         if [ ! -d node_exporter ];then
             mv node_exporter-1.8.2.linux-amd64 node_exporter
         fi
 #- name: 推送配置文件
 #  copy: src=config.yml dest=/usr/local/node_exporter
  - name: 拷贝systemd文件
    copy: src=node_exporter.service dest=/usr/lib/systemd/system
  - name: 启动服务
    systemd: name=node_exporter state=restarted enabled=yes daemon_reload=yes
  - name: 推送注册脚本
    copy: src=consul-register.sh dest=/usr/local/bin/
  - name: 注册当前节点   
    # 服务名 实例名 IP 端口 
    shell: /bin/bash /usr/local/bin/consul-register.sh {{ group_names[0] }} {{ name }} {{ inventory_hostname }} {{ exporter_port }}
    # bash 1.sh  node-exporter-10.10.0.5 10.10.0.5  9100 node-exporter

ansible-playbook -i hosts playbook.yaml -uroot -k 

基于 consul 的服务发现

将 node_exporter 所在机器的 IP 和 端口 注册到 consul 里

prometheus 从 consul 获取所有 ip 和端口自动加入监控

https://github.com/ansible-collections/ansible-consul