Prometheus-AlertManager-Supervisor-钉钉告警

前提

工欲善其事，必先利其器

钉钉添加机器人

文档 - 钉钉开放平台

钉钉开放平台开发文档

https://developers.dingtalk.com/document/app#/serverapi2/qf2nxq

Prometheus-webhook-dingtalk

GitHub - timonwong/prometheus-webhook-dingtalk: DingTalk integration for Prometheus Alertmanager

Generating DingTalk notification from Prometheus AlertManager WebHooks. Precompiled binaries for released versions are available in release page: It's always recommended to use latest stable version available. You can deploy this tool using the Docker image from following registry: Prerequisites Build Clone the repository and build manually: usage: prometheus-webhook-dingtalk [ ] Flags: -h, --help Show context-sensitive help (also try --help-long and --help-man).

https://github.com/timonwong/prometheus-webhook-dingtalk

wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v1.4.0/prometheus-webhook-dingtalk-1.4.0.linux-amd64.tar.gz
tar -zxvf prometheus-webhook-dingtalk-1.4.0.linux-amd64.tar.gz

vim prometheus-webhook-dingtalk.conf

• /xxx/prometheus_webhook_dingtalk/prometheus-webhook-dingtalk：prometheus-webhook-dingtalk启动脚本路径

• -web.enable-ui：启动ui，默认端口为9093

• -ding.profile：钉钉机器人的webhook地址，前面加上webhook1

[program:prometheus_webhook_dingtalk]

command = /xxx/prometheus_webhook_dingtalk/prometheus-webhook-dingtalk --web.enable-ui --ding.profile="webhook1=https://oapi.dingtalk.com/robot/send?access_token=xxx"
directory = /xxx/prometheus_webhook_dingtalk
user = root
startsecs = 5

redirect_stderr = true
stdout_logfile_maxbytes = 10MB
stdout_logfile_backups  = 3
stdout_logfile = /xxx/supervisor/logs/prometheus_webhook_dingtalk.log
autostart=true
autorestart=true

Prometheus

1. 对Alertmanager进行配置，填写alerting对应的targets地址（本文的Alertmanager和Prometheus安装在同一台机子上），例如localhost:9093，默认的端口为9093。

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: [localhost:9093]
      # - alertmanager:9093

rule_files:
   - "first_rules.yml"
  # - "second_rules.yml"

AlterManager

Releases · prometheus/alertmanager

Prometheus Alertmanager. Contribute to prometheus/alertmanager development by creating an account on GitHub.

https://github.com/prometheus/alertmanager/releases

• alert：告警规则的名称。

• expr：基于PromQL表达式告警触发条件，用于计算是否有时间序列满足该条件。

• for：评估等待时间，可选参数。用于表示只有当触发条件持续一段时间后才发送告警。在等待期间新产生告警的状态为pending。

• labels：自定义标签，允许用户指定要附加到告警上的一组附加标签。

• annotations：用于指定一组附加信息，比如用于描述告警详细信息的文字等，annotations的内容在告警产生时会一同作为参数发送到Alertmanager。summary描述告警的概要信息，description用于描述告警的详细信息。同时Alertmanager的UI也会根据这两个标签值，显示告警信息。

groups:
  - name: example
    rules:
    - alert: Server Down
      expr: up{job="xxx"} == 0
      for: 10s
      labels:
        severity: critical
      annotations:
        description: "实例：{{ $labels.instance }} 宕机了"
        summary: "Instance {{ $labels.instance }} down"

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'ops_dingding' #和receivers中定义的name对应
  
receivers:
- name: 'ops_dingding'
  webhook_configs:
  - url: 'http://localhost:8060/dingtalk/webhook1/send' #prometheus_webhook_dingtalk启动后从日志中可以看到用于接收告警信息的webhook的地址
    send_resolved: true
    
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

vim alertmanager.conf

• /xxx/alertmanager/alertmanager：alertmanager启动脚本路径

• -config.file：启动的配置文件

#!bin/sh
[program:alertmanager]

command  = /xxx/alertmanager/alertmanager --config.file=/xxx/alertmanager/alertmanager.yml
directory = /xxx/alertmanager
user = root
startsecs = 5

redirect_stderr = true
stdout_logfile_maxbytes = 10MB
stdout_logfile_backups  = 3
stdout_logfile = /xxx/supervisor/logs/alertmanager.log
autostart=true
autorestart=true