7. Alerting with Alertmanager on Prometheus Basics Training

7.1 Tasks: Enable and configure Alertmanager

Mon, 01 Jan 0001 00:00:00 +0000

Task 7.1.1: Install Alertmanager and Thanosruler

Update your monitoring application (charts/user-monitoring/values.yaml) and update the alertmanager.enabled and ruler.enabled flag to true:

charts/user-monitoring/values.yaml:

user: <user> # Replace me
# prometheus
prometheus:
 enabled: true
# thanos-query
query:
 enabled: true
# grafana
grafana:
 enabled: true
 datasources:
 - name: prometheus
 access: proxy
 editable: false
 type: prometheus
 url: http://prometheus-operated:9090
# blackboxexporter
blackboxexporter:
 enabled: true
# pushgateway
pushgateway:
 enabled: true
# alertmanager
alertmanager:
 enabled: true
# thanos-ruler
ruler:
 enabled: true

Commit and push the changes.

Verify the installation and sync process in the ArgoCD UI . Or execute the following command:

7.1 Tasks: Setup custom alerting rules

Mon, 01 Jan 0001 00:00:00 +0000

Task 7.1.1: Add alerting rules

Note

Alertmanager will automatically send mails to the defined responsible email address in the teams root configuration when you set the label severity=critical in your PrometheusRule. To change this behaviour and/or add Alerting to MS Teams, check the documentation 03 - Setup custom alerting rules in Confluence.

The Prometheus Operator allows you to configure Alerting Rules (PrometheusRules). This enables OpenShift users to configure and maintain alerting rules for their projects. Furthermore it is possible to treat Alerting Rules like any other Kubernetes resource and lets you manage them in Helm or Kustomize. A PrometheusRule has the following form:

7.2 Tasks: Alertmanager

Mon, 01 Jan 0001 00:00:00 +0000

Task 7.2.1: Configure Slack as alert receiver

As part of our lab setup we already configured an alert receiver for Alertmanager alerts.

We deployed the Mail catcher , which is a very simple component. We can send Emails to the server, and those are then displayed in the Web UI. The Emails are not send anywhere, and this setup should only be used for demo purposes.

When we enabled the alertmanager configuration in the charts/user-monitoring/values.yaml we also deployed the following AlertmanagerConfig Custom Resource

7.3 Tasks: Alertrules and alerts

Mon, 01 Jan 0001 00:00:00 +0000

Note

For doing the alerting lab it’s useful to have a “real” application so that alerts can be provoked. You will use the demo app installed in your monitoring-demo namespace for this purpose.

The example app exposes metrics which are already scraped from lab one.

{job="example-web-python"}

The Prometheus Operator allows you to configure Alerting Rules (PrometheusRules). This enables Kubernetes users to configure and maintain alerting rules for their projects. Furthermore it is possible to treat Alerting Rules like any other Kubernetes resource and lets you manage them in Helm or Kustomize. A PrometheusRule has the following form: