8. Advanced Topics on Prometheus Basics Training

8.1 Tasks: Relabeling

Mon, 01 Jan 0001 00:00:00 +0000

Relabeling in Prometheus can be used to perform numerous tasks using regular expressions, such as

adding, modifying or removing labels to/from metrics or alerts,
filtering metrics based on labels, or
enabling horizontal scaling of Prometheus by using hashmod relabeling.

It is a very powerful part of the Prometheus configuration, but it can also get quite complex and confusing. Thus, we will only take a look at some basic/simple examples.

There are four types of relabelings:

8.2 Tasks: Recording Rules

Mon, 01 Jan 0001 00:00:00 +0000

Prometheus recording rules allow you to precompute queries at a defined interval (global.evaluation_interval or interval in rule_group) and save them to a new set of time series.

In this lab you are going to create your first own recording rules. Recording rules are very useful when it comes to queries, which are very complex and take a long time to compute. The naming convention dictates to use the following format when naming recording rules level:metric:operation. Additional information regarding naming best-practices can be found here .

8.3 Tasks: Troubleshoot Kubernetes Service Discovery

Mon, 01 Jan 0001 00:00:00 +0000

Task 8.3.1: Troubleshooting Kubernetes Service Discovery

We will now deploy an application with an error in the monitoring configration.

Deploy Loki in the monitoring namespace.

Create a deployment training_loki-deployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
 app: loki
 name: example-loki
spec:
 replicas: 1
 selector:
 matchLabels:
 app: loki
 template:
 metadata:
 labels:
 app: loki
 spec:
 containers:
 - image: quay.balgroupit.com/acend/loki
 imagePullPolicy: Always
 name: loki
 volumeMounts:
 - mountPath: /loki/chunks
 name: chunks
 - mountPath: /loki/boltdb-shipper-cache
 name: boltdb-shipper-cache
 - mountPath: /loki/boltdb-shipper-active
 name: shipper-active
 - mountPath: /loki/wal
 name: wal
 - mountPath: /loki/compactor
 name: compactor
 restartPolicy: Always
 volumes:
 - name: chunks
 emptyDir: {}
 - name: boltdb-shipper-cache
 emptyDir: {}
 - name: shipper-active
 emptyDir: {}
 - name: wal
 emptyDir: {}
 - name: compactor
 emptyDir: {}

Create a Service training_service-loki.yaml.

8.3 Tasks: Troubleshoot Kubernetes Service Discovery

Mon, 01 Jan 0001 00:00:00 +0000

Task 8.3.1: Troubleshooting Kubernetes Service Discovery

We will now deploy an application with an error in the monitoring configration.

Deploy Loki in your namespace by adding the following files to your git repo in the user-demo/ folder:

---
apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
 app: loki
 name: loki
spec:
 replicas: 1
 selector:
 matchLabels:
 app: loki
 template:
 metadata:
 labels:
 app: loki
 spec:
 containers:
 - image: mirror.gcr.io/grafana/loki:latest
 name: loki

---
apiVersion: v1
kind: Service
metadata:
 labels:
 app: loki
 name: loki
spec:
 ports:
 - name: http
 port: 3100
 protocol: TCP
 targetPort: 3100
 selector:
 app: loki
 type: NodePort

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 labels:
 app.kubernetes.io/name: loki
 name: loki
spec:
 endpoints:
 - interval: 30s
 port: http
 scheme: http
 path: /metrics
 selector:
 matchLabels:
 prometheus-monitoring: 'true'

Add, commit and push changes to your git repository and let ArgoCD sync your app again.

8.4 Tasks: Cleanup your monitoring workspace

Mon, 01 Jan 0001 00:00:00 +0000

Task 8.4.1: Cleanup your monitoring workspace

Make sure to remove all files with the training_ prefix in your monitoring directory.