4.2 Tasks: Pushgateway
Note
As you will be executing some oc commands in the following labs, make sure you are logged in to your OpenShift Cluster.
You can copy the login Command from the OpenShift UI:
- Browse to http://LOCALHOST_OPENSHIFT
- Click on your name in the top right
Copy login command- Replace
6443with443
Task 4.2.2 - Push metrics to Pushgateway
In this task you’re going to push metrics to the Pushgateway. This is what you would normally do, after a cronjob has completed successfully.
Documentation on how the Pushgateway is implemented at Baloise can be found here: 07 - Push metrics to your Prometheus instance .
In order to push
metrics to the Pushgateway, you can simply send an HTTP POST or PUT request, with the actual metric we want to push as content.
When pushing metrics to the Pushgateway, you always have to specify the job, therefore the URL Path looks like this:
http://localhost:9091/metrics/job/<JOB_NAME>{/<LABEL_NAME>/<LABEL_VALUE>}
If we want to push the metric prometheus_training_labs_completed_total with the value 4 and the job prometheus_training to the Pushgateway, we can do that by creating the following Kubernetes Job:
oc -n <team>-monitoring create job --image=quay.balgroupit.com/library/toolkit:ubuntu-20.04 pushgw-example1 -- \
bash -c "echo 'prometheus_training_labs_completed_total 4' | curl --data-binary @- http://pushgateway.<team>-monitoring:9091/metrics/job/prometheus_training"
Command Explanation
If you are not very familiar with oc create job. The above command does the following:
oc -n ... create jobcreates an adhoc kubernetes job--image=specifies, which image the container will use. We will use the toolkit container because it provides bash and curl.pushgw-example1is the name of the jobbash -c "..."is the command, the job should execute
Verify the metric in the Prometheus web UI
. It may take up to 30s ( Depending on the scrape_interval) to be available in Prometheus.
Push
the following metric (notice the instance label) to the Pushgateway and make sure the metric gets scraped by Prometheus
# TYPE some_metric_total counter
# HELP This is just an example metric.
some_metric_total{job="prometheus_training",instance="myinstance"} 42
Hints
To push a metric to the Pushgateway, which will then be scraped by Prometheus, we can simply create the following job. Note the actual content of the HTTP request, is exactly the metric we want Prometheus to scrape.
Execute the following command to push the metric to your Pushgateway:
oc -n <team>-monitoring create job --image=quay.balgroupit.com/library/toolkit:ubuntu-20.04 pushgw-example2 -- \
bash -c "cat <<EOF | curl --data-binary @- http://pushgateway.<team>-monitoring:9091/metrics/job/prometheus_training/instance/myinstance
# TYPE some_metric_total counter
# HELP This is just an example metric.
some_metric_total 42
EOF"
Command Explanation
If you are not very familiar with the Linux shell, the above command does the following:
- the
catcommand reads the actual metric and pipes it tostdin - curl sends a HTTP POST request to the URL http://localhost:9091/metrics/job/prometheus_training/instance/myinstance
with the –data-binary parameter set to
stdin(the actual metric)
Verify the metric in the Prometheus web UI
. It may take up to 30s (depending on the scrape_interval) to be available in Prometheus.
Task 4.2.3 - Delete Pushgateway metrics
By sending HTTP delete requests to the same endpoint, we can delete metrics from the Pushgateway.
Note
Metrics pushed to the Pushgateway are not automatically purged until you manually delete them via the API or the process restarts. If you persist the metrics with--persistence.file, you should ensure that you have set up a job that cleans up the metrics on a regular basis.According to the official Pushgateway documentation you can delete either metrics for specific label combinations (exact match required) or all metrics.
Delete the pushed metrics from the Pushgateway.
Hints
To delete the metrics for the job prometheus_training, you can simply execute the following command:
oc -n <team>-monitoring create job --image=quay.balgroupit.com/library/toolkit:ubuntu-20.04 pushgw-delete -- \
curl -X DELETE http://pushgateway.<team>-monitoring:9091/metrics/job/prometheus_training
Note
This will delete metrics with the label set {job="prometheus_training"} but not {job="prometheus_training",another_label="value"} since the delete methode requires an exact label match.
The Pushgateway pod has no persistence, so you can delete all metrics stored in Pushgateway by deleting the pod.
oc -n <team>-monitoring delete pod -l app.kubernetes.io/component=pushgateway
Remove the created examples jobs.
oc -n <team>-monitoring delete jobs pushgw-delete pushgw-example1 pushgw-example2