3.2 Tasks: Grafana dashboards

• Navigate to https://grafana.com/grafana/dashboards/1860-node-exporter-full/ and download the JSON
• On your Grafana web UI
  • Navigate to Dashboards (Icon with the four squares on the left navigation menu) > New > Import
  • Paste the contents of the downloaded JSON into the Import via panel json field
  • Hit Load
• Choose your thanos-querier data source and hit Import

• Open the dashboard time control (to the upper right)
  • Set From to now-10m
  • Set To to now-5m
  • Hit Apply time range

• Navigate to Dashboards (Icon with the four squares on the left navigation menu)> New > New Dashboard
  • Select Add visualization

• Select the thanos-querier data source

• In general, metrics can be built using the Grafana Query Builder or using “plain” PromQL queries. You can easily switch between these two at the top right of the query window. Going forward, we will use plain PromQL queries.
• Add the expression from above into the text field right next to the Metrics Browser dropdown
• Set the panel title to CPU Utilisation under Panel options > Title (you may need to open the options pane with the < button on the right hand side just below the Apply button)
• Save the dashboard and give it the name happy_little_dashboard

• Hit Add (top navigation menu) > Visualization

• Select the thanos-querier data source
• Add the expression from above into the text field right next to the Metrics Browser dropdown
• Set the panel title to Memory Available under Panel options > Title (you may need to open the options pane with the < button on the right hand side just below the Apply button)

• Define unit under Standard options > Unit > Misc / Percent (0-100)
• Choose Gauge in the dropdown menu just below the Apply button

• Add 20 and 50 thresholds under Thresholds
  • Choose Red for Base
  • Choose Orange for 20
  • Choose Green for 50

• Save the dashboard

• First, we create the variable. Hit the little gear icon on the top right corner of the dashboard
  • Select Variables in the menu on the left > Add Variable
  • As we want the available values for the variable to be calculated dynamically, we will use a PromQL query for this as well. Thus, choose Query in the dropdown menu for variable type
  • Set Name to disk
  • Set Label to Select disk
  • Select the thanos-querier data source
  • Set Query Type to Label values
  • Choose the label mountpoint
  • As we are only interested in the mountpoint label of our linux VM, enter the metric node_filesystem_avail_bytes{instance="prometheus-training.balgroupit.com:9100"}
  • The preview at the bottom should now already show the different mountpoints of the server
  • Leave everything else as is, apply and navigate back to the dashboard
• Now, let’s use the variable in a new panel. Hit Add (top navigation menu) > Visualization
  • Select the thanos-querier data source
  • Add the query 100 - ((node_filesystem_avail_bytes{mountpoint="$disk", instance="prometheus-training.balgroupit.com:9100"} * 100) / node_filesystem_size_bytes{mountpoint="$disk", instance="prometheus-training.balgroupit.com:9100"}) to the Metrics browser field
  • Set the panel title to Disk Available under Panel options > Title (you may need to open the options pane with the < button on the right hand side just below the Apply button)
  • Define unit under Standard options > Unit > Misc / Percent (0-100)
  • Choose Stat in the dropdown menu just below the Apply button
  • Add 10 and 25 thresholds under Thresholds
    • Choose Red for Base
    • Choose Orange for 10
    • Choose Green for 25
• Save the dashboard

• On your Grafana web UI
  • Navigate to your Dashboard happy_little_dashboard
  • Select Dashboard settings (Icon on the top navigation menu that looks like a gear)
  • Select <> JSON Model
  • Copy the JSON content and save the file training_dashboard.yaml in the monitoring folder:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    dashboard: <team>-infra
  name: training-dashboard
data:
  training-dashboard.json: |-
    {
      Your dashboard JSON definition
    }

  training-dashboard.json: |-
    {
      "annotations": {
        "list": [
          {
...

To ensure that the dashboard provisioning works as specified, try deleting the dashboard using the Grafana user interface.

• On your Grafana web UI
  • Navigate to your Dashboard happy_little_dashboard
  • Select Dashboard settings (Icon on the top navigation menu that looks like a gear)
  • Click on Delete Dashboard

You should see a warning message that prevents deleting provisioned dashboards.