4.3 Tasks: Exporter as a sidecar

Task 4.3.1: Deploy a database and use a sidecar container to expose metric

Task description:

As we’ve learned in Lab 4 - Prometheus exporters when applications do not expose metrics in the Prometheus format, there are a lot of exporters available to convert metrics into the correct format. In Kubernetes this is often done by deploying so called sidecar containers along with the actual application.

Use the following command to deploy a MariaDB database your monitoring or application namespace on CAAST.

Create the following deployment (training_baloise_mariadb-deployment.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mariadb
  labels:
    app: mariadb
spec:
  selector:
    matchLabels:
      app: mariadb
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      containers:
        - image: quay.balgroupit.com/acend/mariadb:10.5
          name: mariadb
          args:
            - --ignore-db-dir=lost+found
          env:
            - name: MYSQL_USER
              valueFrom:
                secretKeyRef:
                  key: database-user
                  name: mariadb
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: database-password
                  name: mariadb
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: database-root-password
                  name: mariadb
            - name: MYSQL_DATABASE
              valueFrom:
                secretKeyRef:
                  key: database-name
                  name: mariadb
          livenessProbe:
            tcpSocket:
              port: 3306
          ports:
            - containerPort: 3306
              name: mariadb
          volumeMounts:
            - mountPath: /var/lib/mysql
              name: data
      volumes:
        - name: data
          emptyDir: {}

Create the following secret (training_baloise_mariadb-secret.yaml)

apiVersion: v1
kind: Secret
metadata:
  name: mariadb
  labels:
    app: mariadb
data:
  database-name: YWNlbmRleGFtcGxlZGI= # example
  database-password: bXlzcWxwYXNzd29yZA== # example
  database-root-password: bXlzcWxyb290cGFzc3dvcmQ= # example
  database-user: YWNlbmQtdXNlcg== # example

Create the following service (training_baloise_mariadb-service.yaml)

apiVersion: v1
kind: Service
metadata:
  name: mariadb
  labels:
    app: mariadb
    prometheus-monitoring: 'true'
spec:
  ports:
    - name: mariadb
      port: 3306
      protocol: TCP
      targetPort: 3306
  selector:
    app: mariadb

This will create a Secret (username password to access the database), a Service and the Deployment .

Hints

First we need to alter the MariaDB deployment training_baloise_mariadb-deployment.yaml by adding the MariaDB exporter as a second container.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mariadb
  labels:
    app: mariadb
spec:
  selector:
    matchLabels:
      app: mariadb
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      containers:
        - image: quay.io/prometheus/mysqld-exporter:v0.14.0
          name: mariadb-exporter
          env:
            - name: MYSQL_USER
              valueFrom:
                secretKeyRef:
                  key: database-user
                  name: mariadb
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: database-password
                  name: mariadb
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: database-root-password
                  name: mariadb
            - name: MYSQL_DATABASE
              valueFrom:
                secretKeyRef:
                  key: database-name
                  name: mariadb
            - name: DATA_SOURCE_NAME
              value: $(MYSQL_USER):$(MYSQL_PASSWORD)@(127.0.0.1:3306)/$(MYSQL_DATABASE)
          ports:
            - containerPort: 9104
              name: mariadb-exp
        - image: quay.balgroupit.com/acend/mariadb:10.5
          name: mariadb
          args:
            - --ignore-db-dir=lost+found
          env:
            - name: MYSQL_USER
              valueFrom:
                secretKeyRef:
                  key: database-user
                  name: mariadb
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: database-password
                  name: mariadb
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: database-root-password
                  name: mariadb
            - name: MYSQL_DATABASE
              valueFrom:
                secretKeyRef:
                  key: database-name
                  name: mariadb
          livenessProbe:
            tcpSocket:
              port: 3306
          ports:
            - containerPort: 3306
              name: mariadb
          volumeMounts:
            - mountPath: /var/lib/mysql
              name: data
      volumes:
        - name: data
          emptyDir: {}

Then extend the service training_baloise_mariadb-service.yaml by adding a second port for the MariaDB exporter.

apiVersion: v1
kind: Service
metadata:
  name: mariadb
  labels:
    app: mariadb
    prometheus-monitoring: 'true'
spec:
  ports:
    - name: mariadb
      port: 3306
      protocol: TCP
      targetPort: 3306
    - name: mariadb-exp
      port: 9104
      protocol: TCP
      targetPort: 9104
  selector:
    app: mariadb

Then we also need to create a new ServiceMonitor training_baloise_mariadb-servicemonitor.yaml.

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: mariadb
  name: mariadb
spec:
  endpoints:
    - interval: 60s
      port: mariadb-exp
      scheme: http
      path: /metrics
  selector:
    matchLabels:
      prometheus-monitoring: 'true'

Verify that the target gets scraped in the Prometheus user interface . Target name: serviceMonitor/<team>-monitoring/mariadb/0 (It may take up to a minute for Prometheus to load the new configuration and scrape the metrics).

Task 4.3.2: generic-chart MariaDB deployment (optional)

Task description:

Hints

Create an application on CAAST and deploy the following configuration.

Chart.yaml:

apiVersion: v2
name: example-mariadb
version: 1.0.0
dependencies:
  - name:  generic-chart
    version: 3.13.1
    repository: https://charts.shapp.os1.balgroupit.com/shared/release/
    alias: example-mariadb

values.yaml:

example-mariadb:
  replicaCount: 1
  image:
    repository: quay.balgroupit.com/acend/mariadb
    tag: 10.5
  additionalContainers:
    - name: "mariadb-exporter"
      image: quay.io/prometheus/mysqld-exporter:v0.14.0
      env:
        - name: MYSQL_USER
          valueFrom:
            secretKeyRef:
              key: database-user
              name: mariadb
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              key: database-password
              name: mariadb
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: database-root-password
              name: mariadb
        - name: MYSQL_DATABASE
          valueFrom:
            secretKeyRef:
              key: database-name
              name: mariadb
        - name: DATA_SOURCE_NAME
          value: $(MYSQL_USER):$(MYSQL_PASSWORD)@(127.0.0.1:3306)/$(MYSQL_DATABASE)
  network:
    mariadb-exp:
      servicePort: 9104
      serviceMonitor: {}
    mariadb:
      servicePort: 3306
  env:
    - name: MYSQL_USER
      valueFrom:
        secretKeyRef:
          key: database-user
          name: mariadb
    - name: MYSQL_PASSWORD
      valueFrom:
        secretKeyRef:
          key: database-password
          name: mariadb
    - name: MYSQL_ROOT_PASSWORD
      valueFrom:
        secretKeyRef:
          key: database-root-password
          name: mariadb
    - name: MYSQL_DATABASE
      valueFrom:
        secretKeyRef:
          key: database-name
          name: mariadb
  volumes:
    - name: data
      emptyDir: {}
  volumeMounts:
    - mountPath: /var/lib/mysql
      name: data

templates/secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: mariadb
  labels:
    app: mariadb
data:
  database-name: YWNlbmRleGFtcGxlZGI= # example
  database-password: bXlzcWxwYXNzd29yZA== # example
  database-root-password: bXlzcWxyb290cGFzc3dvcmQ= # example
  database-user: YWNlbmQtdXNlcg== # example

Verify that the target gets scraped in the Prometheus user interface . Target name: application-metrics/mariadb/0 (it may take up to a minute for Prometheus to load the new configuration and scrape the metrics).

Make sure to remove the files Chart.yaml, values.yaml and templates/secret.yaml once finished.