Adding Prometheus metrics to an app

This guide shows you how to collect metrics of a Great Bear application you are developing. Metrics allow you to observe and monitor your application. The metrics are collected by Prometheus and they can be visualized on Grafana. For details, see Observing Great Bear.

The following steps extend the Bear Echo application with basic metrics, and assume that you have already followed the previous sample on Adding Liveness Probe to an app.

Add metrics endpoint to the application

First you need to add a new metrics endpoint to your application so when Prometheus scrapes the HTTP endpoint of the application, the client library sends the current state of all tracked metrics to the Prometheus server.

This example uses the official Python client library for Prometheus and its example on adding metrics to a Flask application.

  1. Add the metrics endpoint to the server.

    from werkzeug.middleware.dispatcher import DispatcherMiddleware
    from prometheus_client import make_wsgi_app
    
    app.wsgi_app = DispatcherMiddleware(app.wsgi_app, {
        '/metrics': make_wsgi_app()
    })
    
  2. Add a metric to the application. This example uses a counter metric type, that is incremented by one every time the base url is accessed.

    from prometheus_client import Counter
    
    _COUNTER_METRIC = Counter('num_of_queries', 'Number of queries')
    
    @app.route(_BASE_URL, methods=['GET'])
    def echo():
        _COUNTER_METRIC.inc()
        return(f'<hr><center><h1>{_ECHO_TEXT}</h1><center><hr>')
    

    For reference, here’s the extended server.py of the Bear Echo application:

    # bear-echo/server.py
    
    import os
    import sys
    import importlib
    from flask import Flask
    from prometheus_client import Counter
    from werkzeug.middleware.dispatcher import DispatcherMiddleware
    from prometheus_client import make_wsgi_app
    
    
    _HTTP_PORT: int = int(os.getenv('HTTP_PORT') or '11777')
    _BASE_URL: str = os.getenv('BASE_URL') or '/'
    _ECHO_TEXT: str = os.getenv('ECHO_TEXT') or 'Hello World'
    
    _COUNTER_METRIC = Counter('num_of_queries', 'Number of queries')
    
    app = Flask(__name__)
    
    app.wsgi_app = DispatcherMiddleware(app.wsgi_app, {
        '/metrics': make_wsgi_app()
    })
    
    
    @app.route(_BASE_URL, methods=['GET'])
    def echo():
        _COUNTER_METRIC.inc()
        return(f'<hr><center><h1>{_ECHO_TEXT}</h1><center><hr>')
    
    
    @app.route("/healthz")
    def healthz():
        return "OK"
    
    
    def _main():
        print('ECHO_TEXT: ' + _ECHO_TEXT)
        print(f'Started on 0.0.0.0:{_HTTP_PORT}{_BASE_URL}')
        app.run(host='0.0.0.0', port=_HTTP_PORT)
    
    
    if __name__ == '__main__':
        _main()
    

Update the Dockerfile

  1. The official example recommends using uwsgi to run the application. Update the Dockerfile of the application to do that.

    # bear-echo/Dockerfile
    
    FROM python:3.8-alpine
    
    WORKDIR /echo
    
    COPY requirements.txt .
    
    RUN set -e; \
        apk add --no-cache --virtual .build-deps \
        gcc \
        libc-dev \
        linux-headers \
        ; \
        pip install --no-cache-dir -r requirements.txt; \
        apk del .build-deps;
    
    COPY server.py ./
    
    EXPOSE 11777
    
    CMD uwsgi --http 0.0.0.0:11777 --wsgi-file server.py --callable app
    
  2. You can see that the Dockerfile now uses a bear-echo/requirements.txt file to keep track of the dependencies used by the application server. Create this file with the following content:

    # bear-echo/requirements.txt
    
    flask
    prometheus-client
    uwsgi
    

Update the Chart to make metrics available

Providing metrics is optional, the user can enable metrics when deploying the app. If metrics are enabled, a podmonitor is deployed. Prometheus discovers the metrics endpoint of your app via the podmonitor. You can customize several settings related to the scraping.

  1. In order to add the podmonitor to the Chart, create the bear-echo/GBAP/templates/podmonitor.yaml file with the following content.

    {{- if eq .Values.config.MONITORING "on" }}
    apiVersion: monitoring.coreos.com/v1
    kind: PodMonitor
    metadata:
      name: {{ include "app.fullname" . }}-prometheus-exporter
      labels: {{- include "app.labels" . | nindent 4 }}
    spec:
      podMetricsEndpoints:
        - interval: 15s
          port: http
          path: /metrics
      jobLabel: {{ template "app.fullname" . }}-prometheus-exporter
      namespaceSelector:
        matchNames:
          - {{ .Release.Namespace }}
      selector:
        matchLabels:
          {{- include "app.selector" . | nindent 6 }}
    {{- end }} 
    
  2. In order to allow users to decide whether to enable metrics when they are deploying the Bear Echo app, we can add a toggle switch for the podmonitor as a deployment parameter within the GBAP inside the gbear/appmetadata.yaml:

        - name: MONITORING
          title: Enable monitoring. Make sure you have enabled observability for the site you're deploying beforehand.
          value: "off"
          choices: ["on", "off"]
    

    For reference, here’s the full bear-echo/GBAP/gbear/appmetadata.yaml file of the Bear Echo application:

    displayName: bear-echo
    description: A simple HTTP echo service, based on Python Flask.
    icon: https://raw.githubusercontent.com/FortAwesome/Font-Awesome/5.15.4/svgs/solid/tools.svg 
    tags:
    - debug
    - sample    
    singleton: true    # Defines whether the app can be deployed to the same site several times
    configuration:
      - name: "ECHO_TEXT"
        title: "Message to say"
        description: "The application will echo this text"
        value: "A Hello from Great Bear."            # Default used if not overridden by the user
        type: String
      - name: "ACCESS_PORT"
        title: "Access HTTP port (30000-32767)"
        value: 31777                               # Default used if not overridden by the user
        type: NodePort
      - name: MONITORING
        title: Enable monitoring. Make sure you have enabled observability for the site you're deploying beforehand.
        value: "off"
        choices: ["on", "off"]
    
  3. Finally, you need to edit the bear-echo/GBAP/values.yaml file and add the monitoring variable to the config section:

    config:
      MONITORING: ""
    

    For reference, here’s the full Values file:

    replicaCount: 1
    strategyType: Recreate
    
    image:
      repository: harbor.eticloud.io/gbear-app-stage/bear-echo
      tag: develop
      pullPolicy: IfNotPresent
    
    imagePullSecrets:
      - name: gbear-harbor-pull
    
    config:
      # Config parameters described in application metadata
      # Their values will be injected here by the Great Bear deployer
      #
      GB_SITE_NAME: |
            Site name with "quotes" and 'quotes'
      GB_NODE_NAMES: |
            {"node1":"Display name 1 with \"quotes\" and 'quotes'","node2":"Display name 2 with \"quotes\" and 'quotes'"}
      ACCESS_PORT: |
            "31777"
      MONITORING: ""
    
    service:
      type: NodePort
      port: 11777
    
    ports:
      http:
        port: 11777
        protocol: TCP
        nodePort: "{{ default 31777 (int .Values.config.ACCESS_PORT)}}"
    
    # ----
    
    nameOverride: ""
    
    fullnameOverride: ""
    
    nodeSelector: {}
    
    affinity: {}
    
    ingress:
      enabled: false
      istioGateway: ""
      servicePath: ""
    
    siteId: ""
    
  4. Update and redeploy your application for the changes to take effect.

Next steps

By following these steps you should now have an application with metrics delivered to Prometheus, which should also be visible in Grafana. For more information on Grafana, see Observing Great Bear.

Bear Echo Metrics Bear Echo Metrics