Adding Prometheus metrics to an app

This guide shows you how to collect metrics of a Great Bear application you are developing. Metrics allow you to observe and monitor your application. The metrics are collected by Prometheus and they can be visualized on Grafana. For details, see Observing Great Bear.

The following steps extend the Bear Echo application with basic metrics, and assume that you have already followed the previous sample on Adding Liveness Probe to an app.

Add metrics endpoint to the application

First you need to add a new metrics endpoint to your application so when Prometheus scrapes the HTTP endpoint of the application, the client library sends the current state of all tracked metrics to the Prometheus server.

This example uses the official Python client library for Prometheus and its example on adding metrics to a Flask application.

  1. Add the metrics endpoint to the server.

    from werkzeug.middleware.dispatcher import DispatcherMiddleware
from prometheus_client import make_wsgi_app

app.wsgi_app = DispatcherMiddleware(app.wsgi_app, {
    '/metrics': make_wsgi_app()
})

  2. Add a metric to the application. This example uses a counter metric type, that is incremented by one every time the base url is accessed.

    from prometheus_client import Counter

_COUNTER_METRIC = Counter('num_of_queries', 'Number of queries')

@app.route(_BASE_URL, methods=['GET'])
def echo():
    _COUNTER_METRIC.inc()
    return(f'<hr><center><h1>{_ECHO_TEXT}</h1><center><hr>')

    For reference, here’s the extended server.py of the Bear Echo application:

    # bear-echo/server.py

import os
import sys
import importlib
from flask import Flask
from prometheus_client import Counter
from werkzeug.middleware.dispatcher import DispatcherMiddleware
from prometheus_client import make_wsgi_app


_HTTP_PORT: int = int(os.getenv('HTTP_PORT') or '11777')
_BASE_URL: str = os.getenv('BASE_URL') or '/'
_ECHO_TEXT: str = os.getenv('ECHO_TEXT') or 'Hello World'

_COUNTER_METRIC = Counter('num_of_queries', 'Number of queries')

app = Flask(__name__)

app.wsgi_app = DispatcherMiddleware(app.wsgi_app, {
    '/metrics': make_wsgi_app()
})


@app.route(_BASE_URL, methods=['GET'])
def echo():
    _COUNTER_METRIC.inc()
    return(f'<hr><center><h1>{_ECHO_TEXT}</h1><center><hr>')


@app.route("/healthz")
def healthz():
    return "OK"


def _main():
    print('ECHO_TEXT: ' + _ECHO_TEXT)
    print(f'Started on 0.0.0.0:{_HTTP_PORT}{_BASE_URL}')
    app.run(host='0.0.0.0', port=_HTTP_PORT)


if __name__ == '__main__':
    _main()

Update the Dockerfile

  1. The official example recommends using uwsgi to run the application. Update the Dockerfile of the application to do that.

    # bear-echo/Dockerfile

FROM python:3.8-alpine

WORKDIR /echo

COPY requirements.txt .

RUN set -e; \
    apk add --no-cache --virtual .build-deps \
    gcc \
    libc-dev \
    linux-headers \
    ; \
    pip install --no-cache-dir -r requirements.txt; \
    apk del .build-deps;

COPY server.py ./

EXPOSE 11777

CMD uwsgi --http 0.0.0.0:11777 --wsgi-file server.py --callable app

  2. You can see that the Dockerfile now uses a bear-echo/requirements.txt file to keep track of the dependencies used by the application server. Create this file with the following content:

    # bear-echo/requirements.txt

flask
prometheus-client
uwsgi

Update the Chart to make metrics available

Providing metrics is optional, the user can enable metrics when deploying the app. If metrics are enabled, a podmonitor is deployed. Prometheus discovers the metrics endpoint of your app via the podmonitor. You can customize several settings related to the scraping.

  1. In order to add the podmonitor to the Chart, create the bear-echo/GBAP/templates/podmonitor.yaml file with the following content.

    {{- if eq .Values.config.MONITORING "on" }}
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: {{ include "app.fullname" . }}-prometheus-exporter
  labels: {{- include "app.labels" . | nindent 4 }}
spec:
  podMetricsEndpoints:
    - interval: 15s
      port: http
      path: /metrics
  jobLabel: {{ template "app.fullname" . }}-prometheus-exporter
  namespaceSelector:
    matchNames:
      - {{ .Release.Namespace }}
  selector:
    matchLabels:
      {{- include "app.selector" . | nindent 6 }}
{{- end }}

  2. In order to allow users to decide whether to enable metrics when they are deploying the Bear Echo app, we can add a toggle switch for the podmonitor as a deployment parameter within the GBAP inside the gbear/appmetadata.yaml:

        - name: MONITORING
      title: Enable monitoring. Make sure you have enabled observability for the site you're deploying beforehand.
      value: "off"
      choices: ["on", "off"]

    For reference, here’s the full bear-echo/GBAP/gbear/appmetadata.yaml file of the Bear Echo application:

    displayName: bear-echo
description: A simple HTTP echo service, based on Python Flask.
icon: https://raw.githubusercontent.com/FortAwesome/Font-Awesome/5.15.4/svgs/solid/tools.svg 
tags:
- debug
- sample    
singleton: true    # Defines whether the app can be deployed to the same site several times
configuration:
  - name: "ECHO_TEXT"
    title: "Message to say"
    description: "The application will echo this text"
    value: "A Hello from Great Bear."            # Default used if not overridden by the user
    type: String
  - name: "ACCESS_PORT"
    title: "Access HTTP port (30000-32767)"
    value: 31777                               # Default used if not overridden by the user
    type: NodePort
  - name: MONITORING
    title: Enable monitoring. Make sure you have enabled observability for the site you're deploying beforehand.
    value: "off"
    choices: ["on", "off"]

  3. Finally, you need to edit the bear-echo/GBAP/values.yaml file and add the monitoring variable to the config section:

    config:
  MONITORING: ""

    For reference, here’s the full Values file:

    replicaCount: 1
strategyType: Recreate

image:
  repository: harbor.eticloud.io/gbear-app-stage/bear-echo
  tag: develop
  pullPolicy: IfNotPresent

imagePullSecrets:
  - name: gbear-harbor-pull

config:
  # Config parameters described in application metadata
  # Their values will be injected here by the Great Bear deployer
  #
  GB_SITE_NAME: |
        Site name with "quotes" and 'quotes'
  GB_NODE_NAMES: |
        {"node1":"Display name 1 with \"quotes\" and 'quotes'","node2":"Display name 2 with \"quotes\" and 'quotes'"}
  ACCESS_PORT: |
        "31777"
  MONITORING: ""

service:
  type: NodePort
  port: 11777

ports:
  http:
    port: 11777
    protocol: TCP
    nodePort: "{{ default 31777 (int .Values.config.ACCESS_PORT)}}"

# ----

nameOverride: ""

fullnameOverride: ""

nodeSelector: {}

affinity: {}

ingress:
  enabled: false
  istioGateway: ""
  servicePath: ""

siteId: ""

  4. Update and redeploy your application for the changes to take effect.

Next steps

By following these steps you should now have an application with metrics delivered to Prometheus, which should also be visible in Grafana. For more information on Grafana, see Observing Great Bear.

Bear Echo Metrics Bear Echo Metrics