Adding Prometheus metrics to an app
This guide shows you how to collect metrics of a Great Bear application you are developing. Metrics allow you to observe and monitor your application. The metrics are collected by Prometheus and they can be visualized on Grafana. For details, see Observing Great Bear.
The following steps extend the Bear Echo application with basic metrics, and assume that you have already followed the previous sample on Adding Liveness Probe to an app.
Add metrics endpoint to the application
First you need to add a new metrics
endpoint to your application so when Prometheus scrapes the HTTP endpoint of the application, the client library sends the current state of all tracked metrics to the Prometheus server.
This example uses the official Python client library for Prometheus and its example on adding metrics to a Flask application.
-
Add the
metrics
endpoint to the server.from werkzeug.middleware.dispatcher import DispatcherMiddleware from prometheus_client import make_wsgi_app app.wsgi_app = DispatcherMiddleware(app.wsgi_app, { '/metrics': make_wsgi_app() })
-
Add a metric to the application. This example uses a
counter
metric type, that is incremented by one every time the base url is accessed.from prometheus_client import Counter _COUNTER_METRIC = Counter('num_of_queries', 'Number of queries') @app.route(_BASE_URL, methods=['GET']) def echo(): _COUNTER_METRIC.inc() return(f'<hr><center><h1>{_ECHO_TEXT}</h1><center><hr>')
For reference, here’s the extended
server.py
of the Bear Echo application:# bear-echo/server.py import os import sys import importlib from flask import Flask from prometheus_client import Counter from werkzeug.middleware.dispatcher import DispatcherMiddleware from prometheus_client import make_wsgi_app _HTTP_PORT: int = int(os.getenv('HTTP_PORT') or '11777') _BASE_URL: str = os.getenv('BASE_URL') or '/' _ECHO_TEXT: str = os.getenv('ECHO_TEXT') or 'Hello World' _COUNTER_METRIC = Counter('num_of_queries', 'Number of queries') app = Flask(__name__) app.wsgi_app = DispatcherMiddleware(app.wsgi_app, { '/metrics': make_wsgi_app() }) @app.route(_BASE_URL, methods=['GET']) def echo(): _COUNTER_METRIC.inc() return(f'<hr><center><h1>{_ECHO_TEXT}</h1><center><hr>') @app.route("/healthz") def healthz(): return "OK" def _main(): print('ECHO_TEXT: ' + _ECHO_TEXT) print(f'Started on 0.0.0.0:{_HTTP_PORT}{_BASE_URL}') app.run(host='0.0.0.0', port=_HTTP_PORT) if __name__ == '__main__': _main()
Update the Dockerfile
-
The official example recommends using
uwsgi
to run the application. Update the Dockerfile of the application to do that.# bear-echo/Dockerfile FROM python:3.8-alpine WORKDIR /echo COPY requirements.txt . RUN set -e; \ apk add --no-cache --virtual .build-deps \ gcc \ libc-dev \ linux-headers \ ; \ pip install --no-cache-dir -r requirements.txt; \ apk del .build-deps; COPY server.py ./ EXPOSE 11777 CMD uwsgi --http 0.0.0.0:11777 --wsgi-file server.py --callable app
-
You can see that the Dockerfile now uses a
bear-echo/requirements.txt
file to keep track of the dependencies used by the application server. Create this file with the following content:# bear-echo/requirements.txt flask prometheus-client uwsgi
Update the Chart to make metrics available
Providing metrics is optional, the user can enable metrics when deploying the app. If metrics are enabled, a podmonitor is deployed. Prometheus discovers the metrics
endpoint of your app via the podmonitor. You can customize several settings related to the scraping.
-
In order to add the podmonitor to the Chart, create the
bear-echo/GBAP/templates/podmonitor.yaml
file with the following content.{{- if eq .Values.config.MONITORING "on" }} apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: {{ include "app.fullname" . }}-prometheus-exporter labels: {{- include "app.labels" . | nindent 4 }} spec: podMetricsEndpoints: - interval: 15s port: http path: /metrics jobLabel: {{ template "app.fullname" . }}-prometheus-exporter namespaceSelector: matchNames: - {{ .Release.Namespace }} selector: matchLabels: {{- include "app.selector" . | nindent 6 }} {{- end }}
-
In order to allow users to decide whether to enable metrics when they are deploying the Bear Echo app, we can add a toggle switch for the podmonitor as a deployment parameter within the GBAP inside the
gbear/appmetadata.yaml
:- name: MONITORING title: Enable monitoring. Make sure you have enabled observability for the site you're deploying beforehand. value: "off" choices: ["on", "off"]
For reference, here’s the full
bear-echo/GBAP/gbear/appmetadata.yaml
file of the Bear Echo application:displayName: bear-echo description: A simple HTTP echo service, based on Python Flask. icon: https://raw.githubusercontent.com/FortAwesome/Font-Awesome/5.15.4/svgs/solid/tools.svg tags: - debug - sample singleton: true # Defines whether the app can be deployed to the same site several times configuration: - name: "ECHO_TEXT" title: "Message to say" description: "The application will echo this text" value: "A Hello from Great Bear." # Default used if not overridden by the user type: String - name: "ACCESS_PORT" title: "Access HTTP port (30000-32767)" value: 31777 # Default used if not overridden by the user type: NodePort - name: MONITORING title: Enable monitoring. Make sure you have enabled observability for the site you're deploying beforehand. value: "off" choices: ["on", "off"]
-
Finally, you need to edit the
bear-echo/GBAP/values.yaml
file and add the monitoring variable to the config section:config: MONITORING: ""
For reference, here’s the full Values file:
replicaCount: 1 strategyType: Recreate image: repository: harbor.eticloud.io/gbear-app-stage/bear-echo tag: develop pullPolicy: IfNotPresent imagePullSecrets: - name: gbear-harbor-pull config: # Config parameters described in application metadata # Their values will be injected here by the Great Bear deployer # GB_SITE_NAME: | Site name with "quotes" and 'quotes' GB_NODE_NAMES: | {"node1":"Display name 1 with \"quotes\" and 'quotes'","node2":"Display name 2 with \"quotes\" and 'quotes'"} ACCESS_PORT: | "31777" MONITORING: "" service: type: NodePort port: 11777 ports: http: port: 11777 protocol: TCP nodePort: "{{ default 31777 (int .Values.config.ACCESS_PORT)}}" # ---- nameOverride: "" fullnameOverride: "" nodeSelector: {} affinity: {} ingress: enabled: false istioGateway: "" servicePath: "" siteId: ""
-
Update and redeploy your application for the changes to take effect.
Next steps
By following these steps you should now have an application with metrics delivered to Prometheus, which should also be visible in Grafana. For more information on Grafana, see Observing Great Bear.