Extending an App with the Sync Server

This guide shows you how to extend the Bear Echo sample app to include the Sync Server. The Sync Server allows the App Control Service to communicate with the app in real time, for example, to change the configuration of the deployed app.

Outline

  1. Goal
  2. Architecture
  3. Extending Bear Echo with the Sync Server
  4. Docker Image
  5. Helm Packaging
  6. Great Bear Metadata

Goal

The Developing an App for Great Bear tutorial has shown you how to create the Bear Echo application. But what if you want to interact with the application at the edge after it has already been deployed? This demo shows you how to modify the ECHO_TEXT variable at runtime, without triggering new deployments.

The goal is to deploy the same Bear Echo application, and display an HTML page with the most recent data payload it has received. At first, it will be the same ECHO_TEXT that was provided as an environment variable at deploy time, as in the original sample.

You can use the Application Control Service dashboard to connect to the edge site and send data payloads by using the Sync Server.

Architecture

The Sync Server is deployed on the edge, along side your application (Bear Echo in this case). Once it is running, it connects up to the Application Control Service using a WebSocket connection. It then creates listeners and exposes events for your application to use.

Sync Server Architecture Sync Server Architecture

Practically, your application needs to provide a few configurations when connecting to the Sync Server, and then add some Socket.io listeners for the events the Sync Server emits to you.

These events are outlined in the bear-echo/app/syncer.py SyncClient that is shown later.

Extending Bear Echo

Let’s revisit the Bear Echo app, and add some additional calls to connect to the Sync Server, and do something with one of the event handlers.

To configure connecting to the Sync Server, you need:

  1. The URL of the Sync Server for the Sync Client to connect to.
  2. The ID of the Great Bear Node where the application (Bear Echo) is running.
  3. The name of the application itself.

The last two are used by the App Control Service to identity the particular application running on the particular node. This should look mostly similar, but the important parts will be highlighted:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

# bear-echo/app/main.py

from flask import Flask
from syncer import SyncClient
import json
import os
import uuid

HTTP_PORT: int = int(os.getenv('HTTP_PORT') or '11777')
BASE_URL: str = os.getenv('BASE_URL') or '/'
ECHO_TEXT: str = os.getenv('ECHO_TEXT') or 'Hello World'

SYNC_SERVER_HOST: str = os.getenv('SYNC_SERVER_HOST')
SYNC_SERVER_PORT: str = os.getenv('SYNC_SERVER_PORT')
GB_NODE_ID: str = os.getenv('GB_NODE_ID') or ''
GB_NODE_IDS: str = os.getenv('GB_NODE_IDS') or '{}'
APP_NAME: str = os.getenv('APP_NAME') or 'bear-echo-runtime'

class EchoSyncer(SyncClient):
    """extends the SyncClient and overrides the "onNewData" listener"""
    def onNewData(self, data):
        global ECHO_TEXT
        print(f'ECHO_TEXT updated from {ECHO_TEXT} to {data}')
        ECHO_TEXT = data

app = Flask(__name__)

@app.route(BASE_URL, methods=['GET'])
def echo():
    return(f'<hr><center><h1>{ECHO_TEXT}</h1><center><hr>')

def main():
    print('ECHO_TEXT: ' + ECHO_TEXT)
    print(f'Started on 0.0.0.0:{HTTP_PORT}{BASE_URL}')
    nodeIDMap = json.loads(GB_NODE_IDS)
    nodeID = nodeIDMap[GB_NODE_ID] if GB_NODE_ID in nodeIDMap else str(uuid.uuid4())
    sync = EchoSyncer(
        syncServerHost='http://%s:%s' % (SYNC_SERVER_HOST, SYNC_SERVER_PORT),
        gbNodeID=nodeID,
        appName=APP_NAME,
        deployID='demo',
    )
    sync.start()
    app.run(host='0.0.0.0', port=HTTP_PORT) # blocking
    sync.stop()

if __name__ == '__main__':
    main()

This is almost the same as the original Bear Echo sample, except for:

  • the extra configuration parameters, and
  • the EchoSyncer class.

The EchoSyncer class extends the SyncClient with extra functionality and default websocket handlers, and it overrides the onNewData handler to update the ECHO_TEXT. That way Bear Echo displays the most recent data payload in an HTML page.

Okay, but what does the SyncClient actually do? Let’s take a look:

# bear-echo/app/syncer.py

from datetime import datetime
from enum import Enum
from threading import Thread
import socketio
import logging
import time

logging.basicConfig(level=logging.DEBUG)


class AppStatus(Enum):
    NULL = 0
    ONLINE = 1
    ERROR = 2
    OFFLINE = 3
    INITIALIZING = 4
    PAYLOAD_ERROR = 5


def nowTS():
    # now timestamp as nondelimited year-month-day-hour-minute-second string
    return datetime.now().strftime("%Y%m%d%H%M%S")


class SyncClient:
    """
    To be used directly, or subclassed with desired methods overridden.
    public methods to override:
        onConnect()
        onConnectError(errMsg)
        onDisconnect()
        onNewData(data)
        onRemoveContent()
        onCheckStatus()
        onCatchAll(data)

    Once the SyncClient is initialized, start listening with
        sync.start()

    Stop with:
        sync.stop()
    """
    def __init__(self, syncServerHost, gbNodeID, appName, deployID=nowTS()):
        self.syncServerHost = syncServerHost
        self.gbNodeID = gbNodeID
        self.appName = '%s-%s' % (appName, deployID)
        self.appID = '%s-%s-%s' % (gbNodeID, appName, deployID)
        self.fullSyncServerURL = '%s?nodeId=%s&appName=%s&appId=%s' % (
                self.syncServerHost,
                self.gbNodeID,
                self.appName,
                self.appID)

        logging.debug('initializing sync client...')
        logging.debug('    syncServerHost: <%s>', self.syncServerHost)
        logging.debug('    nodeID:         <%s>', self.gbNodeID)
        logging.debug('    appName:        <%s>', self.appName)
        logging.debug('    appID:          <%s>', self.appID)

        self.running = False
        self.connected = False
        self.connection = Thread(target=self.connect, daemon=False)

    # overide me
    def onConnect(self):
        logging.info('%s connected', self.appName)

    # overide me
    def onConnectError(self, err):
        logging.error('%s connection error: %s', self.appName, err)

    # overide me
    def onDisconnect(self):
        logging.info('%s disconnected', self.appName)

    # overide me
    def onNewData(self, data):
        logging.info('%s received new data: <%s>', self.appName, data)
        # supports an optional return to emit a response back to the server.
        # if an exception is raised, an error response will be emitted back to
        # the server.
        return None

    # overide me
    def onRemoveContent(self):
        logging.info('%s received remove content', self.appName)
        # supports an optional return
        return None

    # overide me
    def onCheckStatus(self, msg=None):
        logging.info('%s received on check status: %s', self.appName, msg)
        # supports an optional return
        return None

    # overide me
    def onCatchAll(self, data=None):
        logging.info('%s received an unexpeected event: <%s>', self.appName, data)

    # not blocking
    def start(self):
        logging.info('%s starting...', self.appName)
        self.running = True
        self.connection.start()

    # blocking
    def join(self):
        self.connection.join()

    def stop(self):
        self.running = False
        self.connected = False

    def connect(self):
        sio = socketio.Client(logger=False, engineio_logger=False, ssl_verify=False)
        logging.info('%s connecting to <%s>...', self.appName, self.fullSyncServerURL)

        @sio.event
        def connect():
            self.onConnect()

        @sio.event
        def connect_error(err):
            self.onConnectError(err)

        @sio.event
        def disconnect():
            self.onDisconnect()
            self.connected = False

        @sio.on('newData')
        def onNewData(data):
            try:
                resp = self.onNewData(data)
                if resp is not None:
                    logging.info('onNewData handler returned a response: <%s>', resp)
                    return resp
                return "ok"
            except Exception as e:
                logging.error('onNewData handler raised an exception: %s', e)
                return "error: %s" % e


        @sio.on('removeContent')
        def onRemoveContent():
            try:
                resp = self.onRemoveContent()
                if resp is not None:
                    logging.info('onRemoveContent handler returned a response: <%s>', resp)
                    return resp
                return "ok"
            except Exception as e:
                logging.error('onRemoveContent handler raised an exception: %s', e)
                return "error: %s" % e

        @sio.on('checkStatus')
        def onCheckStatus(msg=None):
            heartbeat = {'status': AppStatus.ONLINE.value, 'appId': self.appID}
            try:
                statusMsg = self.onCheckStatus(msg)
                if statusMsg is not None:
                    heartbeat['status_msg'] = statusMsg
            except Exception as e:
                heartbeat['status'] = AppStatus.PAYLOAD_ERROR.value
                heartbeat['error'] = 'exception: %s' % e
            logging.debug('status check response %s', heartbeat)

            # legacy
            sio.emit('sendStatus', heartbeat)

            # this return value is delivered as the acknowledgement to the
            # server. note: cannot return a tuple. must return a single val
            return heartbeat

        @sio.on('*')
        def onCatchAll(data=None):
            self.onCatchAll(data)

        while self.running:
            if (not self.connected):
                try:
                    sio.connect(self.fullSyncServerURL)
                    self.connected = True
                    sio.wait()
                # TODO(sam): handle the "Already Connected" exception better
                except Exception as e:
                    logging.error('connection error: %s', e)
                    time.sleep(5)
                    pass
            else:
                logging.debug("already connected. looping...")
                time.sleep(5)

The SyncClient has been written generically, so you should be able to copy-paste this into your project with minimal modification. As shown above, the best way to use the SyncClient is to extend the class and override the methods as necessary.

But let’s go into more detail on how this SyncClient works:

The SyncClient uses Socket.io (which is really just a wrapper around websockets) under the hood to connect to the Sync Server. After an instance of the SyncClient has been initialized, .start() connects and registers Socket.io handlers for all of the events emitted by the Sync Server. If the SyncClient has been subclassed, or any handlers have been overridden, those are used instead of the defaults. The default handlers simply logs that they have been hit, along with any data they have received.

Adding the Docker Image

Now that you have the application code ready to go, create a Docker image that can be included in our Helm charts later on.

First, define your requirements:

# bear-echo/app/requirements.txt

python-socketio==5.7.1
python-engineio==4.3.4
websocket-client==1.4.1
requests==2.28.1
Flask==2.2.2

And now, the Dockerfile:

# bear-echo/Dockerfile

FROM python:3.9

WORKDIR /usr/src/app

COPY app/requirements.txt .

RUN apt-get update && \
    apt-get install libsm6 libxext6 -y && \
    pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir -r requirements.txt

COPY app/. .

EXPOSE 11777

CMD ["python", "-u", "main.py"]

At this point, your “bear-echo” project should look like:

├─ bear-echo/
    ├─ Dockerfile
    ├─ app/
        ├─ main.py
        ├─ syncer.py
        ├─ requirements.txt

Packaging application as a Helm Chart

You should now have a Docker image for our new Bear Echo edge application. Now create the Helm charts.

To keep things organized and easy to understand, and because the Sync Server is a separate, but dependent chart, you can take advantage of Helm subcharts. This means you are going to create a new Helm chart for Bear Echo, and then import the Sync Server chart as a dependency.

  1. Create a new Helm chart, and then wipe out the default YAML files:

    helm create chart
rm -r chart/templates

  2. Copy the following files into your new helm chart:

    # bear-echo/chart/Chart.yaml

apiVersion: v2
name: bear-echo-runtime
description: A Bear Echo (with Sync Server) example helm chart
type: application
version: 0.0.1
appVersion: "0.0.1"

icon: https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.1.2/svgs/solid/volume-high.svg

dependencies: 
  - name: sync-server
    version: 0.0.1
    # alias needed to avoid issue with hyphenated key names in values.yaml
    # https://github.com/helm/helm/issues/2192
    alias: syncserver

    # bear-echo/chart/values.yaml

syncserver:
  fullnameOverride: "bear-echo-syncserver"
  service:
    port: 8010

  config:
    apiKey: ""
    gbSiteID: ""
    gbSiteName: ""
    controlServerURL: ""

  image:
    repository: harbor.eticloud.io/gbear-dev/great-bear-sync-server
    pullPolicy: Always
    tag: develop

  imagePullSecrets:
    - name: gbear-harbor-pull


# for bear-echo helm charts

fullnameOverride: ""
service:
  port: 11777

config:
  requestPath: "/"
  defaultEchoText: "hi"
  # needs to be base64 encoded json. like `echo -n '{}' | base64`
  gbNodeIDs: "e30="
  appName: "bear-echo-runtime"

image:
  repository: harbor.eticloud.io/gbear-dev/great-bear-bear-echo-runtime
  pullPolicy: Always
  tag: develop

imagePullSecrets:
  - name: gbear-harbor-pull

    {{/* bear-echo/chart/templates/_helpers.tpl */}}

{{/*
Expand the name of the chart.
*/}}
{{- define "bear-echo.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "bear-echo.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "bear-echo.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Common labels
*/}}
{{- define "bear-echo.labels" -}}
helm.sh/chart: {{ include "bear-echo.chart" . }}
{{ include "bear-echo.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{/*
Selector labels
*/}}
{{- define "bear-echo.selectorLabels" -}}
app.kubernetes.io/name: {{ include "bear-echo.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

    # bear-echo/chart/templates/all.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "bear-echo.fullname" . }}
  labels:
    {{- include "bear-echo.labels" . | nindent 4 }}
spec:
  replicas: 1
  selector:
    matchLabels:
      {{- include "bear-echo.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "bear-echo.selectorLabels" . | nindent 8 }}
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          env:
            - name: HTTP_PORT
              value: {{ .Values.service.port | quote }}
            - name: BASE_URL
              value: {{ .Values.config.requestPath | quote }}
            - name: ECHO_TEXT
              value: {{ .Values.config.defaultEchoText | quote }}
            - name: SYNC_SERVER_HOST
              value: {{ .Values.syncserver.fullnameOverride | quote }}
            - name: SYNC_SERVER_PORT
              value: {{ .Values.syncserver.service.port | quote }}
            - name: GB_NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: GB_NODE_IDS
              value: |-
                                {{ .Values.config.gbNodeIDs | b64dec }}
            - name: APP_NAME
              value: {{ .Values.config.appName | quote }}
          ports:
            - name: http
              containerPort: {{ .Values.service.port }}
              protocol: TCP
          livenessProbe:
            exec:
              command:
              - cat
              - main.py
            initialDelaySeconds: 10
            periodSeconds: 5

---

apiVersion: v1
kind: Service
metadata:
  name: {{ include "bear-echo.fullname" . }}
  labels:
    {{- include "bear-echo.labels" . | nindent 4 }}
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.port }}
      targetPort: http
      protocol: TCP
      name: http
  selector:
    {{- include "bear-echo.selectorLabels" . | nindent 4 }}

    You should now have a directory structure like:

    ├─ bear-echo/
    ├─ Dockerfile
    ├─ app/
        ├─ main.py
        ├─ syncer.py
        ├─ requirements.txt
    ├─ chart/
        ├─ Chart.yaml
        ├─ values.yaml
        ├─ templates/
            ├─ _helpers.tpl
            ├─ all.yaml

  3. Finally, add the Sync Server subchart. To do this, copy and paste the following simplified YAML files:

    # bear-echo/chart/charts/sync-server/Chart.yaml

apiVersion: v2
name: sync-server
description: A Sync Server Helm Chart
type: application
version: 0.0.1
appVersion: "0.0.1"

    # bear-echo/chart/charts/sync-server/values.yaml

config:
  apiKey: ""
  gbSiteName: ""
  gbSiteID: ""
  controlServerURL: ""

image:
  repository: harbor.eticloud.io/gbear-dev/great-bear-sync-server
  pullPolicy: Always
  tag: develop

imagePullSecrets:
  - name: gbear-harbor-pull

fullnameOverride: "syncserver"

service:
  type: ClusterIP
  port: 8090

    {{/* bear-echo/chart/charts/sync-server/templates/_helpers.tpl */}}

{{/*
Expand the name of the chart.
*/}}
{{- define "sync-server.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "sync-server.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "sync-server.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Common labels
*/}}
{{- define "sync-server.labels" -}}
helm.sh/chart: {{ include "sync-server.chart" . }}
{{ include "sync-server.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{/*
Selector labels
*/}}
{{- define "sync-server.selectorLabels" -}}
app.kubernetes.io/name: {{ include "sync-server.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

    # bear-echo/chart/charts/sync-server/templates/all.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "sync-server.fullname" . }}
  labels:
    {{- include "sync-server.labels" . | nindent 4 }}
spec:
  replicas: 1
  selector:
    matchLabels:
      {{- include "sync-server.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "sync-server.selectorLabels" . | nindent 8 }}
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          env:
            - name: API_KEY
              value: {{ .Values.config.apiKey | quote }}
            - name: PORT_SYNC_SERVER
              value: {{ .Values.service.port | quote }}
            - name: GB_SITE_NAME
              value: {{ .Values.config.gbSiteName | quote }}
            - name: GB_SITE_ID
              value: {{ .Values.config.gbSiteID | quote }}
            - name: CONTROL_SERVER_URL
              value: {{ .Values.config.controlServerURL | quote }}
          ports:
            - name: http
              containerPort: {{ .Values.service.port }}
              protocol: TCP
          livenessProbe:
            exec:
              command:
              - cat
              - app.js
            initialDelaySeconds: 10
            periodSeconds: 5

---

apiVersion: v1
kind: Service
metadata:
  name: {{ include "sync-server.fullname" . }}
  labels:
    {{- include "sync-server.labels" . | nindent 4 }}
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.port }}
      targetPort: http
      protocol: TCP
      name: http
  selector:
    {{- include "sync-server.selectorLabels" . | nindent 4 }}

    This is the complete directory structure now:

    ├─ bear-echo/
    ├─ Dockerfile
    ├─ app/
        ├─ main.py
        ├─ syncer.py
        ├─ requirements.txt
    ├─ chart/
        ├─ Chart.yaml
        ├─ values.yaml
        ├─ templates/
            ├─ _helpers.tpl
            ├─ all.yaml
        ├─ charts/
            ├─ sync-server/
                ├─ Chart.yaml
                ├─ values.yaml
                ├─ templates/
                    ├─ _helpers.tpl
                    ├─ all.yaml

(Optional) Great Bear metadata

You should now have just about everything you need for your new Great Bear application that supports runtime configuration! There’s a bit more metadata that you can add to describe our application in the Great Bear appstore. (For details, see gbear/appmetadata.yaml.)

Here is the kind of metadata you might want for your Bear Echo + Sync Server application.

# bear-echo/chart/gbear/appmetadata.yaml

displayName: Bear Echo (with Sync Server)
description: Helm chart for Bear Echo with Sync Server

icon: https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.1.2/svgs/solid/volume-high.svg

labels:
   kind: EXAMPLE

tags:
   - sample

configuration:
  - name: defaultEchoText
    key: config.defaultEchoText
    title: "Default Echo Text"
    description: "Text to be Echo'd before Runtime Updates"
    value: "default"
    type: String

  - name: apiKey
    key: syncserver.config.apiKey
    title: "API Key (Deployment Token)"
    description: "API Key generated from app-control dashboard"
    type: Secret

  - name: siteID
    key: syncserver.config.gbSiteID
    value: "{{ .GB_SITE_ID }}"
    type: Runtime

  - name: siteName
    key: syncserver.config.gbSiteName
    value: "{{ .GB_SITE_NAME }}"
    type: Runtime

  - name: controlServerURL
    key: syncserver.config.controlServerURL
    title: "Control Server URL"
    description: "The URL of the Application Control Service"
    value: "https://app-control.dev.gbear.scratch.eticloud.io"
    type: String

  - name: nodeIDs
    key: config.gbNodeIDs
    value: "{{ .GB_NODE_IDS | b64enc }}"
    # actually passed in as json
    type: Runtime

├─ bear-echo/
    ├─ Dockerfile
    ├─ app/
        ├─ main.py
        ├─ syncer.py
        ├─ requirements.txt
    ├─ chart/
        ├─ Chart.yaml
        ├─ values.yaml
        ├─ gbear/
            ├─ appmetadata.yaml
        ├─ templates/
            ├─ _helpers.tpl
            ├─ all.yaml
        ├─ charts/
            ├─ sync-server/
                ├─ Chart.yaml
                ├─ values.yaml
                ├─ templates/
                    ├─ _helpers.tpl
                    ├─ all.yaml