Extending an App with the Sync Server

This guide shows you how to extend the Bear Echo sample app to include the Sync Server. The Sync Server allows the App Control Service to communicate with the app in real time, for example, to change the configuration of the deployed app.

Outline

  1. Goal
  2. Architecture
  3. Extending Bear Echo with the Sync Server
  4. Docker Image
  5. Helm Packaging
  6. Great Bear Metadata

Goal

The Developing an App for Great Bear tutorial has shown you how to create the Bear Echo application. But what if you want to interact with the application at the edge after it has already been deployed? This demo shows you how to modify the ECHO_TEXT variable at runtime, without triggering new deployments.

The goal is to deploy the same Bear Echo application, and display an HTML page with the most recent data payload it has received. At first, it will be the same ECHO_TEXT that was provided as an environment variable at deploy time, as in the original sample.

You can use the Application Control Service dashboard to connect to the edge site and send data payloads by using the Sync Server.

Architecture

The Sync Server is deployed on the edge, along side your application (Bear Echo in this case). Once it is running, it connects up to the Application Control Service using a WebSocket connection. It then creates listeners and exposes events for your application to use.

Sync Server Architecture Sync Server Architecture

Practically, your application needs to provide a few configurations when connecting to the Sync Server, and then add some Socket.io listeners for the events the Sync Server emits to you.

These events are outlined in the bear-echo/app/syncer.py SyncClient that is shown later.

Extending Bear Echo

Let’s revisit the Bear Echo app, and add some additional calls to connect to the Sync Server, and do something with one of the event handlers.

To configure connecting to the Sync Server, you need:

  1. The URL of the Sync Server for the Sync Client to connect to.
  2. The ID of the Great Bear Node where the application (Bear Echo) is running.
  3. The name of the application itself.

The last two are used by the App Control Service to identity the particular application running on the particular node. This should look mostly similar, but the important parts will be highlighted:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# bear-echo/app/main.py

from flask import Flask
from syncer import SyncClient
import json
import os
import uuid

HTTP_PORT: int = int(os.getenv('HTTP_PORT') or '11777')
BASE_URL: str = os.getenv('BASE_URL') or '/'
ECHO_TEXT: str = os.getenv('ECHO_TEXT') or 'Hello World'

SYNC_SERVER_HOST: str = os.getenv('SYNC_SERVER_HOST')
SYNC_SERVER_PORT: str = os.getenv('SYNC_SERVER_PORT')
GB_NODE_ID: str = os.getenv('GB_NODE_ID') or ''
GB_NODE_IDS: str = os.getenv('GB_NODE_IDS') or '{}'
APP_NAME: str = os.getenv('APP_NAME') or 'bear-echo-runtime'

class EchoSyncer(SyncClient):
    """extends the SyncClient and overrides the "onNewData" listener"""
    def onNewData(self, data):
        global ECHO_TEXT
        print(f'ECHO_TEXT updated from {ECHO_TEXT} to {data}')
        ECHO_TEXT = data

app = Flask(__name__)

@app.route(BASE_URL, methods=['GET'])
def echo():
    return(f'<hr><center><h1>{ECHO_TEXT}</h1><center><hr>')

def main():
    print('ECHO_TEXT: ' + ECHO_TEXT)
    print(f'Started on 0.0.0.0:{HTTP_PORT}{BASE_URL}')
    nodeIDMap = json.loads(GB_NODE_IDS)
    nodeID = nodeIDMap[GB_NODE_ID] if GB_NODE_ID in nodeIDMap else str(uuid.uuid4())
    sync = EchoSyncer(
        syncServerHost='http://%s:%s' % (SYNC_SERVER_HOST, SYNC_SERVER_PORT),
        gbNodeID=nodeID,
        appName=APP_NAME,
        deployID='demo',
    )
    sync.start()
    app.run(host='0.0.0.0', port=HTTP_PORT) # blocking
    sync.stop()

if __name__ == '__main__':
    main()

This is almost the same as the original Bear Echo sample, except for:

  • the extra configuration parameters, and
  • the EchoSyncer class.

The EchoSyncer class extends the SyncClient with extra functionality and default websocket handlers, and it overrides the onNewData handler to update the ECHO_TEXT. That way Bear Echo displays the most recent data payload in an HTML page.

Okay, but what does the SyncClient actually do? Let’s take a look:

# bear-echo/app/syncer.py

from datetime import datetime
from enum import Enum
from threading import Thread
import socketio
import logging
import time

logging.basicConfig(level=logging.DEBUG)


class AppStatus(Enum):
    NULL = 0
    ONLINE = 1
    ERROR = 2
    OFFLINE = 3
    INITIALIZING = 4
    PAYLOAD_ERROR = 5


def nowTS():
    # now timestamp as nondelimited year-month-day-hour-minute-second string
    return datetime.now().strftime("%Y%m%d%H%M%S")


class SyncClient:
    """
    To be used directly, or subclassed with desired methods overridden.
    public methods to override:
        onConnect()
        onConnectError(errMsg)
        onDisconnect()
        onNewData(data)
        onRemoveContent()
        onCheckStatus()
        onCatchAll(data)

    Once the SyncClient is initialized, start listening with
        sync.start()

    Stop with:
        sync.stop()
    """
    def __init__(self, syncServerHost, gbNodeID, appName, deployID=nowTS()):
        self.syncServerHost = syncServerHost
        self.gbNodeID = gbNodeID
        self.appName = '%s-%s' % (appName, deployID)
        self.appID = '%s-%s-%s' % (gbNodeID, appName, deployID)
        self.fullSyncServerURL = '%s?nodeId=%s&appName=%s&appId=%s' % (
                self.syncServerHost,
                self.gbNodeID,
                self.appName,
                self.appID)

        logging.debug('initializing sync client...')
        logging.debug('    syncServerHost: <%s>', self.syncServerHost)
        logging.debug('    nodeID:         <%s>', self.gbNodeID)
        logging.debug('    appName:        <%s>', self.appName)
        logging.debug('    appID:          <%s>', self.appID)

        self.running = False
        self.connected = False
        self.connection = Thread(target=self.connect, daemon=False)

    # overide me
    def onConnect(self):
        logging.info('%s connected', self.appName)

    # overide me
    def onConnectError(self, err):
        logging.error('%s connection error: %s', self.appName, err)

    # overide me
    def onDisconnect(self):
        logging.info('%s disconnected', self.appName)

    # overide me
    def onNewData(self, data):
        logging.info('%s received new data: <%s>', self.appName, data)
        # supports an optional return to emit a response back to the server.
        # if an exception is raised, an error response will be emitted back to
        # the server.
        return None

    # overide me
    def onRemoveContent(self):
        logging.info('%s received remove content', self.appName)
        # supports an optional return
        return None

    # overide me
    def onCheckStatus(self, msg=None):
        logging.info('%s received on check status: %s', self.appName, msg)
        # supports an optional return
        return None

    # overide me
    def onCatchAll(self, data=None):
        logging.info('%s received an unexpeected event: <%s>', self.appName, data)

    # not blocking
    def start(self):
        logging.info('%s starting...', self.appName)
        self.running = True
        self.connection.start()

    # blocking
    def join(self):
        self.connection.join()

    def stop(self):
        self.running = False
        self.connected = False

    def connect(self):
        sio = socketio.Client(logger=False, engineio_logger=False, ssl_verify=False)
        logging.info('%s connecting to <%s>...', self.appName, self.fullSyncServerURL)

        @sio.event
        def connect():
            self.onConnect()

        @sio.event
        def connect_error(err):
            self.onConnectError(err)

        @sio.event
        def disconnect():
            self.onDisconnect()
            self.connected = False

        @sio.on('newData')
        def onNewData(data):
            try:
                resp = self.onNewData(data)
                if resp is not None:
                    logging.info('onNewData handler returned a response: <%s>', resp)
                    return resp
                return "ok"
            except Exception as e:
                logging.error('onNewData handler raised an exception: %s', e)
                return "error: %s" % e


        @sio.on('removeContent')
        def onRemoveContent():
            try:
                resp = self.onRemoveContent()
                if resp is not None:
                    logging.info('onRemoveContent handler returned a response: <%s>', resp)
                    return resp
                return "ok"
            except Exception as e:
                logging.error('onRemoveContent handler raised an exception: %s', e)
                return "error: %s" % e

        @sio.on('checkStatus')
        def onCheckStatus(msg=None):
            heartbeat = {'status': AppStatus.ONLINE.value, 'appId': self.appID}
            try:
                statusMsg = self.onCheckStatus(msg)
                if statusMsg is not None:
                    heartbeat['status_msg'] = statusMsg
            except Exception as e:
                heartbeat['status'] = AppStatus.PAYLOAD_ERROR.value
                heartbeat['error'] = 'exception: %s' % e
            logging.debug('status check response %s', heartbeat)

            # legacy
            sio.emit('sendStatus', heartbeat)

            # this return value is delivered as the acknowledgement to the
            # server. note: cannot return a tuple. must return a single val
            return heartbeat

        @sio.on('*')
        def onCatchAll(data=None):
            self.onCatchAll(data)

        while self.running:
            if (not self.connected):
                try:
                    sio.connect(self.fullSyncServerURL)
                    self.connected = True
                    sio.wait()
                # TODO(sam): handle the "Already Connected" exception better
                except Exception as e:
                    logging.error('connection error: %s', e)
                    time.sleep(5)
                    pass
            else:
                logging.debug("already connected. looping...")
                time.sleep(5)

The SyncClient has been written generically, so you should be able to copy-paste this into your project with minimal modification. As shown above, the best way to use the SyncClient is to extend the class and override the methods as necessary.

But let’s go into more detail on how this SyncClient works:

The SyncClient uses Socket.io (which is really just a wrapper around websockets) under the hood to connect to the Sync Server. After an instance of the SyncClient has been initialized, .start() connects and registers Socket.io handlers for all of the events emitted by the Sync Server. If the SyncClient has been subclassed, or any handlers have been overridden, those are used instead of the defaults. The default handlers simply logs that they have been hit, along with any data they have received.

Adding the Docker Image

Now that you have the application code ready to go, create a Docker image that can be included in our Helm charts later on.

First, define your requirements:

# bear-echo/app/requirements.txt

python-socketio==5.7.1
python-engineio==4.3.4
websocket-client==1.4.1
requests==2.28.1
Flask==2.2.2

And now, the Dockerfile:

# bear-echo/Dockerfile

FROM python:3.9

WORKDIR /usr/src/app

COPY app/requirements.txt .

RUN apt-get update && \
    apt-get install libsm6 libxext6 -y && \
    pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir -r requirements.txt

COPY app/. .

EXPOSE 11777

CMD ["python", "-u", "main.py"]

At this point, your “bear-echo” project should look like:

├─ bear-echo/
    ├─ Dockerfile
    ├─ app/
        ├─ main.py
        ├─ syncer.py
        ├─ requirements.txt

Packaging application as a Helm Chart

You should now have a Docker image for our new Bear Echo edge application. Now create the Helm charts.

To keep things organized and easy to understand, and because the Sync Server is a separate, but dependent chart, you can take advantage of Helm subcharts. This means you are going to create a new Helm chart for Bear Echo, and then import the Sync Server chart as a dependency.

  1. Create a new Helm chart, and then wipe out the default YAML files:

    helm create chart
    rm -r chart/templates
    
  2. Copy the following files into your new helm chart:

    # bear-echo/chart/Chart.yaml
    
    apiVersion: v2
    name: bear-echo-runtime
    description: A Bear Echo (with Sync Server) example helm chart
    type: application
    version: 0.0.1
    appVersion: "0.0.1"
    
    icon: https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.1.2/svgs/solid/volume-high.svg
    
    dependencies: 
      - name: sync-server
        version: 0.0.1
        # alias needed to avoid issue with hyphenated key names in values.yaml
        # https://github.com/helm/helm/issues/2192
        alias: syncserver
    
    # bear-echo/chart/values.yaml
    
    syncserver:
      fullnameOverride: "bear-echo-syncserver"
      service:
        port: 8010
    
      config:
        apiKey: ""
        gbSiteID: ""
        gbSiteName: ""
        controlServerURL: ""
    
      image:
        repository: harbor.eticloud.io/gbear-dev/great-bear-sync-server
        pullPolicy: Always
        tag: develop
    
      imagePullSecrets:
        - name: gbear-harbor-pull
    
    
    # for bear-echo helm charts
    
    fullnameOverride: ""
    service:
      port: 11777
    
    config:
      requestPath: "/"
      defaultEchoText: "hi"
      # needs to be base64 encoded json. like `echo -n '{}' | base64`
      gbNodeIDs: "e30="
      appName: "bear-echo-runtime"
    
    image:
      repository: harbor.eticloud.io/gbear-dev/great-bear-bear-echo-runtime
      pullPolicy: Always
      tag: develop
    
    imagePullSecrets:
      - name: gbear-harbor-pull
    
    {{/* bear-echo/chart/templates/_helpers.tpl */}}
    
    {{/*
    Expand the name of the chart.
    */}}
    {{- define "bear-echo.name" -}}
    {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
    {{- end }}
    
    {{/*
    Create a default fully qualified app name.
    We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
    If release name contains chart name it will be used as a full name.
    */}}
    {{- define "bear-echo.fullname" -}}
    {{- if .Values.fullnameOverride }}
    {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
    {{- else }}
    {{- $name := default .Chart.Name .Values.nameOverride }}
    {{- if contains $name .Release.Name }}
    {{- .Release.Name | trunc 63 | trimSuffix "-" }}
    {{- else }}
    {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
    {{- end }}
    {{- end }}
    {{- end }}
    
    {{/*
    Create chart name and version as used by the chart label.
    */}}
    {{- define "bear-echo.chart" -}}
    {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
    {{- end }}
    
    {{/*
    Common labels
    */}}
    {{- define "bear-echo.labels" -}}
    helm.sh/chart: {{ include "bear-echo.chart" . }}
    {{ include "bear-echo.selectorLabels" . }}
    {{- if .Chart.AppVersion }}
    app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
    {{- end }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
    {{- end }}
    
    {{/*
    Selector labels
    */}}
    {{- define "bear-echo.selectorLabels" -}}
    app.kubernetes.io/name: {{ include "bear-echo.name" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}
    {{- end }}
    
    # bear-echo/chart/templates/all.yaml
    
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: {{ include "bear-echo.fullname" . }}
      labels:
        {{- include "bear-echo.labels" . | nindent 4 }}
    spec:
      replicas: 1
      selector:
        matchLabels:
          {{- include "bear-echo.selectorLabels" . | nindent 6 }}
      template:
        metadata:
          labels:
            {{- include "bear-echo.selectorLabels" . | nindent 8 }}
        spec:
          {{- with .Values.imagePullSecrets }}
          imagePullSecrets:
            {{- toYaml . | nindent 8 }}
          {{- end }}
          containers:
            - name: {{ .Chart.Name }}
              image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
              imagePullPolicy: {{ .Values.image.pullPolicy }}
              env:
                - name: HTTP_PORT
                  value: {{ .Values.service.port | quote }}
                - name: BASE_URL
                  value: {{ .Values.config.requestPath | quote }}
                - name: ECHO_TEXT
                  value: {{ .Values.config.defaultEchoText | quote }}
                - name: SYNC_SERVER_HOST
                  value: {{ .Values.syncserver.fullnameOverride | quote }}
                - name: SYNC_SERVER_PORT
                  value: {{ .Values.syncserver.service.port | quote }}
                - name: GB_NODE_ID
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                - name: GB_NODE_IDS
                  value: |-
                                    {{ .Values.config.gbNodeIDs | b64dec }}
                - name: APP_NAME
                  value: {{ .Values.config.appName | quote }}
              ports:
                - name: http
                  containerPort: {{ .Values.service.port }}
                  protocol: TCP
              livenessProbe:
                exec:
                  command:
                  - cat
                  - main.py
                initialDelaySeconds: 10
                periodSeconds: 5
    
    ---
    
    apiVersion: v1
    kind: Service
    metadata:
      name: {{ include "bear-echo.fullname" . }}
      labels:
        {{- include "bear-echo.labels" . | nindent 4 }}
    spec:
      type: {{ .Values.service.type }}
      ports:
        - port: {{ .Values.service.port }}
          targetPort: http
          protocol: TCP
          name: http
      selector:
        {{- include "bear-echo.selectorLabels" . | nindent 4 }}
    

    You should now have a directory structure like:

    ├─ bear-echo/
        ├─ Dockerfile
        ├─ app/
            ├─ main.py
            ├─ syncer.py
            ├─ requirements.txt
        ├─ chart/
            ├─ Chart.yaml
            ├─ values.yaml
            ├─ templates/
                ├─ _helpers.tpl
                ├─ all.yaml
    
  3. Finally, add the Sync Server subchart. To do this, copy and paste the following simplified YAML files:

    # bear-echo/chart/charts/sync-server/Chart.yaml
    
    apiVersion: v2
    name: sync-server
    description: A Sync Server Helm Chart
    type: application
    version: 0.0.1
    appVersion: "0.0.1"
    
    # bear-echo/chart/charts/sync-server/values.yaml
    
    config:
      apiKey: ""
      gbSiteName: ""
      gbSiteID: ""
      controlServerURL: ""
    
    image:
      repository: harbor.eticloud.io/gbear-dev/great-bear-sync-server
      pullPolicy: Always
      tag: develop
    
    imagePullSecrets:
      - name: gbear-harbor-pull
    
    fullnameOverride: "syncserver"
    
    service:
      type: ClusterIP
      port: 8090
    
    {{/* bear-echo/chart/charts/sync-server/templates/_helpers.tpl */}}
    
    {{/*
    Expand the name of the chart.
    */}}
    {{- define "sync-server.name" -}}
    {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
    {{- end }}
    
    {{/*
    Create a default fully qualified app name.
    We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
    If release name contains chart name it will be used as a full name.
    */}}
    {{- define "sync-server.fullname" -}}
    {{- if .Values.fullnameOverride }}
    {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
    {{- else }}
    {{- $name := default .Chart.Name .Values.nameOverride }}
    {{- if contains $name .Release.Name }}
    {{- .Release.Name | trunc 63 | trimSuffix "-" }}
    {{- else }}
    {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
    {{- end }}
    {{- end }}
    {{- end }}
    
    {{/*
    Create chart name and version as used by the chart label.
    */}}
    {{- define "sync-server.chart" -}}
    {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
    {{- end }}
    
    {{/*
    Common labels
    */}}
    {{- define "sync-server.labels" -}}
    helm.sh/chart: {{ include "sync-server.chart" . }}
    {{ include "sync-server.selectorLabels" . }}
    {{- if .Chart.AppVersion }}
    app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
    {{- end }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
    {{- end }}
    
    {{/*
    Selector labels
    */}}
    {{- define "sync-server.selectorLabels" -}}
    app.kubernetes.io/name: {{ include "sync-server.name" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}
    {{- end }}
    
    # bear-echo/chart/charts/sync-server/templates/all.yaml
    
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: {{ include "sync-server.fullname" . }}
      labels:
        {{- include "sync-server.labels" . | nindent 4 }}
    spec:
      replicas: 1
      selector:
        matchLabels:
          {{- include "sync-server.selectorLabels" . | nindent 6 }}
      template:
        metadata:
          labels:
            {{- include "sync-server.selectorLabels" . | nindent 8 }}
        spec:
          {{- with .Values.imagePullSecrets }}
          imagePullSecrets:
            {{- toYaml . | nindent 8 }}
          {{- end }}
          containers:
            - name: {{ .Chart.Name }}
              image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
              imagePullPolicy: {{ .Values.image.pullPolicy }}
              env:
                - name: API_KEY
                  value: {{ .Values.config.apiKey | quote }}
                - name: PORT_SYNC_SERVER
                  value: {{ .Values.service.port | quote }}
                - name: GB_SITE_NAME
                  value: {{ .Values.config.gbSiteName | quote }}
                - name: GB_SITE_ID
                  value: {{ .Values.config.gbSiteID | quote }}
                - name: CONTROL_SERVER_URL
                  value: {{ .Values.config.controlServerURL | quote }}
              ports:
                - name: http
                  containerPort: {{ .Values.service.port }}
                  protocol: TCP
              livenessProbe:
                exec:
                  command:
                  - cat
                  - app.js
                initialDelaySeconds: 10
                periodSeconds: 5
    
    ---
    
    apiVersion: v1
    kind: Service
    metadata:
      name: {{ include "sync-server.fullname" . }}
      labels:
        {{- include "sync-server.labels" . | nindent 4 }}
    spec:
      type: {{ .Values.service.type }}
      ports:
        - port: {{ .Values.service.port }}
          targetPort: http
          protocol: TCP
          name: http
      selector:
        {{- include "sync-server.selectorLabels" . | nindent 4 }}
    

    This is the complete directory structure now:

    ├─ bear-echo/
        ├─ Dockerfile
        ├─ app/
            ├─ main.py
            ├─ syncer.py
            ├─ requirements.txt
        ├─ chart/
            ├─ Chart.yaml
            ├─ values.yaml
            ├─ templates/
                ├─ _helpers.tpl
                ├─ all.yaml
            ├─ charts/
                ├─ sync-server/
                    ├─ Chart.yaml
                    ├─ values.yaml
                    ├─ templates/
                        ├─ _helpers.tpl
                        ├─ all.yaml
    

(Optional) Great Bear metadata

You should now have just about everything you need for your new Great Bear application that supports runtime configuration! There’s a bit more metadata that you can add to describe our application in the Great Bear appstore. (For details, see gbear/appmetadata.yaml.)

Here is the kind of metadata you might want for your Bear Echo + Sync Server application.

# bear-echo/chart/gbear/appmetadata.yaml

displayName: Bear Echo (with Sync Server)
description: Helm chart for Bear Echo with Sync Server

icon: https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.1.2/svgs/solid/volume-high.svg

labels:
   kind: EXAMPLE

tags:
   - sample

configuration:
  - name: defaultEchoText
    key: config.defaultEchoText
    title: "Default Echo Text"
    description: "Text to be Echo'd before Runtime Updates"
    value: "default"
    type: String

  - name: apiKey
    key: syncserver.config.apiKey
    title: "API Key (Deployment Token)"
    description: "API Key generated from app-control dashboard"
    type: Secret

  - name: siteID
    key: syncserver.config.gbSiteID
    value: "{{ .GB_SITE_ID }}"
    type: Runtime

  - name: siteName
    key: syncserver.config.gbSiteName
    value: "{{ .GB_SITE_NAME }}"
    type: Runtime

  - name: controlServerURL
    key: syncserver.config.controlServerURL
    title: "Control Server URL"
    description: "The URL of the Application Control Service"
    value: "https://app-control.dev.gbear.scratch.eticloud.io"
    type: String

  - name: nodeIDs
    key: config.gbNodeIDs
    value: "{{ .GB_NODE_IDS | b64enc }}"
    # actually passed in as json
    type: Runtime
├─ bear-echo/
    ├─ Dockerfile
    ├─ app/
        ├─ main.py
        ├─ syncer.py
        ├─ requirements.txt
    ├─ chart/
        ├─ Chart.yaml
        ├─ values.yaml
        ├─ gbear/
            ├─ appmetadata.yaml
        ├─ templates/
            ├─ _helpers.tpl
            ├─ all.yaml
        ├─ charts/
            ├─ sync-server/
                ├─ Chart.yaml
                ├─ values.yaml
                ├─ templates/
                    ├─ _helpers.tpl
                    ├─ all.yaml