Perses

[!WARNING] This plugin is in beta and please report any bugs by creating an issue here.

Table of Contents

Learn more about the Perses Plugin. Use it to visualize Prometheus/Thanos metrics for your Greenhouse remote cluster.

The main terminologies used in this document can be found in core-concepts.

Overview

Observability is often required for the operation and automation of service offerings. Perses is a CNCF project and it aims to become an open-standard for dashboards and visualization. It provides you with tools to display Prometheus metrics on live dashboards with insightful charts and visualizations. In the Greenhouse context, this complements the kube-monitoring plugin, which automatically acts as a Perses data source which is recognized by Perses. In addition, the Plugin provides a mechanism that automates the lifecycle of datasources and dashboards without having to restart Perses.

Perses Architecture

Disclaimer

This is not meant to be a comprehensive package that covers all scenarios. If you are an expert, feel free to configure the Plugin according to your needs.

Contribution is highly appreciated. If you discover bugs or want to add functionality to the plugin, then pull requests are always welcome.

Quick Start

This guide provides a quick and straightforward way how to use Perses as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

  • A running and Greenhouse-managed Kubernetes remote cluster
  • kube-monitoring Plugin should be installed with .spec.kubeMonitoring.prometheus.persesDatasource: true and it should have at least one Prometheus instance running in the cluster

The plugin works by default with anonymous access enabled. This plugin comes with some default dashboards and the kube-monitoring datasource will be automatically discovered by the plugin.

Step 1: Add your dashboards and datasources

Dashboards are selected from ConfigMaps across namespaces. The plugin searches for ConfigMaps with the label perses.dev/resource: "true" and imports them into Perses. The ConfigMap must contain a key like my-dashboard.json with the dashboard JSON content. Please refer this section for more information.

A guide on how to create custom dashboards on the UI can be found here.

Values

KeyTypeDefaultDescription
global.commonLabelsobject{}Labels to add to all resources. This can be used to add a support_group or service label to all resources and alerting rules.
greenhouseobject{"defaultDashboards":{"enabled":true}}By setting this to true, You will get some default dashboards
perses.additionalLabelsobject{}
perses.annotationsobject{}Statefulset Annotations
perses.config.annotationsobject{}Annotations for config
perses.config.api_prefixstring"/perses"
perses.config.databaseobject{"file":{"extension":"json","folder":"/perses"}}Database config based on data base type
perses.config.database.fileobject{"extension":"json","folder":"/perses"}file system configs
perses.config.frontendobject{"important_dashboards":[]}Important dashboards list
perses.config.provisioningobject{"folders":["/etc/perses/provisioning"]}provisioning config
perses.config.schemasobject{"datasources_path":"/etc/perses/cue/schemas/datasources","interval":"5m","panels_path":"/etc/perses/cue/schemas/panels","queries_path":"/etc/perses/cue/schemas/queries","variables_path":"/etc/perses/cue/schemas/variables"}Schemas paths
perses.config.security.cookieobject{"same_site":"lax","secure":false}cookie config
perses.config.security.enable_authboolfalseEnable Authentication
perses.config.security.readonlyboolfalseConfigure Perses instance as readonly
perses.fullnameOverridestring""Override fully qualified app name
perses.imageobject{"name":"persesdev/perses","pullPolicy":"IfNotPresent","version":""}Image of Perses
perses.image.namestring"persesdev/perses"Perses image repository and name
perses.image.pullPolicystring"IfNotPresent"Default image pull policy
perses.image.versionstring""Overrides the image tag whose default is the chart appVersion.
perses.ingressobject{"annotations":{},"enabled":false,"hosts":[{"host":"perses.local","paths":[{"path":"/","pathType":"Prefix"}]}],"ingressClassName":"","tls":[]}Configure the ingress resource that allows you to access Perses Frontend ref: https://kubernetes.io/docs/concepts/services-networking/ingress/
perses.ingress.annotationsobject{}Additional annotations for the Ingress resource. To enable certificate autogeneration, place here your cert-manager annotations. For a full list of possible ingress annotations, please see ref: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md
perses.ingress.enabledboolfalseEnable ingress controller resource
perses.ingress.hostslist[{"host":"perses.local","paths":[{"path":"/","pathType":"Prefix"}]}]Default host for the ingress resource
perses.ingress.ingressClassNamestring""IngressClass that will be be used to implement the Ingress (Kubernetes 1.18+) This is supported in Kubernetes 1.18+ and required if you have more than one IngressClass marked as the default for your cluster . ref: https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/
perses.ingress.tlslist[]Ingress TLS configuration
perses.livenessProbeobject{"enabled":true,"failureThreshold":5,"initialDelaySeconds":10,"periodSeconds":60,"successThreshold":1,"timeoutSeconds":5}Liveness probe configuration Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
perses.logLevelstring"info"Log level for Perses be configured in available options “panic”, “error”, “warning”, “info”, “debug”, “trace”
perses.nameOverridestring""Override name of the chart used in Kubernetes object names.
perses.persistenceobject{"accessModes":["ReadWriteOnce"],"annotations":{},"enabled":false,"labels":{},"securityContext":{"fsGroup":2000},"size":"8Gi"}Persistence parameters
perses.persistence.accessModeslist["ReadWriteOnce"]PVC Access Modes for data volume
perses.persistence.annotationsobject{}Annotations for the PVC
perses.persistence.enabledboolfalseIf disabled, it will use a emptydir volume
perses.persistence.labelsobject{}Labels for the PVC
perses.persistence.securityContextobject{"fsGroup":2000}Security context for the PVC when persistence is enabled
perses.persistence.sizestring"8Gi"PVC Storage Request for data volume
perses.readinessProbeobject{"enabled":true,"failureThreshold":5,"initialDelaySeconds":5,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":5}Readiness probe configuration Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
perses.replicasint1Number of pod replicas.
perses.resourcesobject{}Resource limits & requests. Update according to your own use case as these values might be too low for a typical deployment. ref: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
perses.serviceobject{"annotations":{},"labels":{"greenhouse.sap/expose":"true"},"port":8080,"portName":"http","targetPort":8080,"type":"ClusterIP"}Expose the Perses service to be accessed from outside the cluster (LoadBalancer service). or access it from within the cluster (ClusterIP service). Set the service type and the port to serve it.
perses.service.annotationsobject{}Annotations to add to the service
perses.service.labelsobject{"greenhouse.sap/expose":"true"}Labeles to add to the service
perses.service.portint8080Service Port
perses.service.portNamestring"http"Service Port Name
perses.service.targetPortint8080Perses running port
perses.service.typestring"ClusterIP"Service Type
perses.serviceAccountobject{"annotations":{},"create":true,"name":""}Service account for Perses to use.
perses.serviceAccount.annotationsobject{}Annotations to add to the service account
perses.serviceAccount.createbooltrueSpecifies whether a service account should be created
perses.serviceAccount.namestring""The name of the service account to use. If not set and create is true, a name is generated using the fullname template
perses.sidecarobject{"allNamespaces":true,"enabled":true,"label":"perses.dev/resource","labelValue":"true"}Sidecar configuration that watches for ConfigMaps with the specified label/labelValue and loads them into Perses provisioning
perses.sidecar.allNamespacesbooltruecheck for configmaps from all namespaces. When set to false, it will only check for configmaps in the same namespace as the Perses instance
perses.sidecar.enabledbooltrueEnable the sidecar container for ConfigMap provisioning
perses.sidecar.labelstring"perses.dev/resource"Label key to watch for ConfigMaps containing Perses resources
perses.sidecar.labelValuestring"true"Label value to watch for ConfigMaps containing Perses resources
perses.volumeMountslist[]Additional VolumeMounts on the output StatefulSet definition.
perses.volumeslist[]Additional volumes on the output StatefulSet definition.

Create a custom dashboard

  1. Add a new Project by clicking on ADD PROJECT in the top right corner. Give it a name and click Add.
  2. Add a new dashboard by clicking on ADD DASHBOARD. Give it a name and click Add.
  3. Now you can add variables, panels to your dashboard.
  4. You can group your panels by adding the panels to a Panel Group.
  5. Move and resize the panels as needed.
  6. Watch this gif to learn more.
  7. You do not need to add the kube-monitoring datasource manually. It will be automatically discovered by Perses.
  8. Click Save after you have made changes.
  9. Export the dashboard.
    • Click on the {} icon in the top right corner of the dashboard.
    • Copy the entire JSON model.
    • See the next section for detailed instructions on how and where to paste the copied dashboard JSON model.

Add Dashboards as ConfigMaps

By default, a sidecar container is deployed in the Perses pod. This container watches all configmaps in the cluster and filters out the ones with a label perses.dev/resource: "true". The files defined in those configmaps are written to a folder and this folder is accessed by Perses. Changes to the configmaps are continuously monitored and are reflected in Perses within 10 minutes.

A recommendation is to use one configmap per dashboard. This way, you can easily manage the dashboards in your git repository.

Folder structure:

dashboards/
├── dashboard1.json
├── dashboard2.json
├── prometheusdatasource1.json
├── prometheusdatasource2.json
templates/
├──dashboard-json-configmap.yaml

Helm template to create a configmap for each dashboard:

{{- range $path, $bytes := .Files.Glob "dashboards/*.json" }}
---
apiVersion: v1
kind: ConfigMap

metadata:
  name: {{ printf "%s-%s" $.Release.Name $path | replace "/" "-" | trunc 63 }}
  labels:
    perses.dev/resource: "true"

data:
{{ printf "%s: |-" $path | replace "/" "-" | indent 2 }}
{{ printf "%s" $bytes | indent 4 }}

{{- end }}