Greenhouse documentation

• 1: Getting started
• 1.1: Overview
• 1.2: Quick start guide
• 2: User guides
• 2.1: Organization management
• 2.1.1: SAP ID Service
• 2.1.2: Creating an organization
• 2.2: Cluster management
• 2.2.1: Cluster onboarding
• 2.3: Plugin management
• 2.3.1: Local Plugin Development
• 2.3.2: Testing a Plugin
• 2.3.3: Plugin deployment
• 2.3.4: Managing Plugins for multiple clusters
• 2.3.5: Plugin Catalog
• 2.4: Team management
• 2.4.1: Role-based access control
• 2.4.2: Team creation
• 3: Architecture
• 3.1: High-level architecture
• 3.2: Product design
• 3.3: Monitoring architecture
• 4: Reference
• 4.1: API
• 4.2: Plugin Catalog
• 4.2.1: Alerts
• 4.2.2: Cert-manager
• 4.2.3: Decentralized Observer of Policies (Violations)
• 4.2.4: Designate Ingress CNAME operator (DISCO)
• 4.2.5: DigiCert issuer
• 4.2.6: External DNS
• 4.2.7: Ingress NGINX
• 4.2.8: Kubernetes monitoring
• 4.2.9: Logshipper
• 4.2.10: OpenTelemetry
• 4.2.11: Plutono
• 4.2.12: Service exposure test
• 4.2.13: Teams2Slack
• 4.2.14: Thanos
• 5: Contribute
• 5.1: Local development setup
• 5.2: Contributing a Plugin
• 6:
• 7:

1 - Getting started

1.1 - Overview

What is Greenhouse?

Greenhouse is a cloud operations platform designed to streamline and simplify the management of a large-scale, distributed infrastructure.

It offers a unified interface for organizations to manage various operational aspects efficiently and transparently and operate their cloud infrastructure in compliance with industry standards.
The platform addresses common challenges such as the fragmentation of tools, visibility of application-specific permission concepts and the management of organizational groups. It also emphasizes the harmonization and standardization of authorization concepts to enhance security and scalability. With its operator-friendly dashboard, features and extensive automation capabilities, Greenhouse empowers organizations to optimize their cloud operations, reduce manual efforts, and achieve greater operational efficiency.

Value Propositions

Roadmap

The Roadmap Kanban board provides an overview of ongoing and planned efforts.

Architecture & Design

The Greenhouse design and architecture document describes the various use-cases and user stories.

1.2 - Quick start guide

This section provides a step-by-step walkthrough for new users to navigate the initial stages of the Greenhouse platform.

Sign up

You’re an administrator and organization lead and want to register for Greenhouse?
We got you covered!

During phase 1 and 2 of the roadmap Greenhouse is only open to selected early adopters.
Please reach out to the Greenhouse team to register and create your organization via Slack or DL Greenhouse.

Prerequisites:

1. CAM Profile
   A CAM profile is required to configure the administrators of the organization.
   Please include the name of the profile in the message to the Greenhouse team when signing up.

2. SAP ID service
   The authentication for the users belonging to your organization is based on the OpenID Connect (OIDC) standard.
   For SAP, we recommend using a SAP ID service (IDS) tenant.
   Please include the parameters for your tenant in the message to the Greenhouse team when signing up.

   If you don’t have a SAP ID Service tenant yet, please refer to the SAP ID Service section for more information.

Already a member

You’re a member of an existing organization and want to manage your teams, clusters or plugins?
Please refer to the user guides to find out more.

2 - User guides

2.1 - Organization management

This section provides guides for the management your organization in Greenhouse.

2.1.1 - SAP ID Service

This section provides a step-by-step walkthrough for new users to request an SAP ID Service (IDS) tenant.

NOTE: This document is only available on the SAP-internal documentation page.

2.1.2 - Creating an organization

Before you begin

This guides describes how to create an organization in Greenhouse.

During phase 1 and 2 of the roadmap Greenhouse is only open to selected early adopters.
Please reach out to the Greenhouse team to register and create your organization via Slack or DL Greenhouse.

Creating an organization

An organization within the Greenhouse cloud operations platform is a separate unit with its own configuration, teams, and resources tailored to their requirements.
These organizations can represent different teams, departments, or projects within an enterprise, and they operate independently within the Greenhouse platform. They allow for the isolation and management of resources and configurations specific to their needs.

While the Greenhouse is build on the idea of a self-service API and automation driven platform, the workflow to onboard an organization to Greenhouse currently involves reaching out to the Greenhouse administrators until the official go-live.
This ensures all pre-requisites are met, the organization is configured correctly and the administrators understand the platform capabilities.

Steps

1. CAM Profile
   A CAM profile is required to configure the administrators of the organization.
   Please include the name of the profile in the message to the Greenhouse team when signing up.

2. SAP ID service
   The authentication for the users belonging to your organization is based on the OpenID Connect (OIDC) standard.
   For SAP, we recommend using a SAP ID service (IDS) tenant.
   Please include the parameters for your tenant in the message to the Greenhouse team when signing up.

   If you don’t have a SAP ID Service tenant yet, please refer to the SAP ID Service section for more information.

3. Greenhouse organization
   A Greenhouse administrator applies the following configuration to the central Greenhouse cluster.
   Bear in mind that the name of the organization is immutable and will be part of all URLs.

   apiVersion: v1
   kind: Namespace
   metadata:
     name: my-organization
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: oidc-config
     namespace: my-organization
   type: Opaque
   data:
     clientID: ...
     clientSecret: ...
   ---
   apiVersion: greenhouse.sap/v1alpha1
   kind: Organization
   metadata:
     name: my-organization
   spec:
     authentication:
       oidc:
         clientIDReference:
           key: clientID
           name: oidc-config
         clientSecretReference:
           key: clientSecret
           name: oidc-config
         issuer: https://...
     description: My new organization
     displayName: Short name of the organization
     mappedOrgAdminIdPGroup: Name of the group in the IDP that should be mapped to the organization admin role.
   

2.2 - Cluster management

Greenhouse enables organizations to register their Kubernetes clusters within the platform, providing a centralized interface for managing and monitoring these clusters.
Once registered, users can perform tasks related to cluster management, such as deploying applications, scaling resources, and configuring access control, all within the Greenhouse platform.

This section provides guides for the management of Kubernetes clusters within Greenhouse.

2.2.1 - Cluster onboarding

This guides describes how to onboard an existing Kubernetes cluster to your Greenhouse organization.
If you don’t have an organization yet please reach out to the Greenhouse administrators.

While all members of an organization can see existing clusters, their management requires organization admin privileges.

NOTE: The UI is currently in development. For now this guide describes the onboarding workflow via command line.

Preparation

Download the latest greenhousectl binary from here.

The command line tool requires access to both the Greenhouse and your Kubernetes cluster. Hence, have the kubeconfig files for both clusters at hand. The kubeconfig file for the Greenhouse Kubernetes cluster can be downloaded via the Greenhouse dashboard: Organization > Clusters > Access greenhouse cluster.

For accessing the Greenhouse Kubernetes cluster, the greenhousectl will check whether your local kubectl is connected to the Greenhouse Kubernetes cluster. If not connected, either the environment variables KUBECONFIG and KUBECONTEXT or the greenhousectl flags --kubeconfig and --kubecontext
must point to the respective Greenhouse kubeconfig.

For accessing your Kubernetes cluster, it’s mandatory to provide the kubeconfig to the greenhousectl using the flag --bootstrap-kubeconfig

Onboard

Since Greenhouse generates URLs which contain the cluster name, we highly recommend to choose a short cluster name. In particular for Gardener Clusters setting a short name is mandatory, because Gardener has very long cluster names, e.g. garden-greenhouse--monitoring-external.

greenhousectl cluster bootstrap --bootstrap-kubeconfig <path/to/cluster-kubeconfig-file> --org <greenhouse-organization-name> --cluster-name <name>

A typical output when you ran the command looks like

2024-02-01T09:34:55.522+0100	INFO	setup	Loaded kubeconfig	{"context": "default", "host": "https://api.greenhouse-qa.eu-nl-1.cloud.sap"}
2024-02-01T09:34:55.523+0100	INFO	setup	Loaded client kubeconfig	{"host": "https://api.monitoring.greenhouse.shoot.canary.k8s-hana.ondemand.com"}
2024-02-01T09:34:56.579+0100	INFO	setup	Bootstraping cluster	{"clusterName": "monitoring", "orgName": "ccloud"}
2024-02-01T09:34:56.639+0100	INFO	setup	created namespace	{"name": "ccloud"}
2024-02-01T09:34:56.696+0100	INFO	setup	created serviceAccount	{"name": "greenhouse"}
2024-02-01T09:34:56.810+0100	INFO	setup	created clusterRoleBinding	{"name": "greenhouse"}
2024-02-01T09:34:56.867+0100	INFO	setup	created serviceAccount	{"name": "tailscale"}
2024-02-01T09:34:56.925+0100	INFO	setup	created role	{"name": "tailscale"}
2024-02-01T09:34:56.982+0100	INFO	setup	created roleBinding	{"name": "tailscale"}
2024-02-01T09:34:57.189+0100	INFO	setup	created clusterSecret	{"name": "monitoring"}
2024-02-01T09:34:58.309+0100	INFO	setup	Bootstraping cluster finished	{"clusterName": "monitoring", "orgName": "ccloud"}

After onboarding

1. List all clusters in your Greenhouse organization:

   kubectl --namespace=<greenhouse-organization-name> get clusters

2. Show the details of a cluster:

   kubectl --namespace=<greenhouse-organization-name> get cluster <name> -o yaml

Example:

apiVersion: greenhouse.sap/v1alpha1
kind: Cluster
metadata:
  creationTimestamp: "2024-02-07T10:23:23Z"
  finalizers:
  - greenhouse.sap/cluster
  generation: 1
  name: monitoring
  namespace: ccloud
  resourceVersion: "282792586"
  uid: 0db6e464-ec36-459e-8a05-4ad668b57f42
spec:
  accessMode: direct
status:
  bearerTokenExpirationTimestamp: "2024-02-09T06:28:57Z"
  kubernetesVersion: v1.27.8
...

When the status.kubernetesVersion field shows the correct version of the Kubernetes cluster, the cluster was successfully bootstrapped in Greenhouse. In the remote cluster, a new namespace is created and contains some resources managed by Greenhouse. The namespace has the same name as your organization in Greenhouse.

If the bootstrapping failed, you can delete the Kubernetes cluster from Greenhouse with kubectl --namespace=<greenhouse-organization-name> delete cluster <name> and run the bootstrap command again.

2.3 - Plugin management

Plugins extends the capabilities of the Greenhouse cloud operations platform, adding specific features or functionalities to tailor and enhance the platform for specific organizational needs.
These plugins are integral to Greenhouse’ extensibility, allowing users to customize their cloud operations environment and address unique requirements while operating within the Greenhouse ecosystem.

This section provides guides for the management of plugins for Kubernetes clusters within Greenhouse.

2.3.1 - Local Plugin Development

Introduction

Let’s illustrate how to leverage Greenhouse Plugins to deploy a Helm Chart into a remote cluster within the local development environment.

This guide will walk you through the process of spinning up the local development environment, creating a new Greenhouse PluginDefinition and deploying it to a local kind cluster.

At the end of the guide you will have spun up the local development environment, onboarded a Cluster, created a PluginDefinition and deployed it as a Plugin to the onboarded Cluster.

[!NOTE] This guide assumes you already have a working Helm chart and will not cover how to create a Helm Chart from scratch. For more information on how to create a Helm Chart, please refer to the Helm documentation.

Requirements

• git
• Docker
• kind
• kubectl
• Greenhousectl

Starting the local develoment environment

Follow the Local Development documentation to spin up the local Greenhouse development environment.

This will provide you with a local Greenhouse instance running, filled with some example Greenhouse resources and the Greenhouse UI running on http://localhost:3000.

Onboarding a Cluster

In this step we will create and onboard a new Cluster to the local Greenhouse instance. The local cluster will be created utilizing kind.

In order to onboard a kind cluster follow the onboarding a cluster secton of the dev-env README.

After onboarding the cluster you should see the new Cluster in the Greenhouse UI.

Prepare Helm Chart

For this example we will use the bitnami nginx Helm Chart. The packaged chart can be downloaded with:

helm pull oci://registry-1.docker.io/bitnamicharts/nginx --destination ./

After unpacking the *.tgz file there is a folder named nginx containing the Helm Chart.

Generating a PluginDefinition from a Helm Chart

Using the files of the Helm Chart we will create a new Greenhouse PluginDefinition using the greenhousectl CLI.

greenhousectl plugin generate ./nginx ./nginx-plugin

This will create a new folder nginx-plugin containing the PluginDefinition in a nested structure.

Modifying the PluginDefinition

The generated PluginDefinition contains a plugindefinition.yaml file which defines the PluginDefinition. But there are still a few steps required to make it work.

Specify the Helm Chart repository

After generating the PluginDefinition the .spec.helmChart.repository field in the plugindefinition.yaml contains a TODO comment. This field should be set to the repository where the Helm Chart is stored. For the bitnami nginx Helm Chart this would be oci://registry-1.docker.io/bitnamicharts.

Specify the UI application

A PluginDefinition may specify a UI application that will be integrated into the Greenhouse UI. This tutorial does not cover how to create a UI application. Therefore the section .spec.uiApplication in the plugindefinition.yaml should be removed.

[!INFORMATION] The UI section of the dev-env readme provides a brief introduction developing a frontend application for Greenhouse.

Modify the Options

The PluginDefinition contains a section .spec.options which defines options that can be set when deploying the Plugin to a Cluster. These options have been generated based on the Helm Chart values.yaml file. You can modify the options to fit your needs.

In general the options are defined as follows:

options:
  - default: true
    value: abcd123
    description: automountServiceAccountToken
    name: automountServiceAccountToken
    required: false
    type: ""

default specifies if the option should provide a default value. If this is set to true, the value specified will be used as the default value. The Plugin can still provide a different value for this option. description provides a description for the option. name specifies the Helm Chart value name, as it is used within the Chart’s template files. required specifies if the option is required. This will be used by the Greenhouse Controllers to determine if a Plugin is valid. type specifies the type of the option. This can be any of [string, secret, bool, int, list, map]. This will be used by the Greenhouse Controllers to validate the provided value.

For this tutorial we will remove all options.

Deploying a Plugin to the Kind Cluster

After modifying the PluginDefinition we can deploy it to the local Greenhouse cluster and create a Plugin that will deploy the nginx to the onboarded cluster.

  kubectl --kubeconfig=./envtest/kubeconfig apply -f ./nginx-plugin/nginx/17.3.2/plugindefinition.yaml
  plugindefinition.greenhouse.sap/nginx-17.3.2 created

The Plugin can be configured using the Greenhouse UI running on http://localhost:3000. Follow the following steps to deploy a Plugin for the created PluginDefinition into the onboarded kind cluster:

1. Navigate on the Greenhouse UI to Organization>Plugins.
2. Click on the Add Plugin button.
3. Select the nginx-17.3.2 PluginDefinition.
4. Click on the Configure Plugin button.
5. Select the cluster in the drop-down.
6. Click on the Create Plugin button.

After the Plugin has been created the Plugin Overview page will show the status of the plugin.

Theh deployment can also be verified in the onboarded cluster by checking the pods in the test-org namespace of the kind cluster.

kind export kubeconfig --name remote-cluster
Set kubectl context to "kind-remote-cluster"

k get pods -n test-org
NAME                                    READY   STATUS    RESTARTS   AGE
nginx-remote-cluster-758bf47c77-pz72l   1/1     Running   0          2m11s

Development Tips

Local Helm Charts

Instead of uploading the Helm Chart to a chart repository, it is possible to load it from the filesystem of the Greenhouse container. This can be especially useful if you are developing your own chart for a PluginDefinition, as it speeds up the testing loop. The Docker compose setup mounts the dev-env/helm-charts directory and watches for any changes. This means you can point to this local chart in your plugindefinition.yaml as such:

helmChart:
  name: helm-charts/{filename}.tgz
  repository:

2.3.2 - Testing a Plugin

Plugin Testing Requirements

All plugins contributed to plugin-extensions repository should include comprehensive Helm Chart Tests using the bats/bats-detik testing framework. This ensures our plugins are robust, deployable, and catch potential issues early in the development cycle.

What is bats/bats-detik?

The bats/bats-detik framework simplifies end-to-end (e2e) Testing in Kubernetes. It combines the Bash Automated Testing System (bats) with Kubernetes-specific assertions (detik). This allows you to write test cases using natural language-like syntax, making your tests easier to read and maintain.

Implementing Tests

1. Create a /tests folder inside your Plugin’s Helm Chart templates folder to store your test resources.

2. ConfigMap definition:

   • Create a test-<plugin-name>-config.yaml file in the templates/tests directory to define a ConfigMap that will hold your test script.
   • This ConfigMap contains the test script run.sh that will be executed by the test Pod to run your tests.

{{- if .Values.testFramework.enabled -}}
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-test
  namespace: {{ .Release.Namespace }}
  labels:
    type: integration-test
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-weight": "-5" # Installed and upgraded before the test pod
    "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded"
data:
  run.sh: |-

    #!/usr/bin/env bats

    load "/usr/lib/bats/bats-detik/utils"
    load "/usr/lib/bats/bats-detik/detik"

    DETIK_CLIENT_NAME="kubectl"

    @test "Verify successful deployment and running status of the {{ .Release.Name }}-operator pod" {
        verify "there is 1 deployment named '{{ .Release.Name }}-operator'"
        verify "there is 1 service named '{{ .Release.Name }}-operator'"
        try "at most 2 times every 5s to get pods named '{{ .Release.Name }}-operator' and verify that '.status.phase' is 'running'"
    }

    @test "Verify successful creation and bound status of {{ .Release.Name }} persistent volume claims" {
        try "at most 3 times every 5s to get persistentvolumeclaims named '{{ .Release.Name }}.*' and verify that '.status.phase' is 'Bound'"
    }

    @test "Verify successful creation and available replicas of {{ .Release.Name }} Prometheus resource" {
        try "at most 3 times every 5s to get prometheuses named '{{ .Release.Name }}' and verify that '.status.availableReplicas' is more than '0'"
    }

    @test "Verify creation of required custom resource definitions (CRDs) for {{ .Release.Name }}" {
        verify "there is 1 customresourcedefinition named 'prometheuses'"
        verify "there is 1 customresourcedefinition named 'podmonitors'"
    }
{{- end -}}

Note: You can use this guide for reference when writing your test assertions.

3. Test Pod Definition:

   • Create a test-<plugin-name>.yaml file in the templates/tests directory to define a Pod that will run your tests.
   • This test Pod will mount the ConfigMap created in the previous step and will execute the test script run.sh.

{{- if .Values.testFramework.enabled -}}
apiVersion: v1
kind: Pod
metadata:
  name: {{ .Release.Name }}-test
  namespace: {{ .Release.Namespace }}
  labels:
    type: integration-test
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded"
spec:
  serviceAccountName: {{ .Release.Name }}-test
  containers:
    - name: bats-test
      image: "{{ .Values.testFramework.image.registry}}/{{ .Values.testFramework.image.repository}}:{{ .Values.testFramework.image.tag }}"
      imagePullPolicy: {{ .Values.testFramework.image.pullPolicy }}
      command: ["bats", "-t", "/tests/run.sh"]
      volumeMounts:
        - name: tests
          mountPath: /tests
          readOnly: true   volumes:
    - name: tests
      configMap:
        name: {{ .Release.Name }}-test
  restartPolicy: Never
{{- end -}}

4. RBAC Permissions:

• Create the necessary RBAC resources in the templates/tests folder with a dedicated ServiceAccount and role authorisations so that the test Pod can cover test the cases.
• You can use test-permissions.yaml from the kube-monitoring as a reference to configure RBAC permissions for your test Pod.

5. Configure the Test Framework in Plugin’s values.yaml:
   • Add the following configuration to your Plugin’s values.yaml file:

testFramework:
  enabled: true
  image:
    registry: ghcr.io
    repository: cloudoperators/greenhouse-extensions-integration-test
    tag: main
  imagePullPolicy: IfNotPresent

6. Running the Tests:

Important: Once you have completed all the steps above, you are ready to run the tests. However, before running the tests, ensure that you perform a fresh Helm installation or upgrade of your Plugin’s Helm release against your test Kubernetes cluster (for example, Minikube or Kind) by executing the following command:

# For a new installation
helm install <Release name> <chart-path>

# For an upgrade
helm upgrade <Release name> <chart-path>

• After the Helm installation or upgrade is successful, run the tests against the same test Kubernetes cluster by executing the following command.

helm test <Release name>

Contribution Checklist

Before submitting a pull request:

• Ensure your Plugin’s Helm Chart includes a /tests directory.
• Verify the presence of test-<plugin-name>.yaml, test-<plugin-name>-config.yaml, and test-permissions.yaml files.
• Test your Plugin thoroughly using helm test <release-name> and confirm that all tests pass against a test Kubernetes cluster.
• Include a brief description of the tests in your pull request.
• Make sure that your Plugin’s Chart Directory and the Plugin’s Upstream Chart Repository are added to this greenhouse-extensions helm test config file. This will ensure that your Plugin’s tests are automatically run in the GitHub Actions workflow when you submit a pull request for this Plugin.

Important Notes

• Test Coverage: Aim for comprehensive test coverage to ensure your Plugin’s reliability.
• Test Isolation: Design tests that don’t interfere with other plugins or production environments.

2.3.3 - Plugin deployment

Before you begin

This guides describes how to configure and deploy a Greenhouse plugin.

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: kube-monitoring-martin
  namespace: <organization namespace> # same namespace in remote cluster for resources
spec:
  clusterName: <name of the remote cluster > # k get cluster
  disabled: false
  displayName: <any human readable name>
  pluginDefinition: <plugin name> # k get plugin
  optionValues:
    - name: <from the plugin options>
      value: <from the plugin options>
    - ...

Deploying a Plugin

Create the Plugin resource via the command:

kubectl --namespace=<organization name> create -f plugin.yaml

After deployment

1. Check with kubectl --namespace=<organization name> get plugin has been properly created. When all components of the plugin are successfully created, the plugin should show the state configured.

2. Check in the remote cluster that all plugin resources are created in the organization namespace.

2.3.4 - Managing Plugins for multiple clusters

Managing Plugins for multiple clusters

This guide describes how to configure and deploy a Greenhouse Plugin with the same configuration into multiple clusters.

The PluginPreset resource is used to create and deploy Plugins with a the identical configuration into multiple clusters. The list of clusters the Plugins will be deployed to is determind by a LabelSelector.

As a result, whenever a cluster, that matches the ClusterSelector is onboarded or offboarded, the Controller for the PluginPresets will take care of the Plugin Lifecycle. This means creating or deleting the Plugin for the respective cluster.

The same validation applies to the PluginPreset as to the Plugin. This includes immutable PluginDefinition and ReleaseNamespace fields, as well as the validation of the OptionValues against the PluginDefinition.

In case the PluginPreset is updated all of the Plugin instances that are managed by the PluginPreset will be updated as well. Each Plugin instance that is created from a PluginPreset has a label greenhouse.sap/pluginpreset: <PluginPreset name>. Also the name of the Plugin follows the scheme <PluginPreset name>-<cluster name>.

Changes that are done directly on a Plugin which was created from a PluginPreset will be overwritten immediately by the PluginPreset Controller. All changes must be performed on the PluginPreset itself. If a Plugin already existed with the same name as the PluginPreset would create, this Plugin will be ignored in following reconciliations.

Example PluginPreset

apiVersion: greenhouse.sap/v1alpha1
kind: PluginPreset
metadata:
  name: kube-monitoring-preset
  namespace: <organization namespace>
spec:
  plugin: # this embeds the PluginSpec
    displayName: <any human readable name>
    pluginDefinition: <PluginDefinition name> # k get plugindefinition
    releaseNamespace: <namespace> # namespace where the plugin is deployed to on the remote cluster. Will be created if not exists
    optionValues:
      - name: <from the PluginDefinition options>
        value: <from the PluginDefinition options>
      - ..
  clusterSelector: # LabelSelector for the clusters the Plugin should be deployed to
    matchLabels:
      <label-key>: <label-value>
  clusterOptionOverrides: # allows you to override specific options in a given cluster
    - clusterName: <cluster name where we want to override values>
      overrides:
        - name: <option name to override>
          value: <new value>
        - ..
    - ..

2.3.5 - Plugin Catalog

Before you begin

This guides describes how to explore the catalog of Greenhouse PluginDefinitions.

While all members of an organization can see the Plugin catalog, enabling, disabling and configuration PluginDefinitions for an organization requires organization admin privileges.

Exploring the PluginDefinition catalog

The PluginDefinition resource describes the backend and frontend components as well as mandatory configuration options of a Greenhouse extension.
While the PluginDefinition catalog is managed by the Greenhouse administrators and the respective domain experts, administrators of an organization can configure and tailor Plugins to their specific requirements.

NOTE: The UI also provides a preliminary catalog of Plugins under Organization> Plugin> Add Plugin.

1. Run the following command to see all available PluginDefinitions.

   $ kubectl get plugindefinition
   
   NAME                      VERSION   DESCRIPTION                                                                                                  AGE
   cert-manager              1.1.0     Automated certificate management in Kubernetes                                                               182d
   digicert-issuer           1.2.0     Extensions to the cert-manager for DigiCert support                                                          182d
   disco                     1.0.0     Automated DNS management using the Designate Ingress CNAME operator (DISCO)                                  179d
   doop                      1.0.0     Holistic overview on Gatekeeper policies and violations                                                      177d
   external-dns              1.0.0     The kubernetes-sigs/external-dns plugin.                                                                     186d
   heureka                   1.0.0     Plugin for Heureka, the patch management system.                                                             177d
   ingress-nginx             1.1.0     Ingress NGINX controller                                                                                     187d
   kube-monitoring           1.0.1     Kubernetes native deployment and management of Prometheus, Alertmanager and related monitoring components.   51d
   prometheus-alertmanager   1.0.0     Prometheus alertmanager                                                                                      60d
   supernova                 1.0.0     Supernova, the holistic alert management UI                                                                  187d
   teams2slack               1.1.0     Manage Slack handles and channels based on Greenhouse teams and their members                                115d
   

2.4 - Team management

A team is a group of users with shared responsibilities for managing and operating cloud resources within a Greenhouse organization.
These teams enable efficient collaboration, access control, and task assignment, allowing organizations to effectively organize their users and streamline cloud operations within the Greenhouse platform.

This section provides guides for the management of teams within an organization.

2.4.1 - Role-based access control

Before you begin

This guides describes how to manage roles and permissions in Greenhouse with the help of TeamRoles and TeamRoleBindings.

While all members of an organization can see the permissions configured with TeamRoles & TeamRoleBindings, configuration of these requires OrganizationAdmin privileges.

Greenhouse Team RBAC user guide

Role-Based Access Control (RBAC) in Greenhouse allows organization administrators to regulate access to Kubernetes resources in onboarded Clusters based on the roles of individual users within an Organization. Within Greenhouse the RBAC on remote Clusters is managed using TeamRole and TeamRoleBinding. These two Custom Resource Defintions allow for fine-grained control over the permissions of each Team within each Cluster and Namespace.

Overview

• TeamRole: Defines a set of permissions that can be assigned to teams.
• TeamRoleBinding: Assigns a TeamRole to a specific Team for certain Clusters and (optionally) Namespaces.

Defining TeamRoles

TeamRoles define what actions a team can perform within the Kubernetes cluster. Common roles including the below cluster-admin are pre-defined within each organization.

Cluster administrator

This TeamRole named pod-read grants read access to Pods!.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRole
metadata:
  name: pod-read
spec:
  rules:
    - apiGroups:
        - ""
      resources:
        - "pods"
      verbs:
        - "get"
        - "list"

Defining TeamRoleBindings

TeamRoleBindings define the permissions of a Greenhouse Team within Clusters by linking to a specific TeamRole. TeamRoleBindings have a simple specification that links a Team, a TeamRole, one or more Clusters and optionally a one or more Namespaces together. Once the TeamRoleBinding is created, the Team will have the permissions defined in the TeamRole within the specified Clusters and Namespaces. This allows for fine-grained control over the permissions of each Team within each Cluster. The TeamRoleBinding Controller within Greenhouse deploys rbacv1 resources to the targeted Clusters. The referenced TeamRole is created as a rbacv1.ClusterRole. In case the TeamRoleBinding references a Namespace, the Controller will create a rbacv1.RoleBinding which links the Team with the rbacv1.ClusterRole. In case no Namespace is referenced, the Controller will create a rbacv1.ClusterRoleBinding instead.

Assigning TeamRoles to Teams on a single Cluster

Roles are assigned to teams through the TeamRoleBinding configuration, which links teams to their respective roles within specific clusters.

This TeamRoleBinding assigns the pod-read TeamRole to the Team named my-team in the Cluster named my-cluster.

Example: team-rolebindings.yaml

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRoleBinding
metadata:
  name: my-team-read-access
spec:
  teamRef: my-team
  roleRef: pod-read
  clusterName: my-cluster

Assigning TeamRoles to Teams on multiple Clusters

It is also possible to use a LabelSelector to assign TeamRoleBindings to multiple Clusters at once.

This TeamRoleBinding assigns the pod-read TeamRole to the Team named my-team in all Clusters with the label environment: production.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRoleBinding
metadata:
  name: production-cluster-admins
spec:
  teamRef: my-team
  roleRef: pod-read
  clusterSelector:
    matchLabels:
      environment: production

Aggregating TeamRoles

It is possible with RBAC to aggregate rbacv1.ClusterRoles. This is also supported for TeamRoles. By specifying .spec.Labels on a TeamRole the resulting ClusterRole on the target cluster will have the same labels set. Then it is possible to aggregate multiple ClusterRole resources by using a rbacv1.AggregationRule. This can be specified on a TeamRole by setting .spec.aggregationRule.

More details on the concept of Aggregated ClusterRoles can be found in the Kubernetes documentation: Aggregated ClusterRoles

[!NOTE] A TeamRole is only created on a cluster if it is referenced by a TeamRoleBinding. If a TeamRole is not referenced by a TeamRoleBinding it will not be created on any target cluster. A TeamRoleBinding referencing a TeamRole with an aggregationRule will only provide the correct access, if there is at least one TeamRoleBinding referencing a TeamRole with the corresponding label deployed to the same cluster.

The following example shows how to an AggregationRule can be used with TeamRoles and TeamRoleBindings.

This TeamRole specifies .spec.Labels. The labels will be applied to the resulting ClusterRole on the target cluster.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRole
metadata:
  name: pod-read
spec:
  labels:
    aggregate: "true"
  rules:
    - apiGroups:
        - ""
      resources:
        - "pods"
      verbs:
        - "get"
        - "list"

This TeamRoleBinding assigns the pod-read TeamRole to the Team named my-team in all Clusters with the label environment: production.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRoleBinding
metadata:
  name: production-pod-read
spec:
  teamRef: my-team
  roleRef: pod-read
  clusterSelector:
    matchLabels:
      environment: production

This creates another TeamRole and TeamRoleBinding including the same labels as above.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRole
metadata:
  name: pod-edit
spec:
  labels:
    aggregate: "true"
  rules:
    - apiGroups:
        - ""
      resources:
        - "pod"
      verbs:
        - "update"
        - "patch"
---
apiVersion: greenhouse.sap/v1alpha1
kind: TeamRoleBinding
metadata:
  name: production-pod-edit
spec:
  teamRef: my-team
  roleRef: pod-edit
  clusterSelector:
    matchLabels:
      environment: production

This TeamRole has an aggregationRule set. This aggregationRule will be added to the ClusterRole created on the target clusters. With the aggregationRule set it will aggregate the ClusterRoles created by the TeamRoles with the label aggregate: "true". The team will have the permissions of both TeamRoles and will be able to get, list, update and patch Pods.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRole
metadata:
  name: aggregated-role
spec:
  aggregationRule:
    clusterRoleSelectors:
    - matchLabels:
        "aggregate": "true"

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRoleBinding
metadata:
  name: aggregated-rolebinding
spec:
  teamRef: operators
  roleRef: aggregated-role
  clusterSelector:
    matchLabels:
      environment: production

2.4.2 - Team creation

Before you begin

This guides describes how to create a team in your Greenhouse organization.

While all members of an organization can see existing teams, their management requires organization admin privileges.

Creating a team

The team resource is used to structure members of your organization and assign fine-grained access and permission levels.
Each team must be backed by a group in the identity provider (IdP) of the organization.

NOTE: The UI is currently in development. For now this guides describes the onboarding workflow via command line.

1. To onboard a new cluster provide the kubeconfig file with a static, short-lived token.
   It should look similar to this example:

   cat <<EOF | kubectl apply -f -
      apiVersion: greenhouse.sap/v1alpha1
      kind: Team
      metadata:
      name: <name>
      spec:
         description: My new team
         mappedIdPGroup: <IdP group name>
   EOF
   

3 - Architecture

This section provides an overview of the architecture and design of the Greenhouse platform.

3.1 - High-level architecture

This section provides a high-level overview of the Greenhouse concepts.

Greenhouse components

Conceptually, the Greenhouse platform consists of 2 types of Kubernetes cluster fulfilling specific purposes.

1. Central cluster

   The central cluster accommodates the core components of the Greenhouse platform, providing the holistic API and dashboard to let users manage their entire cloud infrastructure from a single control point.
   Users of the Greenhouse cloud operations platform, depending on their roles, can perform tasks such as managing organizations, configuring and using plugins, monitoring resources, developing custom plugins, and conducting audits to ensure compliance with standards.
   Greenhouse’s flexibility allows users to efficiently manage cloud infrastructure and applications while tailoring their experience to organizational needs.
   The configuration and metadata is persisted in Kubernetes custom resource definitions (CRDs), acted upon by the Greenhouse operator and managed in the customer cluster.

2. Customer cluster

   Managing and operating Kubernetes clusters can be challenging due to the complexity of tasks related to orchestration, scaling, and ensuring high availability in containerized environments.
   By onboarding their Kubernetes clusters into Greenhouse, users can centralize cluster management, streamlining tasks like resource monitoring and access control.
   This integration enhances visibility, simplifies operational tasks, and ensures better compliance, enabling users to efficiently manage and optimize their Kubernetes infrastructure. While the central cluster contains the user configuration and metadata, all workloads of user-selected Plugins are run in the customer cluster and managed by Greenhouse.

A simplified architecture of the Greenhouse platform is illustrated below.

3.2 - Product design

Introduction

Vision

“Greenhouse is an extendable Platform that enables Organizations to operate their Infrastructure in a compliant, efficient and transparent manner”

We want to build a Platform that is capable to integrate all the tools that are required to operate services in our private cloud environment in a compliant, effective and transparent manner. Greenhouse aims to be the single stop for Operators in the GCS PlusOne Organization. The primary focus of Greenhouse is to provide a unified interface for all operational aspects and tools, providing a simple shared data model describing the support organization.

As every organization is different, using different tools and has different requirements the platform is build in an extendable fashion that allows a distributed development of plugins.

While initially developed for the GCS PlusOne Organization the platform is explicitly designed to be of generic use and can be consumed by multiple organizations and teams.

Problem Statements

Consolidation of Toolsuite

The operation of cloud infrastructure and applications include a large amount of tasks that are supported by different tools and automations.
Due to the high complexity of cloud environments often times a conglomerate of tools is used to cover all operational aspects. Confguration and setup of the operations toolchain is a complex and time-consuming task that often times lacks automation when it comes to on- and off-boarding people and setting up new teams for new projects.

Visibility of Application Specific permission concepts

At SAP, we are managing identities and access centrally. The Converged Cloud is utilizing Cloud Access Manager for this task.
While it is true that we manage who has access to which access level is defined in there it starts getting complicated if you want to figure out the actual Permission Levels on individual Applications those Access Levels are mapped to.

Management of organizational Groups of People

You often have groups of people that are fulfilling a organizational purpose:

• Support Groups
• Working Groups
• etc.

We have currently no way to represent and manage those groups.

Harmonization and Standardization of Authorization Concepts

We are missing a tool that supports teams on creating access levels and profiles following a standardized naming scheme that is enforced by design and takes away the burden of coming up with names for access levels and user profiles/roles.

Single Point of Truth for Operations Metadata of an Organization

For automations, it is often critical to retrieve Metadata about an Organizational Unit:

• Who is member of a certain Group of people, that is maybe not reflecting the HR View of a Team?
• Which Tool is giving me access to data x,y,z?
• What action items are due and need to get alerted on?
• Does component x,z,y belong to my organization?
• etc. Currently, we do not have a single point of Truth for this kind of metadata and have to use a vaierity of Tools.

Terms

This section lists down Terms and description to Terms to ensure a common languague when talking in context of Greenhouse.

User Profiles

Every Application has Users / Stakeholders, so has Greenhouse. The User Profiles mentioned here give a abstract overview of considered Users / Stakeholders for the Application and the Goals and Tasks of them in context of the Platform.

Greenhouse admin

Administrator of a Greenhouse installation.

Goals

• Ensure overall function of the Greenhouse plattform

Tasks

• Create Organizations
• Enables Plugins
• Operates central infrastructure (Kubernetes cluster, operator, etc.)
• Assign initial organization admins

Organization admin

Administrator of a Greenhouse Organization

Goals

• Manage organization-wide resources

Tasks

• activate/configure plugins for the organization
• Create and manage teams for the organization
• Onboard and manage Kubernetes clusters for the organization

Organization member

Member of on organization that accesses the UI to do ops/support tasks. Is member of one ore more teams of the organization. By default members have view permissions on organization resources.

Goals

• Provide ops/support for the services of the organization

Tasks

• Highly dependend on team membership and plugins configured for the organization Examples:                  
  • Check alerts for teams user is assigned
  • Check policy violations for deployed resources owned by users team
  • Check for known vulnerabilites in services

Plugin developer

A plugin developer is developing plugins for Greenhouse.

Goals

• Must be easy to create plugins
• Can create and test plugins independently
• Greenhouse provides tooling to assist creating, validating, etc. plugins
• Publishing the plugin to Greenhouse requires admin permissions.

Tasks

• Plugin Development      
  • Juno UI Development
  • Plugin backend development

Auditor

An Auditor audits Greenhouse and/or Greenhouse Plugins for compliance with industry or regulatory standards.

Goals

• Wants to see that we have a record for all changes happening in greenhouse
• Wants to have access to resources required to audit the respective Plugin

Tasks

• Performs Audits

Greenhouse Developer

Develops parts of the Greenhouse platform (Kubernetes, Greenhouse app, Greenhouse backend, …)

Goals

• Provide Greenhouse framework

Tasks

• Provides framework for plugin developers to develop plugins
• Develops Greenhouse framework components (UI or backend)

User Stories

The User Stories outlined in this Section have the target to archive a common Understanding of the capabilities/functionalities the Platform wants to archive and the functional requirements that come with those. The Integration / Development of Functionalities is not going to be strictly bound to User Stories and they are rather used as an orientation to ensure that envisioned capabilities are not getting Blocked due to implementation details.

The details of all User Stories are subject to change based on the results of Proof of Concept implementations, User feedback or other unforseen factors.

Auditor

Auditor 01 - Audit Logging

As an Auditor, I want to see who did which action and when to verify that the Vulnerability and Patch management process is followed according to company policies and that the platform is functioning as expected.

Acceptance Criteria

• Every state-changing action is logged into an immutable log collection, including:
  • What action was performed
  • Who performed the action
  • When was the action performed
• Every authentication to the platform is logged into an immutable log collection, including:
  • Who logged in
  • When was the login

Greenhouse Admin

Greenhouse Admin 01 - Greenhouse Management Dashboard

As an Greenhouse Admin, I want a central Greenhouse Management Dashboard that allows me to navigate through all organization-wide resources to be able to manage the Platform.

Acceptance Criteria

• Assuming I am on the Greenhouse Management Dashboard view, i can:
  • See all Plugins, including the enabled version
  • Order not enabled Plugins by last Update Date
  • Plugins are Ordered by the Order Attribute
  • The order attribute is a numeric value that can be changed to reflect a different ordering of the Plugin:
    • 1 is ordered before 2 etc.
    • The order attribute is used as well to order the Plugins on the Navigation Bar
  • Navigate to “Plugin Detail View” by clicking a Plugin
  • See all Organizations, including:
    • Number of Organization Admins
    • Number of Organization Members
  • Navigate to organization creation view by clicking “Create Organization”
  • Navigate to Organization Detail View by clicking a Organization
• Only Greenhouse Admin’s should be able to see the Dashboard
• The Navigation item to the Greenhouse Management Dashboard should only be visible to Greenhouse admins

Greenhouse Admin 02 - Organization Creation View

As a Greenhouse Admin, I want a Organization Creation view that allows me to create a new Organization

Acceptance Criteria

• Assuming I am on the Organization creation View, i can:
  • Give a unique name for the organisation
  • Provide a short description for the organization
  • Provide a OIDC Group that gives Organization Admin Privileges

Greenhouse Admin & Organization Admin

Greenhouse Admin & Organization Admin 01 - Organization Detail View

As a Greenhouse Admin or Organization Admin, I want an Organization Detail view that allows me to view details about an organization

Acceptance Criteria

• Assuming I am on the organization detail View, i can:
  • Can see the organization details (name/description)
  • See a list of teams created for this organization
  • See the list of active plugins
  • Add Plugins to the organization by clicking “add Plugin”
  • Change the Organization Admin Role Name

Greenhouse Admin & Organization Admin 02 - Plugin Detail View

As an Greenhouse Admin or Organizatioin Admin, I want a Plugin Detail view that allows me to see  Plugin Details to be able to see details about the plugin.

Acceptance Criteria

• Assuming I am on the Plugin Detail View, I can:
  • see the plugin name
  • see the plugin description
  • see the last update date
  • see the release reference
  • see the ui release refrence
  • see the helm chart reference
  • see the ordering attribute
  • see configuration values for the plugin
  • set the configuration values for the current organization
  • see a change log
  • see the actually released (deployed to greenhouse) version

Organization Admin

Organization Admin 01 - Organization Managment Dashboard

As an Organization Admin, I want to have an Dashboard showing me the most relevant information about my Organization to be able to manage it efficently.

Acceptance Criteria

• Assuming I am on the organisation management dashboard
  • I can see a list of all teams in my organization
  • I can see a list of configured plugins
  • I can click a “add plugin” button to add a new plugin
  • I can see a list of registered clusters
  • I can click a “add cluster” button to register a cluster

Organization Admin 02 - Plugin Configuration View

As an organization admin, I want a Plugin configuration view that allows me to enable and configure greenhouse plugins for my organization

Acceptance Criteria

• Assuming I am on the Plugin configuration View, I can:
  • select the type of plugin I want to configure
  • enable/disable the plugin (for my org)
  • remove the plugin (when already added)
  • manage configuration options specific for the plugin

Organization Admin 03 - Cluster registration View

As an organization admin, I want a Cluster registration view to onboard kubernetes clusters into my organization.

Acceptance Criteria

• Assuming I am on the cluster registration view, I can:
  • Get instructions how to register a kubernetes clusters
  • give a name and description for the registered cluster
• After executing the provided instructions I get feedback that the cluster has been successfully registered
• A cluster can be registered to exactly one organization

Organization Admin 04 - Cluster detail View

As an organization admin, I want a Cluster detail view to get some information about a registered cluster

Acceptance Criteria

• Assuming I am on the cluster detail view, I can:
  • see basic details about the cluster:
    • name
    • api url
    • version
    • node status
  • de-register the cluster from my organization

Organization Admin 05 - Team Detail View

As an organization admin, i want to have a Team Detail View, with the option to configure teams based on role mapping to be to manage teams within my organization without managing the permission administration itself on the Platform

Acceptance Criteria

• Assuming I am on the Team detail view, i can:
  • Change the Name of the Team
  • change the description of the team
  • Define a single OIDC Group  that assign you this team
  • Define The Greenhouse Role that you get within Greenhouse if you are a member of the team
• On Login of a User into an Organization the Platform verifies if the User has ALL required roles

Organization Admin 05 - Team Creation View

As an organization admin, i want to have a Team Creation View, to be able to create a new Team

Acceptance Criteria

• Assuming I am on the Team Creation view, I can::
  • Set the name of the Team
  • Set a description of the Team
  • Set a OIDC Group Name that assigns users to this team

Organization Member

Organization Member 01 - Unified task inbox

As an organization member, I want a task inbox that shows my open tasks from all enabled plugins that need my attention to be on top of my tasks to fulfill across all plugins

Acceptance Criteria

• Assuming I am on the task inbox:
  • I can a list of open task accross all plugins that need attention
  • clicking on an open task jumps in the plugin specific UI the task belong to
  • I can sort open tasks by name, plugin or date

Plugin Developer

As a Plugin Developer, I want to have a seperate Repository for my Plugin which I can own and use to configure plugin internals to have control over the Development efforts and configuration of the Plugin

Plugin Developer 01 - Decentrally Managed Plugin

As a Plugin Developer, I want to have a seperate Repository for my Plugin which I can own and use to configure plugin internals to have control over the Development efforts and configuration of the Plugin

Acceptance Criteria

• Plugin lives on his own Github Repository
• Versions are managed via Github Releases using Tags and the release to Greenhouse is managed by the Plugin:
  • The version to be pulled by Greenhouse is managed by the Plugin Developer.
• I can configure the Plugin Configuration over a greenhouse.yml in the root of the repository, which at least includes (mandatory):
  • description: …
  • version: …
• There are optional attributes in the greenhouse.yml:
  • icon: which if it has a valid absolute path to an image file on the repository makes the icon selectable as an icon in the plugin detail view (GA02)
  • describes available configuration options that attributes that are required for the plugin to function
  • I can specify a reference to a UI App
  • I can specify a reference to Helm Charts

Plugin Developer 02 - Plugin Role Config

As a Plugin Developer i want a section within the Greenhouse.yml metadata, named “Roles” where i can setup Roles used by my Plugin

Acceptance Criteria

Plugin Developer 03 - Spec Schema Validation

As a Plugin Developer I want to have the possibility to validate the schema of my greenhouse.yaml to be able to catch errors within my specification early.

Acceptance Criteria

• The schema check should support IDE’s
• The schema check should be automate-able and be integrate-able to pre-commit hooks and quality gates
• A version with a broken schema should not be release on greenhouse even when pushing for a pull of the release
• It should be visible on the Plugin detail view when an invalid schema was released with a recent version

Plugin Developer 04 - Config Value Validation

As a Plugin Developer I want to have the possibility to write custom regex checks for configuration options of my plugin that include the check to be performed on a field and an error message to be shown if configured wrong by an organization to support organization admins on configuring my plugin

Acceptance Criteria

• The validation rules should be controlled by Plugin Developer
• The validation should happen on the frontend before submitting a configuration
• The error message should be shown when a config value is provided wrong

Plugin Developer 05 - Plugin development tooling

As a plugin developer I want to have an easy setup for developing and testing greenhouse plugins

Acceptance Criteria

• Dev environment available within X Time
• Possible to have a working local setup with a “mock greenhouse apiserver”
• Has a fully working Bootstrap Project that includes Backend and Frontend which can be run locally immediately
• Has documentation

Product Stages

Overview

This Section gives an overview of the different early stages of the Platform that are beeing developed and which functional requirements need to get fulfilled within those stages.

Proof of Concept (POC)

The Proof of Concept is the stage where fundamental Framework/Platform decisions are proven to be working as intended. At this Stage the Platform is not suitable to be used by the intended audience yet but most necessary core functionalities are implemented.

The desired functionalities in this phase are:

• Frontend with Authentication
• Authorization within Greenhouse (Greenhouse Admin, Org Admin, Org Member)
• Team Management (without UI)
• Org Management (without UI)
• Greenhouse Admin User Stories (without UI)
• Dummy Plugin
  • with configuration spec
• Plugin Development Setup (without Documentation)
• Plugin Versioning & Provisioning (without UI)

Minimal viable product (MVP)

This stage is considered to be the earliest stage to open the Platform for General use.
In addition to the PoC functionalities we expect the following requirements to be fulfilled:

• Integrated 3 Plugins:
  • Supernova (Alerts)
  • DOOP (Violations)
  • Heureka
    NOTE: Heureka was excluded from MVP as the Heureka API is only available at a later point in time.
• Team management
• Organization management

3.3 - Monitoring architecture

This section provides an overview of the Greenhouse monitoring components.

Monitoring components

1. Metrics

   The kube-monitoring plugin is intended for monitoring remote Kubernetes clusters and is preconfigured to collect metrics from all Kubernetes components. It also comes with a default set of Kubernetes alerting rules. The plugin includes Prometheus, Prometheus operator, Prometheus node exporter and kube-state-metrics. The plugin is provided in its own namespace, which is managed by Greenhouse. Prometheus and Prometheus operator configurations can also be extended to process customer metrics and alerts.

2. Visualization/Dashboard

   With the plutono plugin, you can deploy Plutono, a fork of Grafana 7.5.17 in the Greenhouse central cluster, to query Prometheus metrics and visualize them in dynamic dashboards.

3. Alerts

   The alerts plugin contains the Prometheus Alertmanager and Supernova as user interface. It is deployed in the Greenhouse central cluster in the customer org namespace. The Alertmanager receives alerts from the Prometheis in the remote Kubernetes clusters.
   Supernova is a user-friendly interface for the Alertmanager. It provides a grouped overview of the alerts with additional simple filter options.

4. Notifications

   The Alertmanager can send out notifications to different receivers (e.g. Slack, Email, …) based on alert labels.

The Greenhouse monitoring components and their interactions are illustrated below.

4 - Reference

This section contains reference documentation for Greenhouse.

4.1 - API

Packages:

• greenhouse.sap/v1alpha1

greenhouse.sap/v1alpha1

Authentication

(Appears on: OrganizationSpec)

Cluster

Cluster is the Schema for the clusters API

ClusterAccessMode (string alias)

(Appears on: ClusterSpec)

ClusterAccessMode configures the access mode to the customer cluster.

ClusterConditionType (string alias)

ClusterConditionType is a valid condition of a cluster.

ClusterKubeconfig

ClusterKubeconfig is the Schema for the clusterkubeconfigs API ObjectMeta.OwnerReferences is used to link the ClusterKubeconfig to the Cluster ObjectMeta.Generation is used to detect changes in the ClusterKubeconfig and sync local kubeconfig files ObjectMeta.Name is designed to be the same with the Cluster name

ClusterKubeconfigAuthInfo

(Appears on: ClusterKubeconfigAuthInfoItem)

ClusterKubeconfigAuthInfoItem

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigCluster

(Appears on: ClusterKubeconfigClusterItem)

ClusterKubeconfigClusterItem

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigContext

(Appears on: ClusterKubeconfigContextItem)

ClusterKubeconfigContextItem

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigData

(Appears on: ClusterKubeconfigSpec)

ClusterKubeconfigData stores the kubeconfig data ready to use kubectl or other local tooling It is a simplified version of clientcmdapi.Config: https://pkg.go.dev/k8s.io/client-go/tools/clientcmd/api#Config

ClusterKubeconfigPreferences

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigSpec

(Appears on: ClusterKubeconfig)

ClusterKubeconfigSpec stores the kubeconfig data for the cluster The idea is to use kubeconfig data locally with minimum effort (with local tools or plain kubectl): kubectl get cluster-kubeconfig $NAME -o yaml | yq -y .spec.kubeconfig

ClusterOptionOverride

(Appears on: PluginPresetSpec)

ClusterOptionOverride defines which plugin option should be override in which cluster

ClusterSpec

(Appears on: Cluster)

ClusterSpec defines the desired state of the Cluster.

ClusterStatus

(Appears on: Cluster)

ClusterStatus defines the observed state of Cluster

Condition

(Appears on: PropagationStatus, StatusConditions)

Condition contains additional information on the state of a resource.

ConditionReason (string alias)

(Appears on: Condition)

ConditionReason is a valid reason for a condition of a resource.

ConditionType (string alias)

(Appears on: Condition)

ConditionType is a valid condition of a resource.

HeadScaleMachineStatus

(Appears on: ClusterStatus)

HeadScaleMachineStatus is the status of a Headscale machine.

HelmChartReference

(Appears on: PluginDefinitionSpec, PluginStatus)

HelmChartReference references a Helm Chart in a chart repository.

HelmReleaseStatus

(Appears on: PluginStatus)

HelmReleaseStatus reflects the status of a Helm release.

NodeStatus

(Appears on: ClusterStatus)

OIDCConfig

(Appears on: Authentication)

Organization

Organization is the Schema for the organizations API

OrganizationSpec

(Appears on: Organization)

OrganizationSpec defines the desired state of Organization

OrganizationStatus

(Appears on: Organization)

OrganizationStatus defines the observed state of an Organization

Plugin

Plugin is the Schema for the plugins API

PluginDefinition

PluginDefinition is the Schema for the PluginDefinitions API

PluginDefinitionSpec

(Appears on: PluginDefinition)

PluginDefinitionSpec defines the desired state of PluginDefinitionSpec

PluginDefinitionStatus

(Appears on: PluginDefinition)

PluginDefinitionStatus defines the observed state of PluginDefinition

PluginOption

(Appears on: PluginDefinitionSpec)

PluginOptionType (string alias)

(Appears on: PluginOption)

PluginOptionType specifies the type of PluginOption.

PluginOptionValue

(Appears on: ClusterOptionOverride, PluginSpec)

PluginOptionValue is the value for a PluginOption.

PluginPreset

PluginPreset is the Schema for the PluginPresets API

PluginPresetSpec

(Appears on: PluginPreset)

PluginPresetSpec defines the desired state of PluginPreset

PluginPresetStatus

(Appears on: PluginPreset)

PluginPresetStatus defines the observed state of PluginPreset

PluginSpec

(Appears on: Plugin, PluginPresetSpec)

PluginSpec defines the desired state of Plugin

PluginStatus

(Appears on: Plugin)

PluginStatus defines the observed state of Plugin

PreAuthKey

(Appears on: HeadScaleMachineStatus)

PreAuthKey reflects the status of the pre-authentication key used by the Headscale machine.

PropagationStatus

(Appears on: TeamRoleBindingStatus)

PropagationStatus defines the observed state of the TeamRoleBinding’s associated rbacv1 resources on a Cluster

SCIMConfig

(Appears on: Authentication)

SecretKeyReference

(Appears on: OIDCConfig, ValueFromSource)

SecretKeyReference specifies the secret and key containing the value.

Service

(Appears on: PluginStatus)

Service references a Kubernetes service of a Plugin.

StatusConditions

(Appears on: ClusterStatus, NodeStatus, PluginPresetStatus, PluginStatus, TeamMembershipStatus, TeamRoleBindingStatus)

A StatusConditions contains a list of conditions. Only one condition of a given type may exist in the list.

Team

Team is the Schema for the teams API

TeamMembership

TeamMembership is the Schema for the teammemberships API

TeamMembershipSpec

(Appears on: TeamMembership)

TeamMembershipSpec defines the desired state of TeamMembership

TeamMembershipStatus

(Appears on: TeamMembership)

TeamMembershipStatus defines the observed state of TeamMembership

TeamRole

TeamRole is the Schema for the TeamRoles API

TeamRoleBinding

TeamRoleBinding is the Schema for the rolebindings API

TeamRoleBindingSpec

(Appears on: TeamRoleBinding)

TeamRoleBindingSpec defines the desired state of a TeamRoleBinding

TeamRoleBindingStatus

(Appears on: TeamRoleBinding)

TeamRoleBindingStatus defines the observed state of the TeamRoleBinding

TeamRoleSpec

(Appears on: TeamRole)

TeamRoleSpec defines the desired state of a TeamRole

TeamRoleStatus

(Appears on: TeamRole)

TeamRoleStatus defines the observed state of a TeamRole

TeamSpec

(Appears on: Team)

TeamSpec defines the desired state of Team

TeamStatus

(Appears on: Team)

TeamStatus defines the observed state of Team

UIApplicationReference

(Appears on: PluginDefinitionSpec, PluginStatus)

UIApplicationReference references the UI pluginDefinition to use.

User

(Appears on: TeamMembershipSpec)

User specifies a human person.

ValueFromSource

(Appears on: PluginOptionValue, SCIMConfig)

ValueFromSource is a valid source for a value.

4.2 - Plugin Catalog

This section provides an overview of the available PluginDefinitions in Greenhouse.

4.2.1 - Alerts

This plugin extension contains Prometheus Alertmanager via prometheus-operator and Supernova, the holistic alert management UI.

Owner

1. Richard Tief (@richardtief)
2. Tommy Sauer (@viennaa)
3. Martin Vossen (@artherd42)

alerts alertmanager parameters

alerts supernova parameters

theme: Override the default theme. Possible values are "theme-light" or "theme-dark" (default)

endpoint: Alertmanager API Endpoint URL /api/v2. Should be one of alerts.alertmanager.ingress.hosts

silenceExcludedLabels: SilenceExcludedLabels are labels that are initially excluded by default when creating a silence. However, they can be added if necessary when utilizing the advanced options in the silence form.The labels must be an array of strings. Example: ["pod", "pod_name", "instance"]

filterLabels: FilterLabels are the labels shown in the filter dropdown, enabling users to filter alerts based on specific criteria. The ‘Status’ label serves as a default filter, automatically computed from the alert status attribute and will be not overwritten. The labels must be an array of strings. Example: ["app", "cluster", "cluster_type"]

predefinedFilters: PredefinedFilters are filters applied through in the UI to differentiate between contexts through matching alerts with regular expressions. They are loaded by default when the application is loaded. The format is a list of objects including name, displayname and matchers (containing keys corresponding value). Example:

[
  {
    "name": "prod",
    "displayName": "Productive System",
    "matchers": {
      "region": "^prod-.*"
    }
  }
]

silenceTemplates: SilenceTemplates are used in the Modal (schedule silence) to allow pre-defined silences to be used to scheduled maintenance windows. The format consists of a list of objects including description, editable_labels (array of strings specifying the labels that users can modify), fixed_labels (map containing fixed labels and their corresponding values), status, and title. Example:

"silenceTemplates": [
    {
      "description": "Description of the silence template",
      "editable_labels": ["region"],
      "fixed_labels": {
        "name": "Marvin",
      },
      "status": "active",
      "title": "Silence"
    }
  ]

Managing Alertmanager configuration

ref:

• https://prometheus.io/docs/alerting/configuration/#configuration-file
• https://prometheus.io/webtools/alerting/routing-tree-editor/

By default, the Alertmanager instances will start with a minimal configuration which isn’t really useful since it doesn’t send any notification when receiving alerts.

You have multiple options to provide the Alertmanager configuration:

1. You can use alerts.alertmanager.config to define a Alertmanager configuration. Example below.

config:
  global:
    resolve_timeout: 5m
  inhibit_rules:
    - source_matchers:
        - "severity = critical"
      target_matchers:
        - "severity =~ warning|info"
      equal:
        - "namespace"
        - "alertname"
    - source_matchers:
        - "severity = warning"
      target_matchers:
        - "severity = info"
      equal:
        - "namespace"
        - "alertname"
    - source_matchers:
        - "alertname = InfoInhibitor"
      target_matchers:
        - "severity = info"
      equal:
        - "namespace"
  route:
    group_by: ["namespace"]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 12h
    receiver: "null"
    routes:
      - receiver: "null"
        matchers:
          - alertname =~ "InfoInhibitor|Watchdog"
  receivers:
    - name: "null"
  templates:
    - "/etc/alertmanager/config/*.tmpl"

2. You can discover AlertmanagerConfig objects. The spec.alertmanagerConfigSelector is always set to matchLabels: plugin: <name> to tell the operator which AlertmanagerConfigs objects should be selected and merged with the main Alertmanager configuration. Note: The default strategy for a AlertmanagerConfig object to match alerts is OnNamespace.

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: config-example
  labels:
    alertmanagerConfig: example
    pluginDefinition: alerts-example
spec:
  route:
    groupBy: ["job"]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: "webhook"
  receivers:
    - name: "webhook"
      webhookConfigs:
        - url: "http://example.com/"

3. You can use alerts.alertmanager.alertmanagerSpec.alertmanagerConfiguration to reference an AlertmanagerConfig object in the same namespace which defines the main Alertmanager configuration.

# Example with select a global alertmanagerconfig
alertmanagerConfiguration:
  name: global-alertmanager-configuration

Examples

Deploy alerts with Alertmanager

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: alerts
spec:
  pluginDefinition: alerts
  disabled: false
  displayName: Alerts
  optionValues:
    - name: alerts.alertmanager.enabled
      value: true
    - name: alerts.alertmanager.ingress.enabled
      value: true
    - name: alerts.alertmanager.ingress.hosts
      value:
        - alertmanager.dns.example.com
    - name: alerts.alertmanager.ingress.tls
      value:
        - hosts:
            - alertmanager.dns.example.com
          secretName: tls-alertmanager-dns-example-com
    - name: alerts.alertmanagerConfig.slack.routes
      value:
        - channel: slack-warning-channel
          webhookURL: https://hooks.slack.com/services/some-id
          matchers:
            - name: severity
              matchType: "="
              value: "warning"
        - channel: slack-critical-channel
          webhookURL: https://hooks.slack.com/services/some-id
          matchers:
            - name: severity
              matchType: "="
              value: "critical"
    - name: alerts.alertmanagerConfig.webhook.routes
      value:
        - name: webhook-route
          url: https://some-webhook-url
          matchers:
            - name: alertname
              matchType: "=~"
              value: ".*"
    - name: alerts.alertmanager.serviceMonitor.additionalLabels
      value:
        plugin: kube-monitoring
    - name: alerts.defaultRules.create
      value: true
    - name: alerts.defaultRules.labels
      value:
        plugin: kube-monitoring
    - name: endpoint
      value: https://alertmanager.dns.example.com/api/v2
    - name: filterLabels
      value:
        - job
        - severity
        - status
    - name: silenceExcludedLabels
      value:
        - pod
        - pod_name
        - instance

Deploy alerts without Alertmanager (Bring your own Alertmanager - Supernova UI only)

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: alerts
spec:
  pluginDefinition: alerts
  disabled: false
  displayName: Alerts
  optionValues:
    - name: alerts.alertmanager.enabled
      value: false
    - name: alerts.alertmanager.ingress.enabled
      value: false
    - name: alerts.defaultRules.create
      value: false
    - name: endpoint
      value: https://alertmanager.dns.example.com/api/v2
    - name: filterLabels
      value:
        - job
        - severity
        - status
    - name: silenceExcludedLabels
      value:
        - pod
        - pod_name
        - instance

4.2.2 - Cert-manager

This Plugin provides the cert-manager to automate the management of TLS certificates.

Configuration

This section highlights configuration of selected Plugin features.
All available configuration options are described in the plugin.yaml.

Ingress shim

An Ingress resource in Kubernetes configures external access to services in a Kubernetes cluster.
Securing ingress resources with TLS certificates is a common use-case and the cert-manager can be configured to handle these via the ingress-shim component.
It can be enabled by deploying an issuer in your organization and setting the following options on this plugin.

4.2.3 - Decentralized Observer of Policies (Violations)

This directory contains the Greenhouse plugin for the Decentralized Observer of Policies (DOOP).

DOOP

To perform automatic validations on Kubernetes objects, we run a deployment of OPA Gatekeeper in each cluster. This dashboard aggregates all policy violations reported by those Gatekeeper instances.

4.2.4 - Designate Ingress CNAME operator (DISCO)

This Plugin provides the Designate Ingress CNAME operator (DISCO) to automate management of DNS entries in OpenStack Designate for Ingress and Services in Kubernetes.

4.2.5 - DigiCert issuer

This Plugin provides the digicert-issuer, an external Issuer extending the cert-manager with the DigiCert cert-central API.

4.2.6 - External DNS

This Plugin provides the external DNS operator) which synchronizes exposed Kubernetes Services and Ingresses with DNS providers.

4.2.7 - Ingress NGINX

This plugin contains the ingress NGINX controller.

Example

To instantiate the plugin create a Plugin like:

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: ingress-nginx
spec:
  pluginDefinition: ingress-nginx-v4.4.0
  values:
    - name: controller.service.loadBalancerIP
      value: 1.2.3.4

4.2.8 - Kubernetes monitoring

This Plugin is intended for monitoring Kubernetes clusters and is preconfigured to collect metrics from all Kubernetes components. It provides a standard set of alerting rules. Many of the useful alerts come from the kubernetes-mixin project.

Components included in this Plugin:

• Prometheus
• Prometheus Operator
• Prometheus adapter for Kubernetes metrics APIs (kubelet, apiserver, coredns, etcd)
• Prometheus node exporter
• kube-state-metrics

Owner

1. Richard Tief (@richardtief)
2. Tommy Sauer (@viennaa)
3. Martin Vossen (@artherd42)

kube-monitoring prometheus-operator parameters

kube-monitoring Kubernetes components scraper configuration

kube-monitoring Prometheus parameters

kube-monitoring Alertmanager config parameters

Examples

Deploy kube-monitoring into a remote cluster

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: kube-monitoring
spec:
  pluginDefinition: kube-monitoring
  disabled: false
  optionValues:
    - name: kubeMonitoring.prometheus.prometheusSpec.retention
      value: 30d
    - name: kubeMonitoring.prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage
      value: 100Gi
    - name: kubeMonitoring.prometheus.service.labels
      value:
        greenhouse.sap/expose: "true"
    - name: kubeMonitoring.prometheus.prometheusSpec.externalLabels
      value:
        cluster: example-cluster
        organization: example-org
        region: example-region
    - name: alerts.enabled
      value: true
    - name: alerts.alertmanagers.hosts
      value:
        - alertmanager.dns.example.com
    - name: alerts.alertmanagers.tlsConfig.cert
      valueFrom:
        secret:
          key: tls.crt
          name: tls-<org-name>-prometheus-auth
    - name: alerts.alertmanagers.tlsConfig.key
      valueFrom:
        secret:
          key: tls.key
          name: tls-<org-name>-prometheus-auth

Deploy Prometheus only

Example Plugin to deploy Prometheus with the kube-monitoring Plugin.

NOTE: If you are using kube-monitoring for the first time in your cluster, it is necessary to set kubeMonitoring.prometheusOperator.enabled to true.

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: example-prometheus-name
spec:
  pluginDefinition: kube-monitoring
  disabled: false
  optionValues:
    - name: kubeMonitoring.defaultRules.create
      value: false
    - name: kubeMonitoring.kubernetesServiceMonitors.enabled
      value: false
    - name: kubeMonitoring.prometheusOperator.enabled
      value: false
    - name: kubeMonitoring.kubeStateMetrics.enabled
      value: false
    - name: kubeMonitoring.nodeExporter.enabled
      value: false
    - name: kubeMonitoring.prometheus.prometheusSpec.retention
      value: 30d
    - name: kubeMonitoring.prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage
      value: 100Gi
    - name: kubeMonitoring.prometheus.service.labels
      value:
        greenhouse.sap/expose: "true"
    - name: kubeMonitoring.prometheus.prometheusSpec.externalLabels
      value:
        cluster: example-cluster
        organization: example-org
        region: example-region
    - name: alerts.enabled
      value: true
    - name: alerts.alertmanagers.hosts
      value:
        - alertmanager.dns.example.com
    - name: alerts.alertmanagers.tlsConfig.cert
      valueFrom:
        secret:
          key: tls.crt
          name: tls-<org-name>-prometheus-auth
    - name: alerts.alertmanagers.tlsConfig.key
      valueFrom:
        secret:
          key: tls.key
          name: tls-<org-name>-prometheus-auth

4.2.9 - Logshipper

This Plugin is intended for shipping container and systemd logs to an Elasticsearch/ OpenSearch cluster. It uses fluentbit to collect logs. The default configuration can be found under chart/templates/fluent-bit-configmap.yaml.

Components included in this Plugin:

• fluentbit

Owner

1. @ivogoman

Parameters

Custom Configuration

To add custom configuration to the fluent-bit configuration please check the fluentbit documentation here. The fluent-bit.customConfig.inputs, fluent-bit.customConfig.filters and fluent-bit.customConfig.outputs parameters can be used to add custom configuration to the default configuration. The configuration should be added as a multi-line string. Inputs are rendered after the default inputs, filters are rendered after the default filters and before the additional values are added. Outputs are rendered after the default outputs. The additional values are added to all logs disregaring the source.

Example Input configuration:

fluent-bit:
  config:
    inputs: |
      [INPUT]
          Name             tail-audit
          Path             /var/log/containers/greenhouse-controller*.log
          Parser           {{ default "cri" ( index .Values "fluent-bit" "parser" ) }}
          Tag              audit.*
          Refresh_Interval 5
          Mem_Buf_Limit    50MB
          Skip_Long_Lines  Off
          Ignore_Older     1m
          DB               /var/log/fluent-bit-tail-audit.pos.db      

Logs collected by the default configuration are prefixed with default_. In case that logs from additional inputs are to be send and processed by the same filters and outputs, the prefix should be used as well.

In case additional secrets are required the fluent-bit.env field can be used to add them to the environment of the fluent-bit container. The secrets should be created by adding them to the fluent-bit.backend field.

fluent-bit:
  backend:
    audit:
      http_user: top-secret-audit
      http_password: top-secret-audit
      host: "audit.test"
      tls:
        enabled: true
        verify: true
        debug: false

4.2.10 - OpenTelemetry

This Plugin is intended for ingesting, generating, collecting, and exporting telemetry data (metrics, logs, and traces).

Components included in this Plugin:

• Collector
• Filelog Receiver
• OpenSearch Exporter
• ServiceMonitor to ingest the metrics of the OpenTelemetry Collector in Prometheus

The Plugin will deploy the OpenTelemetry Operator which works as a manager for the collectors and auto-instrumentation of the workload. By default the Plugin will also

1. Collect logs via the Filelog Receiver, process and export them to OpenSearch
   • You can disable the collecting of logs by setting open_telemetry.LogCollector.enabled to false.
2. Collect and expose metrics using a Prometheus interface which defaults to port 8888
   • You can disable the collecting of metrics by setting open_telemetry.MetricsCollector.enabled to false.

Contributors are welcome to provide additional collector configurations.

Owner

1. Timo Johner (@timojohlo)
2. Olaf Heydorn (@kuckkuck)
3. Tommy Sauer (@viennaa)

Parameters

Examples

TBD

4.2.11 - Plutono

Installs the web dashboarding system Plutono to easily collect, correlate, and visualize Prometheus metrics with dashboards.

Owner

1. Richard Tief (@richardtief)

Quick Start

This guide provides a quick and straightforward way how to use Plutono as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

• A running and Greenhouse-managed Kubernetes cluster
• kube-monitoring Plugin installed to have at least one Prometheus instance running in the cluster

The plugin works by factory default with anonymous access enabled. If you use the standard configuration in the kube-monitoring plugin, the data source and some k8s-monitoring dashboards are already pre-installed.

Step 1: Add your dashboards

Dashboards are selected from ConfigMaps across namespaces. The plugin searches for ConfigMaps with the label plutono-dashboard: "true" and imports them into Plutono. The ConfigMap must contain a key like my-dashboard.json with the dashboard JSON content. Example

Step 2: Add your datasources

Data sources are selected from Secrets across namespaces. The plugin searches for Secrets with the label plutono-dashboard: "true" and imports them into Plutono. The Secrets should contain valid datasource configuration YAML. Example

Configuration

Example ingress with path

With plutono 6.3 and above

plutono.ini:
  server:
    domain: monitoring.example.com
    root_url: "%(protocol)s://%(domain)s/plutono"
    serve_from_sub_path: true
ingress:
  enabled: true
  hosts:
    - "monitoring.example.com"
  path: "/plutono"

Example of extraVolumeMounts and extraVolumes

Configure additional volumes with extraVolumes and volume mounts with extraVolumeMounts.

Example for extraVolumeMounts and corresponding extraVolumes:

extraVolumeMounts:
  - name: plugins
    mountPath: /var/lib/plutono/plugins
    subPath: configs/plutono/plugins
    readOnly: false
  - name: dashboards
    mountPath: /var/lib/plutono/dashboards
    hostPath: /usr/shared/plutono/dashboards
    readOnly: false

extraVolumes:
  - name: plugins
    existingClaim: existing-plutono-claim
  - name: dashboards
    hostPath: /usr/shared/plutono/dashboards

Volumes default to emptyDir. Set to persistentVolumeClaim, hostPath, csi, or configMap for other types. For a persistentVolumeClaim, specify an existing claim name with existingClaim.

Import dashboards

There are a few methods to import dashboards to Plutono. Below are some examples and explanations as to how to use each method:

dashboards:
  default:
    some-dashboard:
      json: |
        {
          "annotations":

          ...
          # Complete json file here
          ...

          "title": "Some Dashboard",
          "uid": "abcd1234",
          "version": 1
        }        
    custom-dashboard:
      # This is a path to a file inside the dashboards directory inside the chart directory
      file: dashboards/custom-dashboard.json
    prometheus-stats:
      # Ref: https://plutono.com/dashboards/2
      gnetId: 2
      revision: 2
      datasource: Prometheus
    loki-dashboard-quick-search:
      gnetId: 12019
      revision: 2
      datasource:
      - name: DS_PROMETHEUS
        value: Prometheus
      - name: DS_LOKI
        value: Loki
    local-dashboard:
      url: https://raw.githubusercontent.com/user/repository/master/dashboards/dashboard.json

BASE64 dashboards

Dashboards could be stored on a server that does not return JSON directly and instead of it returns a Base64 encoded file (e.g. Gerrit) A new parameter has been added to the url use case so if you specify a b64content value equals to true after the url entry a Base64 decoding is applied before save the file to disk. If this entry is not set or is equals to false not decoding is applied to the file before saving it to disk.

Gerrit use case

Gerrit API for download files has the following schema: https://yourgerritserver/a/{project-name}/branches/{branch-id}/files/{file-id}/content where {project-name} and {file-id} usually has ‘/’ in their values and so they MUST be replaced by %2F so if project-name is user/repo, branch-id is master and file-id is equals to dir1/dir2/dashboard the url value is https://yourgerritserver/a/user%2Frepo/branches/master/files/dir1%2Fdir2%2Fdashboard/content

Sidecar for dashboards

If the parameter sidecar.dashboards.enabled is set, a sidecar container is deployed in the plutono pod. This container watches all configmaps (or secrets) in the cluster and filters out the ones with a label as defined in sidecar.dashboards.label. The files defined in those configmaps are written to a folder and accessed by plutono. Changes to the configmaps are monitored and the imported dashboards are deleted/updated.

A recommendation is to use one configmap per dashboard, as a reduction of multiple dashboards inside one configmap is currently not properly mirrored in plutono.

NOTE: Configure your data sources in your dashboards as variables to keep them portable across clusters.

Example dashboard config:

Folder structure:

dashboards/
├── dashboard1.json
├── dashboard2.json
templates/
├──dashboard-json-configmap.yaml

Helm template to create a configmap for each dashboard:

{{- range $path, $bytes := .Files.Glob "dashboards/*.json" }}
---
apiVersion: v1
kind: ConfigMap

metadata:
  name: {{ printf "%s-%s" $.Release.Name $path | replace "/" "-" | trunc 63 }}
  labels:
    plutono-dashboard: "true"

data:
{{ printf "%s: |-" $path | replace "/" "-" | indent 2 }}
{{ printf "%s" $bytes | indent 4 }}

{{- end }}

Sidecar for datasources

If the parameter sidecar.datasources.enabled is set, an init container is deployed in the plutono pod. This container lists all secrets (or configmaps, though not recommended) in the cluster and filters out the ones with a label as defined in sidecar.datasources.label. The files defined in those secrets are written to a folder and accessed by plutono on startup. Using these yaml files, the data sources in plutono can be imported.

Should you aim for reloading datasources in Plutono each time the config is changed, set sidecar.datasources.skipReload: false and adjust sidecar.datasources.reloadURL to http://<svc-name>.<namespace>.svc.cluster.local/api/admin/provisioning/datasources/reload.

Secrets are recommended over configmaps for this usecase because datasources usually contain private data like usernames and passwords. Secrets are the more appropriate cluster resource to manage those.

Example datasource config:

apiVersion: v1
kind: Secret
metadata:
  name: plutono-datasources
  labels:
    # default value for: sidecar.datasources.label
    plutono-datasource: "true"
stringData:
  datasources.yaml: |-
    apiVersion: 1
    datasources:
      - name: my-prometheus 
        type: prometheus
        access: proxy
        orgId: 1
        url: my-url-domain:9090
        isDefault: false
        jsonData:
          httpMethod: 'POST'
        editable: false