Greenhouse documentation

• 1: Getting started
• 1.1: Core Concepts
• 1.1.1: Organizations
• 1.1.2: Teams
• 1.1.3: Clusters
• 1.1.4: PluginDefinitions, Plugins and PluginPresets
• 1.2: Overview
• 1.3: Quick start guide
• 2: User guides
• 2.1: Organization management
• 2.1.1: SAP ID Service
• 2.1.2: Creating an organization
• 2.2: Cluster management
• 2.2.1: Cluster onboarding
• 2.3: Plugin management
• 2.3.1: Local Plugin Development
• 2.3.2: Testing a Plugin
• 2.3.3: Plugin deployment
• 2.3.4: Managing Plugins for multiple clusters
• 2.3.5: Plugin Catalog
• 2.4: Team management
• 2.4.1: Role-based access control
• 2.4.2: Team creation
• 3: Architecture
• 3.1: High-level architecture
• 3.2: Product design
• 3.3: Building Observability Architectures
• 4: Reference
• 4.1: API
• 4.2: Plugin Catalog
• 4.2.1: Alerts
• 4.2.2: Cert-manager
• 4.2.3: Decentralized Observer of Policies (Violations)
• 4.2.4: Designate Ingress CNAME operator (DISCO)
• 4.2.5: DigiCert issuer
• 4.2.6: External DNS
• 4.2.7: Github Guard
• 4.2.8: Ingress NGINX
• 4.2.9: Kubernetes Monitoring
• 4.2.10: Logshipper
• 4.2.11: OpenTelemetry
• 4.2.12: Plutono
• 4.2.13: Service exposure test
• 4.2.14: Teams2Slack
• 4.2.15: Thanos
• 5: Contribute
• 5.1: Local development setup
• 5.2: Contributing a Plugin
• 6:
• 7:

1 - Getting started

1.1 - Core Concepts

1.1.1 - Organizations

What are Organizations?

Organizations are the top-level entities in Greenhouse. Each Organization gets a dedicated Namespace, that contains all resources bound to the Organization. Greenhouse expects an Organization to provide it’s own Identity Provider and currently supports OIDC Identity Providers. Greenhouse also supports SCIM for syncing users and groups from an Identity Provider.

See creating an Organization for more details.

Organization Namespace and Permissions

The Organization’s Namespace in the Greenhouse cluster contains all resources bound to the Organization. This Namespace is automatically provisioned when a new Organization is created and shares the Organization’s name. Once the Namespace is created, Greenhouse will automatically seed RBAC Roles and ClusterRoles for the Organization. These are used to grant permissions for the Organization’s resources to Teams.

• The Administrators of an Organization are specified via a identity provider (IDP) group during the creation of the Organization.
• The Administrators for Plugins and Clusters need to be defined by the Organization Admins via RoleBindings for the seeded Roles role:<org-name>:cluster-admin and role:<org-name>:plugin-admin.
• All authenticated users are considered members of the Organization and are granted the organization:<org-name> Role.

The following roles are seeded for each Organization:

OIDC

Each Organization must specify the OIDC configuration for the Organization’s IDP. This configuration is used together with DEXIDP to authenticate users in the Organization.

SCIM

Each Organization can specify SCIM credentials which are used to syncronize users and groups from an Identity Provider. This makes it possible to view the members of a Team in the Greenhouse dashboard.

1.1.2 - Teams

What are Teams?

Teams are used to manage access to resources in Greenhouse and managed Kubernetes clusters. Each Team must be backed by a group in the identity provider (IdP) of the Organization. Teams are used to structure members of your Organization and assign fine-grained access and permission levels. The Greenhouse Dashboard is showing the members of a Team.

Team RBAC

TeamRoles and TeamRoleBindings provide a mechanism to control the permissions of Teams to onboarded Clusters of an Organization.

Team role-based access control (RBAC) wraps the concept of Kubernetes RBAC in TeamRoles and TeamRoleBindings. TeamRoles are used to define a set of RBAC permissions. These permissions can be granted to Teams with TeamRoleBindings. A TeamRoleBinding refers to a Team, a TeamRole, Cluster(s) and optional Namespaces. Depending on the latter, Greenhouse will create the appropriate rbacv1 resources on the targeted cluster(s) in either Cluster or Namespace scope. More information about how this can be configured is mentioned in this user guide.

Example of a TeamRoleBinding for a observability-admin which grants the cluster-admin role on the observability cluster in the logs and metrics namespaces. The TeamRoleBinding contains a list of namespaces and a label selector to select the cluster(s) to target. If no Namespaces are provided, then Greenhouse will create a ClusterRoleBinding instead of a RoleBinding.

1.1.3 - Clusters

What are Clusters?

In the context of Greenhouse a Cluster represents a Kubernetes cluster that is onboarded to Greenhouse. Onboarded in this context means that Greenhouse can handle the management of role-based access control (RBAC) and the provisioning of operating tools (e.g. logging, monitoring, ingress etc.). The Greenhouse dashboard provides an overview of all onboarded clusters. Throughout Greenhouse the reference to a Cluster is used to target it for configuration and deployments.

Cluster access

During the initial onboarding of a cluster, Greenhouse will create a dedicated ServiceAccount inside the onboarded cluster. This ServiceAccount’s token is rotated automatically by Greenhouse.

Cluster registry (coming soon)

Once a Cluster is onboarded to Greenhouse a ClusterKubeConfig is generated for the Cluster based on the OIDC configuration of the Organization. This enables members of an Organization to access the fleet of onboarded Clusters via the common Identity Provider. on the respective Clusters can be managed via Greenhouse Team RBAC.

In order to make it convenient to use these ClusterKubeConfigs and to easily switch between multiple context locally there will be a CLI provided by Greenhouse.

1.1.4 - PluginDefinitions, Plugins and PluginPresets

What are PluginDefinitions and Plugins?

PluginDefinitons and Plugins are the Greenhouse way to extend the core functionality with domain specific features. PluginDefinitions, as the name suggests, are the definition of a Plugin, whereas a Plugin is a concrete instance of a PluginDefinition that is deployed to a Cluster.

The PluginDefinitions are shared between all Organizations in Greenhouse. A PluginDefinition can include a frontend, that is displayed in the Greenhouse dashboard and/or a backend component. The frontend is expected to be a standalone microfrontend created with the Juno framework. The backend components of a PluginDefinition are packaged as a Helm Chart and provide sane and opinionated default values. This allows Greenhouse to package and distribute tools such as Prometheus with a sensible default configuration, as well as giving the user a list of configurable values.

A Plugin is used to deploy the Helm Chart referenced in a PluginDefinition to a Cluster. The Plugin can be considered as an instance of a PluginDefinition, this instance specifies the PluginDefinition, the desired Cluster and additional values to set. Depending on the PluginDefinition, it may be necessary to specify required values (e.g. credentials, endpoints, etc.), but in general the PluginDefinition provides well-established default values.

[!NOTE] In this example the Plugin ‘openTelemetry-cluster-a’ is used to deploy the PluginDefinition ‘openTelemetry’ to the cluster ‘cluster-a’.

PluginPresets

PluginPresets are a mechanism to configure Plugins for multiple clusters at once. They are used to define a common configuration for a PluginDefinition that can be applied to multiple clusters, while allowing to override the configuration for individual clusters.

[!NOTE] In this example the PluginPreset ’example-obs’ references the PluginDefinition ’example’ and contains a clusterSelector that matches the clusters ‘cluster-a’ and ‘cluster-b’. The PluginPreset creates two Plugins ’example-obs-cluster-a’ and ’example-obs-cluster-b’ for the respective clusters.

1.2 - Overview

What is Greenhouse?

Greenhouse is a cloud operations platform designed to streamline and simplify the management of a large-scale, distributed infrastructure.

It offers a unified interface for organizations to manage various operational aspects efficiently and transparently and operate their cloud infrastructure in compliance with industry standards.
The platform addresses common challenges such as the fragmentation of tools, visibility of application-specific permission concepts and the management of organizational groups. It also emphasizes the harmonization and standardization of authorization concepts to enhance security and scalability. With its operator-friendly dashboard, features and extensive automation capabilities, Greenhouse empowers organizations to optimize their cloud operations, reduce manual efforts, and achieve greater operational efficiency.

Value Propositions

Roadmap

The Roadmap Kanban board provides an overview of ongoing and planned efforts.

Architecture & Design

The Greenhouse design and architecture document describes the various use-cases and user stories.

1.3 - Quick start guide

This section provides a step-by-step walkthrough for new users to navigate the initial stages of the Greenhouse platform.

Sign up

You’re an administrator and organization lead and want to register for Greenhouse?
We got you covered!

During phase 1 and 2 of the roadmap Greenhouse is only open to selected early adopters.
Please reach out to the Greenhouse team to register and create your organization via Slack or DL Greenhouse.

Prerequisites:

1. CAM Profile
   A CAM profile is required to configure the administrators of the organization.
   Please include the name of the profile in the message to the Greenhouse team when signing up.

2. SAP ID service
   The authentication for the users belonging to your organization is based on the OpenID Connect (OIDC) standard.
   For SAP, we recommend using a SAP ID service (IDS) tenant.
   Please include the parameters for your tenant in the message to the Greenhouse team when signing up.

   If you don’t have a SAP ID Service tenant yet, please refer to the SAP ID Service section for more information.

Already a member

You’re a member of an existing organization and want to manage your teams, clusters or plugins?
Please refer to the user guides to find out more.

2 - User guides

2.1 - Organization management

This section provides guides for the management your organization in Greenhouse.

2.1.1 - SAP ID Service

This section provides a step-by-step walkthrough for new users to request an SAP ID Service (IDS) tenant.

NOTE: This document is only available on the SAP-internal documentation page.

2.1.2 - Creating an organization

Before you begin

This guides describes how to create an organization in Greenhouse.

During phase 1 and 2 of the roadmap Greenhouse is only open to selected early adopters.
Please reach out to the Greenhouse team to register and create your organization via Slack or DL Greenhouse.

Creating an organization

An organization within the Greenhouse cloud operations platform is a separate unit with its own configuration, teams, and resources tailored to their requirements.
These organizations can represent different teams, departments, or projects within an enterprise, and they operate independently within the Greenhouse platform. They allow for the isolation and management of resources and configurations specific to their needs.

While the Greenhouse is build on the idea of a self-service API and automation driven platform, the workflow to onboard an organization to Greenhouse currently involves reaching out to the Greenhouse administrators until the official go-live.
This ensures all pre-requisites are met, the organization is configured correctly and the administrators understand the platform capabilities.

Steps

1. CAM Profile
   A CAM profile is required to configure the administrators of the organization.
   Please include the name of the profile in the message to the Greenhouse team when signing up.

2. SAP ID service
   The authentication for the users belonging to your organization is based on the OpenID Connect (OIDC) standard.
   For SAP, we recommend using a SAP ID service (IDS) tenant.
   Please include the parameters for your tenant in the message to the Greenhouse team when signing up.

   If you don’t have a SAP ID Service tenant yet, please refer to the SAP ID Service section for more information.

3. Greenhouse organization
   A Greenhouse administrator applies the following configuration to the central Greenhouse cluster.
   Bear in mind that the name of the organization is immutable and will be part of all URLs.

   apiVersion: v1
   kind: Namespace
   metadata:
     name: my-organization
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: oidc-config
     namespace: my-organization
   type: Opaque
   data:
     clientID: ...
     clientSecret: ...
   ---
   apiVersion: greenhouse.sap/v1alpha1
   kind: Organization
   metadata:
     name: my-organization
   spec:
     authentication:
       oidc:
         clientIDReference:
           key: clientID
           name: oidc-config
         clientSecretReference:
           key: clientSecret
           name: oidc-config
         issuer: https://...
       scim:
         baseURL: URL to the SCIM server.
         basicAuthUser:
           secret:
             name: Name of the secret in the same namespace.
             key: Key in the secret holding the user value.
         basicAuthPw:
           secret:
             name: Name of the secret in the same namespace.
             key: Key in the secret holding the password value.
     description: My new organization
     displayName: Short name of the organization
     mappedOrgAdminIdPGroup: Name of the group in the IDP that should be mapped to the organization admin role.
   

Setting up Team Membership synchronization with Greenhouse

Team Membership synchronization with Greenhouse requires access to SCIM API.

For the Team Memberships to be created Organization needs to be configured with URL and credentials of the SCIM API. SCIM API is used to get members for teams in the organization based on the IDP groups set for teams.

IDP group for the organization admin team must be set to the mappedOrgAdminIdPGroup field in the Organization configuration. It is required for the synchronization to work. IDP groups for remaining teams in the organization should be set in their respective configurations.

2.2 - Cluster management

Greenhouse enables organizations to register their Kubernetes clusters within the platform, providing a centralized interface for managing and monitoring these clusters.
Once registered, users can perform tasks related to cluster management, such as deploying applications, scaling resources, and configuring access control, all within the Greenhouse platform.

This section provides guides for the management of Kubernetes clusters within Greenhouse.

2.2.1 - Cluster onboarding

Content Overview

• Preparation
• Onboard
• After onboarding
• Trouble shooting

This guides describes how to onboard an existing Kubernetes cluster to your Greenhouse organization.
If you don’t have an organization yet please reach out to the Greenhouse administrators.

While all members of an organization can see existing clusters, their management requires org-admin or cluster-admin privileges.

NOTE: The UI is currently in development. For now this guide describes the onboarding workflow via command line.

Preparation

Download the latest greenhousectl binary from here.

Onboarding a Cluster to Greenhouse will require you to authenticate to two different Kubernetes clusters via respective kubeconfig files:

• greenhouse: The cluster your Greenhouse installation is running on. You need organization-admin or cluster-admin priviledges.
• bootstrap: The cluster you want to onboard. You need system:masters privileges.

For consistency we will refer to those two clusters by their names from now on.

You need to have the kubeconfig files for both the greenhouse and the bootstrap cluster at hand. The kubeconfig file for the greenhouse cluster can be downloaded via the Greenhouse dashboard:

Organization > Clusters > Access Greenhouse cluster.

Onboard

For accessing the bootstrap cluster, the greenhousectl will expect your default Kubernetes kubeconfig file and context to be set to bootstrap. This can be achieved by passing the --kubeconfig flag or by setting the KUBECONFIG env var.

The location of the kubeconfig file to the greenhouse cluster is passed via the --greenhouse-kubeconfig flag.

greenhousectl cluster bootstrap --kubeconfig=<path/to/bootstrap-kubeconfig-file> --greenhouse-kubeconfig <path/to/greenhouse-kubeconfig-file> --org <greenhouse-organization-name> --cluster-name <name>

Since Greenhouse generates URLs which contain the cluster name, we highly recommend to choose a short cluster name. In particular for Gardener Clusters setting a short name is mandatory, because Gardener has very long cluster names, e.g. garden-greenhouse--monitoring-external.

A typical output when you ran the command looks like

2024-02-01T09:34:55.522+0100	INFO	setup	Loaded kubeconfig	{"context": "default", "host": "https://api.greenhouse-qa.eu-nl-1.cloud.sap"}
2024-02-01T09:34:55.523+0100	INFO	setup	Loaded client kubeconfig	{"host": "https://api.monitoring.greenhouse.shoot.canary.k8s-hana.ondemand.com"}
2024-02-01T09:34:56.579+0100	INFO	setup	Bootstraping cluster	{"clusterName": "monitoring", "orgName": "ccloud"}
2024-02-01T09:34:56.639+0100	INFO	setup	created namespace	{"name": "ccloud"}
2024-02-01T09:34:56.696+0100	INFO	setup	created serviceAccount	{"name": "greenhouse"}
2024-02-01T09:34:56.810+0100	INFO	setup	created clusterRoleBinding	{"name": "greenhouse"}
2024-02-01T09:34:57.189+0100	INFO	setup	created clusterSecret	{"name": "monitoring"}
2024-02-01T09:34:58.309+0100	INFO	setup	Bootstraping cluster finished	{"clusterName": "monitoring", "orgName": "ccloud"}

After onboarding

1. List all clusters in your Greenhouse organization:

   kubectl --namespace=<greenhouse-organization-name> get clusters

2. Show the details of a cluster:

   kubectl --namespace=<greenhouse-organization-name> get cluster <name> -o yaml

Example:

apiVersion: greenhouse.sap/v1alpha1
kind: Cluster
metadata:
  creationTimestamp: "2024-02-07T10:23:23Z"
  finalizers:
    - greenhouse.sap/cleanup
  generation: 1
  name: monitoring
  namespace: ccloud
  resourceVersion: "282792586"
  uid: 0db6e464-ec36-459e-8a05-4ad668b57f42
spec:
  accessMode: direct
  maxTokenValidity: 72h
status:
  bearerTokenExpirationTimestamp: "2024-02-09T06:28:57Z"
  kubernetesVersion: v1.27.8
  statusConditions:
    conditions:
      - lastTransitionTime: "2024-02-09T06:28:57Z"
        status: "True"
        type: Ready

When the status.kubernetesVersion field shows the correct version of the Kubernetes cluster, the cluster was successfully bootstrapped in Greenhouse. Then status.conditions will contain a Condition with type=Ready and status="true""

In the remote cluster, a new namespace is created and contains some resources managed by Greenhouse. The namespace has the same name as your organization in Greenhouse.

Trouble shooting

If the bootstrapping failed, you can find details about why it failed in the Cluster.statusConditions. More precisely there will be a condition of type=KubeConfigValid which might have hints in the message field. This is also displayed in the UI on the Cluster details view. Reruning the onboarding command with an updated kubeConfig file will fix these issues.

2.3 - Plugin management

Plugins extends the capabilities of the Greenhouse cloud operations platform, adding specific features or functionalities to tailor and enhance the platform for specific organizational needs.
These plugins are integral to Greenhouse’ extensibility, allowing users to customize their cloud operations environment and address unique requirements while operating within the Greenhouse ecosystem.

This section provides guides for the management of plugins for Kubernetes clusters within Greenhouse.

2.3.1 - Local Plugin Development

Introduction

Let’s illustrate how to leverage Greenhouse Plugins to deploy a Helm Chart into a remote cluster within the local development environment.

This guide will walk you through the process of spinning up the local development environment, creating a new Greenhouse PluginDefinition and deploying it to a local kind cluster.

At the end of the guide you will have spun up the local development environment, onboarded a Cluster, created a PluginDefinition and deployed it as a Plugin to the onboarded Cluster.

[!NOTE] This guide assumes you already have a working Helm chart and will not cover how to create a Helm Chart from scratch. For more information on how to create a Helm Chart, please refer to the Helm documentation.

Requirements

• git
• Docker
• kind
• kubectl
• Greenhousectl

Starting the local develoment environment

Follow the Local Development documentation to spin up the local Greenhouse development environment.

This will provide you with a local Greenhouse instance running, filled with some example Greenhouse resources and the Greenhouse UI running on http://localhost:3000.

Onboarding a Cluster

In this step we will create and onboard a new Cluster to the local Greenhouse instance. The local cluster will be created utilizing kind.

In order to onboard a kind cluster follow the onboarding a cluster secton of the dev-env README.

After onboarding the cluster you should see the new Cluster in the Greenhouse UI.

Prepare Helm Chart

For this example we will use the bitnami nginx Helm Chart. The packaged chart can be downloaded with:

helm pull oci://registry-1.docker.io/bitnamicharts/nginx --destination ./

After unpacking the *.tgz file there is a folder named nginx containing the Helm Chart.

Generating a PluginDefinition from a Helm Chart

Using the files of the Helm Chart we will create a new Greenhouse PluginDefinition using the greenhousectl CLI.

greenhousectl plugin generate ./nginx ./nginx-plugin

This will create a new folder nginx-plugin containing the PluginDefinition in a nested structure.

Modifying the PluginDefinition

The generated PluginDefinition contains a plugindefinition.yaml file which defines the PluginDefinition. But there are still a few steps required to make it work.

Specify the Helm Chart repository

After generating the PluginDefinition the .spec.helmChart.repository field in the plugindefinition.yaml contains a TODO comment. This field should be set to the repository where the Helm Chart is stored. For the bitnami nginx Helm Chart this would be oci://registry-1.docker.io/bitnamicharts.

Specify the UI application

A PluginDefinition may specify a UI application that will be integrated into the Greenhouse UI. This tutorial does not cover how to create a UI application. Therefore the section .spec.uiApplication in the plugindefinition.yaml should be removed.

[!INFORMATION] The UI section of the dev-env readme provides a brief introduction developing a frontend application for Greenhouse.

Modify the Options

The PluginDefinition contains a section .spec.options which defines options that can be set when deploying the Plugin to a Cluster. These options have been generated based on the Helm Chart values.yaml file. You can modify the options to fit your needs.

In general the options are defined as follows:

options:
  - default: true
    value: abcd123
    description: automountServiceAccountToken
    name: automountServiceAccountToken
    required: false
    type: ""

default specifies if the option should provide a default value. If this is set to true, the value specified will be used as the default value. The Plugin can still provide a different value for this option. description provides a description for the option. name specifies the Helm Chart value name, as it is used within the Chart’s template files. required specifies if the option is required. This will be used by the Greenhouse Controllers to determine if a Plugin is valid. type specifies the type of the option. This can be any of [string, secret, bool, int, list, map]. This will be used by the Greenhouse Controllers to validate the provided value.

For this tutorial we will remove all options.

Deploying a Plugin to the Kind Cluster

After modifying the PluginDefinition we can deploy it to the local Greenhouse cluster and create a Plugin that will deploy the nginx to the onboarded cluster.

  kubectl --kubeconfig=./envtest/kubeconfig apply -f ./nginx-plugin/nginx/17.3.2/plugindefinition.yaml
  plugindefinition.greenhouse.sap/nginx-17.3.2 created

The Plugin can be configured using the Greenhouse UI running on http://localhost:3000. Follow the following steps to deploy a Plugin for the created PluginDefinition into the onboarded kind cluster:

1. Navigate on the Greenhouse UI to Organization>Plugins.
2. Click on the Add Plugin button.
3. Select the nginx-17.3.2 PluginDefinition.
4. Click on the Configure Plugin button.
5. Select the cluster in the drop-down.
6. Click on the Create Plugin button.

After the Plugin has been created the Plugin Overview page will show the status of the plugin.

Theh deployment can also be verified in the onboarded cluster by checking the pods in the test-org namespace of the kind cluster.

kind export kubeconfig --name remote-cluster
Set kubectl context to "kind-remote-cluster"

k get pods -n test-org
NAME                                    READY   STATUS    RESTARTS   AGE
nginx-remote-cluster-758bf47c77-pz72l   1/1     Running   0          2m11s

Development Tips

Local Helm Charts

Instead of uploading the Helm Chart to a chart repository, it is possible to load it from the filesystem of the Greenhouse container. This can be especially useful if you are developing your own chart for a PluginDefinition, as it speeds up the testing loop. The Docker compose setup mounts the dev-env/helm-charts directory and watches for any changes. This means you can point to this local chart in your plugindefinition.yaml as such:

helmChart:
  name: helm-charts/{filename}.tgz
  repository:

2.3.2 - Testing a Plugin

Plugin Testing Requirements

All plugins contributed to plugin-extensions repository should include comprehensive Helm Chart Tests using the bats/bats-detik testing framework. This ensures our plugins are robust, deployable, and catch potential issues early in the development cycle.

What is bats/bats-detik?

The bats/bats-detik framework simplifies end-to-end (e2e) Testing in Kubernetes. It combines the Bash Automated Testing System (bats) with Kubernetes-specific assertions (detik). This allows you to write test cases using natural language-like syntax, making your tests easier to read and maintain.

Implementing Tests

1. Create a /tests folder inside your Plugin’s Helm Chart templates folder to store your test resources.

2. ConfigMap definition:

   • Create a test-<plugin-name>-config.yaml file in the templates/tests directory to define a ConfigMap that will hold your test script.
   • This ConfigMap contains the test script run.sh that will be executed by the test Pod to run your tests.

{{- if .Values.testFramework.enabled -}}
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-test
  namespace: {{ .Release.Namespace }}
  labels:
    type: integration-test
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-weight": "-5" # Installed and upgraded before the test pod
    "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded"
data:
  run.sh: |-

    #!/usr/bin/env bats

    load "/usr/lib/bats/bats-detik/utils"
    load "/usr/lib/bats/bats-detik/detik"

    DETIK_CLIENT_NAME="kubectl"

    @test "Verify successful deployment and running status of the {{ .Release.Name }}-operator pod" {
        verify "there is 1 deployment named '{{ .Release.Name }}-operator'"
        verify "there is 1 service named '{{ .Release.Name }}-operator'"
        try "at most 2 times every 5s to get pods named '{{ .Release.Name }}-operator' and verify that '.status.phase' is 'running'"
    }

    @test "Verify successful creation and bound status of {{ .Release.Name }} persistent volume claims" {
        try "at most 3 times every 5s to get persistentvolumeclaims named '{{ .Release.Name }}.*' and verify that '.status.phase' is 'Bound'"
    }

    @test "Verify successful creation and available replicas of {{ .Release.Name }} Prometheus resource" {
        try "at most 3 times every 5s to get prometheuses named '{{ .Release.Name }}' and verify that '.status.availableReplicas' is more than '0'"
    }

    @test "Verify creation of required custom resource definitions (CRDs) for {{ .Release.Name }}" {
        verify "there is 1 customresourcedefinition named 'prometheuses'"
        verify "there is 1 customresourcedefinition named 'podmonitors'"
    }
{{- end -}}

Note: You can use this guide for reference when writing your test assertions.

3. Test Pod Definition:

   • Create a test-<plugin-name>.yaml file in the templates/tests directory to define a Pod that will run your tests.
   • This test Pod will mount the ConfigMap created in the previous step and will execute the test script run.sh.

{{- if .Values.testFramework.enabled -}}
apiVersion: v1
kind: Pod
metadata:
  name: {{ .Release.Name }}-test
  namespace: {{ .Release.Namespace }}
  labels:
    type: integration-test
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded"
spec:
  serviceAccountName: {{ .Release.Name }}-test
  containers:
    - name: bats-test
      image: "{{ .Values.testFramework.image.registry}}/{{ .Values.testFramework.image.repository}}:{{ .Values.testFramework.image.tag }}"
      imagePullPolicy: {{ .Values.testFramework.image.pullPolicy }}
      command: ["bats", "-t", "/tests/run.sh"]
      volumeMounts:
        - name: tests
          mountPath: /tests
          readOnly: true   volumes:
    - name: tests
      configMap:
        name: {{ .Release.Name }}-test
  restartPolicy: Never
{{- end -}}

4. RBAC Permissions:

• Create the necessary RBAC resources in the templates/tests folder with a dedicated ServiceAccount and role authorisations so that the test Pod can cover test the cases.
• You can use test-permissions.yaml from the kube-monitoring as a reference to configure RBAC permissions for your test Pod.

5. Configure the Test Framework in Plugin’s values.yaml:
   • Add the following configuration to your Plugin’s values.yaml file:

testFramework:
  enabled: true
  image:
    registry: ghcr.io
    repository: cloudoperators/greenhouse-extensions-integration-test
    tag: main
  imagePullPolicy: IfNotPresent

6. Running the Tests:

Important: Once you have completed all the steps above, you are ready to run the tests. However, before running the tests, ensure that you perform a fresh Helm installation or upgrade of your Plugin’s Helm release against your test Kubernetes cluster (for example, Minikube or Kind) by executing the following command:

# For a new installation
helm install <Release name> <chart-path>

# For an upgrade
helm upgrade <Release name> <chart-path>

• After the Helm installation or upgrade is successful, run the tests against the same test Kubernetes cluster by executing the following command.

helm test <Release name>

Contribution Checklist

Before submitting a pull request:

• Ensure your Plugin’s Helm Chart includes a /tests directory.
• Verify the presence of test-<plugin-name>.yaml, test-<plugin-name>-config.yaml, and test-permissions.yaml files.
• Test your Plugin thoroughly using helm test <release-name> and confirm that all tests pass against a test Kubernetes cluster.
• Include a brief description of the tests in your pull request.
• Make sure that your Plugin’s Chart Directory and the Plugin’s Upstream Chart Repository are added to this greenhouse-extensions helm test config file. This will ensure that your Plugin’s tests are automatically run in the GitHub Actions workflow when you submit a pull request for this Plugin.
• Note that the dependencies of your Plugin’s helm chart might also have their own tests. If so, ensure that the tests of the dependencies are also passing.

Important Notes

• Test Coverage: Aim for comprehensive test coverage to ensure your Plugin’s reliability.
• Test Isolation: Design tests that don’t interfere with other plugins or production environments.

2.3.3 - Plugin deployment

Before you begin

This guides describes how to configure and deploy a Greenhouse plugin.

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: kube-monitoring-martin
  namespace: <organization namespace> # same namespace in remote cluster for resources
spec:
  clusterName: <name of the remote cluster > # k get cluster
  disabled: false
  displayName: <any human readable name>
  pluginDefinition: <plugin name> # k get plugin
  optionValues:
    - name: <from the plugin options>
      value: <from the plugin options>
    - ...

Exposed services

Plugins deploying Helm Charts into remote clusters support exposed services.

By adding the following label to a service in helm chart it will become accessible from the central greenhouse system via a service proxy:

greenhouse.sap/expose: "true"

Deploying a Plugin

Create the Plugin resource via the command:

kubectl --namespace=<organization name> create -f plugin.yaml

After deployment

1. Check with kubectl --namespace=<organization name> get plugin has been properly created. When all components of the plugin are successfully created, the plugin should show the state configured.

2. Check in the remote cluster that all plugin resources are created in the organization namespace.

URLs for exposed services

After deploying the plugin to a remote cluster, ExposedServices section in Plugin’s status provides an overview of the Plugins services that are centrally exposed. It maps the exposed URL to the service found in the manifest.

• The URLs for exposed services are created in the following pattern: $https://$cluster--$hash.$organisation.$basedomain. The $hash is computed from service--$namespace.
• When deploying a plugin to the central cluster, the exposed services won’t have their URLs defined, which will be reflected in the Plugin’s Status.

2.3.4 - Managing Plugins for multiple clusters

Managing Plugins for multiple clusters

This guide describes how to configure and deploy a Greenhouse Plugin with the same configuration into multiple clusters.

The PluginPreset resource is used to create and deploy Plugins with a the identical configuration into multiple clusters. The list of clusters the Plugins will be deployed to is determind by a LabelSelector.

As a result, whenever a cluster, that matches the ClusterSelector is onboarded or offboarded, the Controller for the PluginPresets will take care of the Plugin Lifecycle. This means creating or deleting the Plugin for the respective cluster.

The same validation applies to the PluginPreset as to the Plugin. This includes immutable PluginDefinition and ReleaseNamespace fields, as well as the validation of the OptionValues against the PluginDefinition.

In case the PluginPreset is updated all of the Plugin instances that are managed by the PluginPreset will be updated as well. Each Plugin instance that is created from a PluginPreset has a label greenhouse.sap/pluginpreset: <PluginPreset name>. Also the name of the Plugin follows the scheme <PluginPreset name>-<cluster name>.

Changes that are done directly on a Plugin which was created from a PluginPreset will be overwritten immediately by the PluginPreset Controller. All changes must be performed on the PluginPreset itself. If a Plugin already existed with the same name as the PluginPreset would create, this Plugin will be ignored in following reconciliations.

Example PluginPreset

apiVersion: greenhouse.sap/v1alpha1
kind: PluginPreset
metadata:
  name: kube-monitoring-preset
  namespace: <organization namespace>
spec:
  plugin: # this embeds the PluginSpec
    displayName: <any human readable name>
    pluginDefinition: <PluginDefinition name> # k get plugindefinition
    releaseNamespace: <namespace> # namespace where the plugin is deployed to on the remote cluster. Will be created if not exists
    optionValues:
      - name: <from the PluginDefinition options>
        value: <from the PluginDefinition options>
      - ..
  clusterSelector: # LabelSelector for the clusters the Plugin should be deployed to
    matchLabels:
      <label-key>: <label-value>
  clusterOptionOverrides: # allows you to override specific options in a given cluster
    - clusterName: <cluster name where we want to override values>
      overrides:
        - name: <option name to override>
          value: <new value>
        - ..
    - ..

2.3.5 - Plugin Catalog

Before you begin

This guides describes how to explore the catalog of Greenhouse PluginDefinitions.

While all members of an organization can see the Plugin catalog, enabling, disabling and configuration PluginDefinitions for an organization requires organization admin privileges.

Exploring the PluginDefinition catalog

The PluginDefinition resource describes the backend and frontend components as well as mandatory configuration options of a Greenhouse extension.
While the PluginDefinition catalog is managed by the Greenhouse administrators and the respective domain experts, administrators of an organization can configure and tailor Plugins to their specific requirements.

NOTE: The UI also provides a preliminary catalog of Plugins under Organization> Plugin> Add Plugin.

1. Run the following command to see all available PluginDefinitions.

   $ kubectl get plugindefinition
   
   NAME                      VERSION   DESCRIPTION                                                                                                  AGE
   cert-manager              1.1.0     Automated certificate management in Kubernetes                                                               182d
   digicert-issuer           1.2.0     Extensions to the cert-manager for DigiCert support                                                          182d
   disco                     1.0.0     Automated DNS management using the Designate Ingress CNAME operator (DISCO)                                  179d
   doop                      1.0.0     Holistic overview on Gatekeeper policies and violations                                                      177d
   external-dns              1.0.0     The kubernetes-sigs/external-dns plugin.                                                                     186d
   heureka                   1.0.0     Plugin for Heureka, the patch management system.                                                             177d
   ingress-nginx             1.1.0     Ingress NGINX controller                                                                                     187d
   kube-monitoring           1.0.1     Kubernetes native deployment and management of Prometheus, Alertmanager and related monitoring components.   51d
   prometheus-alertmanager   1.0.0     Prometheus alertmanager                                                                                      60d
   supernova                 1.0.0     Supernova, the holistic alert management UI                                                                  187d
   teams2slack               1.1.0     Manage Slack handles and channels based on Greenhouse teams and their members                                115d
   

2.4 - Team management

A team is a group of users with shared responsibilities for managing and operating cloud resources within a Greenhouse organization.
These teams enable efficient collaboration, access control, and task assignment, allowing organizations to effectively organize their users and streamline cloud operations within the Greenhouse platform.

This section provides guides for the management of teams within an organization.

2.4.1 - Role-based access control

Contents

• Before you begin
• Greenhouse Team RBAC user guide
• Overview
• Defining TeamRoles
  • Example
• Seeded default TeamRoles
• Defining TeamRoleBindings
  • Assigning TeamRoles to Teams on a single Cluster
  • Assigning TeamRoles to Teams on multiple Clusters
  • Aggregating TeamRoles

Before you begin

This guides describes how to manage roles and permissions in Greenhouse with the help of TeamRoles and TeamRoleBindings.

While all members of an organization can see the permissions configured with TeamRoles & TeamRoleBindings, configuration of these requires OrganizationAdmin privileges.

Greenhouse Team RBAC user guide

Role-Based Access Control (RBAC) in Greenhouse allows organization administrators to regulate access to Kubernetes resources in onboarded Clusters based on the roles of individual users within an Organization. Within Greenhouse the RBAC on remote Clusters is managed using TeamRole and TeamRoleBinding. These two Custom Resource Defintions allow for fine-grained control over the permissions of each Team within each Cluster and Namespace.

Overview

• TeamRole: Defines a set of permissions that can be assigned to teams.
• TeamRoleBinding: Assigns a TeamRole to a specific Team for certain Clusters and (optionally) Namespaces.

Defining TeamRoles

TeamRoles define what actions a team can perform within the Kubernetes cluster. Common roles including the below cluster-admin are pre-defined within each organization.

Example

This TeamRole named pod-read grants read access to Pods!.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRole
metadata:
  name: pod-read
spec:
  rules:
    - apiGroups:
        - ""
      resources:
        - "pods"
      verbs:
        - "get"
        - "list"

Seeded default TeamRoles

Greenhouse provides a set of default TeamRoles that are seeded to all clusters:

Defining TeamRoleBindings

TeamRoleBindings define the permissions of a Greenhouse Team within Clusters by linking to a specific TeamRole. TeamRoleBindings have a simple specification that links a Team, a TeamRole, one or more Clusters and optionally a one or more Namespaces together. Once the TeamRoleBinding is created, the Team will have the permissions defined in the TeamRole within the specified Clusters and Namespaces. This allows for fine-grained control over the permissions of each Team within each Cluster. The TeamRoleBinding Controller within Greenhouse deploys rbacv1 resources to the targeted Clusters. The referenced TeamRole is created as a rbacv1.ClusterRole. In case the TeamRoleBinding references a Namespace, the Controller will create a rbacv1.RoleBinding which links the Team with the rbacv1.ClusterRole. In case no Namespace is referenced, the Controller will create a rbacv1.ClusterRoleBinding instead.

Assigning TeamRoles to Teams on a single Cluster

Roles are assigned to teams through the TeamRoleBinding configuration, which links teams to their respective roles within specific clusters.

This TeamRoleBinding assigns the pod-read TeamRole to the Team named my-team in the Cluster named my-cluster.

Example: team-rolebindings.yaml

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRoleBinding
metadata:
  name: my-team-read-access
spec:
  teamRef: my-team
  roleRef: pod-read
  clusterName: my-cluster

Assigning TeamRoles to Teams on multiple Clusters

It is also possible to use a LabelSelector to assign TeamRoleBindings to multiple Clusters at once.

This TeamRoleBinding assigns the pod-read TeamRole to the Team named my-team in all Clusters with the label environment: production.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRoleBinding
metadata:
  name: production-cluster-admins
spec:
  teamRef: my-team
  roleRef: pod-read
  clusterSelector:
    matchLabels:
      environment: production

Aggregating TeamRoles

It is possible with RBAC to aggregate rbacv1.ClusterRoles. This is also supported for TeamRoles. By specifying .spec.Labels on a TeamRole the resulting ClusterRole on the target cluster will have the same labels set. Then it is possible to aggregate multiple ClusterRole resources by using a rbacv1.AggregationRule. This can be specified on a TeamRole by setting .spec.aggregationRule.

More details on the concept of Aggregated ClusterRoles can be found in the Kubernetes documentation: Aggregated ClusterRoles

[!NOTE] A TeamRole is only created on a cluster if it is referenced by a TeamRoleBinding. If a TeamRole is not referenced by a TeamRoleBinding it will not be created on any target cluster. A TeamRoleBinding referencing a TeamRole with an aggregationRule will only provide the correct access, if there is at least one TeamRoleBinding referencing a TeamRole with the corresponding label deployed to the same cluster.

The following example shows how to an AggregationRule can be used with TeamRoles and TeamRoleBindings.

This TeamRole specifies .spec.Labels. The labels will be applied to the resulting ClusterRole on the target cluster.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRole
metadata:
  name: pod-read
spec:
  labels:
    aggregate: "true"
  rules:
    - apiGroups:
        - ""
      resources:
        - "pods"
      verbs:
        - "get"
        - "list"

This TeamRoleBinding assigns the pod-read TeamRole to the Team named my-team in all Clusters with the label environment: production.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRoleBinding
metadata:
  name: production-pod-read
spec:
  teamRef: my-team
  roleRef: pod-read
  clusterSelector:
    matchLabels:
      environment: production

This creates another TeamRole and TeamRoleBinding including the same labels as above.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRole
metadata:
  name: pod-edit
spec:
  labels:
    aggregate: "true"
  rules:
    - apiGroups:
        - ""
      resources:
        - "pod"
      verbs:
        - "update"
        - "patch"
---
apiVersion: greenhouse.sap/v1alpha1
kind: TeamRoleBinding
metadata:
  name: production-pod-edit
spec:
  teamRef: my-team
  roleRef: pod-edit
  clusterSelector:
    matchLabels:
      environment: production

This TeamRole has an aggregationRule set. This aggregationRule will be added to the ClusterRole created on the target clusters. With the aggregationRule set it will aggregate the ClusterRoles created by the TeamRoles with the label aggregate: "true". The team will have the permissions of both TeamRoles and will be able to get, list, update and patch Pods.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRole
metadata:
  name: aggregated-role
spec:
  aggregationRule:
    clusterRoleSelectors:
    - matchLabels:
        "aggregate": "true"

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRoleBinding
metadata:
  name: aggregated-rolebinding
spec:
  teamRef: operators
  roleRef: aggregated-role
  clusterSelector:
    matchLabels:
      environment: production

2.4.2 - Team creation

Before you begin

This guides describes how to create a team in your Greenhouse organization.

While all members of an organization can see existing teams, their management requires organization admin privileges.

Creating a team

The team resource is used to structure members of your organization and assign fine-grained access and permission levels.

Each Team must be backed by a group in the identity provider (IdP) of the Organization.

• IdP group should be set on the mappedIdPGroup field in Team configuration.
• This, along with SCIM API configured in the Organization, allows for synchronization of TeamMemberships with Greenhouse.

NOTE: The UI is currently in development. For now this guides describes the onboarding workflow via command line.

1. To onboard a new cluster provide the kubeconfig file with a static, short-lived token.
   It should look similar to this example:

   cat <<EOF | kubectl apply -f -
      apiVersion: greenhouse.sap/v1alpha1
      kind: Team
      metadata:
      name: <name>
      spec:
         description: My new team
         mappedIdPGroup: <IdP group name>
   EOF
   

3 - Architecture

This section provides an overview of the architecture and design of the Greenhouse platform.

3.1 - High-level architecture

This section provides a high-level overview of the Greenhouse concepts.

Greenhouse components

Conceptually, the Greenhouse platform consists of 2 types of Kubernetes cluster fulfilling specific purposes.

1. Central cluster

   The central cluster accommodates the core components of the Greenhouse platform, providing the holistic API and dashboard to let users manage their entire cloud infrastructure from a single control point.
   Users of the Greenhouse cloud operations platform, depending on their roles, can perform tasks such as managing organizations, configuring and using plugins, monitoring resources, developing custom plugins, and conducting audits to ensure compliance with standards.
   Greenhouse’s flexibility allows users to efficiently manage cloud infrastructure and applications while tailoring their experience to organizational needs.
   The configuration and metadata is persisted in Kubernetes custom resource definitions (CRDs), acted upon by the Greenhouse operator and managed in the customer cluster.

2. Customer cluster

   Managing and operating Kubernetes clusters can be challenging due to the complexity of tasks related to orchestration, scaling, and ensuring high availability in containerized environments.
   By onboarding their Kubernetes clusters into Greenhouse, users can centralize cluster management, streamlining tasks like resource monitoring and access control.
   This integration enhances visibility, simplifies operational tasks, and ensures better compliance, enabling users to efficiently manage and optimize their Kubernetes infrastructure. While the central cluster contains the user configuration and metadata, all workloads of user-selected Plugins are run in the customer cluster and managed by Greenhouse.

A simplified architecture of the Greenhouse platform is illustrated below.

3.2 - Product design

Introduction

Vision

“Greenhouse is an extendable Platform that enables Organizations to operate their Infrastructure in a compliant, efficient and transparent manner”

We want to build a Platform that is capable to integrate all the tools that are required to operate services in our private cloud environment in a compliant, effective and transparent manner. Greenhouse aims to be the single stop for Operators in the GCS PlusOne Organization. The primary focus of Greenhouse is to provide a unified interface for all operational aspects and tools, providing a simple shared data model describing the support organization.

As every organization is different, using different tools and has different requirements the platform is build in an extendable fashion that allows a distributed development of plugins.

While initially developed for the GCS PlusOne Organization the platform is explicitly designed to be of generic use and can be consumed by multiple organizations and teams.

Problem Statements

Consolidation of Toolsuite

The operation of cloud infrastructure and applications include a large amount of tasks that are supported by different tools and automations.
Due to the high complexity of cloud environments often times a conglomerate of tools is used to cover all operational aspects. Confguration and setup of the operations toolchain is a complex and time-consuming task that often times lacks automation when it comes to on- and off-boarding people and setting up new teams for new projects.

Visibility of Application Specific permission concepts

At SAP, we are managing identities and access centrally. The Converged Cloud is utilizing Cloud Access Manager for this task.
While it is true that we manage who has access to which access level is defined in there it starts getting complicated if you want to figure out the actual Permission Levels on individual Applications those Access Levels are mapped to.

Management of organizational Groups of People

You often have groups of people that are fulfilling a organizational purpose:

• Support Groups
• Working Groups
• etc.

We have currently no way to represent and manage those groups.

Harmonization and Standardization of Authorization Concepts

We are missing a tool that supports teams on creating access levels and profiles following a standardized naming scheme that is enforced by design and takes away the burden of coming up with names for access levels and user profiles/roles.

Single Point of Truth for Operations Metadata of an Organization

For automations, it is often critical to retrieve Metadata about an Organizational Unit:

• Who is member of a certain Group of people, that is maybe not reflecting the HR View of a Team?
• Which Tool is giving me access to data x,y,z?
• What action items are due and need to get alerted on?
• Does component x,z,y belong to my organization?
• etc. Currently, we do not have a single point of Truth for this kind of metadata and have to use a vaierity of Tools.

Terms

This section lists down Terms and description to Terms to ensure a common languague when talking in context of Greenhouse.

User Profiles

Every Application has Users / Stakeholders, so has Greenhouse. The User Profiles mentioned here give a abstract overview of considered Users / Stakeholders for the Application and the Goals and Tasks of them in context of the Platform.

Greenhouse admin

Administrator of a Greenhouse installation.

Goals

• Ensure overall function of the Greenhouse plattform

Tasks

• Create Organizations
• Enables Plugins
• Operates central infrastructure (Kubernetes cluster, operator, etc.)
• Assign initial organization admins

Organization admin

Administrator of a Greenhouse Organization

Goals

• Manage organization-wide resources

Tasks

• activate/configure plugins for the organization
• Create and manage teams for the organization
• Onboard and manage Kubernetes clusters for the organization

Organization member

Member of on organization that accesses the UI to do ops/support tasks. Is member of one ore more teams of the organization. By default members have view permissions on organization resources.

Goals

• Provide ops/support for the services of the organization

Tasks

• Highly dependend on team membership and plugins configured for the organization Examples:                  
  • Check alerts for teams user is assigned
  • Check policy violations for deployed resources owned by users team
  • Check for known vulnerabilites in services

Plugin developer

A plugin developer is developing plugins for Greenhouse.

Goals

• Must be easy to create plugins
• Can create and test plugins independently
• Greenhouse provides tooling to assist creating, validating, etc. plugins
• Publishing the plugin to Greenhouse requires admin permissions.

Tasks

• Plugin Development      
  • Juno UI Development
  • Plugin backend development

Auditor

An Auditor audits Greenhouse and/or Greenhouse Plugins for compliance with industry or regulatory standards.

Goals

• Wants to see that we have a record for all changes happening in greenhouse
• Wants to have access to resources required to audit the respective Plugin

Tasks

• Performs Audits

Greenhouse Developer

Develops parts of the Greenhouse platform (Kubernetes, Greenhouse app, Greenhouse backend, …)

Goals

• Provide Greenhouse framework

Tasks

• Provides framework for plugin developers to develop plugins
• Develops Greenhouse framework components (UI or backend)

User Stories

The User Stories outlined in this Section have the target to archive a common Understanding of the capabilities/functionalities the Platform wants to archive and the functional requirements that come with those. The Integration / Development of Functionalities is not going to be strictly bound to User Stories and they are rather used as an orientation to ensure that envisioned capabilities are not getting Blocked due to implementation details.

The details of all User Stories are subject to change based on the results of Proof of Concept implementations, User feedback or other unforseen factors.

Auditor

Auditor 01 - Audit Logging

As an Auditor, I want to see who did which action and when to verify that the Vulnerability and Patch management process is followed according to company policies and that the platform is functioning as expected.

Acceptance Criteria

• Every state-changing action is logged into an immutable log collection, including:
  • What action was performed
  • Who performed the action
  • When was the action performed
• Every authentication to the platform is logged into an immutable log collection, including:
  • Who logged in
  • When was the login

Greenhouse Admin

Greenhouse Admin 01 - Greenhouse Management Dashboard

As an Greenhouse Admin, I want a central Greenhouse Management Dashboard that allows me to navigate through all organization-wide resources to be able to manage the Platform.

Acceptance Criteria

• Assuming I am on the Greenhouse Management Dashboard view, i can:
  • See all Plugins, including the enabled version
  • Order not enabled Plugins by last Update Date
  • Plugins are Ordered by the Order Attribute
  • The order attribute is a numeric value that can be changed to reflect a different ordering of the Plugin:
    • 1 is ordered before 2 etc.
    • The order attribute is used as well to order the Plugins on the Navigation Bar
  • Navigate to “Plugin Detail View” by clicking a Plugin
  • See all Organizations, including:
    • Number of Organization Admins
    • Number of Organization Members
  • Navigate to organization creation view by clicking “Create Organization”
  • Navigate to Organization Detail View by clicking a Organization
• Only Greenhouse Admin’s should be able to see the Dashboard
• The Navigation item to the Greenhouse Management Dashboard should only be visible to Greenhouse admins

Greenhouse Admin 02 - Organization Creation View

As a Greenhouse Admin, I want a Organization Creation view that allows me to create a new Organization

Acceptance Criteria

• Assuming I am on the Organization creation View, i can:
  • Give a unique name for the organisation
  • Provide a short description for the organization
  • Provide a OIDC Group that gives Organization Admin Privileges

Greenhouse Admin & Organization Admin

Greenhouse Admin & Organization Admin 01 - Organization Detail View

As a Greenhouse Admin or Organization Admin, I want an Organization Detail view that allows me to view details about an organization

Acceptance Criteria

• Assuming I am on the organization detail View, i can:
  • Can see the organization details (name/description)
  • See a list of teams created for this organization
  • See the list of active plugins
  • Add Plugins to the organization by clicking “add Plugin”
  • Change the Organization Admin Role Name

Greenhouse Admin & Organization Admin 02 - Plugin Detail View

As an Greenhouse Admin or Organizatioin Admin, I want a Plugin Detail view that allows me to see  Plugin Details to be able to see details about the plugin.

Acceptance Criteria

• Assuming I am on the Plugin Detail View, I can:
  • see the plugin name
  • see the plugin description
  • see the last update date
  • see the release reference
  • see the ui release refrence
  • see the helm chart reference
  • see the ordering attribute
  • see configuration values for the plugin
  • set the configuration values for the current organization
  • see a change log
  • see the actually released (deployed to greenhouse) version

Organization Admin

Organization Admin 01 - Organization Managment Dashboard

As an Organization Admin, I want to have an Dashboard showing me the most relevant information about my Organization to be able to manage it efficently.

Acceptance Criteria

• Assuming I am on the organisation management dashboard
  • I can see a list of all teams in my organization
  • I can see a list of configured plugins
  • I can click a “add plugin” button to add a new plugin
  • I can see a list of registered clusters
  • I can click a “add cluster” button to register a cluster

Organization Admin 02 - Plugin Configuration View

As an organization admin, I want a Plugin configuration view that allows me to enable and configure greenhouse plugins for my organization

Acceptance Criteria

• Assuming I am on the Plugin configuration View, I can:
  • select the type of plugin I want to configure
  • enable/disable the plugin (for my org)
  • remove the plugin (when already added)
  • manage configuration options specific for the plugin

Organization Admin 03 - Cluster registration View

As an organization admin, I want a Cluster registration view to onboard kubernetes clusters into my organization.

Acceptance Criteria

• Assuming I am on the cluster registration view, I can:
  • Get instructions how to register a kubernetes clusters
  • give a name and description for the registered cluster
• After executing the provided instructions I get feedback that the cluster has been successfully registered
• A cluster can be registered to exactly one organization

Organization Admin 04 - Cluster detail View

As an organization admin, I want a Cluster detail view to get some information about a registered cluster

Acceptance Criteria

• Assuming I am on the cluster detail view, I can:
  • see basic details about the cluster:
    • name
    • api url
    • version
    • node status
  • de-register the cluster from my organization

Organization Admin 05 - Team Detail View

As an organization admin, i want to have a Team Detail View, with the option to configure teams based on role mapping to be to manage teams within my organization without managing the permission administration itself on the Platform

Acceptance Criteria

• Assuming I am on the Team detail view, i can:
  • Change the Name of the Team
  • change the description of the team
  • Define a single OIDC Group  that assign you this team
  • Define The Greenhouse Role that you get within Greenhouse if you are a member of the team
• On Login of a User into an Organization the Platform verifies if the User has ALL required roles

Organization Admin 05 - Team Creation View

As an organization admin, i want to have a Team Creation View, to be able to create a new Team

Acceptance Criteria

• Assuming I am on the Team Creation view, I can::
  • Set the name of the Team
  • Set a description of the Team
  • Set a OIDC Group Name that assigns users to this team

Organization Member

Organization Member 01 - Unified task inbox

As an organization member, I want a task inbox that shows my open tasks from all enabled plugins that need my attention to be on top of my tasks to fulfill across all plugins

Acceptance Criteria

• Assuming I am on the task inbox:
  • I can a list of open task accross all plugins that need attention
  • clicking on an open task jumps in the plugin specific UI the task belong to
  • I can sort open tasks by name, plugin or date

Plugin Developer

As a Plugin Developer, I want to have a seperate Repository for my Plugin which I can own and use to configure plugin internals to have control over the Development efforts and configuration of the Plugin

Plugin Developer 01 - Decentrally Managed Plugin

As a Plugin Developer, I want to have a seperate Repository for my Plugin which I can own and use to configure plugin internals to have control over the Development efforts and configuration of the Plugin

Acceptance Criteria

• Plugin lives on his own Github Repository
• Versions are managed via Github Releases using Tags and the release to Greenhouse is managed by the Plugin:
  • The version to be pulled by Greenhouse is managed by the Plugin Developer.
• I can configure the Plugin Configuration over a greenhouse.yml in the root of the repository, which at least includes (mandatory):
  • description: …
  • version: …
• There are optional attributes in the greenhouse.yml:
  • icon: which if it has a valid absolute path to an image file on the repository makes the icon selectable as an icon in the plugin detail view (GA02)
  • describes available configuration options that attributes that are required for the plugin to function
  • I can specify a reference to a UI App
  • I can specify a reference to Helm Charts

Plugin Developer 02 - Plugin Role Config

As a Plugin Developer i want a section within the Greenhouse.yml metadata, named “Roles” where i can setup Roles used by my Plugin

Acceptance Criteria

Plugin Developer 03 - Spec Schema Validation

As a Plugin Developer I want to have the possibility to validate the schema of my greenhouse.yaml to be able to catch errors within my specification early.

Acceptance Criteria

• The schema check should support IDE’s
• The schema check should be automate-able and be integrate-able to pre-commit hooks and quality gates
• A version with a broken schema should not be release on greenhouse even when pushing for a pull of the release
• It should be visible on the Plugin detail view when an invalid schema was released with a recent version

Plugin Developer 04 - Config Value Validation

As a Plugin Developer I want to have the possibility to write custom regex checks for configuration options of my plugin that include the check to be performed on a field and an error message to be shown if configured wrong by an organization to support organization admins on configuring my plugin

Acceptance Criteria

• The validation rules should be controlled by Plugin Developer
• The validation should happen on the frontend before submitting a configuration
• The error message should be shown when a config value is provided wrong

Plugin Developer 05 - Plugin development tooling

As a plugin developer I want to have an easy setup for developing and testing greenhouse plugins

Acceptance Criteria

• Dev environment available within X Time
• Possible to have a working local setup with a “mock greenhouse apiserver”
• Has a fully working Bootstrap Project that includes Backend and Frontend which can be run locally immediately
• Has documentation

Product Stages

Overview

This Section gives an overview of the different early stages of the Platform that are beeing developed and which functional requirements need to get fulfilled within those stages.

Proof of Concept (POC)

The Proof of Concept is the stage where fundamental Framework/Platform decisions are proven to be working as intended. At this Stage the Platform is not suitable to be used by the intended audience yet but most necessary core functionalities are implemented.

The desired functionalities in this phase are:

• Frontend with Authentication
• Authorization within Greenhouse (Greenhouse Admin, Org Admin, Org Member)
• Team Management (without UI)
• Org Management (without UI)
• Greenhouse Admin User Stories (without UI)
• Dummy Plugin
  • with configuration spec
• Plugin Development Setup (without Documentation)
• Plugin Versioning & Provisioning (without UI)

Minimal viable product (MVP)

This stage is considered to be the earliest stage to open the Platform for General use.
In addition to the PoC functionalities we expect the following requirements to be fulfilled:

• Integrated 3 Plugins:
  • Supernova (Alerts)
  • DOOP (Violations)
  • Heureka
    NOTE: Heureka was excluded from MVP as the Heureka API is only available at a later point in time.
• Team management
• Organization management

3.3 - Building Observability Architectures

The main terminologies used in this document can be found in core-concepts.

Introduction to Observability

Observability in software and infrastructure has become essential for operating the complex modern cloud environments. The concept centers on understanding the internal states of systems based on the data they produce, enabling teams to:

• Detect and troubleshoot issues quickly,
• Maintain performance and reliability,
• Make data-driven improvements.

Core Signals and Open Source Tools

Key pillars of observability are metrics, logs, and traces, each providing unique insights that contribute to a comprehensive view of a system’s health.

• Metrics:

  • Metrics are numerical data points representing system health over time (e.g., CPU usage, memory usage, request latency).
  • Prometheus is a widely used tool for collecting and querying metrics. It uses a time-series database optimized for real-time data, making it ideal for gathering system health data, enabling alerting, and visualizing trends.

• Logs:

  • Logs capture detailed, structured/unstructured records of system events, which are essential for post-incident analysis and troubleshooting.
  • OpenSearch provides a robust, scalable platform for log indexing, search, and analysis, enabling teams to sift through large volumes of logs to identify issues and understand system behavior over time.

• Traces:

  • Traces follow a request’s journey through the system, capturing latency and failures across microservices. Traces are key for understanding dependencies and diagnosing bottlenecks.
  • Jaeger is a popular open-source tool for distributed tracing, providing a detailed view of request paths and performance across services.

To provide a unified approach to open source observability, OpenTelemetry was developed as a framework for instrumenting applications and infrastructures to collect metrics, logs and traces. In addition to providing a unified API and SDKs for multiple programming languages, OpenTelemetry also simplifies the integration of various backend systems such as Prometheus, OpenSearch and Jaeger.

Observability in Greenhouse

Greenhouse provides a suite of Plugins which consist of pre-packaged configurations for monitoring and logging tools. These Plugins are designed to simplify the setup and configuration of observability components, enabling users to quickly deploy and manage monitoring and logging for their Greenhouse-onboarded Kubernetes clusters.

The following Plugins are available currently:

• Kubernetes Monitoring: Prometheus, to collect custom and Kubernetes specific metrics with standard Kubernetes alerting enabled.
• Thanos: Thanos, to enable long term metric retention and unified metric accessibility.
• Plutono: Grafana fork, to create dynamic dashboards for metrics.
• Alerts: Prometheus Alertmanager and Supernova, to manage and visualize alerts sent by Prometheus.
• OpenTelemetry: OpenTelemetry Pipelines, to collect metrics and logs from applications and forward them to backends like Prometheus and OpenSearch.

Overview Architecture

4 - Reference

This section contains reference documentation for Greenhouse.

4.1 - API

Packages:

• greenhouse.sap/v1alpha1

greenhouse.sap/v1alpha1

Authentication

(Appears on: OrganizationSpec)

Cluster

Cluster is the Schema for the clusters API

ClusterAccessMode (string alias)

(Appears on: ClusterSpec)

ClusterAccessMode configures the access mode to the customer cluster.

ClusterConditionType (string alias)

ClusterConditionType is a valid condition of a cluster.

ClusterKubeConfig

(Appears on: ClusterSpec)

ClusterKubeConfig configures kube config values.

ClusterKubeconfig

ClusterKubeconfig is the Schema for the clusterkubeconfigs API ObjectMeta.OwnerReferences is used to link the ClusterKubeconfig to the Cluster ObjectMeta.Generation is used to detect changes in the ClusterKubeconfig and sync local kubeconfig files ObjectMeta.Name is designed to be the same with the Cluster name

ClusterKubeconfigAuthInfo

(Appears on: ClusterKubeconfigAuthInfoItem)

ClusterKubeconfigAuthInfoItem

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigCluster

(Appears on: ClusterKubeconfigClusterItem)

ClusterKubeconfigClusterItem

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigContext

(Appears on: ClusterKubeconfigContextItem)

ClusterKubeconfigContextItem

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigData

(Appears on: ClusterKubeconfigSpec)

ClusterKubeconfigData stores the kubeconfig data ready to use kubectl or other local tooling It is a simplified version of clientcmdapi.Config: https://pkg.go.dev/k8s.io/client-go/tools/clientcmd/api#Config

ClusterKubeconfigPreferences

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigSpec

(Appears on: ClusterKubeconfig)

ClusterKubeconfigSpec stores the kubeconfig data for the cluster The idea is to use kubeconfig data locally with minimum effort (with local tools or plain kubectl): kubectl get cluster-kubeconfig $NAME -o yaml | yq -y .spec.kubeconfig

ClusterKubeconfigStatus

(Appears on: ClusterKubeconfig)

ClusterOptionOverride

(Appears on: PluginPresetSpec)

ClusterOptionOverride defines which plugin option should be override in which cluster

ClusterSpec

(Appears on: Cluster)

ClusterSpec defines the desired state of the Cluster.

ClusterStatus

(Appears on: Cluster)

ClusterStatus defines the observed state of Cluster

Condition

(Appears on: PropagationStatus, StatusConditions)

Condition contains additional information on the state of a resource.

ConditionReason (string alias)

(Appears on: Condition)

ConditionReason is a valid reason for a condition of a resource.

ConditionType (string alias)

(Appears on: Condition)

ConditionType is a valid condition of a resource.

HelmChartReference

(Appears on: PluginDefinitionSpec, PluginStatus)

HelmChartReference references a Helm Chart in a chart repository.

HelmReleaseStatus

(Appears on: PluginStatus)

HelmReleaseStatus reflects the status of a Helm release.

NodeStatus

(Appears on: ClusterStatus)

OIDCConfig

(Appears on: Authentication)

Organization

Organization is the Schema for the organizations API

OrganizationSpec

(Appears on: Organization)

OrganizationSpec defines the desired state of Organization

OrganizationStatus

(Appears on: Organization)

OrganizationStatus defines the observed state of an Organization

Plugin

Plugin is the Schema for the plugins API

PluginDefinition

PluginDefinition is the Schema for the PluginDefinitions API

PluginDefinitionSpec

(Appears on: PluginDefinition)

PluginDefinitionSpec defines the desired state of PluginDefinitionSpec

PluginDefinitionStatus

(Appears on: PluginDefinition)

PluginDefinitionStatus defines the observed state of PluginDefinition

PluginOption

(Appears on: PluginDefinitionSpec)

PluginOptionType (string alias)

(Appears on: PluginOption)

PluginOptionType specifies the type of PluginOption.

PluginOptionValue

(Appears on: ClusterOptionOverride, PluginSpec)

PluginOptionValue is the value for a PluginOption.

PluginPreset

PluginPreset is the Schema for the PluginPresets API

PluginPresetSpec

(Appears on: PluginPreset)

PluginPresetSpec defines the desired state of PluginPreset

PluginPresetStatus

(Appears on: PluginPreset)

PluginPresetStatus defines the observed state of PluginPreset

PluginSpec

(Appears on: Plugin, PluginPresetSpec)

PluginSpec defines the desired state of Plugin

PluginStatus

(Appears on: Plugin)

PluginStatus defines the observed state of Plugin

PropagationStatus

(Appears on: TeamRoleBindingStatus)

PropagationStatus defines the observed state of the TeamRoleBinding’s associated rbacv1 resources on a Cluster

SCIMConfig

(Appears on: Authentication)

SecretKeyReference

(Appears on: OIDCConfig, ValueFromSource)

SecretKeyReference specifies the secret and key containing the value.

Service

(Appears on: PluginStatus)

Service references a Kubernetes service of a Plugin.

StatusConditions

(Appears on: ClusterKubeconfigStatus, ClusterStatus, NodeStatus, OrganizationStatus, PluginPresetStatus, PluginStatus, TeamMembershipStatus, TeamRoleBindingStatus, TeamStatus)

A StatusConditions contains a list of conditions. Only one condition of a given type may exist in the list.

Team

Team is the Schema for the teams API

TeamMembership

TeamMembership is the Schema for the teammemberships API

TeamMembershipSpec

(Appears on: TeamMembership)

TeamMembershipSpec defines the desired state of TeamMembership

TeamMembershipStatus

(Appears on: TeamMembership)

TeamMembershipStatus defines the observed state of TeamMembership

TeamRole

TeamRole is the Schema for the TeamRoles API

TeamRoleBinding

TeamRoleBinding is the Schema for the rolebindings API

TeamRoleBindingSpec

(Appears on: TeamRoleBinding)

TeamRoleBindingSpec defines the desired state of a TeamRoleBinding

TeamRoleBindingStatus

(Appears on: TeamRoleBinding)

TeamRoleBindingStatus defines the observed state of the TeamRoleBinding

TeamRoleSpec

(Appears on: TeamRole)

TeamRoleSpec defines the desired state of a TeamRole

TeamRoleStatus

(Appears on: TeamRole)

TeamRoleStatus defines the observed state of a TeamRole

TeamSpec

(Appears on: Team)

TeamSpec defines the desired state of Team

TeamStatus

(Appears on: Team)

TeamStatus defines the observed state of Team

UIApplicationReference

(Appears on: PluginDefinitionSpec, PluginStatus)

UIApplicationReference references the UI pluginDefinition to use.

User

(Appears on: TeamMembershipSpec)

User specifies a human person.

ValueFromSource

(Appears on: PluginOptionValue, SCIMConfig)

ValueFromSource is a valid source for a value.

4.2 - Plugin Catalog

This section provides an overview of the available PluginDefinitions in Greenhouse.

4.2.1 - Alerts

Learn more about the alerts plugin. Use it to activate Prometheus alert management for your Greenhouse organisation.

The main terminologies used in this document can be found in core-concepts.

Overview

This Plugin includes a preconfigured Prometheus Alertmanager, which is deployed and managed via the Prometheus Operator, and Supernova, an advanced user interface for Prometheus Alertmanager. Certificates are automatically generated to enable sending alerts from Prometheus to Alertmanager. These alerts can too be sent as Slack notifications with a provided set of notification templates.

Components included in this Plugin:

• Prometheus Alertmanager
• Supernova

This Plugin usually is deployed along the kube-monitoring Plugin and does not deploy the Prometheus Operator itself. However, if you are intending to use it stand-alone, you need to explicitly enable the deployment of Prometheus Operator, otherwise it will not work. It can be done in the configuration interface of the plugin.

Disclaimer

This is not meant to be a comprehensive package that covers all scenarios. If you are an expert, feel free to configure the plugin according to your needs.

The Plugin is a deeply configured kube-prometheus-stack Helm chart which helps to keep track of versions and community updates.

It is intended as a platform that can be extended by following the guide.

Contribution is highly appreciated. If you discover bugs or want to add functionality to the plugin, then pull requests are always welcome.

Quick start

This guide provides a quick and straightforward way to use alerts as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

• A running and Greenhouse-onboarded Kubernetes cluster. If you don’t have one, follow the Cluster onboarding guide.
• kube-monitoring plugin (which brings in Prometheus Operator) OR stand alone: awareness to enable the deployment of Prometheus Operator with this plugin

Step 1:

You can install the alerts package in your cluster with Helm manually or let the Greenhouse platform lifecycle it for you automatically. For the latter, you can either:

1. Go to Greenhouse dashboard and select the Alerts Plugin from the catalog. Specify the cluster and required option values.
2. Create and specify a Plugin resource in your Greenhouse central cluster according to the examples.

Step 2:

After the installation, you can access the Supernova UI by navigating to the Alerts tab in the Greenhouse dashboard.

Step 3:

Greenhouse regularly performs integration tests that are bundled with alerts. These provide feedback on whether all the necessary resources are installed and continuously up and running. You will find messages about this in the plugin status and also in the Greenhouse dashboard.

Configuration

Prometheus Alertmanager options

Supernova options

theme: Override the default theme. Possible values are "theme-light" or "theme-dark" (default)

endpoint: Alertmanager API Endpoint URL /api/v2. Should be one of alerts.alertmanager.ingress.hosts

silenceExcludedLabels: SilenceExcludedLabels are labels that are initially excluded by default when creating a silence. However, they can be added if necessary when utilizing the advanced options in the silence form.The labels must be an array of strings. Example: ["pod", "pod_name", "instance"]

filterLabels: FilterLabels are the labels shown in the filter dropdown, enabling users to filter alerts based on specific criteria. The ‘Status’ label serves as a default filter, automatically computed from the alert status attribute and will be not overwritten. The labels must be an array of strings. Example: ["app", "cluster", "cluster_type"]

predefinedFilters: PredefinedFilters are filters applied through in the UI to differentiate between contexts through matching alerts with regular expressions. They are loaded by default when the application is loaded. The format is a list of objects including name, displayname and matchers (containing keys corresponding value). Example:

[
  {
    "name": "prod",
    "displayName": "Productive System",
    "matchers": {
      "region": "^prod-.*"
    }
  }
]

silenceTemplates: SilenceTemplates are used in the Modal (schedule silence) to allow pre-defined silences to be used to scheduled maintenance windows. The format consists of a list of objects including description, editable_labels (array of strings specifying the labels that users can modify), fixed_labels (map containing fixed labels and their corresponding values), status, and title. Example:

"silenceTemplates": [
    {
      "description": "Description of the silence template",
      "editable_labels": ["region"],
      "fixed_labels": {
        "name": "Marvin",
      },
      "status": "active",
      "title": "Silence"
    }
  ]

Managing Alertmanager configuration

ref:

• https://prometheus.io/docs/alerting/configuration/#configuration-file
• https://prometheus.io/webtools/alerting/routing-tree-editor/

By default, the Alertmanager instances will start with a minimal configuration which isn’t really useful since it doesn’t send any notification when receiving alerts.

You have multiple options to provide the Alertmanager configuration:

1. You can use alerts.alertmanager.config to define a Alertmanager configuration. Example below.

config:
  global:
    resolve_timeout: 5m
  inhibit_rules:
    - source_matchers:
        - "severity = critical"
      target_matchers:
        - "severity =~ warning|info"
      equal:
        - "namespace"
        - "alertname"
    - source_matchers:
        - "severity = warning"
      target_matchers:
        - "severity = info"
      equal:
        - "namespace"
        - "alertname"
    - source_matchers:
        - "alertname = InfoInhibitor"
      target_matchers:
        - "severity = info"
      equal:
        - "namespace"
  route:
    group_by: ["namespace"]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 12h
    receiver: "null"
    routes:
      - receiver: "null"
        matchers:
          - alertname =~ "InfoInhibitor|Watchdog"
  receivers:
    - name: "null"
  templates:
    - "/etc/alertmanager/config/*.tmpl"

2. You can discover AlertmanagerConfig objects. The spec.alertmanagerConfigSelector is always set to matchLabels: plugin: <name> to tell the operator which AlertmanagerConfigs objects should be selected and merged with the main Alertmanager configuration. Note: The default strategy for a AlertmanagerConfig object to match alerts is OnNamespace.

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: config-example
  labels:
    alertmanagerConfig: example
    pluginDefinition: alerts-example
spec:
  route:
    groupBy: ["job"]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: "webhook"
  receivers:
    - name: "webhook"
      webhookConfigs:
        - url: "http://example.com/"

3. You can use alerts.alertmanager.alertmanagerSpec.alertmanagerConfiguration to reference an AlertmanagerConfig object in the same namespace which defines the main Alertmanager configuration.

# Example with select a global alertmanagerconfig
alertmanagerConfiguration:
  name: global-alertmanager-configuration

Examples

Deploy alerts with Alertmanager

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: alerts
spec:
  pluginDefinition: alerts
  disabled: false
  displayName: Alerts
  optionValues:
    - name: alerts.alertmanager.enabled
      value: true
    - name: alerts.alertmanager.ingress.enabled
      value: true
    - name: alerts.alertmanager.ingress.hosts
      value:
        - alertmanager.dns.example.com
    - name: alerts.alertmanager.ingress.tls
      value:
        - hosts:
            - alertmanager.dns.example.com
          secretName: tls-alertmanager-dns-example-com
    - name: alerts.alertmanagerConfig.slack.routes
      value:
        - channel: slack-warning-channel
          webhookURL: https://hooks.slack.com/services/some-id
          matchers:
            - name: severity
              matchType: "="
              value: "warning"
        - channel: slack-critical-channel
          webhookURL: https://hooks.slack.com/services/some-id
          matchers:
            - name: severity
              matchType: "="
              value: "critical"
    - name: alerts.alertmanagerConfig.webhook.routes
      value:
        - name: webhook-route
          url: https://some-webhook-url
          matchers:
            - name: alertname
              matchType: "=~"
              value: ".*"
    - name: alerts.alertmanager.serviceMonitor.additionalLabels
      value:
        plugin: kube-monitoring
    - name: alerts.defaultRules.create
      value: true
    - name: alerts.defaultRules.labels
      value:
        plugin: kube-monitoring
    - name: endpoint
      value: https://alertmanager.dns.example.com/api/v2
    - name: filterLabels
      value:
        - job
        - severity
        - status
    - name: silenceExcludedLabels
      value:
        - pod
        - pod_name
        - instance

Deploy alerts without Alertmanager (Bring your own Alertmanager - Supernova UI only)

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: alerts
spec:
  pluginDefinition: alerts
  disabled: false
  displayName: Alerts
  optionValues:
    - name: alerts.alertmanager.enabled
      value: false
    - name: alerts.alertmanager.ingress.enabled
      value: false
    - name: alerts.defaultRules.create
      value: false
    - name: endpoint
      value: https://alertmanager.dns.example.com/api/v2
    - name: filterLabels
      value:
        - job
        - severity
        - status
    - name: silenceExcludedLabels
      value:
        - pod
        - pod_name
        - instance

4.2.2 - Cert-manager

This Plugin provides the cert-manager to automate the management of TLS certificates.

Configuration

This section highlights configuration of selected Plugin features.
All available configuration options are described in the plugin.yaml.

Ingress shim

An Ingress resource in Kubernetes configures external access to services in a Kubernetes cluster.
Securing ingress resources with TLS certificates is a common use-case and the cert-manager can be configured to handle these via the ingress-shim component.
It can be enabled by deploying an issuer in your organization and setting the following options on this plugin.

4.2.3 - Decentralized Observer of Policies (Violations)

This directory contains the Greenhouse plugin for the Decentralized Observer of Policies (DOOP).

DOOP

To perform automatic validations on Kubernetes objects, we run a deployment of OPA Gatekeeper in each cluster. This dashboard aggregates all policy violations reported by those Gatekeeper instances.

4.2.4 - Designate Ingress CNAME operator (DISCO)

This Plugin provides the Designate Ingress CNAME operator (DISCO) to automate management of DNS entries in OpenStack Designate for Ingress and Services in Kubernetes.

4.2.5 - DigiCert issuer

This Plugin provides the digicert-issuer, an external Issuer extending the cert-manager with the DigiCert cert-central API.

4.2.6 - External DNS

This Plugin provides the external DNS operator) which synchronizes exposed Kubernetes Services and Ingresses with DNS providers.

4.2.7 - Github Guard

Github Guard Greenhouse Plugin manages Github teams, team memberships and repository & team assignments.

Hierarchy of Custom Resources

Custom Resources

Github – an installation of Github App

apiVersion: githubguard.sap/v1
kind: Github
metadata:
  name: com
spec:
  webURL: https://github.com
  v3APIURL: https://api.github.com
  integrationID: 420328
  clientUserAgent: greenhouse-github-guard
  secret: github-com-secret

GithubOrganization with Feature & Action Flags

apiVersion: githubguard.sap/v1
kind: GithubOrganization
metadata:
  name: com--greenhouse-sandbox
  labels:
    githubguard.sap/addTeam: "true"
    githubguard.sap/removeTeam: "true"
    githubguard.sap/addOrganizationOwner: "true"
    githubguard.sap/removeOrganizationOwner: "true"
    githubguard.sap/addRepositoryTeam: "true"
    githubguard.sap/removeRepositoryTeam: "true"
    githubguard.sap/dryRun: "false"

Default team & repository assignments:

GithubTeamRepository for exception team & repository assignments

GithubUsername for external username matching

apiVersion: githubguard.sap/v1
kind: GithubUsername
metadata:
  annotations: 
    last-check-timestamp: 1681614602 
   name: com-I313226 
spec:
  userID: greenhouse_onuryilmaz
  githubUsername: onuryilmaz
  github: com

4.2.8 - Ingress NGINX

This plugin contains the ingress NGINX controller.

Example

To instantiate the plugin create a Plugin like:

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: ingress-nginx
spec:
  pluginDefinition: ingress-nginx-v4.4.0
  values:
    - name: controller.service.loadBalancerIP
      value: 1.2.3.4

4.2.9 - Kubernetes Monitoring

Learn more about the kube-monitoring plugin. Use it to activate Kubernetes monitoring for your Greenhouse cluster.

The main terminologies used in this document can be found in core-concepts.

Overview

Observability is often required for operation and automation of service offerings. To get the insights provided by an application and the container runtime environment, you need telemetry data in the form of metrics or logs sent to backends such as Prometheus or OpenSearch. With the kube-monitoring Plugin, you will be able to cover the metrics part of the observability stack.

This Plugin includes a pre-configured package of components that help make getting started easy and efficient. At its core, an automated and managed Prometheus installation is provided using the prometheus-operator. This is complemented by Prometheus target configuration for the most common Kubernetes components providing metrics by default. In addition, Cloud operators curated Prometheus alerting rules and Plutono dashboards are included to provide a comprehensive monitoring solution out of the box.

Components included in this Plugin:

• Prometheus
• Prometheus Operator
• Prometheus target configuration for Kubernetes metrics APIs (e.g. kubelet, apiserver, coredns, etcd)
• Prometheus node exporter
• kube-state-metrics
• kubernetes-operations

Disclaimer

It is not meant to be a comprehensive package that covers all scenarios. If you are an expert, feel free to configure the plugin according to your needs.

The Plugin is a deeply configured kube-prometheus-stack Helm chart which helps to keep track of versions and community updates.

It is intended as a platform that can be extended by following the guide.

Contribution is highly appreciated. If you discover bugs or want to add functionality to the plugin, then pull requests are always welcome.

Quick start

This guide provides a quick and straightforward way to use kube-monitoring as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

• A running and Greenhouse-onboarded Kubernetes cluster. If you don’t have one, follow the Cluster onboarding guide.

Step 1:

You can install the kube-monitoring package in your cluster by installing it with Helm manually or let the Greenhouse platform lifecycle it for you automatically. For the latter, you can either:

1. Go to Greenhouse dashboard and select the Kubernetes Monitoring plugin from the catalog. Specify the cluster and required option values.
2. Create and specify a Plugin resource in your Greenhouse central cluster according to the examples.

Step 2:

After installation, Greenhouse will provide a generated link to the Prometheus user interface. This is done via the annotation greenhouse.sap/expose: “true” at the Prometheus Service resource.

Step 3:

Greenhouse regularly performs integration tests that are bundled with kube-monitoring. These provide feedback on whether all the necessary resources are installed and continuously up and running. You will find messages about this in the plugin status and also in the Greenhouse dashboard.

Configuration

Global options

Prometheus-operator options

Kubernetes component scraper options

Prometheus options

Alertmanager options

Examples

Deploy kube-monitoring into a remote cluster

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: kube-monitoring
spec:
  pluginDefinition: kube-monitoring
  disabled: false
  optionValues:
    - name: kubeMonitoring.prometheus.prometheusSpec.retention
      value: 30d
    - name: kubeMonitoring.prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage
      value: 100Gi
    - name: kubeMonitoring.prometheus.service.labels
      value:
        greenhouse.sap/expose: "true"
    - name: kubeMonitoring.prometheus.prometheusSpec.externalLabels
      value:
        cluster: example-cluster
        organization: example-org
        region: example-region
    - name: alerts.enabled
      value: true
    - name: alerts.alertmanagers.hosts
      value:
        - alertmanager.dns.example.com
    - name: alerts.alertmanagers.tlsConfig.cert
      valueFrom:
        secret:
          key: tls.crt
          name: tls-<org-name>-prometheus-auth
    - name: alerts.alertmanagers.tlsConfig.key
      valueFrom:
        secret:
          key: tls.key
          name: tls-<org-name>-prometheus-auth

Deploy Prometheus only

Example Plugin to deploy Prometheus with the kube-monitoring Plugin.

NOTE: If you are using kube-monitoring for the first time in your cluster, it is necessary to set kubeMonitoring.prometheusOperator.enabled to true.

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: example-prometheus-name
spec:
  pluginDefinition: kube-monitoring
  disabled: false
  optionValues:
    - name: kubeMonitoring.defaultRules.create
      value: false
    - name: kubeMonitoring.kubernetesServiceMonitors.enabled
      value: false
    - name: kubeMonitoring.prometheusOperator.enabled
      value: false
    - name: kubeMonitoring.kubeStateMetrics.enabled
      value: false
    - name: kubeMonitoring.nodeExporter.enabled
      value: false
    - name: kubeMonitoring.prometheus.prometheusSpec.retention
      value: 30d
    - name: kubeMonitoring.prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage
      value: 100Gi
    - name: kubeMonitoring.prometheus.service.labels
      value:
        greenhouse.sap/expose: "true"
    - name: kubeMonitoring.prometheus.prometheusSpec.externalLabels
      value:
        cluster: example-cluster
        organization: example-org
        region: example-region
    - name: alerts.enabled
      value: true
    - name: alerts.alertmanagers.hosts
      value:
        - alertmanager.dns.example.com
    - name: alerts.alertmanagers.tlsConfig.cert
      valueFrom:
        secret:
          key: tls.crt
          name: tls-<org-name>-prometheus-auth
    - name: alerts.alertmanagers.tlsConfig.key
      valueFrom:
        secret:
          key: tls.key
          name: tls-<org-name>-prometheus-auth

Extension of the plugin

kube-monitoring can be extended with your own Prometheus alerting rules and target configurations via the Custom Resource Definitions (CRDs) of the Prometheus operator. The user-defined resources to be incorporated with the desired configuration are defined via label selections.

The CRD PrometheusRule enables the definition of alerting and recording rules that can be used by Prometheus or Thanos Rule instances. Alerts and recording rules are reconciled and dynamically loaded by the operator without having to restart Prometheus or Thanos Rule.

kube-monitoring Prometheus will automatically discover and load the rules that match labels plugin: <plugin-name>.

Example:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: example-prometheus-rule
  labels:
    plugin: <metadata.name> 
    ## e.g plugin: kube-monitoring
spec:
 groups:
   - name: example-group
     rules:
     ...

The CRDs PodMonitor, ServiceMonitor, Probe and ScrapeConfig allow the definition of a set of target endpoints to be scraped by Prometheus. The operator will automatically discover and load the configurations that match labels plugin: <plugin-name>.

Example:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: example-pod-monitor
  labels:
    plugin: <metadata.name> 
    ## e.g plugin: kube-monitoring
spec:
  selector:
    matchLabels:
      app: example-app
  namespaceSelector:
    matchNames:
      - example-namespace
  podMetricsEndpoints:
    - port: http
  ...

4.2.10 - Logshipper

This Plugin is intended for shipping container and systemd logs to an Elasticsearch/ OpenSearch cluster. It uses fluentbit to collect logs. The default configuration can be found under chart/templates/fluent-bit-configmap.yaml.

Components included in this Plugin:

• fluentbit

Owner

1. @ivogoman

Parameters

Custom Configuration

To add custom configuration to the fluent-bit configuration please check the fluentbit documentation here. The fluent-bit.customConfig.inputs, fluent-bit.customConfig.filters and fluent-bit.customConfig.outputs parameters can be used to add custom configuration to the default configuration. The configuration should be added as a multi-line string. Inputs are rendered after the default inputs, filters are rendered after the default filters and before the additional values are added. Outputs are rendered after the default outputs. The additional values are added to all logs disregaring the source.

Example Input configuration:

fluent-bit:
  config:
    inputs: |
      [INPUT]
          Name             tail-audit
          Path             /var/log/containers/greenhouse-controller*.log
          Parser           {{ default "cri" ( index .Values "fluent-bit" "parser" ) }}
          Tag              audit.*
          Refresh_Interval 5
          Mem_Buf_Limit    50MB
          Skip_Long_Lines  Off
          Ignore_Older     1m
          DB               /var/log/fluent-bit-tail-audit.pos.db      

Logs collected by the default configuration are prefixed with default_. In case that logs from additional inputs are to be send and processed by the same filters and outputs, the prefix should be used as well.

In case additional secrets are required the fluent-bit.env field can be used to add them to the environment of the fluent-bit container. The secrets should be created by adding them to the fluent-bit.backend field.

fluent-bit:
  backend:
    audit:
      http_user: top-secret-audit
      http_password: top-secret-audit
      host: "audit.test"
      tls:
        enabled: true
        verify: true
        debug: false

4.2.11 - OpenTelemetry

Learn more about the OpenTelemetry Plugin. Use it to enable the ingestion, collection and export of telemetry signals (logs and metrics) for your Greenhouse cluster.

The main terminologies used in this document can be found in core-concepts.

Overview

OpenTelemetry is an observability framework and toolkit for creating and managing telemetry data such as metrics, logs and traces. Unlike other observability tools, OpenTelemetry is vendor and tool agnostic, meaning it can be used with a variety of observability backends, including open source tools such as OpenSearch and Prometheus.

The focus of the plugin is to provide easy-to-use configurations for common use cases of receiving, processing and exporting telemetry data in Kubernetes. The storage and visualization of the same is intentionally left to other tools.

Components included in this Plugin:

• Operator
• Collector
• Receivers
  • Filelog Receiver
  • k8events Receiver
  • journald Receiver
  • prometheus/internal
• OpenSearch Exporter

Architecture

TBD: Architecture picture

Note

It is the intention to add more configuration over time and contributions of your very own configuration is highly appreciated. If you discover bugs or want to add functionality to the plugin, feel free to create a pull request.

Quick Start

This guide provides a quick and straightforward way to use OpenTelemetry as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

• A running and Greenhouse-onboarded Kubernetes cluster. If you don’t have one, follow the Cluster onboarding guide.
• For logs, a OpenSearch instance to store. If you don’t have one, reach out to your observability team to get access to one.
• For metrics, a Prometheus instance to store. If you don’t have one, install a kube-monitoring Plugin first.

Step 1:

You can install the OpenTelemetry package in your cluster by installing it with Helm manually or let the Greenhouse platform lifecycle do it for you automatically. For the latter, you can either:

1. Go to Greenhouse dashboard and select the OpenTelemetry plugin from the catalog. Specify the cluster and required option values.
2. Create and specify a Plugin resource in your Greenhouse central cluster according to the examples.

Step 2:

The package will deploy the OpenTelemetry Operator which works as a manager for the collectors and auto-instrumentation of the workload. By default, the package will include a configuration for collecting metrics and logs. The log-collector is currently processing data from the preconfigured receivers:

• Files via the Filelog Receiver
• Kubernetes Events from the Kubernetes API server
• Journald events from systemd journal
• its own metrics

You can disable the collection of logs by setting open_telemetry.LogCollector.enabled to false. The same is true for disabling metrics: open_telemetry.MetricsCollector.enabled to false.

Based on the backend selection the telemetry data will be exporter to the backend.

Step 3:

Greenhouse regularly performs integration tests that are bundled with OpenTelemetry. These provide feedback on whether all the necessary resources are installed and continuously up and running. You will find messages about this in the plugin status and also in the Greenhouse dashboard.

Configuration

Examples

TBD

4.2.12 - Plutono

Learn more about the plutono Plugin. Use it to install the web dashboarding system Plutono to collect, correlate, and visualize Prometheus metrics for your Greenhouse cluster.

The main terminologies used in this document can be found in core-concepts.

Overview

Observability is often required for the operation and automation of service offerings. Plutono provides you with tools to display Prometheus metrics on live dashboards with insightful charts and visualizations. In the Greenhouse context, this complements the kube-monitoring plugin, which automatically acts as a Plutono data source which is recognized by Plutono. In addition, the Plugin provides a mechanism that automates the lifecycle of datasources and dashboards without having to restart Plutono.

Disclaimer

This is not meant to be a comprehensive package that covers all scenarios. If you are an expert, feel free to configure the Plugin according to your needs.

Contribution is highly appreciated. If you discover bugs or want to add functionality to the plugin, then pull requests are always welcome.

Quick Start

This guide provides a quick and straightforward way how to use Plutono as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

• A running and Greenhouse-managed Kubernetes cluster
• kube-monitoring Plugin installed to have at least one Prometheus instance running in the cluster

The plugin works by default with anonymous access enabled. If you use the standard configuration in the kube-monitoring plugin, the data source and some kubernetes-operations dashboards are already pre-installed.

Step 1: Add your dashboards

Dashboards are selected from ConfigMaps across namespaces. The plugin searches for ConfigMaps with the label plutono-dashboard: "true" and imports them into Plutono. The ConfigMap must contain a key like my-dashboard.json with the dashboard JSON content. Example

A guide on how to create dashboards can be found here.

Step 2: Add your datasources

Data sources are selected from Secrets across namespaces. The plugin searches for Secrets with the label plutono-dashboard: "true" and imports them into Plutono. The Secrets should contain valid datasource configuration YAML. Example

Configuration