Greenhouse documentation

• 1: Getting started
• 1.1: Core Concepts
• 1.1.1: Organizations
• 1.1.2: Teams
• 1.1.3: Clusters
• 1.1.4: PluginDefinitions, Plugins and PluginPresets
• 1.2: Overview
• 1.3: Installation
• 1.4: Operational Processes
• 1.5: Ownership
• 2: User guides
• 2.1: Organization management
• 2.1.1: Creating an organization
• 2.2: Cluster management
• 2.2.1: Cluster onboarding
• 2.2.2: Remote Cluster Connectivity with OIDC
• 2.2.3: Cluster offboarding
• 2.3: Plugin management
• 2.3.1: Testing a Plugin
• 2.3.2: Plugin deployment
• 2.3.3: Managing Plugins for multiple clusters
• 2.3.4: Plugin Catalog
• 2.4: Team management
• 2.4.1: Role-based access control on remote clusters
• 2.4.2: Team creation
• 3: Architecture
• 3.1: Architecture
• 3.2: Product design
• 3.3: Building Observability Architectures
• 4: Reference
• 4.1: API
• 4.2: Greenhouse API
• 4.2.1: Organizations
• 4.2.2: PluginDefinitions
• 4.2.3: PluginPresets
• 4.2.4: Plugins
• 4.2.5: Catalogs
• 4.3: Plugin Catalog
• 4.3.1:
• 4.3.2: Alerts
• 4.3.3: Audit Logs Plugin
• 4.3.4: Cert-manager
• 4.3.5: Decentralized Observer of Policies (Violations)
• 4.3.6: Designate Ingress CNAME operator (DISCO)
• 4.3.7: DigiCert issuer
• 4.3.8: External DNS
• 4.3.9: Ingress NGINX
• 4.3.10: Kubernetes Monitoring
• 4.3.11: Logs Plugin
• 4.3.12: Logshipper
• 4.3.13: OpenSearch
• 4.3.14: Perses
• 4.3.15: Plutono
• 4.3.16: Prometheus
• 4.3.17: Reloader
• 4.3.18: Repo Guard
• 4.3.19: Service exposure test
• 4.3.20: Thanos
• 5: Contribute
• 5.1: Local development setup
• 5.2: Contributing a Plugin
• 5.3: Greenhouse Controller Development
• 6:
• 7:

1 - Getting started

1.1 - Core Concepts

1.1.1 - Organizations

What are Organizations?

Organizations are the top-level entities in Greenhouse. Each Organization gets a dedicated Namespace, that contains all resources bound to the Organization. Greenhouse expects an Organization to provide its own Identity Provider and currently supports OIDC Identity Providers. Greenhouse also supports SCIM for syncing users and groups from an Identity Provider.

See Creating an Organization for more details.

Organization Namespace and Permissions

The Organization’s Namespace in the Greenhouse cluster contains all resources bound to the Organization. This Namespace is automatically provisioned when a new Organization is created and shares the Organization’s name. Once the Namespace is created, Greenhouse will automatically seed RBAC Roles and ClusterRoles for the Organization. These are used to grant permissions for the Organization’s resources to Teams.

• The Administrators of an Organization are specified via a identity provider (IDP) group during the creation of the Organization. Greenhouse automatically creates a Team called <org-name>-admin. This Team is also a support group and all alerts created by the Greenhouse controller are routed to this admin Team, if no other ownership is provided. See operational processes and ownership for details.
• The Administrators for Plugins and Clusters need to be defined by the Organization Admins via RoleBindings for the seeded Roles role:<org-name>:cluster-admin and role:<org-name>:plugin-admin.
• All authenticated users are considered members of the Organization and are granted the organization:<org-name> Role.

See Working with Organizations for details on the seeded Roles and ClusterRoles.

OIDC

Each Organization must specify the OIDC configuration for the Organization’s IDP. This configuration is used together with DEXIDP to authenticate users in the Organization.

SCIM

Each Organization can specify SCIM credentials which are used to syncronize users and groups from an Identity Provider. This makes it possible to view the members of a Team in the Greenhouse dashboard.

Next Steps

• Creating an Organization
• Organization reference

1.1.2 - Teams

What are Teams?

Teams in a Greenhouse Organization are used to group users by group claims on the token provided by the upstream identity provider (IdP).

This can be used, for example, for

• organizational management
• access and permission management
• identifying Ownership of resources

The Greenhouse UI is showing the members of a Team.

Team RBAC

TeamRoles and TeamRoleBindings provide a mechanism to control the permissions of Teams to onboarded Clusters of an Organization.

Team role-based access control (RBAC) wraps the concept of Kubernetes RBAC in TeamRoles and TeamRoleBindings . TeamRoles are used to define a set of RBAC permissions. These permissions can be granted to Teams with TeamRoleBindings . A TeamRoleBinding refers to a Team, a TeamRole , Cluster(s) and optional Namespaces. Depending on the latter, Greenhouse will create the appropriate rbacv1 resources on the targeted cluster(s) in either Cluster or Namespace scope. More information about how this can be configured is mentioned in this user guide.

Example of a TeamRoleBinding for a observability-admin which grants the cluster-admin TeamRole on the observability Cluster in the logs and metrics Namespaces. The TeamRoleBinding contains a list of Namespaces and a ClusterSelector to select the cluster(s) to target. Setting the .spec.namespaces field decides whether the created RBAC resources will be namespace-scoped or cluster-scoped.

apiVersion: greenhouse.sap/v1alpha2
kind: TeamRoleBinding
metadata:
  name: observability-admin
spec:
  teamRef: observability
  roleRef: cluster-admin
  clusterSelector:
    clusterName: observability
  namespaces:
    - logs
    - metrics

Support Groups

Support Groups in Greenhouse are a subset of Teams in an Organization. These Teams are used to identify response groups for operational tasks and to prefilter UI content.

Since a user can be part of many Teams the expectation is that they are only part of one Support Group.

To identify a Team as a Support Group in Greenhouse it needs to be labeled with greenhouse.sap/support-group: "true".

1.1.3 - Clusters

What are Clusters?

In the context of Greenhouse a Cluster represents a Kubernetes cluster that is onboarded to Greenhouse. Onboarded in this context means that Greenhouse can handle the management of role-based access control (RBAC) and the provisioning of operating tools (e.g. logging, monitoring, ingress etc.). The Greenhouse dashboard provides an overview of all onboarded clusters. Throughout Greenhouse the reference to a Cluster is used to target it for configuration and deployments.

Cluster access

During the initial onboarding of a cluster, Greenhouse will create a dedicated ServiceAccount inside the onboarded cluster. This ServiceAccount’s token is rotated automatically by Greenhouse.

Cluster registry (coming soon)

Once a Cluster is onboarded to Greenhouse a ClusterKubeConfig is generated for the Cluster based on the OIDC configuration of the Organization. This enables members of an Organization to access the fleet of onboarded Clusters via the common Identity Provider. on the respective Clusters can be managed via Greenhouse Team RBAC.

In order to make it convenient to use these ClusterKubeConfigs and to easily switch between multiple context locally there will be a CLI provided by Greenhouse.

1.1.4 - PluginDefinitions, Plugins and PluginPresets

What are PluginDefinitions and Plugins?

PluginDefinitons and Plugins are the Greenhouse way to extend the core functionality with domain specific features. PluginDefinitions, as the name suggests, are the definition of a Plugin, whereas a Plugin is a concrete instance of a PluginDefinition that is deployed to a Cluster.

The PluginDefinitions are shared between all Organizations in Greenhouse. A PluginDefinition can include a frontend, that is displayed in the Greenhouse dashboard and/or a backend component. The frontend is expected to be a standalone microfrontend created with the Juno framework. The backend components of a PluginDefinition are packaged as a Helm Chart and provide sane and opinionated default values. This allows Greenhouse to package and distribute tools such as Prometheus with a sensible default configuration, as well as giving the user a list of configurable values.

A Plugin is used to deploy the Helm Chart referenced in a PluginDefinition to a Cluster. The Plugin can be considered as an instance of a PluginDefinition, this instance specifies the PluginDefinition, the desired Cluster and additional values to set. Depending on the PluginDefinition, it may be necessary to specify required values (e.g. credentials, endpoints, etc.), but in general the PluginDefinition provides well-established default values.

[!NOTE] In this example the Plugin ‘openTelemetry-cluster-a’ is used to deploy the PluginDefinition ‘openTelemetry’ to the cluster ‘cluster-a’.

PluginPresets

PluginPresets are a mechanism to configure Plugins for multiple clusters at once. They are used to define a common configuration for a PluginDefinition that can be applied to multiple clusters, while allowing to override the configuration for individual clusters.

[!NOTE] In this example the PluginPreset ’example-obs’ references the PluginDefinition ’example’ and contains a clusterSelector that matches the clusters ‘cluster-a’ and ‘cluster-b’. The PluginPreset creates two Plugins ’example-obs-cluster-a’ and ’example-obs-cluster-b’ for the respective clusters.

Next Steps

• Creating an Organization
• Plugin reference
• PluginPreset reference
• PluginDefinition reference

1.2 - Overview

What is Greenhouse?

Greenhouse is a cloud operations platform designed to streamline and simplify the management of a large-scale, distributed infrastructure.

It offers a unified interface for organizations to manage various operational aspects efficiently and transparently and operate their cloud infrastructure in compliance with industry standards.
The platform addresses common challenges such as the fragmentation of tools, visibility of application-specific permission concepts and the management of organizational groups. It also emphasizes the harmonization and standardization of authorization concepts to enhance security and scalability. With its operator-friendly dashboard, features and extensive automation capabilities, Greenhouse empowers organizations to optimize their cloud operations, reduce manual efforts, and achieve greater operational efficiency.

Value Propositions

1. Holistic dashboard Unified dashboard for infrastructure, alert, security, compliance, and organizational management. (Juno)
2. Organization management Greenhouse allows to manage organizational groups as Teams. Teams can be provided with fine-grained access control to resources and tools. (e.g. Github Repositories, Kubernetes RBAC, etc.)
3. Automation Greenhouse allows to configure tools and access control in a declarative way, that is auto-deployed across a fleet of Kubernetes clusters.
4. Security & Compliance With Heureka, Greenhouse integrates a Security Posture Management tool that focuses on remediation of security issues (vulnerabilities, security events, policy violations), while ensuring compliance and auditability.
5. Extensibility Greenhouse provides a plugin system that provides a curated catalog of plugins with sane defaults. Furthermore, it is possible to extend the platform with self-developed plugins.

Roadmap

The Roadmap Kanban board provides an overview of ongoing and planned efforts.

Architecture & Design

The Greenhouse design and architecture document describes the various use-cases and user stories.

1.3 - Installation

This section provides a step-by-step guide to install Greenhouse on a Gardner shoot cluster.

Prerequisites

Before you start the installation, make sure you have the following prerequisites:

• Helm & Kubernetes CLI
• OAuth2/OpenID provider (see Authentik)
• Gardener Shoot Cluster configured to use the OIDC provider
• nginx-ingress deployed in the cluster

Installation

To install Greenhouse on your Gardener shoot cluster, follow these steps:

1. Create a values file called values.yaml with the following content:

     global:
       dnsDomain: tld.domain # Shoot.spec.dns.domain
       kubeAPISubDomain: myapi # api is already used by Gardener
       oidc:
         enabled: true
         issuer: <issuer-url>
         clientID: <client-ID>
         clientSecret: <top-secret>
   
     organization:
       enabled: false # disable, because the greenhouse webhook is not running yet
   
     teams:
       admin:
         mappedIdPGroup: greenhouse-admins
   
     # gardener specifics
     dashboard:
       ingress:
         annotations:
           dns.gardener.cloud/dnsnames: "*"
           dns.gardener.cloud/ttl: "600"
           dns.gardener.cloud/class: garden
           cert.gardener.cloud/purpose: managed
   
     idproxy:
       enabled: false # disable because no organization is created yet
       ingress:
         annotations:
           dns.gardener.cloud/dnsnames: "*"
           dns.gardener.cloud/ttl: "600"
           dns.gardener.cloud/class: garden
           cert.gardener.cloud/purpose: managed
   
     cors-proxy:
       ingress:
         annotations:
           dns.gardener.cloud/dnsnames: "*"
           dns.gardener.cloud/ttl: "600"
           dns.gardener.cloud/class: garden
           cert.gardener.cloud/purpose: managed
   
     # disable Plugins for the greenhouse organization, PluginDefinitions are missing
     plugins:
       enabled: false
   
     # disable, Prometheus CRDs are missing
     manager:
       alerts:
         enabled: false
   

2. Install the Greenhouse Helm chart:

   helm install greenhouse oci://ghcr.io/cloudoperators/greenhouse/charts/greenhouse --version <greenhouse-release-version> -f values.yaml
   

3. Enable Greenhouse OIDC

   Now set organization.enabled and idproxy.enabled to true in the values.yaml file and upgrade the Helm release:

   helm upgrade greenhouse oci://ghcr.io/cloudoperators/greenhouse/charts/greenhouse --version <greenhouse-release-version> -f values.yaml
   

   This will create the initial Greenhouse Organization and the Greenhouse Admin Team. This Organization will receive the greenhouse namespace, which is used to manage the Greenhouse installation and allows to administer other organizations. Enabling the idproxy will deploy the idproxy service which handles the OIDC authentication.

1.4 - Operational Processes

Operational Processes in Greenhouse

Greenhouse provides a couple of predefined operational processes.

Operational processes facilitated via Greenhouse heavily rely on the Ownership principle. It is used to route operational tasks to Support Groups.

Examples for these operational tasks are:

• Alert routing based on metrics
• Lifecycle management of k8s Clusters
• Security posture and vulnerability patch management
• Secret rotation and management

Labels Used

Greenhouse focuses on labels in three different places:

• On resources (e.g. PluginPresets, Clusters but also k8s Deployments, Pods, etc.)
• On metrics exposed by those resources
• On Prometheus alerts based on metrics

The following labels are used by Greenhouse automation:

Alert Routing

Monitoring and alert routing is achieved through a combination of Plugins running on the remote Clusters and the Greenhouse central cluster.

All alerts processed with Greenhouse need the support_group label that can be extracted from the greenhouse.sap/owned-by.

With the Alerts Plugin a holistic alerts dashboard is integrated to the Greenhouse UI. This dashboard is prefiltered on the support group a user is member of. It directly displays alerts by region and severity. Also service is prominently displayed.

It is good practice to also route alerts by support_group and/or severity to specific Alertmanager receivers (e.g. Slack channels).

Lifecycle management of k8s Clusters

All Cluster related alerts, including version expiration and other maintenance tasks are routed to the owning support_group of the Cluster.

Security Management

Security posture and vulnerability management is achieved through the heureka Plugin. It scans for security violations in running k8s containers and displays these by support_group and service.

Secret Management

With secret management Greenhouse wants to have alerts on expiring Secrets in need of rotation. These alerts will be routed to the respective support_groups. See roadmap item for further information.

1.5 - Ownership

What is Ownership within Greenhouse

Ownership in Greenhouse is the combination of two of the core features:

• User and Organization management via Teams
• Deployment of resources (Plugins, TeamRoleBindings) to remote Clusters

Greenhouse provides a 1:1 relationship between a Team and

• PluginPresets
• Plugins
• Clusters
• TeamRoleBindings
• Secrets

Within the context of Greenhouse this relationship is called Ownership.

Why Ownership of Resources

Operational processes facilitated via Greenhouse rely on Ownership:

By identifying the owner of a resource it is possible to route operational tasks on the resource to the owner.

How is Ownership achieved

Greenhouse expects a label with the key greenhouse.sap/owned-by with a value matching an existing Team on the following resources in the Greenhouse central cluster:

• PluginPresets
• Plugins
• Clusters
• TeamRoleBindings
• Secrets

Missing greenhouse.sap/owned-by label results in a StatusCondition called OwnerLabelSetCondition set to false. A greenhouse_owned_by_label_missing metric on missing owner labels is exposed and alerted on.

The owner label is also expected on k8s resources (e.g. Deployments, Pods, …) exposing metrics on the remote clusters.

Label Transport

On Greenhouse central cluster

The Greenhouse controller transports labels from a source resource to a target resource on the Greenhouse cluster. This is currently active for:

• Secrets that are used to bootstrap a Cluster
• PluginPresets creating Plugins

The transport works via metadata.annotation on the source:

metadata:
  ...
  labels:
    foo: bar
    qux: baz
    greenhouse.sap/owned_by: foo-team
    ...
  annotations:
    greenhouse.sap/propagate-labels: "foo, greenhouse.sap/owned_by"
  ...

which results in metadata.labels and a state in metadata.annotations added to the target:

metadata:
  annotations:
   ...
    greenhouse.sap/last-applied-propagator: '{"labelKeys":["foo","greenhouse.sap/owned_by"]}'
  labels:
    foo: bar
    greenhouse.sap/owned_by: foo-team
   ...

On Resources on Remote Clusters

Greenhouse will provide the automation to label all resources created by a Plugin on the remote Cluster in the future: https://github.com/cloudoperators/greenhouse-extensions/issues/704

Currently Greenhouse provides the owned-by label as a OptionValue to be consumed by the underlying helm chart of the Plugin.

2 - User guides

2.1 - Organization management

This section provides guides for the management your organization in Greenhouse.

2.1.1 - Creating an organization

Before you begin

This guides describes how to create an organization in Greenhouse.

Creating an organization

An organization within the Greenhouse cloud operations platform is a separate unit with its own configuration, teams, and resources tailored to their requirements. These organizations can represent different teams, departments, or projects within an enterprise, and they operate independently within the Greenhouse platform. They allow for the isolation and management of resources and configurations specific to their needs. Since each Organization will receive its own Kubernetes Namespace within the Greenhouse cluster.

While Greenhouse is build on the idea of a self-service API and automation driven platform, the workflow to onboard an organization to Greenhouse involves reaching out to the Greenhouse administrators. This ensures all pre-requisites are met, the organization is configured correctly and the administrators of the Organization understand the platform capabilities.

Steps

1. IdP Group An IdP Group is required to configure the administrators of the organization. Onboarding without an Group is not possible, as this would leave the organization without any administrators having access. Please include the name of the IdP Group in the message to the Greenhouse team when signing up.

2. Identity Provider
   The authentication for the users belonging to your organization is based on the OpenID Connect (OIDC) standard.
   Please include the parameters for your OIDC provider in the message to the Greenhouse team when signing up.

3. Greenhouse organization
   A Greenhouse administrator applies the following configuration to the central Greenhouse cluster.
   Bear in mind that the name of the organization is immutable and will be part of all URLs.

   apiVersion: v1
   kind: Namespace
   metadata:
     name: my-organization
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: oidc-config
     namespace: my-organization
   type: Opaque
   data:
     clientID: ...
     clientSecret: ...
   ---
   apiVersion: greenhouse.sap/v1alpha1
   kind: Organization
   metadata:
     name: my-organization
   spec:
     authentication:
       oidc:
         clientIDReference:
           key: clientID
           name: oidc-config
         clientSecretReference:
           key: clientSecret
           name: oidc-config
         issuer: https://...
       scim:
         baseURL: URL to the SCIM server.
         basicAuthUser:
           secret:
             name: Name of the secret in the same namespace.
             key: Key in the secret holding the user value.
         basicAuthPw:
           secret:
             name: Name of the secret in the same namespace.
             key: Key in the secret holding the password value.
     description: My new organization
     displayName: Short name of the organization
     mappedOrgAdminIdPGroup: Name of the group in the IDP that should be mapped to the organization admin role.
   

Setting up Team members synchronization with Greenhouse

Team members synchronization with Greenhouse requires access to SCIM API.

For the members to be reflected in a Team’s status, the created Organization needs to be configured with URL and credentials of the SCIM API. SCIM API is used to get members for teams in the organization based on the IDP groups set for teams.

IDP group for the organization admin team must be set to the mappedOrgAdminIdPGroup field in the Organization configuration. It is required for the synchronization to work. IDP groups for remaining teams in the organization should be set in their respective configurations - Team’s mappedIdpGroup field.

Next Steps

• Organization reference

2.2 - Cluster management

Greenhouse enables organizations to register their Kubernetes clusters within the platform, providing a centralized interface for managing and monitoring these clusters.
Once registered, users can perform tasks related to cluster management, such as deploying applications, scaling resources, and configuring access control, all within the Greenhouse platform.

This section provides guides for the management of Kubernetes clusters within Greenhouse.

2.2.1 - Cluster onboarding

Content Overview

• Preparation
• Onboard
• After onboarding
• Troubleshooting

This guides describes how to onboard an existing Kubernetes cluster to your Greenhouse organization.
If you don’t have an organization yet please reach out to the Greenhouse administrators.

While all members of an organization can see existing clusters, their management requires org-admin or cluster-admin privileges. See RBAC within the Organization namespace for details.

:information_source: The UI is currently in development. For now this guide describes the onboarding workflow via command line.

Preparation

Download the latest greenhousectl binary from the Greenhouse releases.

Onboarding a Cluster to Greenhouse will require you to authenticate to two different Kubernetes clusters via respective kubeconfig files:

• greenhouse: The cluster your Greenhouse installation is running on. You need organization-admin or cluster-admin privileges.
• bootstrap: The cluster you want to onboard. You need system:masters privileges.

For consistency, we will refer to those two clusters by their names from now on.

You need to have the kubeconfig files for both the greenhouse and the bootstrap cluster at hand. The kubeconfig file for the greenhouse cluster can be downloaded via the Greenhouse dashboard:

Organization > Clusters > Access Greenhouse cluster.

Onboard

For accessing the bootstrap cluster, the greenhousectl will expect your default Kubernetes kubeconfig file and context to be set to bootstrap. This can be achieved by passing the --kubeconfig flag or by setting the KUBECONFIG env var.

The location of the kubeconfig file to the greenhouse cluster is passed via the --greenhouse-kubeconfig flag.

greenhousectl cluster bootstrap --kubeconfig=<path/to/bootstrap-kubeconfig-file> --greenhouse-kubeconfig <path/to/greenhouse-kubeconfig-file> --org <greenhouse-organization-name> --cluster-name <name>

Since Greenhouse generates URLs which contain the cluster name, we highly recommend to choose a short cluster name. In particular for Gardener Clusters setting a short name is mandatory, because Gardener has very long cluster names, e.g. garden-greenhouse--monitoring-external.

A typical output when you run the command looks like

2024-02-01T09:34:55.522+0100	INFO	setup	Loaded kubeconfig	{"context": "default", "host": "https://api.greenhouse.tld"}
2024-02-01T09:34:55.523+0100	INFO	setup	Loaded client kubeconfig	{"host": "https://api.remote.tld"}
2024-02-01T09:34:56.579+0100	INFO	setup	Bootstraping cluster	{"clusterName": "monitoring", "orgName": "ccloud"}
2024-02-01T09:34:56.639+0100	INFO	setup	created namespace	{"name": "ccloud"}
2024-02-01T09:34:56.696+0100	INFO	setup	created serviceAccount	{"name": "greenhouse"}
2024-02-01T09:34:56.810+0100	INFO	setup	created clusterRoleBinding	{"name": "greenhouse"}
2024-02-01T09:34:57.189+0100	INFO	setup	created clusterSecret	{"name": "monitoring"}
2024-02-01T09:34:58.309+0100	INFO	setup	Bootstraping cluster finished	{"clusterName": "monitoring", "orgName": "ccloud"}

After onboarding

1. List all clusters in your Greenhouse organization:

   kubectl --namespace=<greenhouse-organization-name> get clusters

2. Show the details of a cluster:

   kubectl --namespace=<greenhouse-organization-name> get cluster <name> -o yaml

Example:

apiVersion: greenhouse.sap/v1alpha1
kind: Cluster
metadata:
  creationTimestamp: "2024-02-07T10:23:23Z"
  finalizers:
    - greenhouse.sap/cleanup
  generation: 1
  name: monitoring
  namespace: ccloud
  resourceVersion: "282792586"
  uid: 0db6e464-ec36-459e-8a05-4ad668b57f42
spec:
  accessMode: direct
  maxTokenValidity: 72h
status:
  bearerTokenExpirationTimestamp: "2024-02-09T06:28:57Z"
  kubernetesVersion: v1.27.8
  statusConditions:
    conditions:
      - lastTransitionTime: "2024-02-09T06:28:57Z"
        status: "True"
        type: Ready

When the status.kubernetesVersion field shows the correct version of the Kubernetes cluster, the cluster was successfully bootstrapped in Greenhouse. Then status.conditions will contain a Condition with type=Ready and status="true""

In the remote cluster, a new namespace is created and contains some resources managed by Greenhouse. The namespace has the same name as your organization in Greenhouse.

Troubleshooting

If bootstrapping fails, you can inspect the Cluster.statusConditions for more details. The type=KubeConfigValid condition may contain hints in the message field. Additional insights can be found in the type=PermissionsVerified and type=ManagedResourcesDeployed conditions, which indicate whether ServiceAccount has valid permissions and whether required resources were successfully deployed. These conditions are also visible in the UI on the Cluster details view. Reruning the onboarding command with an updated kubeConfig file will fix these issues.

2.2.2 - Remote Cluster Connectivity with OIDC

Content Overview

• OIDC Overview
• Preparation
• Onboard
• Troubleshooting

This guide describes how to onboard an existing Kubernetes cluster to your Greenhouse Organization with OIDC configuration. If you don’t have a Greenhouse Organization yet, please reach out to the Greenhouse administrators.

While all members of an Organization can see existing Clusters, their management requires org-admin or cluster-admin privileges. See RBAC with the Organization namespace for details.

:information_source: The UI is currently in development. For now this guide describes the onboarding workflow via command line.

OIDC Overview

Starting from Kubernetes v1.21, the feature Service Account Issuer Discovery transforms the Kubernetes API server into an OIDC identity provider. This setup facilitates the issuance of tokens, via service accounts to pods, which are recognizable by external services outside the Kubernetes cluster, thereby establishing an authentication pathway between the pod within the cluster and external services including those on Azure, AWS, etc.

Starting from Kubernetes v1.30, Structured Authentication Configuration moved to beta and the feature gate is enabled by default. This feature allows configuring multiple OIDC issuers and passing them as a configuration file to the Kubernetes API server.

More information on Structured Authentication Configuration can be found at https://kubernetes.io/docs/reference/access-authn-authz/authentication/#using-authentication-configuration

With the combination of Service Account Issuer Discovery and Structured Authentication Configuration, Cluster to Cluster trust can be established.

A remote cluster can add the Greenhouse cluster’s Service Account Issuer as an OIDC issuer in its Structured Authentication Configuration. This allows the Greenhouse cluster to authenticate against said remote cluster, using an in-cluster service account token.

The OIDC Remote Cluster Connectivity is illustrated below -

Preparation

The Greenhouse cluster should expose the /.well-known/openid-configuration over an unauthenticated endpoint to allow remote clusters to fetch the OIDC configuration.

Some cloud providers or managed Kubernetes services might not expose the Service Account Issuer Discovery as an unauthenticated endpoint. In such cases, you can serve this configuration from a different endpoint and set this as the discoveryURL in structured authentication configuration.

Check out https://kubernetes.io/docs/reference/access-authn-authz/authentication/#using-authentication-configuration for more information.

Configure the OIDC issuer in the Structured Authentication Configuration of the remote cluster.

Example Structured Authentication Configuration file

apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
  - issuer:
      url: https://<greenhouse-service-account-issuer>
      audiences:
        - greenhouse # audience should be greenhouse
    claimMappings:
      username:
        claim: 'sub' # claim to be used as username
        prefix: 'greenhouse:' # prefix to be added to the username to prevent impersonation (can be any string of your choice)
# additional trusted issuers
#     - issuer:

Add RBAC rules to the remote cluster, authorizing Greenhouse to manage Kubernetes resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: greenhouse-<cluster-name>-oidc-access
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: greenhouse:system:serviceaccount:<your-organization-namespace>:<cluster-name>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin

The subject kind User name must follow the pattern of <prefix>:system:serviceaccount:<your-organization-namespace>:<cluster-name>.

<prefix> is the prefix used in the Structured Authentication Configuration file for the username claim mapping.

For convenience purposes, the `prefix` is set to `greenhouse:` in the example `Structured Authentication Configuration`
but it can be any string identifier of your choice.

If you use '-' in prefix, for example, `identifier-` then the subject name should be `identifier-system:serviceaccount:<your-organization-namespace>:<cluster-name>`.

Onboard

You can now onboard the remote Cluster to your Greenhouse Organization by applying a Secret in the following format:

apiVersion: v1
kind: Secret
metadata:
  annotations:
    "oidc.greenhouse.sap/api-server-url": "https://<remote-cluster-api-server-url>"
  name: <cluster-name> # ensure the name provided here is the same as the <cluster-name> in the ClusterRoleBinding
  namespace: <organization-namespace>
data:
  ca.crt: <double-encoded-ca.crt> # remote cluster CA certificate base64 encoded
type: greenhouse.sap/oidc # secret type

Mandatory fields:

• the annotation oidc.greenhouse.sap/api-server-url must have a valid URL pointing to the remote cluster’s API server
• the ca.crt field must contain the remote cluster’s CA certificate
• the type of the Secret must be greenhouse.sap/oidc
• the name of the secret must equal the <cluster-name> used in the ClusterRoleBinding Subject

ca.crt is the certificate-authority-data from the kubeconfig file of the remote cluster.

The certificate-authority-data can be extracted from the ConfigMap kube-root-ca.crt. This ConfigMap is present in every Namespace.

If the certificate is extracted from kube-root-ca.crt then it should be base64 encoded twice before adding it to the secret.

example:

$ kubectl get configmap kube-root-ca.crt -n kube-system -o jsonpath='{.data.ca\.crt}' | base64 | base64

If the certificate is extracted from the KubeConfig file then the certificate is already base64 encoded, so the encoding is needed only once.

Apply the Secret to the Organization Namespace to onboard the remote cluster.

$ kubectl apply -f <oidc-secret-file>.yaml

Troubleshooting

If the bootstrapping failed, you can find details about why it failed in the Cluster.status.statusConditions. More precisely, there will be conditions of type=KubeConfigValid, type=PermissionsVerified, type=ManagedResourcesDeployed, and type=Ready, each of which may provide helpful context in the message field. These conditions are also displayed in the UI on the Cluster details view.

If there is any error message regarding RBAC then check the ClusterRoleBinding and ensure the subject name is correct

If there is any authentication error then you might see a message similar to the server has asked for the client to provide credentials, in such cases verify the Structured Authentication Configuration and ensure the issuer and audiences are correct.

The API Server logs in the remote cluster will provide more information on the authentication errors.

2.2.3 - Cluster offboarding

Content Overview

• Pre-requisites
• Schedule Deletion
• Impact
• Immediate Deletion
• Troubleshooting

This guides describes how to off-board an existing Kubernetes cluster in your Greenhouse organization.

While all members of an organization can see existing clusters, their management requires org-admin or cluster-admin privileges. See RBAC with the Organization namespace for details.

:information_source: The UI is currently in development. For now this guide describes the onboarding workflow via command line.

Pre-requisites

Offboarding a Cluster in Greenhouse requires authenticating to the greenhouse cluster via kubeconfig file:

• greenhouse: The cluster where Greenhouse installation is running on.
• organization-admin or cluster-admin privileges is needed for deleting a Cluster resource.

Schedule Deletion

By default Cluster resource deletion is blocked by ValidatingWebhookConfiguration in Greenhouse. This is done to prevent accidental deletion of cluster resources.

List the clusters in your Greenhouse organization:

kubectl --namespace=<greenhouse-organization-name> get clusters

A typical output when you run the command looks like

NAME          AGE    ACCESSMODE   READY
mycluster-1   15d    direct       True
mycluster-2   35d    direct       True
mycluster-3   108d   direct       True

Delete a Cluster resource by annotating it with greenhouse.sap/delete-cluster: "true".

Example:

kubectl annotate cluster mycluster-1 greenhouse.sap/delete-cluster=true --namespace=my-org

Once the Cluster resource is annotated, the Cluster will be scheduled for deletion in 48 hours (UTC time). This is reflected in the Cluster resource annotations and in the status conditions.

View the deletion schedule by inspecting the Cluster resource:

kubectl get cluster mycluster-1 --namespace=my-org -o yaml

A typical output when you run the command looks like

apiVersion: greenhouse.sap/v1alpha1
kind: Cluster
metadata:
  annotations:
    greenhouse.sap/delete-cluster: "true"
    greenhouse.sap/deletion-schedule: "2025-01-17 11:16:40"
  finalizers:
  - greenhouse.sap/cleanup
  name: mycluster-1
  namespace: my-org
spec:
  accessMode: direct
  kubeConfig:
    maxTokenValidity: 72
status:
  ...
  statusConditions:
    conditions:
    ...
    - lastTransitionTime: "2025-01-15T11:16:40Z"
      message: deletion scheduled at 2025-01-17 11:16:40
      reason: ScheduledDeletion
      status: "False"
      type: Delete

In order to cancel the deletion, you can remove the greenhouse.sap/delete-cluster annotation:

kubectl annotate cluster mycluster-1 greenhouse.sap/delete-cluster- --namespace=my-org

the - at the end of the annotation name is used to remove the annotation.

Impact

When a Cluster resource is scheduled for deletion, all Plugin resources associated with the Cluster resource will skip the reconciliation process.

When the deletion schedule is reached, the Cluster resource will be deleted and all associated resources Plugin resources will be deleted as well.

Immediate Deletion

In order to delete a Cluster resource immediately -

1. annotate the Cluster resource with greenhouse.sap/delete-cluster. (see Schedule Deletion)
2. update the greenhouse.sap/deletion-schedule annotation to the current date and time.

You can also annotate the Cluster resource with greenhouse.sap/delete-cluster and greenhouse.sap/deletion-schedule at the same time and set the current date and time for deletion.

The time and date should be in YYYY-MM-DD HH:MM:SS format or golang’s time.DateTime format. The time should be in UTC timezone.

Troubleshooting

If the cluster deletion has failed, you can troubleshoot the issue by inspecting -

1. Cluster resource status conditions, specifically the KubeConfigValid condition.
2. status conditions of the Plugin resources associated with the Cluster resource. There will be a clear indication of the issue in HelmReconcileFailed condition.

2.3 - Plugin management

Plugins extends the capabilities of the Greenhouse cloud operations platform, adding specific features or functionalities to tailor and enhance the platform for specific organizational needs.
These plugins are integral to Greenhouse’ extensibility, allowing users to customize their cloud operations environment and address unique requirements while operating within the Greenhouse ecosystem.

This section provides guides for the management of plugins for Kubernetes clusters within Greenhouse.

2.3.1 - Testing a Plugin

Overview

Plugin Testing Requirements

All Plugins contributed to Plugin-Extensions repository should include comprehensive Helm Chart Tests using the bats/bats-detik testing framework. This ensures our Plugins are robust, deployable, and catch potential issues early in the development cycle.

What is bats/bats-detik?

The bats/bats-detik framework simplifies end-to-end (e2e) Testing in Kubernetes. It combines the Bash Automated Testing System (bats) with Kubernetes-specific assertions (detik). This allows you to write test cases using natural language-like syntax, making your tests easier to read and maintain.

Implementing Tests

1. Create a /tests folder inside your Plugin’s Helm Chart templates folder to store your test resources.

2. ConfigMap definition:

   • Create a test-<plugin-name>-config.yaml file in the templates/tests directory to define a ConfigMap that will hold your test script.
   • This ConfigMap contains the test script run.sh that will be executed by the test Pod to run your tests.

{{- if .Values.testFramework.enabled -}}
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-test
  namespace: {{ .Release.Namespace }}
  labels:
    type: integration-test
data:
  run.sh: |-

    #!/usr/bin/env bats

    load "/usr/lib/bats/bats-detik/utils"
    load "/usr/lib/bats/bats-detik/detik"

    DETIK_CLIENT_NAME="kubectl"

    @test "Verify successful deployment and running status of the {{ .Release.Name }}-operator pod" {
        verify "there is 1 deployment named '{{ .Release.Name }}-operator'"
        verify "there is 1 service named '{{ .Release.Name }}-operator'"
        try "at most 2 times every 5s to get pods named '{{ .Release.Name }}-operator' and verify that '.status.phase' is 'running'"
    }

    @test "Verify successful creation and bound status of {{ .Release.Name }} persistent volume claims" {
        try "at most 3 times every 5s to get persistentvolumeclaims named '{{ .Release.Name }}.*' and verify that '.status.phase' is 'Bound'"
    }

    @test "Verify successful creation and available replicas of {{ .Release.Name }} Prometheus resource" {
        try "at most 3 times every 5s to get prometheuses named '{{ .Release.Name }}' and verify that '.status.availableReplicas' is more than '0'"
    }

    @test "Verify creation of required custom resource definitions (CRDs) for {{ .Release.Name }}" {
        verify "there is 1 customresourcedefinition named 'prometheuses'"
        verify "there is 1 customresourcedefinition named 'podmonitors'"
    }
{{- end -}}

Note: You can use this guide for reference when writing your test assertions.

3. Test Pod Definition:

   • Create a test-<plugin-name>.yaml file in the templates/tests directory to define a Pod that will run your tests.
   • This test Pod will mount the ConfigMap created in the previous step and will execute the test script run.sh.

{{- if .Values.testFramework.enabled -}}
apiVersion: v1
kind: Pod
metadata:
  name: {{ .Release.Name }}-test
  namespace: {{ .Release.Namespace }}
  labels:
    type: integration-test
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded"
spec:
  serviceAccountName: {{ .Release.Name }}-test
  containers:
    - name: bats-test
      image: "{{ .Values.testFramework.image.registry}}/{{ .Values.testFramework.image.repository}}:{{ .Values.testFramework.image.tag }}"
      imagePullPolicy: {{ .Values.testFramework.image.pullPolicy }}
      command: ["bats", "-t", "/tests/run.sh"]
      volumeMounts:
        - name: tests
          mountPath: /tests
          readOnly: true   volumes:
    - name: tests
      configMap:
        name: {{ .Release.Name }}-test
  restartPolicy: Never
{{- end -}}

4. RBAC Permissions:

• Create the necessary RBAC resources in the templates/tests folder with a dedicated ServiceAccount and role authorisations so that the test Pod can cover test the cases.
• You can use test-permissions.yaml from the kube-monitoring as a reference to configure RBAC permissions for your test Pod.

5. Configure the Test Framework in Plugin’s values.yaml:
   • Add the following configuration to your Plugin’s values.yaml file:

testFramework:
  enabled: true
  image:
    registry: ghcr.io
    repository: cloudoperators/greenhouse-extensions-integration-test
    tag: main
  imagePullPolicy: IfNotPresent

6. Running the Tests:

Important: Once you have completed all the steps above, you are ready to run the tests. However, before running the tests, ensure that you perform a fresh Helm installation or upgrade of your Plugin’s Helm release against your test Kubernetes cluster (for example, Minikube or Kind) by executing the following command:

# For a new installation
helm install <Release name> <chart-path>

# For an upgrade
helm upgrade <Release name> <chart-path>

• After the Helm installation or upgrade is successful, run the tests against the same test Kubernetes Cluster by executing the following command.

helm test <Release name>

Plugin Testing with dependencies during Pull Requests

Overview

Some Plugins require other Plugins to be installed in the Cluster for their tests to run successfully. To support this, each Plugin can declare required dependencies using a test-dependencies.yaml file.

[!NOTE]
The test-dependencies.yaml file is required if other Plugins need to be installed in the Kind Cluster created by the GitHub Actions workflow before running tests during a Pull Request for the Plugin.

How It Works

• Each Plugin can optionally include a test-dependencies.yaml file in the Plugin’s root directory (e.g., Thanos/test-dependencies.yaml).
• This file defines both the dependencies (other Plugins) that should be installed before testing begins and custom values for these dependencies.

Example test-dependencies.yaml

dependencies:
  - kube-monitoring
values:
  kubeMonitoring:
    prometheus:
      enabled: true
      serviceMonitor:
        enabled: false
      prometheusSpec:
        thanos:
          objectStorageConfig:
            secret:
              type: FILESYSTEM
              config:
                directory: "/test"
              prefix: ""

In this example, the Plugin:

• Declares kube-monitoring as a dependency that must be installed first
• Provides custom values for this dependent Plugin, specifically configuring Prometheus settings

Dependency Structure

The test-dependencies.yaml file supports:

• dependencies: A list of Plugin names that should be installed before testing the current Plugin.
• values: Custom configuration values to be applied when installing dependencies

Automation during Pull Requests

The GitHub Actions workflow automatically:

1. Detects Plugins that are changed in the Pull Request.
2. Parses the test-dependencies.yaml for each changed Plugin if present.
3. Installs listed dependencies in order
4. Proceeds with Helm Chart linting and testing

Testing Values Configuration

Parent Plugin Configuration

• A Plugin may optionally provide a <plugin-name>/ci/test-values.yaml file
• The GitHub Actions workflow will use this values file for testing the Plugin if it exists
• This allows you to customize values specifically for CI testing, without modifying the default values.yaml

Dependent Plugin Configuration

• Values for dependent Plugins should be specified in the values section of your Plugin’s test-dependencies.yaml file.
• This allows you to customize the configuration of dependent Plugins when they are installed for testing.
• The values specified in the test-dependencies.yaml file will override the default values of the dependent Plugins.

Example File Structure:

alert/
├── charts/
├── ci/
│   └── test-values.yaml
└── test-dependencies.yaml

Contribution Checklist

Before submitting a Pull Request:

• Ensure your Plugin’s Helm Chart includes a /tests directory.
• Verify the presence of test-<plugin-name>.yaml, test-<plugin-name>-config.yaml, and test-permissions.yaml files.
• Test your Plugin thoroughly using helm test <release-name> and confirm that all tests pass against a test Kubernetes Cluster.
• Include a brief description of the tests in your Pull Request.
• Make sure that your Plugin’s Chart Directory and the Plugin’s Upstream Chart Repository are added to this Greenhouse-Extensions Helm Test Config File. This will ensure that your Plugin’s tests are automatically run in the GitHub Actions workflow when you submit a Pull Request for this Plugin.
• Note that the dependencies of your Plugin’s Helm Chart might also have their own tests. If so, ensure that the tests of the dependencies are also passing.
• If your Plugin relies on other Plugins for testing, please follow the Plugin Testing with Dependencies section for declaring those dependencies.

Important Notes

• Test Coverage: Aim for comprehensive test coverage to ensure your Plugin’s reliability.

Next Steps

• Plugin reference
• PluginPreset reference
• PluginDefinition reference

2.3.2 - Plugin deployment

Before you begin

This guides describes how to configure and deploy a Greenhouse plugin.

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: kube-monitoring-martin
  namespace: <organization namespace> # same namespace in remote cluster for resources
spec:
  clusterName: <name of the remote cluster >
  disabled: false
  displayName: <any human readable name>
  pluginDefinition: <pluginDefinition name>
  releaseNamespace: <namespace> # namespace in remote cluster where the plugin is deployed
  releaseName: <helm release name> # name of the helm release that will be created
  optionValues:
    - name: <from the plugin options>
      value: <from the plugin options>
    - ...

Exposed services and ingresses

Plugins deploying Helm Charts into remote clusters support exposing their services and ingresses in two ways:

Service exposure via service-proxy

Services can be exposed through Greenhouse’s central service-proxy by adding this annotation:

annotations:
  greenhouse.sap/expose: "true"

For services with multiple ports, you can specify which port to expose:

annotations:
  greenhouse.sap/expose: "true"
  greenhouse.sap/exposed-named-port: "https"  # optional, defaults to first port

Direct ingress exposure

Ingresses can be exposed directly using their external URLs:

annotations:
  greenhouse.sap/expose: "true"
  greenhouse.sap/exposed-host: "api.example.com"  # optional, for multi-host ingresses

Both types of exposures appear in the Plugin’s status.exposedServices with different types: service or ingress.

Deploying a Plugin

Create the Plugin resource via the command:

kubectl --namespace=<organization name> create -f plugin.yaml

After deployment

1. Check with kubectl --namespace=<organization name> get plugin has been properly created. When all components of the plugin are successfully created, the plugin should show the state configured.

2. Check in the remote cluster that all plugin resources are created in the organization namespace.

URLs for exposed services and ingresses

After deploying the plugin to a remote cluster, the ExposedServices section in Plugin’s status provides an overview of the exposed resources. It maps URLs to both services and ingresses found in the manifest.

Service-proxy URLs (for services)

• Services exposed through service-proxy use the pattern: https://$cluster--$hash.$organization.$basedomain
• The $hash is computed from service--$namespace

Direct ingress URLs (for ingresses)

• Ingresses are exposed using their actual hostnames: https://api.example.com or http://internal.service.com
• Protocol (http/https) is automatically detected from the ingress TLS configuration
• The host is taken from greenhouse.sap/exposed-host annotation or defaults to the first host rule

Both types are listed together in status.exposedServices with their respective types for easy identification.

Next Steps

• Plugin reference
• PluginPreset reference
• PluginDefinition reference

2.3.3 - Managing Plugins for multiple clusters

Managing Plugins for multiple clusters

This guide describes how to configure and deploy a Greenhouse Plugin with the same configuration into multiple clusters.

The PluginPreset resource is used to create and deploy Plugins with a the identical configuration into multiple clusters. The list of clusters the Plugins will be deployed to is determind by a LabelSelector.

As a result, whenever a cluster, that matches the ClusterSelector is onboarded or offboarded, the Controller for the PluginPresets will take care of the Plugin Lifecycle. This means creating or deleting the Plugin for the respective cluster.

The same validation applies to the PluginPreset as to the Plugin. This includes immutable PluginDefinition and ReleaseNamespace fields, as well as the validation of the OptionValues against the PluginDefinition.

In case the PluginPreset is updated all of the Plugin instances that are managed by the PluginPreset will be updated as well. Each Plugin instance that is created from a PluginPreset has a label greenhouse.sap/pluginpreset: <PluginPreset name>. Also the name of the Plugin follows the scheme <PluginPreset name>-<cluster name>.

Changes that are done directly on a Plugin which was created from a PluginPreset will be overwritten immediately by the PluginPreset Controller. All changes must be performed on the PluginPreset itself. If a Plugin already existed with the same name as the PluginPreset would create, this Plugin will be ignored in following reconciliations.

A PluginPreset with the annotation greenhouse.sap/prevent-deletion may not be deleted. This is to prevent the accidental deletion of a PluginPreset including the managed Plugins and their deployed Helm releases. Only after removing the annotation it is possible to delete a PluginPreset.

Example PluginPreset

apiVersion: greenhouse.sap/v1alpha1
kind: PluginPreset
metadata:
  name: kube-monitoring-preset
  namespace: <organization namespace>
spec:
  plugin: # this embeds the PluginSpec
    displayName: <any human readable name>
    pluginDefinition: <PluginDefinition name> # k get plugindefinition
    releaseNamespace: <namespace> # namespace where the plugin is deployed to on the remote cluster. Will be created if not exists
    optionValues:
      - name: <from the PluginDefinition options>
        value: <from the PluginDefinition options>
      - ..
  clusterSelector: # LabelSelector for the clusters the Plugin should be deployed to
    matchLabels:
      <label-key>: <label-value>
  clusterOptionOverrides: # allows you to override specific options in a given cluster
    - clusterName: <cluster name where we want to override values>
      overrides:
        - name: <option name to override>
          value: <new value>
        - ..
    - ..

See Writing a PluginPreset Spec for more details on how to configure a PluginPreset.

Next Steps

• Plugin reference
• PluginPreset reference
• PluginDefinition reference

2.3.4 - Plugin Catalog

Before you begin

This guides describes how to explore the catalog of Greenhouse PluginDefinitions.

While all members of an organization can see the Plugin catalog, enabling, disabling and configuration PluginDefinitions for an organization requires organization admin privileges.

Exploring the PluginDefinition catalog

The PluginDefinition resource describes the backend and frontend components as well as mandatory configuration options of a Greenhouse extension.
While the PluginDefinition catalog is managed by the Greenhouse administrators and the respective domain experts, administrators of an organization can configure and tailor Plugins to their specific requirements.

NOTE: The UI also provides a preliminary catalog of Plugins under Organization> Plugin> Add Plugin.

1. Run the following command to see all available PluginDefinitions.

   $ kubectl get plugindefinition
   
   NAME                      VERSION   DESCRIPTION                                                                                                  AGE
   cert-manager              1.1.0     Automated certificate management in Kubernetes                                                               182d
   digicert-issuer           1.2.0     Extensions to the cert-manager for DigiCert support                                                          182d
   disco                     1.0.0     Automated DNS management using the Designate Ingress CNAME operator (DISCO)                                  179d
   doop                      1.0.0     Holistic overview on Gatekeeper policies and violations                                                      177d
   external-dns              1.0.0     The kubernetes-sigs/external-dns plugin.                                                                     186d
   heureka                   1.0.0     Plugin for Heureka, the patch management system.                                                             177d
   ingress-nginx             1.1.0     Ingress NGINX controller                                                                                     187d
   kube-monitoring           1.0.1     Kubernetes native deployment and management of Prometheus, Alertmanager and related monitoring components.   51d
   prometheus-alertmanager   1.0.0     Prometheus alertmanager                                                                                      60d
   supernova                 1.0.0     Supernova, the holistic alert management UI                                                                  187d
   teams2slack               1.1.0     Manage Slack handles and channels based on Greenhouse teams and their members                                115d
   

Next Steps

• PluginDefinition reference

2.4 - Team management

A team is a group of users with shared responsibilities for managing and operating cloud resources within a Greenhouse organization.
These teams enable efficient collaboration, access control, and task assignment, allowing organizations to effectively organize their users and streamline cloud operations within the Greenhouse platform.

This section provides guides for the management of teams within an organization.

2.4.1 - Role-based access control on remote clusters

Greenhouse Team RBAC user guide

Role-Based Access Control (RBAC) in Greenhouse allows Organization administrators to manage the access of Teams on Clusters. TeamRole and TeamRoleBindings are used to manage the RBAC on remote Clusters. These two Custom Resource Definitions allow for fine-grained control over the permissions of each Team within each Cluster and Namespace.

Contents

• Before you begin
• Overview
• Defining TeamRoles
  • Example
• Seeded default TeamRoles
• Defining TeamRoleBindings
  • Assigning TeamRoles to Teams on a single Cluster
  • Assigning TeamRoles to Teams on multiple Clusters
  • Aggregating TeamRoles

Before you begin

This guide is intended for users who want to manage Role-Based Access Control (RBAC) for Teams on remote clusters managed by Greenhouse. It assumes you have a basic understanding of Kubernetes RBAC concepts and the Greenhouse platform.

🔑 Permissions

1. Create/Update TeamRoles and TeamRoleBindings in the Organization namespace.
2. View Teams and Clusters in the Organization namespace

By default the necessary authorizations are provieded via the role:<organization>:admin RoleBinding that is granted to members of the Organizations Admin Team. You can check the permissions inside the Organization namespace by running the following command:

kubectl auth can-i --list --namespace=<organization-namespace>

💻 Software

1. kubectl: The Kubernetes command-line tool which allows you to manage Kubernetes cluster resources.

Overview

• TeamRole: Defines a set of permissions that can be assigned to teams and individual users.
• TeamRoleBinding: Assigns a TeamRole to a specific Team and/or a list of users fo Clusters and (optionally) Namespaces.

Defining TeamRoles

TeamRoles define the actions a Team can perform on a Kubernetes cluster. For each Organziation a set of TeamRoles is seeded. The syntax of the TeamRole’s .spec is following the Kubernetes RBAC API.

Example

This TeamRole named pod-read grants read access to Pods.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRole
metadata:
  name: pod-read
spec:
  rules:
    - apiGroups:
        - ""
      resources:
        - "pods"
      verbs:
        - "get"
        - "list"

Seeded default TeamRoles

Greenhouse provides a set of default TeamRoles that are seeded to all clusters:

Defining TeamRoleBindings

TeamRoleBindings link a Team, a TeamRole, one or more Clusters and optionally one or more Namespaces together. Once the TeamRoleBinding is created, the Team will have the permissions defined in the TeamRole within the specified Clusters and Namespaces. This allows for fine-grained control over the permissions of each Team within each Cluster. The TeamRoleBinding Controller within Greenhouse deploys RBAC resources to the targeted Clusters. The referenced TeamRole is created as a rbacv1.ClusterRole. In case the TeamRoleBinding references a Namespace, it is considered to be namespace-scoped. Hence, the controller will create a rbacv1.RoleBinding which links the Team with the rbacv1.ClusterRole. In case no Namespace is referenced, the Controller will create a cluster-scoped rbacv1.ClusterRoleBinding instead.

Assigning TeamRoles to Teams on a single Cluster

Roles are assigned to Teams through the TeamRoleBinding configuration, which links Teams to their respective roles within specific clusters.

This TeamRoleBinding assigns the pod-read TeamRole to the Team named my-team in the Cluster named my-cluster.

Example: team-rolebindings.yaml

apiVersion: greenhouse.sap/v1alpha2
kind: TeamRoleBinding
metadata:
  name: my-team-read-access
spec:
  teamRef: my-team
  roleRef: pod-read
  clusterSelector:
    clusterName: my-cluster

Assigning TeamRoles to Teams on multiple Clusters

A LabelSelector can be used to assign a TeamRoleBinding to multiple Clusters.

This TeamRoleBinding assigns the pod-read TeamRole to the Team named my-team in all Clusters that have the label environment: production set.

apiVersion: greenhouse.sap/v1alpha2
kind: TeamRoleBinding
metadata:
  name: production-cluster-admins
spec:
  teamRef: my-team
  roleRef: pod-read
  clusterSelector:
    labelSelector:
      matchLabels:
        environment: production

Aggregating TeamRoles

It is possible with Kubernetes RBAC to aggregate rbacv1.ClusterRoles. This is also supported for TeamRoles. All label specified on a TeamRole’s .spec.Labels will be set on the rbacv1.ClusterRole created on the target cluster. This makes it possible to aggregate multiple rbacv1.ClusterRole resources by using a rbacv1.AggregationRule. This can be specified on a TeamRole by setting .spec.aggregationRule.

More details on the concept of Aggregated ClusterRoles can be found in the Kubernetes documentation: Aggregated ClusterRoles

[!NOTE] A TeamRole is only created on a cluster if it is referenced by a TeamRoleBinding. If a TeamRole is not referenced by a TeamRoleBinding it will not be created on any target cluster. A TeamRoleBinding referencing a TeamRole with an aggregationRule will only provide the correct access, if there is at least one TeamRoleBinding referencing a TeamRole with the corresponding label deployed to the same cluster.

The following example shows how an AggregationRule can be used with TeamRoles and TeamRoleBindings.

This TeamRole specifies .spec.Labels. The labels will be applied to the resulting ClusterRole on the target cluster.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRole
metadata:
  name: pod-read
spec:
  labels:
    aggregate: "true"
  rules:
    - apiGroups:
        - ""
      resources:
        - "pods"
      verbs:
        - "get"
        - "list"

This TeamRoleBinding assigns the pod-read TeamRole to the Team named my-team in all Clusters with the label environment: production.

apiVersion: greenhouse.sap/v1alpha2
kind: TeamRoleBinding
metadata:
  name: production-pod-read
spec:
  teamRef: my-team
  roleRef: pod-read
  clusterSelector:
    labelSelector:
      matchLabels:
        environment: production

Access granted by TeamRoleBinding can also be restricted to specified Namespaces. This can be achieved by specifying the .spec.namespaces field in the TeamRoleBinding.

Setting dedicated Namespaces results in RoleBindings being created in the specified Namespaces. The Team will then only have access to the Pods in the specified Namespaces. The TeamRoleBinding controller will create a non-existing Namespace, only if the field .spec.createNamespaces is set to true on the TeamRoleBinding. If this field is not set, the TeamRoleBinding controller will not create the Namespace or the RBAC resources. Deleting a TeamRoleBinding will only result in the deletion of the RBAC resources but will never result in the deletion of the Namespace.

apiVersion: greenhouse.sap/v1alpha2
kind: TeamRoleBinding
metadata:
  name: production-pod-read
spec:
  teamRef: my-team
  roleRef: pod-read
  clusterSelector:
    labelSelector:
      matchLabels:
        environment: production
  namespaces:
    - kube-system
  # createNamespaces: true # optional, if set the TeamRoleBinding will create the namespaces if they do not exist

This TeamRole has a .spec.aggregationRule set. This aggregationRule will be added to the ClusterRole created on the target clusters. With the aggregationRule set it will aggregate the ClusterRoles created by the TeamRoles with the label aggregate: "true". The Team will have the permissions of both TeamRoles and will be able to get, list, update and patch Pods.

apiVersion: greenhouse.sap/v1alpha1
kind: TeamRole
metadata:
  name: aggregated-role
spec:
  aggregationRule:
    clusterRoleSelectors:
    - matchLabels:
        "aggregate": "true"

apiVersion: greenhouse.sap/v1alpha2
kind: TeamRoleBinding
metadata:
  name: aggregated-rolebinding
spec:
  teamRef: operators
  roleRef: aggregated-role
  clusterSelector:
    labelSelector:
      matchLabels:
        environment: production

Updating TeamRoleBindings

Updating the RoleRef of a ClusterRoleBinding and RoleBinding is not allowed, but requires recreating the TeamRoleBinding resources. See ClusterRoleBinding docs for more information. This is to allow giving out permissions to update the subjects, while avoiding that privileges are changed. Furthermore, changing the TeamRole can change the extent of a binding significantly. Therefore it needs to be recreated.

After the TeamRoleBinding has been created, it can be updated with some limitations. Similarly to RoleBindings, the .spec.roleRef and .spec.teamRef can not be changed. The TeamRoleBinding’s .spec.namespaces can be amended to include more namespaces. However, the scope of the TeamRoleBinding cannot be changed. If a TeamRoleBinding has been created with .spec.namespaces specified, it is namespace-scoped, and cannot be changed to cluster-scoped by removing the .spec.namespaces. The reverse is true for a cluster-scoped TeamRoleBinding, where it is not possible to add .spec.namespaces once created.

2.4.2 - Team creation

Before you begin

This guides describes how to create a team in your Greenhouse organization.

While all members of an organization can see existing teams, their management requires organization admin privileges.

Creating a team

The team resource is used to structure members of your organization and assign fine-grained access and permission levels.

Each Team must be backed by a group in the identity provider (IdP) of the Organization.

• IdP group should be set on the mappedIdPGroup field in Team configuration.
• This, along with SCIM API configured in the Organization, allows for synchronization of Team members with Greenhouse.

NOTE: The UI is currently in development. For now this guides describes the onboarding workflow via command line.

1. To onboard a new cluster provide the kubeconfig file with a static, short-lived token.
   It should look similar to this example:

   cat <<EOF | kubectl apply -f -
      apiVersion: greenhouse.sap/v1alpha1
      kind: Team
      metadata:
      name: <name>
      spec:
         description: My new team
         mappedIdPGroup: <IdP group name>
   EOF
   

3 - Architecture

This section provides an overview of the architecture and design of the Greenhouse platform.

3.1 - Architecture

Greenhouse components

From a high-level perspective, the Greenhouse platform consists of these main components:

1. Greenhouse Central Cluster: The API Server of the central Greenhouse cluster serves as the API endpoint that acts as the primary interface for users to interact with the Greenhouse platform. The API consists of the Greenhouse Custom Resource Definitions (CRDs). Interactions with the API are possible via the Greenhouse dashboard and the kubectl command line tool. The Greenhouse operators run inside this central cluster.
2. Remote Clusters: The are the clusters that are onboarded into Greenhouse, so that the Greenhouse operators can manage the lifecycle of Greenhouse resources in these clusters.

Greenhouse API

The Greenhouse API serves as the backbone of the platform, offering a familiar interface for interacting with Greenhouse. It is deliberately not exposing all the Kubernetes APIs to the users, but uses RBAC to limit the available resources to the ones that are relevant for the use of Greenhouse. For example it is not permitted to run arbitrary workload resources inside of the Greenhouse central clusters.

Greenhouse Clusters

When talking about clusters in the context of Greenhouse, we are referring to two different types of clusters:

1. Central cluster

   This is the cluster where all of the Greenhouse components are running. It is the central point of access for users to interact with the platform. The dashboard, Kubernetes API and the operator are all running in this cluster. It is possible to mutliple organizations on one central cluster. They are isolated by dedicated Kubernetes Namespaces created for each Organization. The Kubernetes API access is limited to the Custom Resource Definitions (CRDs) that are relevant for the Greenhouse platform, such as Organizations, Teams, Clusters, TeamRoleBindings, PluginPresets and more.

   Users of the Greenhouse cloud operations platform, depending on their roles, can perform tasks such as managing Organizations, configuring and using Plugins, monitoring resources, developing custom PluginDefinitions, and conducting audits to ensure compliance with standards. Greenhouse’s flexibility allows users to efficiently manage cloud infrastructure and applications while tailoring their experience to organizational needs. The configuration and metadata is persisted in Kubernetes custom resource definitions (CRDs), acted upon by the Greenhouse operator and managed in the remote cluster.

2. Remote cluster

   When referring to a remote cluster we are talking about the clusters that are onboarded into Greenhouse. Onboarding means that a valid KubeConfig is provided so that the Greenhouse operator can access the cluster and manage the resources in it. Managing and operating Kubernetes clusters can be challenging due to the complexity of tasks related to orchestration, scaling, and ensuring high availability in containerized environments.
   By onboarding their Kubernetes clusters into Greenhouse, users can centralize cluster management, streamlining tasks like resource monitoring and access control. This integration enhances visibility, simplifies operational tasks, and ensures better compliance, enabling users to efficiently manage and optimize their Kubernetes infrastructure. While the central cluster contains the user configuration and metadata, all workloads of user-selected Plugins are run in the remote cluster and managed by Greenhouse. More information about the Cluster resources can be found here.

Organizations & Authentication

The Greenhouse platform is designed to support multiple organizations, each with its own set of users and permissions. Each organization can have multiple teams, and each team can have its own set of roles and permissions. The Greenhouse API provides a way to manage these organizations, teams, and roles. The Organization is a cluster-scoped resource for which a namespace with the same name will be created. When creating an Organization an identity provider (IdP) needs to be specified. All users in this IdP have read access to the resources in the Organization’s namespace. Greenhouse will automatically provision a set of RBAC roles, view RBAC within the Organization namespace for details.

In order to make the Kubernetes API available for multiple organizations, Greenhouse provides an idproxy build on-top of Dex, which allows to handle different identity providers (IdP) when authenticating users against the Kubernetes API.

Remote Cluster management with Plugins & Team RBAC

Greenhouse provides a way to manage access and tooling (e.g. monitoring, security, compliance, etc.) on a Kubernetes cluster through the use of Plugins and TeameRoleBindings. Plugins are used to deploy and manage workloads (e.g. Prometheus, Open Telemetry, Cert-Manager, etc.) in the remote clusters, while TeamRoleBindings are used to manage access to the remote clusters through Kubernetes RBAC. Details about Team RBAC can be found here.

The Plugin API is a key feature of Greenhouse, allowing domain experts to extend the core platform with custom functionality. Greenhouse provides rollout and lifecyle management of Plugins to onboarded Kubernetes Clusters. PluginDefinitions define a Helm chart and/or UI (to be displayed on the Greenhouse dashboard) with pre-configured default values. A PluginPreset can be used to create and configure Plugins for a set of Clusters. Here you can find more information about the Plugin API.

Access in the remote clusters is managed via TeamRole and TeamRoleBinding resources. TeamRoles define the permissions, similar to RBAC Roles & ClusterRoles. The Greenhouse operator will create the necessary Kubernetes RBAC resources in the remote clusters based on the TeamRoleBinding. The TeamRoleBinding combines the TeamRole with a Team and defines the target Clusters and Namespaces.

3.2 - Product design

Introduction

Vision

“Greenhouse is an extendable Platform that enables Organizations to operate their Infrastructure in a compliant, efficient and transparent manner”

We want to build a Platform that is capable to integrate all the tools that are required to operate services in a cloud environment in a compliant, effective and transparent manner. Greenhouse aims to be the single stop for Operators of cloud-native infrastructure. The primary focus of Greenhouse is to provide a unified interface for all operational aspects and tools, providing a simple shared data model describing the support organization.

As every organization is different, using different tools and has different requirements the platform is build in an extendable fashion that allows a distributed development of plugins.

While initially developed for ApeiroRA the platform is explicitly designed to be of generic use and can be consumed by multiple organizations and teams.

Problem Statements

Consolidation of Toolsuite

The operation of cloud infrastructure and applications include a large amount of tasks that are supported by different tools and automations. Due to the high complexity of cloud environments often times a conglomerate of tools is used to cover all operational aspects. Confguration and setup of the operations toolchain is a complex and time-consuming task that often times lacks automation when it comes to on- and off-boarding people and setting up new teams for new projects.

Visibility of Application Specific permission concepts

At SAP, we are managing identities and access centrally. The Converged Cloud is utilizing Cloud Access Manager for this task. While it is true that we manage who has access to which access level is defined in there it starts getting complicated if you want to figure out the actual Permission Levels on individual Applications those Access Levels are mapped to.

Management of organizational Groups of People

You often have groups of people that are fulfilling a organizational purpose:

• Support Groups
• Working Groups
• etc.

We have currently no way to represent and manage those groups.

Harmonization and Standardization of Authorization Concepts

We are missing a tool that supports teams on creating access levels and profiles following a standardized naming scheme that is enforced by design and takes away the burden of coming up with names for access levels and user profiles/roles.

Single Point of Truth for Operations Metadata of an Organization

For automations, it is often critical to retrieve Metadata about an Organizational Unit:

• Who is member of a certain Group of people, that is maybe not reflecting the HR View of a Team?
• Which Tool is giving me access to data x,y,z?
• What action items are due and need to get alerted on?
• Does component x,z,y belong to my organization?
• etc. Currently, we do not have a single point of Truth for this kind of metadata and have to use a vaierity of Tools.

Terms

This section lists down Terms and description to Terms to ensure a common languague when talking in context of Greenhouse.

User Profiles

Every Application has Users / Stakeholders, so has Greenhouse. The User Profiles mentioned here give a abstract overview of considered Users / Stakeholders for the Application and the Goals and Tasks of them in context of the Platform.

Greenhouse admin

Administrator of a Greenhouse installation.

Goals

• Ensure overall function of the Greenhouse plattform

Tasks

• Create Organizations
• Enable PluginDefinitions
• Operates central infrastructure (Kubernetes cluster, operator, etc.)
• Assign initial Organization admins

Organization admin

Administrator of a Greenhouse Organization

Goals

• Manage organization-wide resources

Tasks

• configure Plugins for the Organization
• Create and manage Teams for the Organization
• Onboard and manage Kubernetes clusters for the Organization

Organization Cluster admin

Administrator of a Greenhouse Organization’s Kubernetes cluster

Goals

• Manage an Organizations Clusters

Tasks

• configure Plugins for the Organization
• Create and manage Team access to the Organization Cluster
• Onboard and manage Kubernetes clusters for the Organization

Organization Plugin admin

Administrator of a Greenhouse Organization’s Plugins

Goals

• Manage an Organizations Plugins

Tasks

• configure Plugins for the Organization

Organization member

Member of an Organization that accesses the UI to do ops/support tasks. Is member of one ore more teams of the organization. By default members have view permissions on Organization resources.

Goals

• Provide ops/support for the services of the organization

Tasks

• Highly dependend on Team membership and Plugins configured for the Organization
• Examples:
  • Check alerts for Teams the user is assigned to
  • Check policy violations for deployed resources owned by the Users Team
  • Check for known vulnerabilites in services

Plugin developer

A plugin developer is developing PluginDefinitions for Greenhouse.

Goals

• Must be easy to create PluginDefinitions
• Can create and test PluginDefinitions independently
• Greenhouse provides tooling to assist creating, validating, etc. of PluginDefinitions
• Publishing the PluginDefintion to Greenhouse requires admin permissions.

Tasks

• PluginDefintion Development
  • Juno UI Development
  • PluginDefinition backend development

Auditor

An Auditor audits Greenhouse and/or Greenhouse PluginDefinitions for compliance with industry or regulatory standards.

Goals

• Wants to see that we have a record for all changes happening in greenhouse
• Wants to have access to resources required to audit the respective Plugin

Tasks

• Performs Audits

Greenhouse Developer

Develops parts of the Greenhouse platform (Kubernetes, Greenhouse app, Greenhouse backend, …)

Goals

• Provide Greenhouse framework

Tasks

• Provides framework for plugin developers to develop plugins
• Develops Greenhouse framework components (UI or backend)

User Stories

The User Stories outlined in this Section have the target to archive a common Understanding of the capabilities/functionalities the Platform wants to archive and the functional requirements that come with those. The Integration / Development of Functionalities is not going to be strictly bound to User Stories and they are rather used as an orientation to ensure that envisioned capabilities are not getting Blocked due to implementation details.

The details of all User Stories are subject to change based on the results of Proof of Concept implementations, User feedback or other unforseen factors.

Auditor

Auditor 01 - Audit Logging

As an Auditor, I want to see who did which action and when to verify that the Vulnerability and Patch management process is followed according to company policies and that the platform is functioning as expected.

Acceptance Criteria

• Every state-changing action is logged into an immutable log collection, including:
  • What action was performed
  • Who performed the action
  • When was the action performed
• Every authentication to the platform is logged into an immutable log collection, including:
  • Who logged in
  • When was the login

Greenhouse Admin

Greenhouse Admin 01 - Greenhouse Management Dashboard

As an Greenhouse Admin, I want a central Greenhouse Management Dashboard that allows me to navigate through all organization-wide resources to be able to manage the Platform.

Acceptance Criteria

• Assuming I am on the Greenhouse Management Dashboard view, i can:
  • See all Plugins, including the enabled version
  • Order not enabled Plugins by last Update Date
  • Plugins are Ordered by the Order Attribute
  • The order attribute is a numeric value that can be changed to reflect a different ordering of the Plugin:
    • 1 is ordered before 2 etc.
    • The order attribute is used as well to order the Plugins on the Navigation Bar
  • Navigate to “Plugin Detail View” by clicking a Plugin
  • See all Organizations, including:
    • Number of Organization Admins
    • Number of Organization Members
  • Navigate to organization creation view by clicking “Create Organization”
  • Navigate to Organization Detail View by clicking a Organization
• Only Greenhouse Admin’s should be able to see the Dashboard
• The Navigation item to the Greenhouse Management Dashboard should only be visible to Greenhouse admins

Greenhouse Admin 02 - Organization Creation View

As a Greenhouse Admin, I want a Organization Creation view that allows me to create a new Organization

Acceptance Criteria

• Assuming I am on the Organization creation View, i can:
  • Give a unique name for the organisation
  • Provide a short description for the organization
  • Provide a OIDC Group that gives Organization Admin Privileges

Greenhouse Admin & Organization Admin

Greenhouse Admin & Organization Admin 01 - Organization Detail View

As a Greenhouse Admin or Organization Admin, I want an Organization Detail view that allows me to view details about an organization

Acceptance Criteria

• Assuming I am on the organization detail View, i can:
  • Can see the organization details (name/description)
  • See a list of teams created for this organization
  • See the list of active plugins
  • Add Plugins to the organization by clicking “add Plugin”
  • Change the Organization Admin Role Name

Greenhouse Admin & Organization Admin 02 - Plugin Detail View

As an Greenhouse Admin or Organizatioin Admin, I want a Plugin Detail view that allows me to see  Plugin Details to be able to see details about the plugin.

Acceptance Criteria

• Assuming I am on the Plugin Detail View, I can:
  • see the plugin name
  • see the plugin description
  • see the last update date
  • see the release reference
  • see the ui release refrence
  • see the helm chart reference
  • see the ordering attribute
  • see configuration values for the plugin
  • set the configuration values for the current organization
  • see a change log
  • see the actually released (deployed to greenhouse) version

Organization Admin

Organization Admin 01 - Organization Managment Dashboard

As an Organization Admin, I want to have an Dashboard showing me the most relevant information about my Organization to be able to manage it efficently.

Acceptance Criteria

• Assuming I am on the organisation management dashboard
  • I can see a list of all teams in my organization
  • I can see a list of configured plugins
  • I can click a “add plugin” button to add a new plugin
  • I can see a list of registered clusters
  • I can click a “add cluster” button to register a cluster

Organization Admin 02 - Plugin Configuration View

As an organization admin, I want a Plugin configuration view that allows me to enable and configure greenhouse plugins for my organization

Acceptance Criteria

• Assuming I am on the Plugin configuration View, I can:
  • select the type of plugin I want to configure
  • enable/disable the plugin (for my org)
  • remove the plugin (when already added)
  • manage configuration options specific for the plugin

Organization Admin 03 - Cluster registration View

As an organization admin, I want a Cluster registration view to onboard kubernetes clusters into my organization.

Acceptance Criteria

• Assuming I am on the cluster registration view, I can:
  • Get instructions how to register a kubernetes clusters
  • give a name and description for the registered cluster
• After executing the provided instructions I get feedback that the cluster has been successfully registered
• A cluster can be registered to exactly one organization

Organization Admin 04 - Cluster detail View

As an organization admin, I want a Cluster detail view to get some information about a registered cluster

Acceptance Criteria

• Assuming I am on the cluster detail view, I can:
  • see basic details about the cluster:
    • name
    • api url
    • version
    • node status
  • de-register the cluster from my organization

Organization Admin 05 - Team Detail View

As an organization admin, i want to have a Team Detail View, with the option to configure teams based on role mapping to be to manage teams within my organization without managing the permission administration itself on the Platform

Acceptance Criteria

• Assuming I am on the Team detail view, i can:
  • Change the Name of the Team
  • change the description of the team
  • Define a single OIDC Group  that assign you this team
  • Define The Greenhouse Role that you get within Greenhouse if you are a member of the team
• On Login of a User into an Organization the Platform verifies if the User has ALL required roles

Organization Admin 05 - Team Creation View

As an organization admin, i want to have a Team Creation View, to be able to create a new Team

Acceptance Criteria

• Assuming I am on the Team Creation view, I can::
  • Set the name of the Team
  • Set a description of the Team
  • Set a OIDC Group Name that assigns users to this team

Organization Member

Organization Member 01 - Unified task inbox

As an organization member, I want a task inbox that shows my open tasks from all enabled plugins that need my attention to be on top of my tasks to fulfill across all plugins

Acceptance Criteria

• Assuming I am on the task inbox:
  • I can a list of open task accross all plugins that need attention
  • clicking on an open task jumps in the plugin specific UI the task belong to
  • I can sort open tasks by name, plugin or date

Plugin Developer

As a Plugin Developer, I want to have a seperate Repository for my Plugin which I can own and use to configure plugin internals to have control over the Development efforts and configuration of the Plugin

Plugin Developer 01 - Decentrally Managed Plugin

As a Plugin Developer, I want to have a seperate Repository for my Plugin which I can own and use to configure plugin internals to have control over the Development efforts and configuration of the Plugin

Acceptance Criteria

• Plugin lives on his own Github Repository
• Versions are managed via Github Releases using Tags and the release to Greenhouse is managed by the Plugin:
  • The version to be pulled by Greenhouse is managed by the Plugin Developer.
• I can configure the Plugin Configuration over a greenhouse.yml in the root of the repository, which at least includes (mandatory):
  • description: …
  • version: …
• There are optional attributes in the greenhouse.yml:
  • icon: which if it has a valid absolute path to an image file on the repository makes the icon selectable as an icon in the plugin detail view (GA02)
  • describes available configuration options that attributes that are required for the plugin to function
  • I can specify a reference to a UI App
  • I can specify a reference to Helm Charts

Plugin Developer 02 - Plugin Role Config

As a Plugin Developer i want a section within the Greenhouse.yml metadata, named “Roles” where i can setup Roles used by my Plugin

Acceptance Criteria

Plugin Developer 03 - Spec Schema Validation

As a Plugin Developer I want to have the possibility to validate the schema of my greenhouse.yaml to be able to catch errors within my specification early.

Acceptance Criteria

• The schema check should support IDE’s
• The schema check should be automate-able and be integrate-able to pre-commit hooks and quality gates
• A version with a broken schema should not be release on greenhouse even when pushing for a pull of the release
• It should be visible on the Plugin detail view when an invalid schema was released with a recent version

Plugin Developer 04 - Config Value Validation

As a Plugin Developer I want to have the possibility to write custom regex checks for configuration options of my plugin that include the check to be performed on a field and an error message to be shown if configured wrong by an organization to support organization admins on configuring my plugin

Acceptance Criteria

• The validation rules should be controlled by Plugin Developer
• The validation should happen on the frontend before submitting a configuration
• The error message should be shown when a config value is provided wrong

Plugin Developer 05 - Plugin development tooling

As a plugin developer I want to have an easy setup for developing and testing greenhouse plugins

Acceptance Criteria

• Dev environment available within X Time
• Possible to have a working local setup with a “mock greenhouse apiserver”
• Has a fully working Bootstrap Project that includes Backend and Frontend which can be run locally immediately
• Has documentation

Product Stages

Overview

This Section gives an overview of the different early stages of the Platform that are beeing developed and which functional requirements need to get fulfilled within those stages.

Proof of Concept (POC)

The Proof of Concept is the stage where fundamental Framework/Platform decisions are proven to be working as intended. At this Stage the Platform is not suitable to be used by the intended audience yet but most necessary core functionalities are implemented.

The desired functionalities in this phase are:

• Frontend with Authentication
• Authorization within Greenhouse (Greenhouse Admin, Org Admin, Org Member)
• Team Management (without UI)
• Org Management (without UI)
• Greenhouse Admin User Stories (without UI)
• Dummy Plugin
  • with configuration spec
• Plugin Development Setup (without Documentation)
• Plugin Versioning & Provisioning (without UI)

Minimal viable product (MVP)

This stage is considered to be the earliest stage to open the Platform for General use.
In addition to the PoC functionalities we expect the following requirements to be fulfilled:

• Integrated 3 Plugins:
  • Supernova (Alerts)
  • DOOP (Violations)
  • Heureka
    NOTE: Heureka was excluded from MVP as the Heureka API is only available at a later point in time.
• Team management
• Organization management

3.3 - Building Observability Architectures

The main terminologies used in this document can be found in core-concepts.

Introduction to Observability

Observability in software and infrastructure is essential for operating a complex cloud environment. High-quality observability will enable teams to:

• Detect and troubleshoot issues quickly,
• Maintain performance and reliability,
• Make data-driven improvements.

Core Signals

The three key signals of observability are metrics, logs, and traces, each providing unique insights contributing to a comprehensive view of a system’s health.

Metrics

Metrics are numerical data points reporting on a system health over time. Examples of metrics can be CPU usage, memory usage, request latency.

Logs

Logs are detailed records of system or application events. These can be in various formats like structured/unstructured, single-/multi-line or written to files/streamed. Logs provide text-based data for post-incident analysis and crucial for auditing, troubleshooting or improving a system.

Traces

Traces follow a request’s journey through the system, capturing latency and failures across microservices. Traces are key for understanding dependencies, diagnosing bottlenecks and debugging the source code with real system data.

Tools

Common Open Source Tools to capture these signals are:

• Prometheus is a tool for collecting and querying metrics. It uses a time-series database optimized for real-time data, making it ideal for gathering system health data, enabling alerting, and visualizing trends.
• OpenSearch provides a scalable platform for log indexing, search, and analysis of logs. Enabling teams to sift through large volumes of logs to identify issues and understand system behaviour over time.
• Jaeger is a tool for distributed tracing, providing a detailed view of request paths and performance across services.
• OpenTelemetry was developed as a framework for instrumenting applications and infrastructures to collect metrics, logs and traces. It defines a standard for unifying the processing of all three types of signals. In addition to providing an API and SDKs for multiple programming languages, OpenTelemetry also simplifies the integration with backend systems such as Prometheus, OpenSearch and Jaeger.

Observability in Greenhouse

Greenhouse provides a suite of Plugins, including pre-packaged configurations for monitoring and logging tools. These Plugins are designed to simplify the setup and configuration of observability components. It enables users to quickly deploy monitoring and logging solutions to their Greenhouse clusters.

The following Plugins are currently available:

• Kube-Monitoring: installs Prometheus to collect custom and Kubernetes specific metrics with standard Kubernetes alerting enabled.
• Thanos: installs Thanos to enable long term metric retention and unified metric accessibility.
• Perses: installs Perses an open cloud native dashboard tool for Prometheus and other data sources
• Alerts: installs Prometheus AlertManager and Supernova to manage and visualize alerts sent by Prometheus.
• Logs: installs OpenTelemetry in the form of OpenTelemetryCollectors to collect metrics and logs from applications and forward them to backends like Prometheus and OpenSearch.
• Audit-Logs: installs OpenTelemetry in the form of OpenTelemetryCollectors to collect audit-relevant logs from applications and forward them to backends like Prometheus and OpenSearch.

Overview Architecture

4 - Reference

This section contains reference documentation for Greenhouse.

4.1 - API

Packages:

• greenhouse.sap/v1alpha1
• greenhouse.sap/v1alpha2

greenhouse.sap/v1alpha1

Authentication

(Appears on: OrganizationSpec)

Catalog

Catalog is the Schema for the catalogs API.

CatalogOverrides

(Appears on: CatalogSource)

CatalogSource

(Appears on: CatalogSpec)

  GitHub App Example:
-------------------
githubAppID: "<app-id>"
githubAppInstallationID: "<app-installation-id>"
githubAppPrivateKey: |
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
githubAppBaseURL: "<github-enterprise-api-url>" #optional, required only for GitHub Enterprise Server users
ca.crt: | #optional, for GitHub Enterprise Server users
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
GitHub Token Example:
-------------------
username: <BASE64>
password: <BASE64>
ca.crt: <BASE64> #optional, for GitHub Enterprise Server users

CatalogSpec

(Appears on: Catalog)

CatalogSpec defines the desired state of Catalog.

CatalogStatus

(Appears on: Catalog)

CatalogStatus defines the observed state of Catalog.

Cluster

Cluster is the Schema for the clusters API

ClusterAccessMode (string alias)

(Appears on: ClusterSpec)

ClusterAccessMode configures the access mode to the customer cluster.

ClusterConditionType (string alias)

ClusterConditionType is a valid condition of a cluster.

ClusterKubeConfig

(Appears on: ClusterSpec)

ClusterKubeConfig configures kube config values.

ClusterKubeconfig

ClusterKubeconfig is the Schema for the clusterkubeconfigs API ObjectMeta.OwnerReferences is used to link the ClusterKubeconfig to the Cluster ObjectMeta.Generation is used to detect changes in the ClusterKubeconfig and sync local kubeconfig files ObjectMeta.Name is designed to be the same with the Cluster name

ClusterKubeconfigAuthInfo

(Appears on: ClusterKubeconfigAuthInfoItem)

ClusterKubeconfigAuthInfoItem

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigCluster

(Appears on: ClusterKubeconfigClusterItem)

ClusterKubeconfigClusterItem

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigContext

(Appears on: ClusterKubeconfigContextItem)

ClusterKubeconfigContextItem

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigData

(Appears on: ClusterKubeconfigSpec)

ClusterKubeconfigData stores the kubeconfig data ready to use kubectl or other local tooling It is a simplified version of clientcmdapi.Config: https://pkg.go.dev/k8s.io/client-go/tools/clientcmd/api#Config

ClusterKubeconfigPreferences

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigSpec

(Appears on: ClusterKubeconfig)

ClusterKubeconfigSpec stores the kubeconfig data for the cluster The idea is to use kubeconfig data locally with minimum effort (with local tools or plain kubectl): kubectl get cluster-kubeconfig $NAME -o yaml | yq -y .spec.kubeconfig

ClusterKubeconfigStatus

(Appears on: ClusterKubeconfig)

ClusterOptionOverride

(Appears on: PluginPresetSpec)

ClusterOptionOverride defines which plugin option should be override in which cluster

ClusterPluginDefinition

ClusterPluginDefinition is the Schema for the clusterplugindefinitions API.

ClusterPluginDefinitionStatus

(Appears on: ClusterPluginDefinition)

ClusterPluginDefinitionStatus defines the observed state of ClusterPluginDefinition.

ClusterSpec

(Appears on: Cluster)

ClusterSpec defines the desired state of the Cluster.

ClusterStatus

(Appears on: Cluster)

ClusterStatus defines the observed state of Cluster

GitRef

(Appears on: CatalogSource)

HelmChartReference

(Appears on: PluginDefinitionSpec, PluginStatus)

HelmChartReference references a Helm Chart in a chart repository.

HelmReleaseStatus

(Appears on: PluginStatus)

HelmReleaseStatus reflects the status of a Helm release.

IgnoreDifference

(Appears on: PluginSpec)

IgnoreDifference defines a set of paths to ignore for matching resources.

ManagedPluginStatus

(Appears on: PluginPresetStatus)

ManagedPluginStatus defines the Ready condition of a managed Plugin identified by its name.

NodeStatus

(Appears on: Nodes)

NodeStatus represents a status of non-ready node.

Nodes

(Appears on: ClusterStatus)

Nodes contains node count metrics and tracks non-ready nodes in a cluster.

OIDCConfig

(Appears on: Authentication)

Organization

Organization is the Schema for the organizations API

OrganizationSpec

(Appears on: Organization)

OrganizationSpec defines the desired state of Organization

OrganizationStatus

(Appears on: Organization)

OrganizationStatus defines the observed state of an Organization

Plugin

Plugin is the Schema for the plugins API

PluginDefinition

PluginDefinition is the Schema for the PluginDefinitions API

PluginDefinitionReference

(Appears on: PluginSpec)

PluginDefinitionReference defines the reference to the PluginDefinition or ClusterPluginDefinition.

PluginDefinitionSpec

(Appears on: ClusterPluginDefinition, PluginDefinition)

PluginDefinitionSpec defines the desired state of PluginDefinitionSpec

PluginDefinitionStatus

(Appears on: PluginDefinition)

PluginDefinitionStatus defines the observed state of PluginDefinition

PluginOption

(Appears on: PluginDefinitionSpec)

PluginOptionType (string alias)

(Appears on: PluginOption)

PluginOptionType specifies the type of PluginOption.

PluginOptionValue

(Appears on: ClusterOptionOverride, PluginSpec)

PluginOptionValue is the value for a PluginOption.

PluginPreset

PluginPreset is the Schema for the PluginPresets API

PluginPresetSpec

(Appears on: PluginPreset)

PluginPresetSpec defines the desired state of PluginPreset

PluginPresetStatus

(Appears on: PluginPreset)

PluginPresetStatus defines the observed state of PluginPreset

PluginRef

(Appears on: WaitForItem)

PluginRef defines a reference to the Plugin, either by its name or the PluginPreset that it’s created from.

PluginSpec

(Appears on: Plugin, PluginPresetSpec)

PluginSpec defines the desired state of Plugin

PluginStatus

(Appears on: Plugin)

PluginStatus defines the observed state of Plugin

PropagationStatus

(Appears on: TeamRoleBindingStatus)

PropagationStatus defines the observed state of the TeamRoleBinding’s associated rbacv1 resources on a Cluster

SCIMConfig

(Appears on: Authentication)

SecretKeyReference

(Appears on: OIDCConfig, ValueFromSource)

SecretKeyReference specifies the secret and key containing the value.

Service

(Appears on: PluginStatus)

Service references a Kubernetes service of a Plugin.

ServiceType (string alias)

(Appears on: Service)

ServiceType defines the type of exposed service.

SourceStatus

Team

Team is the Schema for the teams API

TeamRole

TeamRole is the Schema for the TeamRoles API

TeamRoleBinding

TeamRoleBinding is the Schema for the rolebindings API

TeamRoleBindingSpec

(Appears on: TeamRoleBinding)

TeamRoleBindingSpec defines the desired state of a TeamRoleBinding

TeamRoleBindingStatus

(Appears on: TeamRoleBinding)

TeamRoleBindingStatus defines the observed state of the TeamRoleBinding

TeamRoleSpec

(Appears on: TeamRole)

TeamRoleSpec defines the desired state of a TeamRole

TeamRoleStatus

(Appears on: TeamRole)

TeamRoleStatus defines the observed state of a TeamRole

TeamSpec

(Appears on: Team)

TeamSpec defines the desired state of Team

TeamStatus

(Appears on: Team)

TeamStatus defines the observed state of Team

UIApplicationReference

(Appears on: PluginDefinitionSpec, PluginStatus)

UIApplicationReference references the UI pluginDefinition to use.

User

(Appears on: TeamStatus)

User specifies a human person.

ValueFromSource

(Appears on: PluginOptionValue, SCIMConfig)

ValueFromSource is a valid source for a value.

WaitForItem

(Appears on: PluginPresetSpec, PluginSpec)

WaitForItem is a wrapper around PluginRef to add context for every WaitFor list item.

greenhouse.sap/v1alpha2

ClusterSelector

(Appears on: TeamRoleBindingSpec)

ClusterSelector specifies a selector for clusters by name or by label

PropagationStatus

(Appears on: TeamRoleBindingStatus)

PropagationStatus defines the observed state of the TeamRoleBinding’s associated rbacv1 resources on a Cluster

TeamRoleBinding

TeamRoleBinding is the Schema for the rolebindings API

TeamRoleBindingSpec

(Appears on: TeamRoleBinding)

TeamRoleBindingSpec defines the desired state of a TeamRoleBinding

TeamRoleBindingStatus

(Appears on: TeamRoleBinding)

TeamRoleBindingStatus defines the observed state of the TeamRoleBinding

4.2 - Greenhouse API

4.2.1 - Organizations

An Organization is the top-level entity within Greenhouse. Each Organization represents a different tenant within Greenhouse and is provided with a dedicated Namespace in the Greenhouse cluster.

Example Organization Spec

apiVersion: greenhouse.sap/v1alpha1
kind: Organization
metadata:
  name: example-organization
spec:
  authentication:
    oidc:
      issuer: https://accounts.example.com
      clientId: example-client-id

Writing an Organization Spec

DisplayName

.spec.displayName is a human-friendly name for the Organization. This field is optional; if not provided, it defaults to the value of metadata.name.

Authentication

OIDC

.spec.authentication.oidc is used to configure how members of the Organization will authenticate to Greenhouse. Greenhouse IDProxy is using Dex to provide the OIDC authentication for multiple Organizations. Each Organization receives their own Dex connector.

The config requires issuer, the URL of the identity provider, and clientIDReference and clientSecretReference, which reference Kubernetes Secrets containing the OIDC client ID and client secret, respectively.

spec:
  authentication:
    oidc:
      issuer: https://accounts.example.com
      clientIdReference:
        name: oidc-client-id-secret
        key: client-id
      clientSecretReference:
        name: oidc-client-secret
        key: client-secret

.authentication.oidc.redirectURI is an optional field to specify a custom redirect URI for OIDC authentication. If not provided, it defaults to the Greenhouse ID Proxy (auth.<greenhouse domain name>).

.authentication.oidc.oauth2ClientRedirectURIs is an optional list of URIs that are added to the Dex connector as allowed redirect URIs for OAuth2 clients.

SCIM

.spec.authentication.scim is used by Greenhouse to retrieve the members of a Team from the Organization’s identity provider. This field is optional; if not provided, Greenhouse will not attempt to sync users via SCIM.

The configuration requires baseURL, the URL of the SCIM endpoint, and authType, which specifies the authentication method to use when connecting to the SCIM endpoint. Supported methods are basic and token.

spec:
  authentication:
    scim:
      baseURL: https://scim.example.com
      authType: token
      bearerToken:
        secret:
          name: scim-bearer-token-secret
          key: bearer-token
      bearerPrefix: Bearer
      bearerHeader: Authorization

.authentication.scim.bearerPrefix is an optional field to specify a custom prefix for the bearer token in the authorization header. If not provided, it defaults to Bearer.

.authentication.scim.bearerHeader is an optional field to specify a custom header name for the bearer token. If not provided, it defaults to Authorization.

MappedOrgAdminIdPGroup

.spec.mappedOrgAdminIdPGroup is an optional field that specifies the name of an identity provider group whose members will be granted Organization Admin privileges within Greenhouse. If this field is not provided, no users will be granted Organization Admin privileges.

Working with Organizations

Role-Based Access Control within the Organization namespace

Greenhouse provisions a set of default Roles and RoleBindings within each Organization’s Namespace to facilitate Role-Based Access Control (RBAC). These roles can be used by the Organization Admins as a starting point to manage access to resources within their Organization.

The following roles are seeded for each Organization:

Next Steps

• Creating an Organization

4.2.2 - PluginDefinitions

A PluginDefinition brings either a UI application, a Helm chart deployment, or both, to the Greenhouse platform. The Helm chart for a PluginDefinition can be used to deploy infrastructure components to a Kubernetes cluster managed with Greenhouse. The PluginDefinition provides an opinionated way to configure, integrate and deploy these components with Greenhouse.

Example PluginDefinition Spec

apiVersion: greenhouse.sap/v1alpha1
kind: PluginDefinition
metadata:
  name: alerts
  namespace: example-organization
spec:
  description: The Alerts Plugin consists of both Prometheus Alertmanager and Supernova,
    the holistic alert management UI
  displayName: Alerts
  docMarkDownUrl: https://raw.githubusercontent.com/cloudoperators/greenhouse-extensions/main/alerts/README.md
  helmChart:
    name: alerts
    repository: oci://ghcr.io/cloudoperators/greenhouse-extensions/charts
    version: 4.0.3
  icon: https://raw.githubusercontent.com/cloudoperators/greenhouse-extensions/main/alerts/logo.png
  uiApplication:
    name: supernova
    version: latest
  version: 5.0.3
  weight: 0

Writing a PluginDefinition Spec

.spec.displayName is a human-readable name for the PluginDefinition. This field is optional; if not provided, it defaults to the value of metadata.name. This name is used in the Greenhouse UI to display the PluginDefinition in the Catalog of available PluginDefinitions.

.spec.version specifies the semantic version of the PluginDefinition. This versions the combination of Helm chart, UI application and any options provided in the PluginDefinition. The version should be incremented whenever any of these fields are updated.

.spec.uiApplication is an optional field that specifies the UI application associated with the PluginDefinition. The UI application will be made available in the Greenhouse UI when a Plugin is created from this PluginDefinition.

spec:
  uiApplication:
    name: supernova
    version: latest
  weight: 0
  icon: https://raw.githubusercontent.com/cloudoperators/greenhouse-extensions/main/alerts/logo.png
  docMarkDownUrl: https://raw.githubusercontent.com/cloudoperators/greenhouse-extensions/main/alerts/README.md

The fields weight and icon are optional and are used to customize the appearance of the Plugin in the Greenhouse UI sidebar. The optional field docMarkDownUrl can be used to provide a link to documentation for the PluginDefinition, which will be displayed in the entry of available PluginDefinitions in the Greenhouse UI.

.spec.helmChart is an optional field that specifies the Helm chart that is deployed when creating a Plugin from this PluginDefinition.

spec:
  helmChart:
    name: alerts
    repository: oci://ghcr.io/cloudoperators/greenhouse-extensions/charts
    version: 4.0.3

.spec.options is an optional field that specifies default configuration options for the PluginDefinition. These options will be pre-filled when creating a Plugin from this PluginDefinition, but can be overridden by the user.

spec:
  options:
  - description: Alertmanager API Endpoint URL
    name: endpoint
    required: true
    type: string
  - description: FilterLabels are the labels shown in the filter dropdown, enabling
      users to filter alerts based on specific criteria. The format is a list of strings.
    name: filterLabels
    required: false
    type: list
  - default: false
    description: Install Prometheus Operator CRDs if kube-monitoring has not already
      installed them.
    name: alerts.crds.enabled
    required: false
    type: bool

.required indicates whether the option is mandatory when creating a Plugin from this PluginDefinition. .default contains the default value for the option if the Plugin does not provide a value for it. .type is used to enforce validation of the value. The following types are supported: string, bool, int, list, map and secret.

| ℹ️ The type secret requires a secret reference, disallowing clear-text credentials. Vault/OpenBao references will be allowed with 1211|

Next Steps

• Creating a PluginDefinition
• PluginPreset reference
• Plugin reference
• Managing PluginDefinitions via Catalog

4.2.3 - PluginPresets

A PluginPreset is used to configure Plugins for a set of Clusters. This allows administrators to define standard configurations for Clusters in the same environment or with similar requirements. Greenhouse will create Plugins based on the PluginPreset for each Cluster that matches the specified selector.

Example PluginPreset Spec

apiVersion: greenhouse.sap/v1alpha1
kind: PluginPreset
metadata:
  name: perses-preset
  namespace: example-organization
spec:
  clusterOptionOverrides:
    - clusterName: example-cluster
      overrides:
      - name: perses.serviceMonitor.selfMonitor
        value: true
      - name: perses.serviceMonitor.labels
        value:
          plugin: kube-monitoring
  clusterSelector:
    matchExpressions:
    - key: cluster-type
      operator: In
      values:
      - observability
  deletionPolicy: Delete
  plugin:
    optionValues:
    - name: perses.sidecar.enabled
      value: true
    pluginDefinitionRef:
      kind: ClusterPluginDefinition
      name: perses
    releaseName: perses
    releaseNamespace: kube-monitoring

Writing a PluginPreset Spec

.spec.plugin is the template for the Plugins that will be created for each matching Cluster. This field has the same structure as the PluginSpec. Only .spec.clusterName is not allowed in the PluginPreset’s Plugin template, as the Cluster name is determined by the matching Clusters.

spec:
  plugin:
    optionValues:
    - name: perses.sidecar.enabled
      value: true
    pluginDefinitionRef:
      kind: ClusterPluginDefinition
      name: perses
    releaseName: perses
    releaseNamespace: kube-monitoring

.spec.clusterSelector is a required field that specifies the label selector used to list the Clusters for which Plugins will be created based on this PluginPreset.

spec:
  clusterSelector:
    matchExpressions:
    - key: cluster-type
      operator: In
      values:
      - observability

| ⚠️ Changing the clusterSelector may result in the creation or deletion of Plugins for Clusters that start or stop matching the selector. |

.spec.clusterOptionOverrides is an optional field that can be used to provide per-Cluster overrides for the Plugin’s OptionValues. This can be used to customize the configuration of the Plugin for specific Clusters.

spec:
  clusterOptionOverrides:
    - clusterName: example-cluster
      overrides:
      - name: perses.serviceMonitor.selfMonitor
        value: true

.spec.deletionPolicy is an optional field that specifies the behaviour when a PluginPreset is deleted. The possible values are Delete and Retain. If set to Delete (the default), all Plugins created by the PluginPreset will also be deleted when the PluginPreset is deleted. If set to Retain, the Plugins will remain after the PluginPreset is deleted or if the Cluster stops matching the selector.

Next Steps

• Managing Plugins for multiple clusters
• Plugin reference
• PluginDefinition reference

4.2.4 - Plugins

A Plugin is an instance of a PluginDefinition and is used to deploy infrastructure components such as observability, compliance or system components to a Kubernetes cluster managed with Greenhouse. A Plugin provides the specific configuration for deploying the Helm chart associated with the referenced PluginDefinition to a specific cluster.

Example Plugin Spec

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: alerts-plugin
  namespace: example-organization
spec:
  clusterName: example-cluster
  displayName: Example Alerts Plugin
  optionValues:
  - name: image.tag
    value: foobar
  pluginDefinitionRef:
    kind: PluginDefinition
    name: alerts
  releaseName: alerts
  releaseNamespace: kube-monitoring

Writing a Plugin Spec

.spec.displayName is an optional human-readable name that is used to display the Plugin in the Greenhouse UI. If not provided, it defaults to the value of metadata.name.

.spec.clusterName is the name of the Cluster resource where the Helm chart associated with the PluginDefinition will be deployed.

.spec.pluginDefinitionRef is the required and immutable reference to a PluginDefinition resource that defines the Helm chart and UI application associated with this Plugin.

spec:
  pluginDefinitionRef:
    kind: PluginDefinition
    name: alerts

.spec.releaseName is the optional and immutable name of the Helm release that will be created when deploying the Plugin to the target cluster. If not provided it defaults to the name of the PluginDefinition’s Helm chart.

.spec.releaseNamespace is the optional and immutable Kubernetes namespace in the target cluster where the Helm release will be deployed. If not provided, it defaults to the name of the Organization.

.spec.optionValues is an optional list of Helm chart values that will be used to customize the deployment of the Helm chart associated with the PluginDefinition. These values are used to set Options required by the PluginDefinition or to override provided default values.

  optionValues:
  - name: image.tag
    value: foobar
  - name: secret
    valueFrom:
      secret:
        name: alerts-secret
        key: secret-key

| ℹ️ A defaulting webhook automatically merges the OptionValues with the defaults set in the PluginDefinition. The defaulting does not update OptionValues when the defaults change and does not remove values when they are removed from the PluginDefinition. |

.spec.waitFor is an optional field that specifies PluginPresets or Plugins which have to be successfully deployed before this Plugin can be deployed. This can be used to express dependencies between Plugins. This can be useful if one Plugin depends on Custom Resource Definitions or other resources created by another Plugin.

spec:
  waitFor:
  - pluginRef:
      pluginPreset: ingress-nginx
  - pluginRef:
      name: cert-manager-example-cluster

| ℹ️ The dependency on a PluginPreset ensures that a Plugin created by this PluginPreset has been deployed to the same cluster. The dependency on a Plugin is fulfilled if the referenced Plugin is deployed to the same cluster. |

.spec.ignoreDifferences is an optional field that is used to suppress specific differences detected by the drift detection of the deployment tool. This can be useful to ignore differences in fields that are managed by other controllers or tools. Example configuration when using Flux as the deployment tool.

spec:
  ignoreDifferences:
  - group: apps
    version: v1
    kind: Deployment
    paths:
    - /spec/replicas

| ⚠️ The ignoreDifferences field is only supported when using Flux as the deployment tool. It is ignored when using the legacy Helm controller. |

Working with Plugins

Choosing the deployment tool

The annotation greenhouse.sap/deployment-tool can be added to a Plugin resource to choose the deployment tool used to deploy the Helm release. Supported values are flux and legacy.

Suspending the Plugin’s reconciliation

The annotation greenhouse.sap/suspend can be added to a Plugin resource to temporarily suspend the reconciliation of the Plugin. This results in no changes on the Plugin or referenced resources being applied until the annotation is removed. This also includes upgrades of the Helm release on the target cluster. This also blocks the deletion of the Plugin resource until the annotation is removed.

Triggering reconciliation of the Plugin’s managed resources

The annotation greenhouse.sap/reconcile can be added to a Plugin resource to trigger a reconciliation of the Plugin and its managed resources. When the Plugin is deployed using FluxCD this annotation is propagated to the Flux HelmRelease resource and triggers a reconciliation of the Helm release on the target cluster. This can be useful to trigger a reconciliation even if no changes were made to the Plugin resource.

Next Steps

• PluginPreset reference
• PluginDefinition reference

4.2.5 - Catalogs

A Catalog is a collection of PluginDefinitions that can be made available to Organizations within Greenhouse. Catalogs allow organization admins to manage the lifecycle of PluginDefinitions by controlling which version of a PluginDefinition is deployed to their cluster fleet.

The Catalog API is currently in alpha and is still under active development and is subjected to change.

Example

The following is an example of a Greenhouse Catalog that reconciles the PluginDefinition manifests stored in a Git Repository.

apiVersion: greenhouse.sap/v1alpha1
kind: Catalog
metadata:
  name: greenhouse-extensions
  namespace: <organization-namespace>
spec:
  sources:
    - repository: https://github.com/cloudoperators/greenhouse-extensions
      resources:
        - alerts/plugindefinition.yaml
        - audit-logs/plugindefinition.yaml
        - cert-manager/plugindefinition.yaml
        - external-dns/plugindefinition.yaml
        - repo-guard/plugindefinition.yaml
        - ingress-nginx/plugindefinition.yaml
        - kube-monitoring/plugindefinition.yaml
        - logs/plugindefinition.yaml
        - perses/plugindefinition.yaml
        - thanos/plugindefinition.yaml
      ref:
        branch: main

In the above example:

• The Greenhouse Catalog references a Git Repository targeting the main branch.
• The Catalog is configured to target specific PluginDefinition manifests stored in a path within the repository specified under .spec.sources[].resources[].
• The Catalog watches for changes in the specified Git Repository branch and reconciles the PluginDefinitions in the Organization namespace accordingly.

Writing a Catalog Spec

Sources

.spec.sources is a list of sources from which the Catalog will fetch PluginDefinition manifests. Currently, only Git repositories are supported as sources. Each source requires a repository URL and a list of resources that specify the paths to the PluginDefinition manifests within the repository. The ref field is used to specify the branch, tag, or commit SHA to fetch from the repository.

⚠️ Each Organization has a dedicated ServiceAccount used to apply the Catalog resources. This ServiceAccount only has permissions to apply PluginDefinitions into the Organization’s Namespace. It will not be possible to bring other Kinds using the Catalog.

Repository

.spec.sources[].repository is used to specify the URL of the Git repository containing the PluginDefinition manifests.

Resources

.spec.sources[].resources is a list of file paths within the repository that point to the PluginDefinition manifests to be included in the Catalog.

Empty resources list will result in an error.

Ref

.spec.sources[].ref is used to specify the branch, tag, or commit SHA to fetch from the repository.

Available fields are:

• sha - The commit SHA to fetch.
• tag - The tag to fetch.
• branch - The branch that is targeted.

The priority order is: sha > tag > branch. If multiple fields are specified, the field with the highest priority will be used.

When multiple sources are defined with the same repository and ref, a duplicate error will be raised.

Interval (Optional)

.spec.sources[].interval is an optional field that specifies how often the Catalog should check for updates in the source repository. The default value is 1h (1 hour) if not specified.

The value must be in a Go recognized duration format, e.g. 5m0s to reconcile the source every 5 minutes.

Secret Reference (Optional)

.spec.sources[].secretName is an optional field that specifies a reference to a Kubernetes Secret name containing credentials for accessing private Git repositories.

The following are the types of authentication supported:

• Basic Authentication
• GitHub App Credentials

The Secret must be in the same namespace as the Catalog resource.

Overrides (Optional)

.spec.sources[].overrides is an optional field that allows specifying overrides for specific PluginDefinitions in the Catalog. This can be used to customize certain fields of PluginDefinitions.

Currently, you can override the following fields:

• alias field to override metadata.name of the PluginDefinition
• repository field to override the helm chart repository in the PluginDefinition spec

Working with Catalog

Configuring Secret for Basic Authentication:

To authenticate towards a Git repository over HTTPS using basic access authentication (in other words: using a username and password), the referenced Secret is expected to contain .data.username and .data.password values.

apiVersion: v1
kind: Secret
metadata:
  name: git-credentials
  namespace: <catalog-namespace>
type: Opaque
data:
  username: <BASE64>
  password: <BASE64>

password is the Personal Access Token (PAT) for accessing the Git repository.

Configure Secret for GitHub App authentication:

Pre-requisites:

• Register the GitHub App with the necessary permissions and generate a private key for the app.

• Install the app in the organization/account configuring access to the necessary repositories.

To authenticate towards a GitHub repository using a GitHub App, the referenced secret is expected to contain the following values:

• Get the App ID from the app settings page at https://github.com/settings/apps/<app-name>.
• Get the App Installation ID from the app installations page at https://github.com/settings/installations. Click the installed app, the URL will contain the installation ID https://github.com/settings/installations/<installation-id>. For organizations, the first part of the URL may be different, but it follows the same pattern.
• The private key that was generated in the pre-requisites.
• (Optional) GitHub Enterprise Server users can set the base URL to http(s)://HOSTNAME/api/v3.
• (Optional) If GitHub Enterprise Server uses a private CA, include its bundle (root and any intermediates) in ca.crt. If the ca.crt is specified, then it will be used for TLS verification for all API / Git over HTTPS requests to the GitHub Enterprise Server.

apiVersion: v1
kind: Secret
metadata:
  name: github-app-credentials
  namespace: <catalog-namespace>
type: Opaque
stringData:
  githubAppID: "5001"
  githubAppInstallationID: "1005"
  githubAppBaseURL: "github.enterprise.example.com/api/v3" #optional, required only for GitHub Enterprise Server users
  githubAppPrivateKey: |
    -----BEGIN RSA PRIVATE KEY-----
    .....
    -----END RSA PRIVATE KEY-----    
  ca.crt: | #optional, for GitHub Enterprise Server users
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

Minimum required permissions for the GitHub App or Personal Access Token (PAT) is content read access for the target repository.

Configuring Overrides for PluginDefinitions

If you want to bring in multiple versions of the same PluginDefinition, you can use the alias option to reference the PluginDefinition under a different name.

Example:

spec:
  sources:
    - repository: https://github.com/cloudoperators/greenhouse-extensions
      resources:
        - perses/plugindefinition.yaml
      ref:
        branch: main
      overrides:
        - name: perses
          alias: perses-latest
    - repository: https://github.com/cloudoperators/greenhouse-extensions
      resources:
        - perses/plugindefinition.yaml
      ref:
        sha: <commit-sha>
      overrides:
        - name: perses
          alias: perses-stable