Reference

• 1: API
• 2: Plugin Catalog
• 2.1: Alerts
• 2.2: Audit Logs Plugin
• 2.3: Cert-manager
• 2.4: Decentralized Observer of Policies (Violations)
• 2.5: Designate Ingress CNAME operator (DISCO)
• 2.6: DigiCert issuer
• 2.7: External DNS
• 2.8: Ingress NGINX
• 2.9: Kubernetes Monitoring
• 2.10: Logs Plugin
• 2.11: Logshipper
• 2.12: OpenSearch
• 2.13: Perses
• 2.14: Plutono
• 2.15: Prometheus
• 2.16: Repo Guard
• 2.17: Service exposure test
• 2.18: Teams2Slack
• 2.19: Thanos

1 - API

Packages:

• greenhouse.sap/v1alpha1
• greenhouse.sap/v1alpha2

greenhouse.sap/v1alpha1

Authentication

(Appears on: OrganizationSpec)

Cluster

Cluster is the Schema for the clusters API

ClusterAccessMode (string alias)

(Appears on: ClusterSpec)

ClusterAccessMode configures the access mode to the customer cluster.

ClusterConditionType (string alias)

ClusterConditionType is a valid condition of a cluster.

ClusterKubeConfig

(Appears on: ClusterSpec)

ClusterKubeConfig configures kube config values.

ClusterKubeconfig

ClusterKubeconfig is the Schema for the clusterkubeconfigs API ObjectMeta.OwnerReferences is used to link the ClusterKubeconfig to the Cluster ObjectMeta.Generation is used to detect changes in the ClusterKubeconfig and sync local kubeconfig files ObjectMeta.Name is designed to be the same with the Cluster name

ClusterKubeconfigAuthInfo

(Appears on: ClusterKubeconfigAuthInfoItem)

ClusterKubeconfigAuthInfoItem

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigCluster

(Appears on: ClusterKubeconfigClusterItem)

ClusterKubeconfigClusterItem

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigContext

(Appears on: ClusterKubeconfigContextItem)

ClusterKubeconfigContextItem

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigData

(Appears on: ClusterKubeconfigSpec)

ClusterKubeconfigData stores the kubeconfig data ready to use kubectl or other local tooling It is a simplified version of clientcmdapi.Config: https://pkg.go.dev/k8s.io/client-go/tools/clientcmd/api#Config

ClusterKubeconfigPreferences

(Appears on: ClusterKubeconfigData)

ClusterKubeconfigSpec

(Appears on: ClusterKubeconfig)

ClusterKubeconfigSpec stores the kubeconfig data for the cluster The idea is to use kubeconfig data locally with minimum effort (with local tools or plain kubectl): kubectl get cluster-kubeconfig $NAME -o yaml | yq -y .spec.kubeconfig

ClusterKubeconfigStatus

(Appears on: ClusterKubeconfig)

ClusterOptionOverride

(Appears on: PluginPresetSpec)

ClusterOptionOverride defines which plugin option should be override in which cluster

ClusterPluginDefinition

ClusterPluginDefinition is the Schema for the clusterplugindefinitions API.

ClusterPluginDefinitionStatus

(Appears on: ClusterPluginDefinition)

ClusterPluginDefinitionStatus defines the observed state of ClusterPluginDefinition.

ClusterSpec

(Appears on: Cluster)

ClusterSpec defines the desired state of the Cluster.

ClusterStatus

(Appears on: Cluster)

ClusterStatus defines the observed state of Cluster

HelmChartReference

(Appears on: PluginDefinitionSpec, PluginStatus)

HelmChartReference references a Helm Chart in a chart repository.

HelmReleaseStatus

(Appears on: PluginStatus)

HelmReleaseStatus reflects the status of a Helm release.

ManagedPluginStatus

(Appears on: PluginPresetStatus)

ManagedPluginStatus defines the Ready condition of a managed Plugin identified by its name.

NodeStatus

(Appears on: ClusterStatus)

OIDCConfig

(Appears on: Authentication)

Organization

Organization is the Schema for the organizations API

OrganizationSpec

(Appears on: Organization)

OrganizationSpec defines the desired state of Organization

OrganizationStatus

(Appears on: Organization)

OrganizationStatus defines the observed state of an Organization

Plugin

Plugin is the Schema for the plugins API

PluginDefinition

PluginDefinition is the Schema for the PluginDefinitions API

PluginDefinitionSpec

(Appears on: ClusterPluginDefinition, PluginDefinition)

PluginDefinitionSpec defines the desired state of PluginDefinitionSpec

PluginDefinitionStatus

(Appears on: PluginDefinition)

PluginDefinitionStatus defines the observed state of PluginDefinition

PluginOption

(Appears on: PluginDefinitionSpec)

PluginOptionType (string alias)

(Appears on: PluginOption)

PluginOptionType specifies the type of PluginOption.

PluginOptionValue

(Appears on: ClusterOptionOverride, PluginSpec)

PluginOptionValue is the value for a PluginOption.

PluginPreset

PluginPreset is the Schema for the PluginPresets API

PluginPresetSpec

(Appears on: PluginPreset)

PluginPresetSpec defines the desired state of PluginPreset

PluginPresetStatus

(Appears on: PluginPreset)

PluginPresetStatus defines the observed state of PluginPreset

PluginSpec

(Appears on: Plugin, PluginPresetSpec)

PluginSpec defines the desired state of Plugin

PluginStatus

(Appears on: Plugin)

PluginStatus defines the observed state of Plugin

PropagationStatus

(Appears on: TeamRoleBindingStatus)

PropagationStatus defines the observed state of the TeamRoleBinding’s associated rbacv1 resources on a Cluster

SCIMConfig

(Appears on: Authentication)

SecretKeyReference

(Appears on: OIDCConfig, ValueFromSource)

SecretKeyReference specifies the secret and key containing the value.

Service

(Appears on: PluginStatus)

Service references a Kubernetes service of a Plugin.

Team

Team is the Schema for the teams API

TeamRole

TeamRole is the Schema for the TeamRoles API

TeamRoleBinding

TeamRoleBinding is the Schema for the rolebindings API

TeamRoleBindingSpec

(Appears on: TeamRoleBinding)

TeamRoleBindingSpec defines the desired state of a TeamRoleBinding

TeamRoleBindingStatus

(Appears on: TeamRoleBinding)

TeamRoleBindingStatus defines the observed state of the TeamRoleBinding

TeamRoleSpec

(Appears on: TeamRole)

TeamRoleSpec defines the desired state of a TeamRole

TeamRoleStatus

(Appears on: TeamRole)

TeamRoleStatus defines the observed state of a TeamRole

TeamSpec

(Appears on: Team)

TeamSpec defines the desired state of Team

TeamStatus

(Appears on: Team)

TeamStatus defines the observed state of Team

UIApplicationReference

(Appears on: PluginDefinitionSpec, PluginStatus)

UIApplicationReference references the UI pluginDefinition to use.

User

(Appears on: TeamStatus)

User specifies a human person.

ValueFromSource

(Appears on: PluginOptionValue, SCIMConfig)

ValueFromSource is a valid source for a value.

greenhouse.sap/v1alpha2

ClusterSelector

(Appears on: TeamRoleBindingSpec)

ClusterSelector specifies a selector for clusters by name or by label

PropagationStatus

(Appears on: TeamRoleBindingStatus)

PropagationStatus defines the observed state of the TeamRoleBinding’s associated rbacv1 resources on a Cluster

TeamRoleBinding

TeamRoleBinding is the Schema for the rolebindings API

TeamRoleBindingSpec

(Appears on: TeamRoleBinding)

TeamRoleBindingSpec defines the desired state of a TeamRoleBinding

TeamRoleBindingStatus

(Appears on: TeamRoleBinding)

TeamRoleBindingStatus defines the observed state of the TeamRoleBinding

2 - Plugin Catalog

This section provides an overview of the available PluginDefinitions in Greenhouse.

2.1 - Alerts

Learn more about the alerts plugin. Use it to activate Prometheus alert management for your Greenhouse organisation.

The main terminologies used in this document can be found in core-concepts.

Overview

This Plugin includes a preconfigured Prometheus Alertmanager, which is deployed and managed via the Prometheus Operator, and Supernova, an advanced user interface for Prometheus Alertmanager. Certificates are automatically generated to enable sending alerts from Prometheus to Alertmanager. These alerts can too be sent as Slack notifications with a provided set of notification templates.

Components included in this Plugin:

• Prometheus Alertmanager
• Supernova

This Plugin usually is deployed along the kube-monitoring Plugin and does not deploy the Prometheus Operator itself. However, if you are intending to use it stand-alone, you need to explicitly enable the deployment of Prometheus Operator, otherwise it will not work. It can be done in the configuration interface of the plugin.

Disclaimer

This is not meant to be a comprehensive package that covers all scenarios. If you are an expert, feel free to configure the plugin according to your needs.

The Plugin is a deeply configured kube-prometheus-stack Helm chart which helps to keep track of versions and community updates.

It is intended as a platform that can be extended by following the guide.

Contribution is highly appreciated. If you discover bugs or want to add functionality to the plugin, then pull requests are always welcome.

Quick start

This guide provides a quick and straightforward way to use alerts as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

• A running and Greenhouse-onboarded Kubernetes cluster. If you don’t have one, follow the Cluster onboarding guide.
• kube-monitoring plugin (which brings in Prometheus Operator) OR stand alone: awareness to enable the deployment of Prometheus Operator with this plugin

Step 1:

You can install the alerts package in your cluster with Helm manually or let the Greenhouse platform lifecycle it for you automatically. For the latter, you can either:

1. Go to Greenhouse dashboard and select the Alerts Plugin from the catalog. Specify the cluster and required option values.
2. Create and specify a Plugin resource in your Greenhouse central cluster according to the examples.

Step 2:

After the installation, you can access the Supernova UI by navigating to the Alerts tab in the Greenhouse dashboard.

Step 3:

Greenhouse regularly performs integration tests that are bundled with alerts. These provide feedback on whether all the necessary resources are installed and continuously up and running. You will find messages about this in the plugin status and also in the Greenhouse dashboard.

Configuration

Prometheus Alertmanager options

cert-manager options

Supernova options

theme: Override the default theme. Possible values are "theme-light" or "theme-dark" (default)

endpoint: Alertmanager API Endpoint URL /api/v2. Should be one of alerts.alertmanager.ingress.hosts

silenceExcludedLabels: SilenceExcludedLabels are labels that are initially excluded by default when creating a silence. However, they can be added if necessary when utilizing the advanced options in the silence form.The labels must be an array of strings. Example: ["pod", "pod_name", "instance"]

filterLabels: FilterLabels are the labels shown in the filter dropdown, enabling users to filter alerts based on specific criteria. The ‘Status’ label serves as a default filter, automatically computed from the alert status attribute and will be not overwritten. The labels must be an array of strings. Example: ["app", "cluster", "cluster_type"]

predefinedFilters: PredefinedFilters are filters applied through in the UI to differentiate between contexts through matching alerts with regular expressions. They are loaded by default when the application is loaded. The format is a list of objects including name, displayname and matchers (containing keys corresponding value). Example:

[
  {
    "name": "prod",
    "displayName": "Productive System",
    "matchers": {
      "region": "^prod-.*"
    }
  }
]

silenceTemplates: SilenceTemplates are used in the Modal (schedule silence) to allow pre-defined silences to be used to scheduled maintenance windows. The format consists of a list of objects including description, editable_labels (array of strings specifying the labels that users can modify), fixed_labels (map containing fixed labels and their corresponding values), status, and title. Example:

"silenceTemplates": [
    {
      "description": "Description of the silence template",
      "editable_labels": ["region"],
      "fixed_labels": {
        "name": "Marvin",
      },
      "status": "active",
      "title": "Silence"
    }
  ]

Managing Alertmanager configuration

ref:

• https://prometheus.io/docs/alerting/configuration/#configuration-file
• https://prometheus.io/webtools/alerting/routing-tree-editor/

By default, the Alertmanager instances will start with a minimal configuration which isn’t really useful since it doesn’t send any notification when receiving alerts.

You have multiple options to provide the Alertmanager configuration:

1. You can use alerts.alertmanager.config to define a Alertmanager configuration. Example below.

config:
  global:
    resolve_timeout: 5m
  inhibit_rules:
    - source_matchers:
        - "severity = critical"
      target_matchers:
        - "severity =~ warning|info"
      equal:
        - "namespace"
        - "alertname"
    - source_matchers:
        - "severity = warning"
      target_matchers:
        - "severity = info"
      equal:
        - "namespace"
        - "alertname"
    - source_matchers:
        - "alertname = InfoInhibitor"
      target_matchers:
        - "severity = info"
      equal:
        - "namespace"
  route:
    group_by: ["namespace"]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 12h
    receiver: "null"
    routes:
      - receiver: "null"
        matchers:
          - alertname =~ "InfoInhibitor|Watchdog"
  receivers:
    - name: "null"
  templates:
    - "/etc/alertmanager/config/*.tmpl"

2. You can discover AlertmanagerConfig objects. The spec.alertmanagerConfigSelector is always set to matchLabels: plugin: <name> to tell the operator which AlertmanagerConfigs objects should be selected and merged with the main Alertmanager configuration. Note: The default strategy for a AlertmanagerConfig object to match alerts is OnNamespace.

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: config-example
  labels:
    alertmanagerConfig: example
    pluginDefinition: alerts-example
spec:
  route:
    groupBy: ["job"]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: "webhook"
  receivers:
    - name: "webhook"
      webhookConfigs:
        - url: "http://example.com/"

3. You can use alerts.alertmanager.alertmanagerSpec.alertmanagerConfiguration to reference an AlertmanagerConfig object in the same namespace which defines the main Alertmanager configuration.

# Example with select a global alertmanagerconfig
alertmanagerConfiguration:
  name: global-alertmanager-configuration

TLS Certificate Requirement

Greenhouse onboarded Prometheus installations need to communicate with the Alertmanager component to enable processing of alerts. If an Alertmanager Ingress is enabled, this requires a TLS certificate to be configured and trusted by Alertmanger to ensure the communication. To enable automatic self-signed TLS certificate provisioning via cert-manager, set the alerts.certManager.enabled value to true.

Note: Prerequisite of this feature is a installed jetstack/cert-manager which can be implemented via the Greenhouse cert-manager Plugin.

Examples

Deploy alerts with Alertmanager

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: alerts
spec:
  pluginDefinition: alerts
  disabled: false
  displayName: Alerts
  optionValues:
    - name: alerts.alertmanager.enabled
      value: true
    - name: alerts.alertmanager.ingress.enabled
      value: true
    - name: alerts.alertmanager.ingress.hosts
      value:
        - alertmanager.dns.example.com
    - name: alerts.alertmanager.ingress.tls
      value:
        - hosts:
            - alertmanager.dns.example.com
          secretName: tls-alertmanager-dns-example-com
    - name: alerts.alertmanagerConfig.slack.routes
      value:
        - channel: slack-warning-channel
          webhookURL: https://hooks.slack.com/services/some-id
          matchers:
            - name: severity
              matchType: "="
              value: "warning"
        - channel: slack-critical-channel
          webhookURL: https://hooks.slack.com/services/some-id
          matchers:
            - name: severity
              matchType: "="
              value: "critical"
    - name: alerts.alertmanagerConfig.webhook.routes
      value:
        - name: webhook-route
          url: https://some-webhook-url
          matchers:
            - name: alertname
              matchType: "=~"
              value: ".*"
    - name: alerts.alertmanager.serviceMonitor.additionalLabels
      value:
        plugin: kube-monitoring
    - name: alerts.defaultRules.create
      value: true
    - name: alerts.defaultRules.labels
      value:
        plugin: kube-monitoring
    - name: endpoint
      value: https://alertmanager.dns.example.com/api/v2
    - name: filterLabels
      value:
        - job
        - severity
        - status
    - name: silenceExcludedLabels
      value:
        - pod
        - pod_name
        - instance

Deploy alerts without Alertmanager (Bring your own Alertmanager - Supernova UI only)

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: alerts
spec:
  pluginDefinition: alerts
  disabled: false
  displayName: Alerts
  optionValues:
    - name: alerts.alertmanager.enabled
      value: false
    - name: alerts.alertmanager.ingress.enabled
      value: false
    - name: alerts.defaultRules.create
      value: false
    - name: endpoint
      value: https://alertmanager.dns.example.com/api/v2
    - name: filterLabels
      value:
        - job
        - severity
        - status
    - name: silenceExcludedLabels
      value:
        - pod
        - pod_name
        - instance

2.2 - Audit Logs Plugin

Learn more about the Audit Logs Plugin. Use it to enable the ingestion, collection and export of telemetry signals (logs and metrics) for your Greenhouse cluster.

The main terminologies used in this document can be found in core-concepts.

Overview

OpenTelemetry is an observability framework and toolkit for creating and managing telemetry data such as metrics, logs and traces. Unlike other observability tools, OpenTelemetry is vendor and tool agnostic, meaning it can be used with a variety of observability backends, including open source tools such as OpenSearch and Prometheus.

The focus of the Plugin is to provide easy-to-use configurations for common use cases of receiving, processing and exporting telemetry data in Kubernetes. The storage and visualization of the same is intentionally left to other tools.

Components included in this Plugin:

• Collector
• Receivers
  • Filelog Receiver
  • k8sevents Receiver
  • journald Receiver
  • prometheus/internal
• Connector
• OpenSearch Exporter

Architecture

Note

It is the intention to add more configuration over time and contributions of your very own configuration is highly appreciated. If you discover bugs or want to add functionality to the Plugin, feel free to create a pull request.

Quick Start

This guide provides a quick and straightforward way to use OpenTelemetry for Logs as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

• A running and Greenhouse-onboarded Kubernetes cluster. If you don’t have one, follow the Cluster onboarding guide.
• For logs, a OpenSearch instance to store. If you don’t have one, reach out to your observability team to get access to one.
• We recommend a running cert-manager in the cluster before installing the Logs Plugin
• To gather metrics, you must have a Prometheus instance in the onboarded cluster for storage and for managing Prometheus specific CRDs. If you don not have an instance, install the kube-monitoring Plugin first.
• The Audit Logs Plugin currently requires the OpenTelemetry Operator bundled in the Logs Plugin to be installed in the same cluster beforehand. This is a technical limitation of the Audit Logs Plugin and will be removed in future releases.

Step 1:

You can install the Logs package in your cluster by installing it with Helm manually or let the Greenhouse platform lifecycle do it for you automatically. For the latter, you can either:

1. Go to Greenhouse dashboard and select the Logs Plugin from the catalog. Specify the cluster and required option values.
2. Create and specify a Plugin resource in your Greenhouse central cluster according to the examples.

Step 2:

The package will deploy the OpenTelemetry collectors and auto-instrumentation of the workload. By default, the package will include a configuration for collecting metrics and logs. The log-collector is currently processing data from the preconfigured receivers:

• Files via the Filelog Receiver
• Kubernetes Events from the Kubernetes API server
• Journald events from systemd journal
• its own metrics

Based on the backend selection the telemetry data will be exporter to the backend.

Failover Connector

The Logs Plugin comes with a Failover Connector for OpenSearch for two users. The connector will periodically try to establish a stable connection for the prefered user (failover_username_a) and in case of a failed try, the connector will try to establish a connection with the fallback user (failover_username_b). This feature can be used to secure the shipping of logs in case of expiring credentials or password rotation.

Values

Examples

TBD

2.3 - Cert-manager

This Plugin provides the cert-manager to automate the management of TLS certificates.

Configuration

This section highlights configuration of selected Plugin features.
All available configuration options are described in the plugin.yaml.

Ingress shim

An Ingress resource in Kubernetes configures external access to services in a Kubernetes cluster.
Securing ingress resources with TLS certificates is a common use-case and the cert-manager can be configured to handle these via the ingress-shim component.
It can be enabled by deploying an issuer in your organization and setting the following options on this plugin.

2.4 - Decentralized Observer of Policies (Violations)

This directory contains the Greenhouse plugin for the Decentralized Observer of Policies (DOOP).

DOOP

To perform automatic validations on Kubernetes objects, we run a deployment of OPA Gatekeeper in each cluster. This dashboard aggregates all policy violations reported by those Gatekeeper instances.

2.5 - Designate Ingress CNAME operator (DISCO)

This Plugin provides the Designate Ingress CNAME operator (DISCO) to automate management of DNS entries in OpenStack Designate for Ingress and Services in Kubernetes.

2.6 - DigiCert issuer

This Plugin provides the digicert-issuer, an external Issuer extending the cert-manager with the DigiCert cert-central API.

2.7 - External DNS

This Plugin provides the external DNS operator) which synchronizes exposed Kubernetes Services and Ingresses with DNS providers.

2.8 - Ingress NGINX

This plugin contains the ingress NGINX controller.

Example

To instantiate the plugin create a Plugin like:

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: ingress-nginx
spec:
  pluginDefinition: ingress-nginx-v4.4.0
  values:
    - name: controller.service.loadBalancerIP
      value: 1.2.3.4

2.9 - Kubernetes Monitoring

Learn more about the kube-monitoring plugin. Use it to activate Kubernetes monitoring for your Greenhouse cluster.

The main terminologies used in this document can be found in core-concepts.

Overview

Observability is often required for operation and automation of service offerings. To get the insights provided by an application and the container runtime environment, you need telemetry data in the form of metrics or logs sent to backends such as Prometheus or OpenSearch. With the kube-monitoring Plugin, you will be able to cover the metrics part of the observability stack.

This Plugin includes a pre-configured package of components that help make getting started easy and efficient. At its core, an automated and managed Prometheus installation is provided using the prometheus-operator. This is complemented by Prometheus target configuration for the most common Kubernetes components providing metrics by default. In addition, Cloud operators curated Prometheus alerting rules and Plutono dashboards are included to provide a comprehensive monitoring solution out of the box.

Components included in this Plugin:

• Prometheus
• Prometheus Operator
• Prometheus target configuration for Kubernetes metrics APIs (e.g. kubelet, apiserver, coredns, etcd)
• Prometheus node exporter
• kube-state-metrics
• kubernetes-operations

Disclaimer

It is not meant to be a comprehensive package that covers all scenarios. If you are an expert, feel free to configure the plugin according to your needs.

The Plugin is a deeply configured kube-prometheus-stack Helm chart which helps to keep track of versions and community updates.

It is intended as a platform that can be extended by following the guide.

Contribution is highly appreciated. If you discover bugs or want to add functionality to the plugin, then pull requests are always welcome.

Quick start

This guide provides a quick and straightforward way to use kube-monitoring as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

• A running and Greenhouse-onboarded Kubernetes cluster. If you don’t have one, follow the Cluster onboarding guide.

Step 1:

You can install the kube-monitoring package in your cluster by installing it with Helm manually or let the Greenhouse platform lifecycle it for you automatically. For the latter, you can either:

1. Go to Greenhouse dashboard and select the Kubernetes Monitoring plugin from the catalog. Specify the cluster and required option values.
2. Create and specify a Plugin resource in your Greenhouse central cluster according to the examples.

Step 2:

After installation, Greenhouse will provide a generated link to the Prometheus user interface. This is done via the annotation greenhouse.sap/expose: “true” at the Prometheus Service resource.

Step 3:

Greenhouse regularly performs integration tests that are bundled with kube-monitoring. These provide feedback on whether all the necessary resources are installed and continuously up and running. You will find messages about this in the plugin status and also in the Greenhouse dashboard.

Values

absent-metrics-operator options

Alertmanager options

Blackbox exporter config

Global options

Kubernetes component scraper options

Prometheus options

Prometheus-operator options

Absent-metrics-operator

The kube-monitoring Plugin can optionally deploy and configure the absent-metrics-operator to help detect missing or absent metrics in your Prometheus setup. This operator automatically generates alerts when expected metrics are not present, improving observability and alerting coverage.

Service Discovery

The kube-monitoring Plugin provides a PodMonitor to automatically discover the Prometheus metrics of the Kubernetes Pods in any Namespace. The PodMonitor is configured to detect the metrics endpoint of the Pods if the following annotations are set:

metadata:
  annotations:
    greenhouse/scrape: “true”
    greenhouse/target: <kube-monitoring plugin name>

Note: The annotations needs to be added manually to have the pod scraped and the port name needs to match.

Examples

Deploy kube-monitoring into a remote cluster

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: kube-monitoring
spec:
  pluginDefinition: kube-monitoring
  disabled: false
  optionValues:
    - name: kubeMonitoring.prometheus.prometheusSpec.retention
      value: 30d
    - name: kubeMonitoring.prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage
      value: 100Gi
    - name: kubeMonitoring.prometheus.service.labels
      value:
        greenhouse.sap/expose: "true"
    - name: kubeMonitoring.prometheus.prometheusSpec.externalLabels
      value:
        cluster: example-cluster
        organization: example-org
        region: example-region
    - name: alerts.enabled
      value: true
    - name: alerts.alertmanagers.hosts
      value:
        - alertmanager.dns.example.com
    - name: alerts.alertmanagers.tlsConfig.cert
      valueFrom:
        secret:
          key: tls.crt
          name: tls-<org-name>-prometheus-auth
    - name: alerts.alertmanagers.tlsConfig.key
      valueFrom:
        secret:
          key: tls.key
          name: tls-<org-name>-prometheus-auth

Deploy Prometheus only

Example Plugin to deploy Prometheus with the kube-monitoring Plugin.

NOTE: If you are using kube-monitoring for the first time in your cluster, it is necessary to set kubeMonitoring.prometheusOperator.enabled to true.

apiVersion: greenhouse.sap/v1alpha1
kind: Plugin
metadata:
  name: example-prometheus-name
spec:
  pluginDefinition: kube-monitoring
  disabled: false
  optionValues:
    - name: kubeMonitoring.defaultRules.create
      value: false
    - name: kubeMonitoring.kubernetesServiceMonitors.enabled
      value: false
    - name: kubeMonitoring.prometheusOperator.enabled
      value: false
    - name: kubeMonitoring.kubeStateMetrics.enabled
      value: false
    - name: kubeMonitoring.nodeExporter.enabled
      value: false
    - name: kubeMonitoring.prometheus.prometheusSpec.retention
      value: 30d
    - name: kubeMonitoring.prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage
      value: 100Gi
    - name: kubeMonitoring.prometheus.service.labels
      value:
        greenhouse.sap/expose: "true"
    - name: kubeMonitoring.prometheus.prometheusSpec.externalLabels
      value:
        cluster: example-cluster
        organization: example-org
        region: example-region
    - name: alerts.enabled
      value: true
    - name: alerts.alertmanagers.hosts
      value:
        - alertmanager.dns.example.com
    - name: alerts.alertmanagers.tlsConfig.cert
      valueFrom:
        secret:
          key: tls.crt
          name: tls-<org-name>-prometheus-auth
    - name: alerts.alertmanagers.tlsConfig.key
      valueFrom:
        secret:
          key: tls.key
          name: tls-<org-name>-prometheus-auth

Extension of the plugin

kube-monitoring can be extended with your own Prometheus alerting rules and target configurations via the Custom Resource Definitions (CRDs) of the Prometheus operator. The user-defined resources to be incorporated with the desired configuration are defined via label selections.

The CRD PrometheusRule enables the definition of alerting and recording rules that can be used by Prometheus or Thanos Rule instances. Alerts and recording rules are reconciled and dynamically loaded by the operator without having to restart Prometheus or Thanos Rule.

kube-monitoring Prometheus will automatically discover and load the rules that match labels plugin: <plugin-name>.

Example:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: example-prometheus-rule
  labels:
    plugin: <metadata.name>
    ## e.g plugin: kube-monitoring
spec:
 groups:
   - name: example-group
     rules:
     ...

The CRDs PodMonitor, ServiceMonitor, Probe and ScrapeConfig allow the definition of a set of target endpoints to be scraped by Prometheus. The operator will automatically discover and load the configurations that match labels plugin: <plugin-name>.

Example:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: example-pod-monitor
  labels:
    plugin: <metadata.name>
    ## e.g plugin: kube-monitoring
spec:
  selector:
    matchLabels:
      app: example-app
  namespaceSelector:
    matchNames:
      - example-namespace
  podMetricsEndpoints:
    - port: http
  ...

2.10 - Logs Plugin

Learn more about the Logs Plugin. Use it to enable the ingestion, collection and export of telemetry signals (logs and metrics) for your Greenhouse cluster.

The main terminologies used in this document can be found in core-concepts.

Overview

OpenTelemetry is an observability framework and toolkit for creating and managing telemetry data such as metrics, logs and traces. Unlike other observability tools, OpenTelemetry is vendor and tool agnostic, meaning it can be used with a variety of observability backends, including open source tools such as OpenSearch and Prometheus.

The focus of the Plugin is to provide easy-to-use configurations for common use cases of receiving, processing and exporting telemetry data in Kubernetes. The storage and visualization of the same is intentionally left to other tools.

Components included in this Plugin:

• Operator
• Collector
• Receivers
  • Filelog Receiver
  • k8sevents Receiver
  • journald Receiver
  • prometheus/internal
• Connector
• OpenSearch Exporter

Architecture

Note

It is the intention to add more configuration over time and contributions of your very own configuration is highly appreciated. If you discover bugs or want to add functionality to the Plugin, feel free to create a pull request.

Quick Start

This guide provides a quick and straightforward way to use OpenTelemetry for Logs as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

• A running and Greenhouse-onboarded Kubernetes cluster. If you don’t have one, follow the Cluster onboarding guide.
• For logs, a OpenSearch instance to store. If you don’t have one, reach out to your observability team to get access to one.
• We recommend a running cert-manager in the cluster before installing the Logs Plugin
• To gather metrics, you must have a Prometheus instance in the onboarded cluster for storage and for managing Prometheus specific CRDs. If you don not have an instance, install the kube-monitoring Plugin first.

Step 1:

You can install the Logs package in your cluster by installing it with Helm manually or let the Greenhouse platform lifecycle do it for you automatically. For the latter, you can either:

1. Go to Greenhouse dashboard and select the Logs Plugin from the catalog. Specify the cluster and required option values.
2. Create and specify a Plugin resource in your Greenhouse central cluster according to the examples.

Step 2:

The package will deploy the OpenTelemetry Operator which works as a manager for the collectors and auto-instrumentation of the workload. By default, the package will include a configuration for collecting metrics and logs. The log-collector is currently processing data from the preconfigured receivers:

• Files via the Filelog Receiver
• Kubernetes Events from the Kubernetes API server
• Journald events from systemd journal
• its own metrics

You can disable the collection of logs by setting openTelemetry.logCollector.enabled to false. The same is true for disabling the collection of metrics by setting openTelemetry.metricsCollector.enabled to false. The logsCollector comes with a standard set of log-processing, such as adding cluster information and common labels for Journald events. In addition we provide default pipelines for common log types. Currently the following log types have default configurations that can be enabled (requires logsCollector.enabled to true):

1. KVM: openTelemetry.logsCollector.kvmConfig: Logs from Kernel-based Virtual Machines (KVMs) providing insights into virtualization activities, resource usage, and system performance
2. Ceph:openTelemetry.logsCollector.cephConfig: Logs from Ceph storage systems, capturing information about cluster operations, performance metrics, and health status

These default configurations provide common labels and Grok parsing for logs emitted through the respective services.

Based on the backend selection the telemetry data will be exporter to the backend.

Step 3:

Greenhouse regularly performs integration tests that are bundled with the Logs Plugin. These provide feedback on whether all the necessary resources are installed and continuously up and running. You will find messages about this in the Plugin status and also in the Greenhouse dashboard.

Failover Connector

The Logs Plugin comes with a Failover Connector for OpenSearch for two users. The connector will periodically try to establish a stable connection for the prefered user (failover_username_a) and in case of a failed try, the connector will try to establish a connection with the fallback user (failover_username_b). This feature can be used to secure the shipping of logs in case of expiring credentials or password rotation.

Values

Examples

TBD

2.11 - Logshipper

This Plugin is intended for shipping container and systemd logs to an Elasticsearch/ OpenSearch cluster. It uses fluentbit to collect logs. The default configuration can be found under chart/templates/fluent-bit-configmap.yaml.

Components included in this Plugin:

• fluentbit

Owner

1. @ivogoman

Parameters

Custom Configuration

To add custom configuration to the fluent-bit configuration please check the fluentbit documentation here. The fluent-bit.customConfig.inputs, fluent-bit.customConfig.filters and fluent-bit.customConfig.outputs parameters can be used to add custom configuration to the default configuration. The configuration should be added as a multi-line string. Inputs are rendered after the default inputs, filters are rendered after the default filters and before the additional values are added. Outputs are rendered after the default outputs. The additional values are added to all logs disregaring the source.

Example Input configuration:

fluent-bit:
  config:
    inputs: |
      [INPUT]
          Name             tail-audit
          Path             /var/log/containers/greenhouse-controller*.log
          Parser           {{ default "cri" ( index .Values "fluent-bit" "parser" ) }}
          Tag              audit.*
          Refresh_Interval 5
          Mem_Buf_Limit    50MB
          Skip_Long_Lines  Off
          Ignore_Older     1m
          DB               /var/log/fluent-bit-tail-audit.pos.db

Logs collected by the default configuration are prefixed with default_. In case that logs from additional inputs are to be send and processed by the same filters and outputs, the prefix should be used as well.

In case additional secrets are required the fluent-bit.env field can be used to add them to the environment of the fluent-bit container. The secrets should be created by adding them to the fluent-bit.backend field.

fluent-bit:
  backend:
    audit:
      http_user: top-secret-audit
      http_password: top-secret-audit
      host: "audit.test"
      tls:
        enabled: true
        verify: true
        debug: false

2.12 - OpenSearch

OpenSearch Plugin

The OpenSearch plugin sets up an OpenSearch environment using the OpenSearch Operator, automating deployment, provisioning, management, and orchestration of OpenSearch clusters and dashboards. It functions as the backend for logs gathered by collectors such as OpenTelemetry collectors, enabling storage and visualization of logs for Greenhouse-onboarded Kubernetes clusters.

The main terminologies used in this document can be found in core-concepts.

Overview

OpenSearch is a distributed search and analytics engine designed for real-time log and event data analysis. The OpenSearch Operator simplifies the management of OpenSearch clusters by providing declarative APIs for configuration and scaling.

Components included in this Plugin:

• OpenSearch Operator
• OpenSearch Cluster Management
• OpenSearch Dashboards Deployment
• OpenSearch Index Management
• OpenSearch Security Configuration

Architecture

The OpenSearch Operator automates the management of OpenSearch clusters within a Kubernetes environment. The architecture consists of:

• OpenSearchCluster CRD: Defines the structure and configuration of OpenSearch clusters, including node roles, scaling policies, and version management.
• OpenSearchDashboards CRD: Manages OpenSearch Dashboards deployments, ensuring high availability and automatic upgrades.
• OpenSearchISMPolicy CRD: Implements index lifecycle management, defining policies for retention, rollover, and deletion.
• OpenSearchIndexTemplate CRD: Enables the definition of index mappings, settings, and template structures.
• Security Configuration via OpenSearchRole and OpenSearchUser: Manages authentication and authorization for OpenSearch users and roles.

Note

More configurations will be added over time, and contributions of custom configurations are highly appreciated. If you discover bugs or want to add functionality to the plugin, feel free to create a pull request.

Quick Start

This guide provides a quick and straightforward way to use OpenSearch as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

• A running and Greenhouse-onboarded Kubernetes cluster. If you don’t have one, follow the Cluster onboarding guide.
• The OpenSearch Operator installed via Helm or Kubernetes manifests.
• An OpenTelemetry or similar log ingestion pipeline configured to send logs to OpenSearch.

Installation

Install via Greenhouse

1. Navigate to the Greenhouse Dashboard.
2. Select the OpenSearch plugin from the catalog.
3. Specify the target cluster and configuration options.

Values

Usage

Once deployed, OpenSearch can be accessed via OpenSearch Dashboards.

kubectl port-forward svc/opensearch-dashboards 5601:5601

Visit http://localhost:5601 in your browser and log in using the configured credentials.

Conclusion

This guide ensures that OpenSearch is fully integrated into the Greenhouse ecosystem, providing scalable log management and visualization. Additional custom configurations can be introduced to meet specific operational needs.

For troubleshooting and further details, check out the OpenSearch documentation.

2.13 - Perses

Table of Contents

• Table of Contents
• Overview
• Disclaimer
• Quick Start
• Configuration
• Create a custom dashboard
• Add Dashboards as ConfigMaps
  • Recommended folder structure

Learn more about the Perses Plugin. Use it to visualize Prometheus/Thanos metrics for your Greenhouse remote cluster.

The main terminologies used in this document can be found in core-concepts.

Overview

Observability is often required for the operation and automation of service offerings. Perses is a CNCF project and it aims to become an open-standard for dashboards and visualization. It provides you with tools to display Prometheus metrics on live dashboards with insightful charts and visualizations. In the Greenhouse context, this complements the kube-monitoring plugin, which automatically acts as a Perses data source which is recognized by Perses. In addition, the Plugin provides a mechanism that automates the lifecycle of datasources and dashboards without having to restart Perses.

Disclaimer

This is not meant to be a comprehensive package that covers all scenarios. If you are an expert, feel free to configure the Plugin according to your needs.

Contribution is highly appreciated. If you discover bugs or want to add functionality to the plugin, then pull requests are always welcome.

Quick Start

This guide provides a quick and straightforward way how to use Perses as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

• A running and Greenhouse-managed Kubernetes remote cluster
• kube-monitoring Plugin will integrate into Perses automatically with its own datasource
• thanos Plugin can be enabled alongside kube-monitoring. Perses then will have both datasources (thanos, kube-monitoring) and will default to thanos to provide access to long term metrics

The plugin works by default with anonymous access enabled. This plugin comes with some default dashboards and datasources will be automatically discovered by the plugin.

Step 1: Add your dashboards and datasources

Dashboards are selected from ConfigMaps across namespaces. The plugin searches for ConfigMaps with the label perses.dev/resource: "true" and imports them into Perses. The ConfigMap must contain a key like my-dashboard.json with the dashboard JSON content. Please refer this section for more information.

A guide on how to create custom dashboards on the UI can be found here.

Values

Create a custom dashboard

1. Add a new Project by clicking on ADD PROJECT in the top right corner. Give it a name and click Add.
2. Add a new dashboard by clicking on ADD DASHBOARD. Give it a name and click Add.
3. Now you can add variables, panels to your dashboard.
4. You can group your panels by adding the panels to a Panel Group.
5. Move and resize the panels as needed.
6. Watch this gif to learn more.
7. You do not need to add the kube-monitoring datasource manually. It will be automatically discovered by Perses.
8. Click Save after you have made changes.
9. Export the dashboard.
   • Click on the {} icon in the top right corner of the dashboard.
   • Copy the entire JSON model.
   • See the next section for detailed instructions on how and where to paste the copied dashboard JSON model.

Dashboard-as-Code

Perses offers the possibility to define dashboards as code (DaC) instead of going through manipulations on the UI. But why would you want to do this? Basically Dashboard-as-Code (DaC) is something that becomes useful at scale, when you have many dashboards to maintain, to keep aligned on certain parts, etc. If you are interested in this, you can check the Perses documentation for more information.

Add Dashboards as ConfigMaps

By default, a sidecar container is deployed in the Perses pod. This container watches all configmaps in the cluster and filters out the ones with a label perses.dev/resource: "true". The files defined in those configmaps are written to a folder and this folder is accessed by Perses. Changes to the configmaps are continuously monitored and are reflected in Perses within 10 minutes.

A recommendation is to use one configmap per dashboard. This way, you can easily manage the dashboards in your git repository.

Recommended folder structure

Folder structure:

dashboards/
├── dashboard1.json
├── dashboard2.json
├── prometheusdatasource1.json
├── prometheusdatasource2.json
templates/
├──dashboard-json-configmap.yaml

Helm template to create a configmap for each dashboard:

{{- range $path, $bytes := .Files.Glob "dashboards/*.json" }}
---
apiVersion: v1
kind: ConfigMap

metadata:
  name: {{ printf "%s-%s" $.Release.Name $path | replace "/" "-" | trunc 63 }}
  labels:
    perses.dev/resource: "true"

data:
{{ printf "%s: |-" $path | replace "/" "-" | indent 2 }}
{{ printf "%s" $bytes | indent 4 }}

{{- end }}

2.14 - Plutono

Learn more about the plutono Plugin. Use it to install the web dashboarding system Plutono to collect, correlate, and visualize Prometheus metrics for your Greenhouse cluster.

The main terminologies used in this document can be found in core-concepts.

Overview

Observability is often required for the operation and automation of service offerings. Plutono provides you with tools to display Prometheus metrics on live dashboards with insightful charts and visualizations. In the Greenhouse context, this complements the kube-monitoring plugin, which automatically acts as a Plutono data source which is recognized by Plutono. In addition, the Plugin provides a mechanism that automates the lifecycle of datasources and dashboards without having to restart Plutono.

Disclaimer

This is not meant to be a comprehensive package that covers all scenarios. If you are an expert, feel free to configure the Plugin according to your needs.

Contribution is highly appreciated. If you discover bugs or want to add functionality to the plugin, then pull requests are always welcome.

Quick Start

This guide provides a quick and straightforward way how to use Plutono as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

• A running and Greenhouse-managed Kubernetes cluster
• kube-monitoring Plugin installed to have at least one Prometheus instance running in the cluster

The plugin works by default with anonymous access enabled. If you use the standard configuration in the kube-monitoring plugin, the data source and some kubernetes-operations dashboards are already pre-installed.

Step 1: Add your dashboards

Dashboards are selected from ConfigMaps across namespaces. The plugin searches for ConfigMaps with the label plutono-dashboard: "true" and imports them into Plutono. The ConfigMap must contain a key like my-dashboard.json with the dashboard JSON content. Example

A guide on how to create dashboards can be found here.

Step 2: Add your datasources

Data sources are selected from Secrets across namespaces. The plugin searches for Secrets with the label plutono-dashboard: "true" and imports them into Plutono. The Secrets should contain valid datasource configuration YAML. Example

Values