Logs Plugin

Learn more about the Logs Plugin. Use it to enable the ingestion, collection and export of telemetry signals (logs and metrics) for your Greenhouse cluster.

The main terminologies used in this document can be found in core-concepts.

Overview

OpenTelemetry is an observability framework and toolkit for creating and managing telemetry data such as metrics, logs and traces. Unlike other observability tools, OpenTelemetry is vendor and tool agnostic, meaning it can be used with a variety of observability backends, including open source tools such as OpenSearch and Prometheus.

The focus of the Plugin is to provide easy-to-use configurations for common use cases of receiving, processing and exporting telemetry data in Kubernetes. The storage and visualization of the same is intentionally left to other tools.

Components included in this Plugin:

• Operator
• Collector
• Receivers
  • Filelog Receiver
  • k8sevents Receiver
  • journald Receiver
  • prometheus/internal
• Connector
• OpenSearch Exporter

Architecture

Note

It is the intention to add more configuration over time and contributions of your very own configuration is highly appreciated. If you discover bugs or want to add functionality to the Plugin, feel free to create a pull request.

Quick Start

This guide provides a quick and straightforward way to use OpenTelemetry for Logs as a Greenhouse Plugin on your Kubernetes cluster.

Prerequisites

• A running and Greenhouse-onboarded Kubernetes cluster. If you don’t have one, follow the Cluster onboarding guide.
• For logs, a OpenSearch instance to store. If you don’t have one, reach out to your observability team to get access to one.
• We recommend a running cert-manager in the cluster before installing the Logs Plugin
• To gather metrics, you must have a Prometheus instance in the onboarded cluster for storage and for managing Prometheus specific CRDs. If you don not have an instance, install the kube-monitoring Plugin first.

Step 1:

You can install the Logs package in your cluster by installing it with Helm manually or let the Greenhouse platform lifecycle do it for you automatically. For the latter, you can either:

1. Go to Greenhouse dashboard and select the Logs Plugin from the catalog. Specify the cluster and required option values.
2. Create and specify a Plugin resource in your Greenhouse central cluster according to the examples.

Step 2:

The package will deploy the OpenTelemetry Operator which works as a manager for the collectors and auto-instrumentation of the workload. By default, the package will include a configuration for collecting metrics and logs. The log-collector is currently processing data from the preconfigured receivers:

• Files via the Filelog Receiver
• Kubernetes Events from the Kubernetes API server
• Journald events from systemd journal
• its own metrics

You can disable the collection of logs by setting openTelemetry.logCollector.enabled to false. The same is true for disabling the collection of metrics by setting openTelemetry.metricsCollector.enabled to false. The logsCollector comes with a standard set of log-processing, such as adding cluster information and common labels for Journald events. In addition we provide default pipelines for common log types. Currently the following log types have default configurations that can be enabled (requires logsCollector.enabled to true):

1. KVM: openTelemetry.logsCollector.kvmConfig: Logs from Kernel-based Virtual Machines (KVMs) providing insights into virtualization activities, resource usage, and system performance
2. Ceph:openTelemetry.logsCollector.cephConfig: Logs from Ceph storage systems, capturing information about cluster operations, performance metrics, and health status

These default configurations provide common labels and Grok parsing for logs emitted through the respective services.

Based on the backend selection the telemetry data will be exporter to the backend.

Step 3:

Greenhouse regularly performs integration tests that are bundled with the Logs Plugin. These provide feedback on whether all the necessary resources are installed and continuously up and running. You will find messages about this in the Plugin status and also in the Greenhouse dashboard.

Failover Connector

The Logs Plugin comes with a Failover Connector for OpenSearch for two users. The connector will periodically try to establish a stable connection for the prefered user (failover_username_a) and in case of a failed try, the connector will try to establish a connection with the fallback user (failover_username_b). This feature can be used to secure the shipping of logs in case of expiring credentials or password rotation.

Values

Examples

TBD