Local development setup

What is Greenhouse?

Greenhouse is a Kubernetes operator build with Kubebuilder and a UI on top of the k8s API.

It expands the Kubernetes API via CustomResourceDefinitions. The different aspects of the CRDs are reconciled by several controllers. It also acts as an admission webhook.

The Greenhouse Dashboard is a UI acting on the k8s apiserver of the cluster Greenhouse is running in. It includes a dashboard and an Organization admin consisting of several Juno micro frontends.

This guide provides the following:

1. Local Setup

   1.1 Mock k8s Server

   1.2 Greenhouse Controller

   1.3 Greenhouse UI

   1.4 docker compose

   1.5 Bootstrap

2. Run And Debug The Code

3. Run The Tests

Local Setup

Quick start the local setup with docker compose

Note: As for the time being, the images published in our registry are linux/amd64 only. Export env var to set your docker to use this architecture, if your default differs:

export DOCKER_DEFAULT_PLATFORM=linux/amd64

Env Var Overview

Mock k8s Server, a.k.a. envtest

The Greenhouse controller needs a Kubernetes API to run it’s reconciliation against. This k8s API needs to know about the Greenhouse CRDs to maintain the state of their respective resources. It also needs to know about any running admission/validation webhooks.

We provide a local mock k8s apiserver and etcd leveraging the envtest package of SIG controller-runtime. This comes with the CRDs and MutatingWebhookConfiguration installed and provides a little bit of utility. Find the docker image on our registry.

Additionally it will bootstrap some users with different permissions in a test-org. The test-org resource does not yet exist on the apiserver, but can be bootstrapped from the test-org.yaml which is done for you if you use the docker compose setup.

Running the image will:

• spin up the apiserver and etcd
• deploy CRDs and the webhook
• create some users with respective contexts and certificates
• finally proxy the apiserver via kubectl proxy to 127.0.0.1:8090.

The latter is done to avoid painful authentication to the local apiserver.

We still can showcase different permission levels on the apiserver, by setting context via the env var DEV_ENV_CONTEXT.

To access the running apiserver instance, some kubeconfig files and client certificates are created on the container in the /envtest folder.

The internal.kubeconfig file uses the different certificates and contextes to directly address the apiserver running on port 6884.

The kubeconfig file uses the proxied context without authentication running on port 8090. It is also scoped to the namespace test-org.

Chose the respective ports to be exposed on your localhost when running the image or expose them all by running in network host mode.

We are reusing the autogenerated certificates of the dev-env for authenticating the webhook server on localhost. The files are stored on /webhook-certs on the container.

It is good practice to mount local volumes to these folders, running the image as such:

docker run --network host -e DEV_ENV_CONTEXT=<your-context> -v ./envtest:/envtest -v /tmp/k8s-webhook-server/serving-certs:/webhook-certs  ghcr.io/cloudoperators/greenhouse-dev-env:main

Greenhouse Controller

Run your local go code from ./cmd/greenhouse with the minimal configuration necessary (this example points the controller to run against the local mock apiserver):

go run . --dns-domain localhost --kubeconfig ./envtest/kubeconfig

Make sure the webhook server certs are placed in /tmp/k8s-webhook-server/serving-certs

Or run our greenhouse image as such:

docker run --network host -e KUBECONFIG=/envtest/kubeconfig -v ./envtest:/envtest -v /tmp/k8s-webhook-server/serving-certs:/tmp/k8s-webhook-server/serving-certs ghcr.io/cloudoperators/greenhouse:main --dns-domain localhost

See all available flags here.

Note that not setting headscaleAPIURL, headscaleAPIKey or tailscaleProxy will prevent running the headscale access controller.

Greenhouse UI

Build the latest dev-ui image by running:

$ docker build --platform linux/amd64 -t ghcr.io/cloudoperators/greenhouse-dev-ui:main -f Dockerfile.dev-ui  .

To run the Greenhouse UI locally you can either run the docker image exposing port 3000 or in host network:

docker run --network host ghcr.io/cloudoperators/greenhouse-dev-ui:main

or run the node process itself via npm (both dependencies need to be installed locally). We use the props template prepared for dev-env to run the app: via:

cp ./dev-env/build/greenhouse-ui/secretProps.json ./ui/dashboard/secretProps.json
npm start --prefix ./ui/dashboard

As you might have noticed we inject a props template prepared for dev-env expecting the k8s api to run on 127.0.0.1:8090, which is the default exported by the mock api server image. Also authentication will be mocked.

Have a look at the props template to point your local UI to other running Greenhouse instances.

Access the UI on localhost:3000.

Run the UI on a different port by exposing env var APP_PORT:

APP_PORT=<your-port> npm start --prefix ./ui/dashboard

Note: Running the code locally only watches and live reloads the local code (changes) of the dashboard micro frontend (MFE). This is not true for the embedded MFEs. Run those separately with respective props pointing to the mock k8s apiserver for development.

docker compose

If you do not need or want to run your local code but want to run a set of Greenhouse images we provide a setup with docker compose and can be started via

docker compose up

from the ./dev-env folder.

You might need to build the dev-ui image by hand:

docker compose build greenhouse-ui

The network-host.docker-compose.yaml provides the same setup starting all containers in host network mode.

Bootstrap

The docker-compose setup per default bootstraps an Organization test-org to your cluster, which is the bare minimum to get the dev-env working.

Running

docker compose run bootstrap kubectl apply -f /bootstrap/additional_resources

or uncommenting the additional resources in the command of the bootstrap container on the docker-compose file will additionally deploy the following:

• test-team-1 through test-team-3 within Organization test-org
• respective dummy teammemberships for both teams
• cluster-1 through cluster-3 and self with different conditions and states
• some dummy nodes for clusters
• some plugindefinitions with plugins across the clusters

Note: These resources are intended to showcase the UI and produce a lot of “noise” on the Greenhouse controller.

Add any additional resources you need to the ./bootstrap folder.

Run And Debug The Code

Spin up the envtest container only, e.g. via:

docker compose up envtest

Reuse the certs created by envtest for locally serving the webhooks by copying them to the default location kubebuilder expects webhook certs at:

cp ./webhook-certs/* /tmp/k8s-webhook-server/serving-certs

Note: use $TMPDIR on MacOS for /tmp

Start your debugging process in respective IDE exposing the envtest kubeconfig at ./envtest/kubeconfig. Do not forget to pass the --dns-domain=localhost flag.

Run The Tests

For running e2e tests see here.

Same as the local setup our unit tests run against an envtest mock cluster. To install the setup-envtest tool run

make envtest

which will install setup-envtest to your $(LOCALBIN), usually ./bin.

To run all tests from cli:

make test

To run tests independently make sure the $(KUBEBUILDER_ASSETS) env var is set. This variable contains the path to the binary to use for starting up the mock controlplane with the respective k8s version on your architecture.

Print the path by executing:

./bin/setup-envtest use <your-preferred-k8s-version> -p path

Env Vars Overview In Testing