Control planes with Crossplane

In our previous article, we highlighted how Kubernetes could be used as a declarative control system. In this part, we’re focusing on Crossplane, a tool that I find very useful in this context.

When we run applications and services supporting our business goals, we must provision infrastructure such as databases, networks, Cloud accounts, clusters, serverless functions and load balancers. There are a few options for achieving what we need:

Manually create infrastructure using Cloud vendor web portal
Manually create infrastructure using Cloud vendor CLI tool
Automate the creation of infrastructure from code using Cloud vendor API’s

For other than the tiniest projects or testing, the first two options are only feasible in the short term. Manually created infrastructure cannot be versioned, controlled sufficiently, tested, and maintained properly.

There are quite a few tools for automating infrastructure provisioning from code (commonly called IaC or "Infrastructure as Code"). For this article, I will look into Crossplane and draw a comparison against the widely used IaC tool, Hashicorp's Terraform.

Crossplane is, according to their own material, a framework for building cloud-native control planes without needing to write code. It has an extensible backend to manage any infrastructure in any environment, and a configurable frontend to expose declarative APIs for developer self-service.

The concept of a control plane sets Crossplane apart from other traditional IaC tools. The control plane is always running and detecting drift in our infrastructure, applies all changes in real time, and allows us to interact with it using an API.

Let’s try it

Assuming we already have a cluster running, e.g. in Docker Desktop, we can install Crossplane:

# install crossplane
kubectl create namespace crossplane-system
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update

helm install crossplane --namespace crossplane-system crossplane-stable/crossplane

# install kubectl plugin
curl -sL https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh | sh

For this demo, we're going to set an example configuration package with GCP:

kubectl crossplane install configuration registry.upbound.io/xp/getting-started-with-gcp:v1.10.1
# wait to be ready
watch kubectl get pkg

We can see that the configuration package pulled provider-gcp, and we now have a new provider pod running in crossplane-system namespace. Providers are an important concept in Crossplane: they bundle a set of managed resources and their respective controllers to allow Crossplane to provision a particular class of managed resources, in this case resources on top of Google Cloud Platform. Managed resources could be any services or resources that can be provisioned and configured using some form of APIs. You can create your own providers using Golang, but common use cases are already covered by contributed providers.

For the next step, we need to get the GCP account key, and install that on the cluster as a secret and configure GCP provider for Crossplane. I’m not going to detail how to do that here, but you can follow official instructions.

Now with everything set up, we can try to do something. Let’s try to create a storage bucket. We can submit the following Kubernetes resource on our cluster:

apiVersion: storage.gcp.crossplane.io/v1alpha3
kind: Bucket
metadata:
  name: bucket-for-demo
	namespace: default
  labels:
    example: "true"
  annotations:
    # Note that this will be the actual bucket name so it has to be globally unique/available.
    crossplane.io/external-name: bucket-for-demo-gdfg34g3
spec:
  location: EU
  storageClass: MULTI_REGIONAL
  providerConfigRef:
		# this needs to match with provider name created earlier
    name: default
  deletionPolicy: Delete

After some time, we can observe in the console our bucket got created:

kubectl get bucket
NAME              READY   SYNCED   STORAGE_CLASS    LOCATION   AGE
bucket-for-demo   True    True     MULTI_REGIONAL   EU         102s

While this was an simplified example, there are a few fundamental points we can take from this:

We created the bucket from within the Kubernetes namespace using a simple and understandable YAML definition. Imagine we are deploying, let's say, WordPress using a Helm chart: this would allow us to create associated storage buckets directly from the chart template using just Kubernetes definitions and connect it to the application.
On the client side (our laptop), we did not need to run any custom tools. We could get things done by submitting Kubernetes manifests to the Kubernetes API. Even the Crossplane kubectl plugin is optional. Everything resides persistently on the server side.
As Crossplane definitions are Kubernetes manifests, we can plug this into any Kubernetes deployment tooling we prefer, such as Argo-CD, Flux, or plain old kubectl. We can also use any standard templating tooling such as Helm, Kustomize, or Jsonnet.
Definitions themselves are quite verbose but straightforward. There is no custom domain-specific language in Crossplane, unlike in Terraform. When advanced generation from code is desired we're free to choose tooling ourselves, one good option being https://cdk8s.io/
We can apply our choice of policy controller to Crossplane manifests, such as Kyverno, OPA Gatekeeper, or Datree. With these and Kubernetes RBAC it is possible to tell the system which actors are allowed to deploy a certain class of managed resources (new clusters for example), and if their configuration matches what is permitted. We can easily apply business rules, such as “all storage buckets must be located; due to GDPR, within EU”
Because we are dealing with Kubernetes entities and API, it is relatively straightforward to answer questions like "how many databases do we have and who owns them, and what are their versions". All entities can have metadata associated with them. Crossplane entities have standard status field that gives out the state of the entity.

We could even integrate other tooling like Panopticon to expose this data as Prometheus metrics, and collect data about resources that exists, and generate to reports for business needs.

Compositions

With providers and managed resources we learned how Crossplane could deploy Cloud resources. There’s another compelling concept in Crossplane: compositions. With them, Platform Builders can create abstractions that users can consume. We use compositions to hide specific detail and complexity our business and users don’t care, and create higher-level abstractions, like an entity representing entire “WordPress” instance, that, in reality, is composed of managed MySQL database, a storage bucket, and a Helm chart, but only exposes what users should be able to modify in it.

We’re going into detail of compositions in the next series of this article, but it is important to note they are major feature and benefit of Crossplane.

Comparison to Terraform

These tools are quite different, while they can do same things. It is worth noting Crossplane has Terraform provider, so everything Terraform can do, Crossplane can also do.

One major drawback of Crossplane is that it requires a running Kubernetes cluster, to which its controllers are installed. Of course, Terraform also needs to be executed from somewhere, but that is often a simpler setup in CI pipeline, or even developer’s own machine. With Crossplane how does the “management cluster”, or the first cluster get bootstrapped? Should it be hand-crafted or created using another IaC tool? There is also the option for managed control plane from Upbound, a commercial part of the Crossplane ecosystem.

	Crossplane	Terraform
Description	Framework for building control planes	Declarative infrastructure as code tool
State storage	In Kubernetes custom objects, human readable.	Usually S3. Not human readable.
API	Uses Kubernetes API	Nothing directly comparable
Drift detection	Constant and realtime reconciliation	Whenever Terraform is configured to run
Language	YAML	Custom DSL
Runs on	Continuously on Kubernetes cluster	Usually Terraform Cloud, or in CI pipeline as one off process

To wrap up

Use Crossplane when you:

Already use Kubernetes, and prefer API’s to control and view state of infrastructure
Need to build self-serviceable abstractions for developers to consume
Don’t like enforced custom DSLs

Use Terraform when:

Don’t have or are not familiar with Kubernetes at all
You, or central Ops team, creates all infrastructure, and you don’t have a need for self-serviceable abstractions
Prefer once-off workflows to manage infrastructure instead of a continuously running control plane