Synthetic Infrastructure Testing With Canary Checker and Flux | Mission Control

Skip to main content

Deploying applications with Kubernetes is easier than ever, yet developers face increasing complexity.

Kubernetes simplifies deployment, but with it comes a labyrinth of potential issues. From resource conflicts to version incompatibilities, a failure in one component can cascade. Understanding application health through metric models like RED (Requests, Errors, Duration) and USE (Utilization, Saturation, Errors) isn't always enough. Latent errors might only surface during deployment or scaling.

For example, consider deploying a stateful PostgreSQL database via Flux on AWS. Problems could arise, including:

Tools like helm template and helm lint can validate chart rendering and syntax, but they don't guarantee compatibility with a specific Kubernetes version or the operators running on the cluster.

ct install on a kind or simulated cluster can verify API compatibility and ensure all resources and operators work correctly in ideal conditions.

Deploying to a staging environment can help catch issues before they reach production, but this approach doesn't detect capacity, performance or latent errors that only surface under load.

Control plane testing can help improve resilience by continuously redeploying workloads, ensuring there is enough capacity within the system and that all operators and external dependencies are working correctly.

Canary checker is a kubernetes-native test platform that continuously runs tests using 30+ check styles against your workloads. In this tutorial, we use it to continuously verify the ability to provision and run stateful workloads in a cluster.

The kubernetesResource check creates kubernetes resources based on the provided manifests & perform checks on them, it has 5 lifecycle stages:

Lifecycle​

Apply Static Resources<br>Applies all staticResources that are required for all tests to pass e.g. namespaces, secrets, etc..

Apply Resources<br>Applies all the workloads defined in resources

Wait - Using the parameters defined in waitFor, wait for the resources to be ready using is-healthy

Run Checks - Run all the checks against the workloads

Cleanup - Delete all the resources that were created during the test.

Tutorial​

Prerequisites​

Prerequisites<br>To follow this tutorial, you need:

A Kubernetes cluster

FluxCD installed

Define the workload under test

Before you can create a canary you should start with a working example of a resource, in this example we use a HelmRelease to deploy a postgres database.

flux.yaml<br>apiVersion: v1<br>kind: Namespace<br>metadata:<br>name: control-plane-tests<br>apiVersion: source.toolkit.fluxcd.io/v1<br>kind: HelmRepository<br>metadata:<br>name: bitnami<br>namespace: control-plane-tests<br>spec:<br>type: oci<br>interval: 1h<br>url: oci://registry-1.docker.io/bitnamicharts<br>apiVersion: helm.toolkit.fluxcd.io/v2<br>kind: HelmRelease<br>metadata:<br>name: postgresql<br>spec:<br>chart:<br>spec:<br>chart: postgresql<br>sourceRef:<br>kind: HelmRepository<br>name: bitnami<br>namespace: control-plane-tests<br>version: "*"<br>interval: 1h<br>values:<br>auth:<br>database: my_database<br>password: qwerty123<br>username: admin<br>primary:<br>persistence:<br>enabled: true<br>size: 8Gi

Once you have verified the helm release is working on its own, you can then begin building the control plane test using canary-checker.

Install the canary-checker binary

Linux (amd64)<br>Linux (arm64)<br>MacOSX (amd64)<br>MacOSX (arm64)<br>Makefile<br>Windows<br>wget https://github.com/flanksource/canary-checker/releases/latest/download/canary-checker_linux_amd64 \<br>-O /usr/bin/canary-checker && \<br>chmod +x /usr/bin/canary-checker

wget https://github.com/flanksource/canary-checker/releases/latest/download/canary-checker_linux_arm64 \<br>-O /usr/bin/canary-checker && \<br>chmod +x /usr/bin/canary-checker

wget https://github.com/flanksource/canary-checker/releases/latest/download/canary-checker_darwin_amd64 \<br>-O /usr/local/bin/canary-checker && \<br>chmod +x /usr/local/bin/canary-checker

wget https://github.com/flanksource/canary-checker/releases/latest/download/canary-checker_darwin_arm64 \<br>-O /usr/local/bin/canary-checker && \<br>chmod +x /usr/local/bin/canary-checker

OS = $(shell uname -s | tr '[:upper:]' '[:lower:]')<br>ARCH = $(shell uname -m | sed 's/x86_64/amd64/')<br>wget -nv -nc https://github.com/flanksource/canary-checker/releases/latest/download/canary-checker_$(OS)_$(ARCH) \\<br>-O /usr/local/bin/canary-checker && \\<br>chmod +x /usr/local/bin/canary-checker

wget -nv -nc -O https://github.com/flanksource/canary-checker/releases/latest/download/canary-checker.exe

Helm Installation<br>This tutorial uses the CLI for faster feedback, in production we recommend installing canary-checker as an operator using the helm chart or as part of the full Mission Control platform.

Next create a Canary CustomResourceDefinition (CRD) using the kubernetesResource check type, the layout of the canary is as follows:

basic-canary.yaml<br>apiVersion: canaries.flanksource.com/v1<br>kind: Canary<br>metadata:<br>name:...