CI/CD workflow: Generate test environments with Gitlab and Kubernetes

If you need some guidelines on how to quickly create a CI/CD workflow with Gitlab CI and Kubernetes, I strongly recommend you read this article.

A common problem in any CI/CD workflow is the bottleneck of the testing environment:

I’ve developed my feature, I have tested it locally, I have pushed it to the repository and I would like to deploy it on a remote environment on our testing Kubernetes cluster so the QA team can test the new feature.

But here’s the catch: since we have only a single testing environment on our Kubernetes cluster, I have to wait for it to be available. And this could take time depending on the time it takes to run the tests.

Since nobody likes to wait, I would like to talk about a solution that allows you to deploy multiple environments on the fly and never find yourself in that situation again.

Gitlab CI

First let’s look into the deploy stage of the gitlab-ci.yml:

This is a quite common helm deployment stage I’ve chosen to split into a feature deployment that will deploy to a testing environment and a production job that will deploy to, well... production.

Nonetheless, I would like to mention a few specificities on those jobs:

• the envsubst function is used on the helm values file to replace every occurrence of the ${CI_COMMIT_REF_SLUG} key with its value. This Gitlab predefined environment variable is what we’re going to use to differentiate between each newly created environment. Its value is the name of the branch with hyphen replacing characters that cannot be used in a DNS record. We use it because Kubernetes can be a little picky on its naming conventions. We’ll look into the Helm values file a bit later to see how this variable is used.
• Each Helm release is deployed into a new namespace. Both the release and the namespace use CI_COMMIT_REF_SLUG as part of their names so there is no conflict between each resource.
• We chose not to use CI_COMMIT_REF_SLUG in the production deployment job, so the production application is always deployed in the same namespace with the same name so we can keep deployment history and allow easier rollbacks.
• We use the environment Gitlab CI key to let Gitlab know about this new environment we’re creating. We will then be able to find it in the environment section of the Gitlab UI.

Here is the values.feature.yml used by Helm when deploying a new feature branch:

As you can see, there are a few occurrences of the CI_COMMIT_REF_SLUG variable. This allows users to customize our testing environment with the name of our feature branch.

However, there is an important drawback to this strategy: this helm values file can not be used as is; the Gitlab runner has to run the envsubst command on the file to have the final values. 

Let’s quickly look into a helm template file, for example, this one declaring ingresses to see how those values are used.

Now that all these pieces are in place, any new commit on a branch named feature/* will be deployed on a new Kubernetes namespace and my app will be accessible via a new api-${CI_COMMIT_REF_SLUG}.<company-dev-domain> DNS record. 

If another developer pushes another branch, there will be no repercussion to my deployment and his feature could be tested alongside mine.

External DNS

The external DNS is a deployment that syncs any ingress you’ve declared in your Kubernetes cluster with the remote DNS server your zone is hosted on.

More info can be found on external DNS.

Here is the external DNS deployment I use which sync the ingresses with the Cloudflare API. 

Now I have no need to interfere with the DNS hosting service. Everything is done automatically by the external DNS deployment.

You get a new testing environment!

With a few changes to your Gitlab-ci.yml, you can deploy multiple testing environments in your Kubernetes cluster with minimal overhead in complexity.

The job is made easy with tools from the Kubernetes ecosystem like external DNS which handles the DNS creation part of the new environment.