cover image

February 24, 2022

A Vercel-like PaaS beyond Jamstack with Kubernetes and GitOps, part II

Gitlab pipeline and CI/CD configuration


This article is the second part of the A Vercel-like PaaS beyond Jamstack with Kubernetes and GitOps series.

A Vercel-like PaaS beyond Jamstack with Kubernetes and GitOps


After installing a k0s Kubernetes cluster in part I, I'm now going to set up repositories and CI/CD configuration. Then, I'll set up a pipeline to build and deploy applications to Kubernetes. This is what it looks like when completed:

gitlab pipeline interface

A Vercel-like pipeline done with GitLab

I'm using GitLab here mostly because I've been using it for 10 years and it's the platform I know the most. In my opinion, it has always been the best platform of its market. However, the competition has closed the gap in the last years and picking one platform instead of another is mostly about preferences or partnership.

I've already set up the same workflow with Azure DevOps in the past so I'm confident it could be set up with many other CI/CD platforms. I'm not using GitLab's Kubernetes integration features here.

In the future, I might use Dagger to implement cross-platform CI workflow.

  1. Introduction
  2. Grant GitLab CI/CD access to the Kubernetes cluster
  3. Grant Kubernetes access to the image registry
  4. Add pipeline configuration file and stage declaration
  5. The package stage configuration
  6. The deploy stage configuration
  7. The promote stage configuration
  8. The delete stage configuration
  9. Next step

Introduction

For the purpose of this experiment, I've created Node.js, PHP, Python and Ruby web applications. These are the applications I'll deploy to the Kubernetes cluster. They all have the same two endpoints:

  • /version returns the commit short hash of the built code, such as 7c77eb36

  • / returns a random name such as distracted_clarke

To generate names, I've copied the code from Docker-CE that generate Docker container names and translated it to the appropriate language.

I also added runtime information in a HTTP header:

$ curl -i https://nodejs.k0s.gaudi.sh | grep env
env: node-16.13.2

For the simplicity of the experiment, I've kept these apps very basic. One might think a higher complexity in the code could impact the reliability of the setup. I'm confident it does not.

I've been running much more complex Next.js and NextJS apps, some with an in-memory MongoDB servers, on AKS using a similar setup. For about two years now, it has been reliable as long as allocated hardware resources follows the cluster workload.


1. Grant GitLab CI/CD access to the Kubernetes cluster

The first thing I want to do is allow GitLab to query the Kubernetes cluster from pipeline jobs. To do so, I need to provide GitLab with the cluster's API URL, and a token to perform requests as an authorized user. I'll store these values in GitLab's CI/CD variables.

I've added every application repository to the k0s-examples group so that they can share the same inherited group CI/CD variables. Then, I'll only have to update the group variables when I recreate a fresh Kubernetes cluster.

Variables must be added in the Settings > CI/CD > Variables section of the group.

Once values are added and revealed, the section looks like this:

gitlab ci/cd variables

GitLab's CI/CD variables interface

These variables will be used to configure kubectl at the deploy stage.

The KUBE_URL variable

The cluster API URL can be retrieved from the k0s server terminal I've opened in part I:

$ sudo k0s kubeconfig admin \
| sed 's/'$(ip r | grep default | awk '{ print $9}')'/'$(curl -s ifconfig.me)'/g' \
| grep 'server: https' \
| awk '{ print $2 }'
WARN[2022-02-07 22:51:36] no config file given, using defaults
https://3.89.90.202:6443

The KUBE_TOKEN variable

I create an admin user for GitLab from the k0s server and copy paste its authentication token from the terminal to GitLab:

# first create gitlab user
$ sudo k0s kubectl create serviceaccount gitlab -n kube-system
$ sudo k0s kubectl create clusterrolebinding gitlab \
--clusterrole=cluster-admin \
--serviceaccount=kube-system:gitlab
# then print the token value
$ sudo k0s kubectl get secrets \
-n kube-system \
-o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='gitlab')].data.token}" \
| base64 --decode
eyJhbGciOiJSUzI1NiIsImtpZCI6IkpNa2lUUTVLTVJiVExDcUY5Y0JEaDY2MEZNT3NEM1Z...

2. Grant Kubernetes access to the image registry

Once GitLab can query the cluster, I have to allow GitLab to push images into a registry and to allow Kubernetes to pull those images.

GitLab automatically provides an image registry for each repository, called Container Registry, where I'll push Docker images built at the package stage of the GitLab pipeline and then pull them from the Kubernetes cluster at the deploy stage.

From Settings > Repository > Deploy token I create a deploy token named gitlab-deploy-token (check documentation to know why this name) and grant read_registry permission only:

gitlab deploy token form

Adding a deploy token to GitLab

Once created, the token is displayed in the active deploy token list:

gitlab deploy token list

Active deploy token list

This token will populate the value of CI_DEPLOY_USER and CI_DEPLOY_PASSWORD variables in pipelines, which are required to log in to the GitLab image registry and are passed to Kubernetes to allow images to be pulled from the cluster at the deploy stage.


3. Add pipeline configuration file and stage declaration

GitLab is set up, now I can create the pipeline configuration and use variables I've added in the two previous sections

The pipeline I'm creating consists of four stages:

  1. Package the application in Docker image.
  2. Deploy this image to a Kubernetes cluster and generate a unique URL to access this application instance, such as https://7c77eb36.nodejs.k0s.gaudi.sh
  3. Promote the instance to production by pointing a non-prefixed URL to it, such as https://nodejs.k0s.gaudi.sh
  4. Remove the application instance.

This is what it looks like in GitLab:

gitlab pipeline interface

A Vercel-like pipeline done with GitLab

In the root folder of each repository, I create a .gitlab-ci.yml file that contains the configuration of the GitLab pipeline.

I add the service to use the Docker-in-docker strategy in pipeline jobs:

# the base image that runs the pipeline
image: docker:19.03.13
# docker service to run docker-in-docker
services:
- docker:19.03.13-dind

Then the stages that will run sequentially:

stages:
- package
- deploy
- promote
- delete

I've highlighted the different stages and jobs in the screenshot below to illustrate how each stage will contain a unique job and will run one after another, the delete stage is not visible here:

gitlab pipeline interface

A pipeline is made of stages and jobs


4. The package stage configuration

Before deploying to the cluster, I have to package the application in a Docker image and push it to the Container Registry.

Since each step has only one job, I use the same stage name and job name. The excerpt below shows the configuration of the package job:

package: # job name
stage: package # attach this "package" job to the "package" stage
when: manual # remove this to deploy automatically
image: docker:19.03.13-dind # use docker-in-docker
script: |
# login to the image registry
echo ${CI_REGISTRY_PASSWORD} \
| docker login -u ${CI_REGISTRY_USER} ${CI_REGISTRY} --password-stdin
# build docker image
docker build -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA} \
--build-arg COMMIT_SHORT_HASH=${CI_COMMIT_SHORT_SHA} .
# push docker image
docker push ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}

I'm not using the CI_DEPLOY_USER and CI_DEPLOY_PASSWORD values I created in the previous section. Instead, I use CI_REGISTRY_USER and the CI_REGISTRY_PASSWORD short living value that is allowed to write to the registry:

The password to push containers to the project's GitLab Container Registry. Only available if the Container Registry is enabled for the project. This password value is the same as the CI_JOB_TOKEN and is valid only as long as the job is running. Use the CI_DEPLOY_PASSWORD for long-lived access to the registry. GitLab documentation

I use the git commit short hash to tag the built image with CI_COMMIT_SHORT_SHA.

I also pass this value to the Docker build command as an argument to store it in the Docker image and return it with the /version application endpoint.


5. The deploy stage configuration

The Docker image has been pushed, now I can trigger a Kubernetes deployment using kubectl and manifest files stored in the repository.

This job starts automatically when the package stage completes. It doesn't require Docker-in-docker. Instead, I use an image provided by Bitnami that contains kubectl:

deploy:
stage: deploy
needs: [package] # needs "package" stage completion before running
image:
name: bitnami/kubectl:latest
entrypoint: ['']
script: |
# declare Kubernetes namespace to use
export KUBE_NAMESPACE=${CI_PROJECT_NAME}-${CI_COMMIT_SHORT_SHA}
# declare the public URL of the deployment
export KUBE_INGRESS_HOST=${CI_COMMIT_SHORT_SHA}.${CI_PROJECT_NAME}.k0s.gaudi.sh
# add KUBE_URL, KUBE_TOKEN and KUBE_NAMESPACE to kubectl
./sh/configure-kubectl.sh
# inject values in manifests and deploy
./sh/deploy.sh

The KUBE_NAMESPACE refers to the Kubernetes namespace where I want to place any resource created by kubectl during the job execution. This way I can group components together and makes removing them easier at the delete stage.

The KUBE_INGRESS_HOST variable refers to the public URL I want to use to route network traffic to the application instance:

$ curl https://7c77eb36.nodejs.k0s.gaudi.sh
vigorous_cray

Once variables are declared, the first thing I want to do is to configure kubectl with the KUBE_URL and KUBE_TOKEN values I've added in section 1.

The configure-kubectl.sh script

The script runs inside the bitnami/kubectl image and calls the configure-kubectl.sh script first

All actions performed with kubectl will be done under this namespace:

#!/bin/sh
kubectl config set-cluster k0s --server=${KUBE_URL} --insecure-skip-tls-verify=true
kubectl config set-credentials gitlab --token=${KUBE_TOKEN}
kubectl config set-context ci --cluster=k0s
kubectl config set-context ci --user=gitlab
kubectl config set-context ci --namespace=${KUBE_NAMESPACE}
kubectl config use-context ci

The deploy.sh script

The deploy.sh file contains 3 subtasks:

  1. It calls kubectl to store the deploy token in a Kubernetes Secret to allow Docker images to be pulled from the GitLab Container Registry by the Kubernetes Deployment later on.

  2. It replaces strings such as __CI_REGISTRY_IMAGE__ with the actual value of the CI_REGISTRY_IMAGE variable in yaml manifest files. I use this solution to make up for the lack of a template engine in this setup.

I do a simple find and replace, using sed and "|" instead of "/" because CI_REGISTRY_IMAGE contains "/" characters.

  1. It calls kubectl to deploy the app to the cluster.
...
# 1. add credentials to pull image from gitlab
kubectl create secret docker-registry gitlab-registry \
--docker-server="${CI_REGISTRY}" \
--docker-username="${CI_DEPLOY_USER}" \
--docker-password="${CI_DEPLOY_PASSWORD}" \
--docker-email="${GITLAB_USER_EMAIL}" \
-o yaml --dry-run=client | kubectl apply -f -
# 2. replace __tokens__ in yaml files with real values
find manifests -type f -exec \
sed -i -e 's|__CI_REGISTRY_IMAGE__|'${CI_REGISTRY_IMAGE}'|g' {} \;
find manifests -type f -exec \
sed -i -e 's|__CI_COMMIT_SHORT_SHA__|'${CI_COMMIT_SHORT_SHA}'|g' {} \;
...
# 3. deploy app to cluster
kubectl apply -f manifests/
...

6. The promote stage

Finally, given an existing deployment such as 7c77eb36.nodejs.k0s.gaudi.sh, I want to point the production domain nodejs.k0s.gaudi.sh to the same deployment.

This stage creates a new Kubernetes Ingress to route the production domain to the same container I've deployed in the previous section:

promote to prod:
when: manual
needs: [deploy]
image:
name: bitnami/kubectl:latest
entrypoint: ['']
stage: promote
script: |
export KUBE_NAMESPACE=${CI_PROJECT_NAME}-${CI_COMMIT_SHORT_SHA}
export KUBE_INGRESS_HOST=${CI_PROJECT_NAME}.k0s.gaudi.sh
./sh/configure-kubectl.sh
./sh/promote.sh

Like in the previous deploy job, kubectl is configured with the configure-kubectl.sh script.

Then, the promote.sh is executed:

...
# delete existing ingress
kubectl delete ingress \
--all-namespaces \
--field-selector metadata.name=api-prod \
-l app=${CI_PROJECT_NAME},tier=backend
# add new ingress
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-prod
labels:
app: ${CI_PROJECT_NAME}
tier: backend
...
rules:
- host: ${KUBE_INGRESS_HOST}
...

Now both URLs return the same version value since they point to the same deployment:

$ curl https://7c77eb36.nodejs.k0s.gaudi.sh/version
7c77eb36
$ curl https://nodejs.k0s.gaudi.sh/version
7c77eb36

7. The delete stage

To clean up deployments, I'm adding a fourth stage to delete Kubernetes resources. Docker images are not removed from the Container Registry, though.

To remove all resources attached to the deployment, I simply delete the Kubernetes namespace:

delete env:
when: manual
needs: [deploy]
image:
name: bitnami/kubectl:latest
entrypoint: ['']
stage: delete
script: |
export KUBE_NAMESPACE=${CI_PROJECT_NAME}-${CI_COMMIT_SHORT_SHA}
./sh/configure-kubectl.sh
kubectl delete namespace $KUBE_NAMESPACE

Next step

All stages of the pipeline are set up, now I must add two things to make it work:

  • the Dockerfile to build the Docker image in the package stage.

  • Kubernetes manifests I'm applying in the deploy.sh with the kubectl apply -f manifests/ command in the deploy stage.

A Vercel-like PaaS beyond Jamstack with Kubernetes and GitOps, part III: Applications and the Dockerfile


About me

Me

Hi, I'm Jonathan Experton.

I help companies start, plan, execute and deliver software development projects on time, on scope and on budget.

Montreal, Canada · GMT -4