code and games
Built with Hugo and Hyde-Y.

Configuring Kubernetes Deployments With Makefiles and Envsubst

· Read in about 4 min · (801 Words)

If you have a project that requires frequent Docker builds & pushes to a Kubernetes cluster, it can be pretty frustrating to come up with a suitably hands-off build+deploy step. After developing on several such projects I’ve come up with an easy system that relies on minimal tooling, making it portable and easy to customize to an individual project’s needs.

Screenshot of before and after Kubernetes YAML files

Dependencies

First, ensure you have make and envsubst on your machine. make is usually in a build-essential package in your distribution’s package manager of choice, while envsubst is in the gettext package.

I will assume you already have kubectl and docker (or your containerization software of choice, this method is flexible).

Example Project Layout

hello-world
│
├── k8s/
│   └── deployment.yaml
├── Dockerfile
├── Makefile
└── main.go

The Magic’s In the Makefile

For convenience, the first things I put in my Makefile are the name of the project (in Docker registry friendly format i.e. snake-case) and a snippet to retrieve the current git tag or commit hash.

PACKAGE ?= hello-world
VERSION ?= $(shell git describe --tags --always --dirty --match="v*" 2> /dev/null || cat $(CURDIR)/.version 2> /dev/null || echo v0)

I then specify current location for the Kubernetes configuration YAML and the location to output the new files to.

K8S_DIR       ?= ./k8s
K8S_BUILD_DIR ?= ./build_k8s
K8S_FILES     := $(shell find $(K8S_DIR) -name '*.yaml' | sed 's:$(K8S_DIR)/::g')

K8S_FILES will recursively find all *.yaml files within K8S_DIR and return the results with the K8S_DIR/ stripped out, making it easy to build the same layout in the output to K8S_BUILD_DIR. It is recommended that the K8S_BUILD_DIR is included in the project’s .gitignore or equivalent.

I then specify the Docker configuration information. With Docker Hub and most Docker tools specifying the domain is unnecessary and defaults to docker.io, however for completeness sake I include it, and to make it easier to use alternative public or private Docker registries.

DOCKER_REGISTRY_DOMAIN ?= docker.io
DOCKER_REGISTRY_PATH   ?= zikes
DOCKER_IMAGE           ?= $(DOCKER_REGISTRY_PATH)/$(PACKAGE):$(VERSION)
DOCKER_IMAGE_DOMAIN    ?= $(DOCKER_REGISTRY_DOMAIN)/$(DOCKER_IMAGE)

Next, the list of environment variables that will be used in the Kubernetes templates.

MAKE_ENV += PACKAGE VERSION DOCKER_IMAGE DOCKER_IMAGE_DOMAIN

If you wish to make use of any additional environment variables include them in the above array. You can break it into multiple lines using additional MAKE_ENV += lines for readability.

HELLO ?= World
MAKE_ENV += HELLO

Then the list is iterated through and a shell compatible output is generated.

SHELL_EXPORT := $(foreach v,$(MAKE_ENV),$(v)='$($(v))' )

Note the space before the closing parentheses, this is required.

The contents of SHELL_EXPORT will end up looking something like this:

PACKAGE='hello-world' VERSION='v1' DOCKER_IMAGE='zikes/hello-world:v1' \
DOCKER_IMAGE_DOMAIN='docker.io/zikes/hello-world:v1' \
HELLO='World'

This is ideal for inserting before a shell command to share these values environment variables.

Finally, define the build and deploy steps:

.PHONY: build
build:
        # build project binaries as normal, e.g.
        go build -o ./bin/hello

.PHONY: build-docker
build-docker: build
        docker build . t "$(DOCKER_IMAGE)"

.PHONY: push-docker
push-docker: build-docker
        docker push "$(DOCKER_IMAGE)"

# Builds the Kubernetes build directory if it does not exist
# The @ symbol prevents Make from echoing the results of the
# command.
$(K8S_BUILD_DIR):
        @mkdir -p $(K8S_BUILD_DIR)

.PHONY: build-k8s
build-k8s: $(K8S_BUILD_DIR)
        @for file in $(K8S_FILES); do \
                mkdir -p `dirname "$(K8S_BUILD_DIR)/$$file"` ; \
                $(SHELL_EXPORT) envsubst <$(K8S_DIR)/$$file >$(K8S_BUILD_DIR)/$$file ;\
        done

.PHONY: deploy
deploy: build-k8s build-docker
        kubectl apply -f $(K8S_BUILD_DIR)

The build, build-docker, and push-docker commands will likely need modified to suit your project’s specific needs.

You can run make build-k8s in your shell to manually run the YAML templating and review the output.

k8s/deployment.yaml before:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world
        image: ${DOCKER_IMAGE_DOMAIN}
        env:
        - name: HELLO
          value: "${HELLO}"

build_k8s/deployment.yaml after:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world
        image: docker.io/zikes/hello-world:v1
        env:
        - name: HELLO
          value: "World"

When you’re all done, you should be able to run make deploy and watch everything automatically build and deploy to whatever cluster your kubectl is currently pointed to!

$ make deploy
go build -o ./bin/hello
docker build . -t "zikes/hello-world:v1"
Sending build context to Docker daemon  2.201MB
Step 1/3 : FROM scratch
---> 
Step 2/3 : ADD ./bin /
---> Using cache
---> d13c64c8f72f
Step 3/3 : CMD ["/hello"]
---> Using cache
---> 718d0c08fb46
Successfully built 718d0c08fb46
Successfully tagged zikes/hello-world:v1
docker push "zikes/hello-world:v1"
The push refers to repository [docker.io/zikes/hello-world]
c59266e118bd: Layer already exists 
v1: digest: sha256:87c26583d89cd5f88c2e19536cb3bd9240cd1fdfa67911937b5d9c586b9c0f12 size: 527
kubectl apply -f ./build_k8s
deployment.apps/hello-world configured
$ kubectl get pods
NAME                           READY     STATUS    RESTARTS   AGE
hello-world-5b8b5dfdc5-hqcdj   1/1       Running   0          1m
$ kubectl logs -f hello-world-5b8b5dfdc5-hqcdj
Hello, World!
Hello, World!
Hello, World!
...

To easily change settings in the deployment, just override your environment variables.

$ HELLO=Jason make deploy
...
$ kubectl logs -f hello-world-64c6bbd876-w2fv7
Hello, Jason!
Hello, Jason!
Hello, Jason!
...