Many CI/CD providers, like GitHub Actions and CircleCI, offer the options to run your CI/CD job using Docker images today.

This is a useful feature since you can ensure your job is always running with the same pre-installed dependencies. Teams may choose to bring their own Docker image, or use readily-available community images on Docker Hub for instance.

One challenge is that the Docker image in use may not have been intended for use in CI/CD automation. Your team may thus find yourselves trying to debug puzzles like:

• I'm pretty sure XYZ is installed. Why does the CI/CD job fail to find XYZ?
• Why is the FOOBAR environment variable different from what we have defined in the Docker image?

Docker image 101

When you execute the docker container run ... command, Docker will, by default, run a Docker container as a process based on the ENTRYPOINT and CMD definitions of your image. Docker will also load the environment variables declared in the ENV definitions.

Your ENTRYPOINT and CMD may be defined to run a long-running process (e.g., web application), or a short process (e.g., running Speccy to validate your OpenAPI spec). This will depend on the intended use of your image.

In addition, Docker images will be designed to come with just-enough tools to run its intended purpose. For example, the wework/speccy image understandably does not come installed with git or curl (see Dockerfile).

Docker images may also be published for specific OS architectures only (e.g., linux/amd64). You will want to confirm which OS and architecture the image can be run on.

These are important contexts, when designing CI/CD jobs using Docker images.

Understanding Docker images for CI/CD

Generally, for CI/CD automation, your job will run a series of shell commands in the build environment.

CI/CD providers like GitLab CI and CircleCI achieve this by override your Docker image's entrypoint with/bin/sh or /bin/bash when executing them as containers.

This is why you would want to use the -debug tag variant for Kaniko's Docker image when using in CircleCI for instance.

Additionally, your Docker image may not come with the required tools for your CI/CD automation. For example, you would require git in order to clone the repository as part of the CI/CD job steps.

Debug Cheatsheet

With this information in mind, here is a list of commands you can run locally to debug your chosen Docker image.

# inspect the "built-in" environment variables of an image
$ docker image inspect docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 | jq ".[0].Config.Env"
[
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "LANG=en_US.UTF-8",
  "PYSPARK_PYTHON=python3",
  "SPARK_HOME=/home/glue_user/spark",
  "SPARK_CONF_DIR=/home/glue_user/spark/conf",
  "PYTHONPATH=/home/glue_user/aws-glue-libs/PyGlue.zip:/home/glue_user/spark/python/lib/py4j-0.10.7-src.zip:/home/glue_user/spark/python/",
  "PYSPARK_PYTHON_DRIVER=python3",
  "HADOOP_CONF_DIR=/home/glue_user/spark/conf"
]

# check the default entrypoint
$ docker image inspect docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 | jq ".[0].Config.Entrypoint"
[
  "bash",
  "-lc"
]

# check the default cmd
$ docker image inspect docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 | jq ".[0].Config.Cmd" 
[
  "pyspark"
]

# check tools installed
$ docker container run --rm docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 "git --version"
...
git version 2.37.1

$ docker container run --rm docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 "python --version"
...
Python 2.7.18

You can also find an example of this debugging for a CircleCI use-case here: https://github.com/kelvintaywl-cci/docker-executor-explore/blob/main/.circleci/config.yml

#docker #cicd #debug #cheatsheet

buy Kelvin a cup of coffee ☕

Before Diving in

This is an attempt to explain and explore how teams can use Docker Buildx for delivering Docker images.

Since we will not be covering all features around Docker Buildx, this is a wide snorkel rather than a deep dive.

This is a quick article for developers who have yet to use Docker Buildx but are curious on its use-cases.

What is Docker Buildx?

Let's take a few steps back before plunging in.

We use Docker Build to build Docker images from Dockerfiles.

Since 18.09, BuildKit was introduced as an improved version of the previous builder. As an example, we can mount secrets when building our images with BuildKit. BuildKit will also ensure that these secrets are not exposed within the built image's layers.

Buildx builds (no pun intended) on top of BuildKit. It comes with more operations besides image-building, as you can see from its available commands. Importantly, Buildx provides features for caching and cross-platform image builds.

Why should we use Docker Buildx?

For software teams shipping Docker images often, Docker Buildx can be an important tool in the box.

Caching image layers ensure a next rebuild of the image will be faster.

Before, teams would need various machines on different platforms to build images for each platform. For example, we would need a ARM64 machine to build a Docker image for ARM64 architectures.

With Docker Buildx's cross-platform feature, we can now use the same AMD64 machine to build both AMD64 and ARM64 Docker images.

Why is it relevant in CI/CD?

Many teams are building Docker images as part of their CI/CD pipelines. Hence, they can lean on the build cache and cross-platform capabilities of Docker Buildx to build various images faster and cheaper.

Let's discuss the two mentioned features a little deeper.

Caching

This pertains to the cache-from and cache-to options with the docker buildx build command.

Docker Buildx allows you to choose your caching strategy (e.g., inline, local, registry and etc), and each comes with its pros and cons.

Your choice will depend largely on your team's philosophy and the CI/CD provider.

For example, you can leverage GitHub's Cache service when running Docker Buildx on GitHub Actions.

For CircleCI users, you may find my exploratory project here useful.

Cross-platform

When building an ARM64 Docker image on a CI/CD pipeline, you would need to do so on an ARM64-based machine runner then (if not using Buildx).

Depending on your CI/CD provider, there may not be ARM64 support.

This can be worked around, if your CI/CD provider allows you to “bring you own runners” (also known as self-hosted runners). GitHub Actions and CircleCI support self-hosted runners. However, it does mean someone in your team now has to manage these runners on your infrastructure.

With Docker Buildx, we can now build cross-platform images within any arbitrary machine runner.

This can be a big win for team that prefers not owning additional infrastructures.

Resurfacing to Shore

We have explored the appeal of Docker Buildx, particularly in a CI/CD context here. As mentioned, it is ultimately a tool. For teams building Docker images in their CI/CD pipelines, I do encourage you to look into Docker Buildx if you have not!

#docker #buildx #cicd #performance

buy Kelvin a cup of coffee ☕

This is an attempt to explain CircleCI's Docker Layer Cache (DLC) feature in simple terms.

What is Docker Layer Cache

Docker creates an image via layers. Each layer is created via Dockerfile commands like RUN, COPY and ADD, and they represent a unit of filesystem change.

When Docker builds an image, the built layers are stored. Rebuilding the same image becomes faster because Docker can now retrieve the stored layers as cache. This is known as layer caching on Docker, or Docker Layer Cache (DLC).

DLC is useful when we are rebuilding the same layers. It works well when we build images on the same machine; The stored layers are still there when we build a second time.

However, this becomes tricky on CI/CD systems, since most CI/CD pipelines run on ephemeral environments. The execution environment is destroyed once your pipeline completes.

How does CircleCI implement DLC

For DLC to work on CI/CD systems, we need to ensure stored/cached layers can be shared between CI/CD pipelines.

To achieve this, CircleCI assigns an external volume (store) to a job when we enable DLC. Think of it like passing around an external hard drive amongst friends.

This external volume will store the built Docker images' layers in a job. When CircleCI assigns same external volume to the next job, we should see faster image builds since the cached layers can be found.

An external volume can be assigned only to one job at a time. Hence, a job may see cache misses when building images due to a different volume being assigned.

A cache miss will mean a longer time taken to build the Docker image, which translates to a longer job duration. However, it does not and should not cause a job on CircleCI to fail.

#docker #dockerlayercache #circleci #explanation

buy Kelvin a cup of coffee ☕