What is Dockers BuildKit and Why Does It Matter?
Docker BuildKit is an opt-in image building engine which offers substantial improvements over the traditional process. BuildKit creates images layers in parallel, accelerating the overall build process. BuildKit was developed as part of the Moby project, a Docker effort to assemble specialized container systems without reinventing the wheel. It was announced in 2017 and began shipping with Docker Engine in 2018s version 18.09. BuildKit focuses on improving build performance, storage management, and extensibility. Its headline claims are parallel processing, more advanced caching, a pluggable architecture, and automatic garbage collection. These combine into a build system thats more efficient and more extensible than the original engine. Layers which dont impact on each other can be built simultaneously, reducing waiting times for stages to complete. BuildKit also optimizes access to the local files you reference with COPY instructions. It tracks the changes you make and only copies files that have been modified since the last build, instead of transferring the entire build context. BuildKit simplifies multi-platform builds too. You can supply the --platform flag to specify targets to build for. BuildKit will automatically assemble an appropriate image manifest to cover all specified architectures. docker buildx --create --platform linux/amd64,linux/arm64 . BuildKits performance enhancements are facilitated by the use of a low-level build definition format, dubbed LLB. Its a graph-based binary format that ties together complex build definitions. Advertisement    As layers are directly linked, BuildKit facilitates quicker comparison of build graphs and the content they include. The default Dockerfile builder needs to rely on imprecise heuristics to determine whether two images are comparable. The LLB is completely separate from the BuildKit frontend. The frontend takes a human-readable representation of an image, such as a Dockerfile , and converts it into an LLB graph. Once an LLBs been generated, it can be exported and moved between environments. Exporting to a registry lets frontend clients acquire an existing LLB to further improve first-time build performance. The LLB technical details are fairly complex. The underlying technology is based on graph theory and checksum comparison. You dont need to understand it to benefit from its power: as an end user, builds work in the same way as ever, with the docker build command. BuildKit still creates OCI-compliant images that are portable across different container environments. You can use BuildKit to create any Docker image with a Linux base. Windows images arent currently supported. Activating BuildKit Support  There are two ways to enable BuildKit. If you want to build a single image with the feature, set the DOCKER_BUILDKIT environment variable in your shell:  DOCKER_BUILDKIT=1 docker build . Advertisement    For long-term use, configure the Docker daemon to use BuildKit by default. Create or edit the /etc/docker/daemon.json file and add the following contents to the top-level config object:  { "features": { "buildkit": true } }  Reload the daemon configuration to apply the change:  systemctl reload docker  BuildKit will now be used instead of the default build engine when you run the docker build command. You can tell when BuildKit is active because it produces different CLI output to the regular engine. BuildKits progress display offers improved readability and clear visualization of when each stage starts and completes. The information includes a breakdown of the time taken to build each layer. docker buildx  You can also interact with BuildKit through docker buildx commands. These will always use BuildKit. The dockerx command group exposes advanced BuildKit functionality including the ability to build on a remote host. A single BuildKit client can interact with several distinct image builder instances. This facilitates multi-platform builds by letting you add a builder for each architecture youre targeting. Heres an example of adding a remote host as a BuildKit target. This assumes the target machine has a Docker socket exposed on TCP port 2375. You can reference the host using any Docker endpoint identifier, or the name of a Docker context (obtained from docker context ls ). The latter lets you build on compatible cloud environments by adding them as a context. docker buildx create --name remote-builder tcp://my-docker-host:2375  Advertisement    You can select the builder to use with the docker buildx use command:  docker buildx use remote-builder  Then you can use the build command to build your image on the selected builder instance:  docker buildx build . You can remove builder instances using docker buildx rm , passing in the builders name. Builders are listed using docker buildx ls ; you can use docker buildx inspect to get more detailed information about a specific builder. If you want to check BuildKits disk usage, run the docker buildx du command. You can clear the build cache to free up storage with docker buildx prune . This may reduce performance next time you rebuild your image, as previously cached layers will be reconstructed. Build Features  BuildKit adds a couple of extra build-time features to simplify your Dockerfile steps. You can pass in secret data using the --secret flag. This lets your Dockerfile access sensitive values without storing them inside the image. The values only available at build time. docker build --secret id=demo-secret,src=demo-secret.txt . FROM my-image:latest RUN --mount=type=secret,id=demo-secret cat /run/secrets/demo-secrets  Advertisement    This Dockerfile references a secret called demo-secret . Its value is read from the demo-secret.txt file when you run docker build . The RUN instruction with the --mount flag provides access to the secret. The file is temporarily mounted into the /run/secrets directory. BuildKit also lets you forward your hosts SSH agent so your build instructions can interact with existing remote connections. Pass the --ssh flag to docker build and add --mount=type=ssh to RUN instructions in your Dockerfile :  docker build --ssh . RUN --mount=type=ssh git clone git@example.com:/project.git  These features make image building more convenient without affecting overall security. SSH agent forwarding and secret mounts are only available in BuildKit; theres no counterpart in the default build engine. Conclusion  BuildKit is the next-generation Docker image builder which uses a graphed binary format to drastically accelerate builds. Although performance will vary considerably for each Dockerfile , you can expect to see substantial speed-ups in cases where parallel processing of image layers is possible. BuildKits architecture optimizes some of the newest Dockerfile features. Multi-stage builds benefit from skipping of unused stages which makes the process more efficient than the standard builder. Although BuildKit is now stable, Docker still doesnt ship with it on by default. Make sure to enable it in your Docker client if you want to use its features. Theres an active proposal to make BuildKit the standard build engine but there are still unresolved issues preventing the switch.