During development it often occurs that with local testing, docker is used for building and running container images. Then, when working with a local kubernetes cluster such as for instance k3d, you also want to test out kubernetes deployment. This requires the local k8s cluster to be able to access the images. Unfortunately, the docker deamon provides its own API and does not provide a registry API. So this setup requires you to run a separate registry (in a docker container) and push images after building them to the local registry after which k3d can access them.
The workflow typically looks like this:
- build docker image and test locally without kubernetes
- push docker image to local registry (e.g. at localhost:5000)
- run tests on local kubernetes cluster
The first step can be automated easily using docker compose and by tagging the images that to be of the form localhost:5000/<imagename>. The second step is then docker compose push. However, in practice it is really easy to forget to do the second step and this can cost some time for troubleshooting. Wouldn’t it be nice if we could make it work without having to push manually every time after building?
A smarter local registry
As it turns out, the second step can be eliminated by having a smarter docker registry that can also retrieve images from docker. However, that also isn’t that easy and writing your own registry from scratch is also not a good idea. There is a simpler solution which is to use a registry proxy that intercepts all requests for a backend registry (which is the regular docker container registry container registry:2). The main functionality of the proxy is to forward all requests to the backend registry. However, when images are requested using a HEAD or GET request, the proxy instructs docker to push the image to the registry when needed. Pushing the image is done in the following situations:
- the registry does not have the requested image but docker has
- the registry has another version of the image than the docker daemon
In other cases, the proxy does nothing and simply proxies to the backend registry, thus ensuring the normal error handling behavior.
The picture below shows what happens when a local kubernetes cluster tries to pull an image from a local registry that does not exist:
First of all (0) the user builds a docker image which is stored by the docker daemon. Then, the kubernetes cluster tries to pull the image (1), this request ends up at the proxy (autoregistry) which checks both the backend registry (2) and the docker daemon (3) for the image id. When it finds that the ids are not equal, it instructs the docker daemon through the docker API to push the image. Since the image name starts with localhost:5000, the push request ends up at the autoregistry again, but since the push related requests do not match a HEAD or GET request for an image, they are proxied transparently to the backend registry which stores the image. Then after this, the original request (1) which was waiting for a reponse can now get the image. This approach works quite well in practice, even for large images of several GB. In case there is a timeout because pushing the image takes too long, then the pull request will be retried later by kubernetes.
A similar flow is present for when an image is requested based on a digest. Note that the whole flow for pushing images through the proxy is completely transparent since this is standard proxying behavior.
Implementation in Go
Go is a programming language developed by google and powers most of modern infrastructure. For instance, docker, kubernetes, grafana, and prometheus are all implemented in Go. The focus of Go is on simplicity but importantly for this problem, there is a lot of focus on networking and concurrency in the Go community which makes implementing a proxy a breeze.
The highlights of the implementation are as follows. We start off defining the AutoRegistry
type AutoRegistry struct {
host string
port string
backendRegistryUrl string
reverseProxy *httputil.ReverseProxy
}
This defines the essential information for the registry such as the host and post it is intercepting traffic for (in the example resp. localhost and :5000), and the registry URL of the backend (I used localhost:4999) and a reverse proxy component from the standard library. The latter is a component that provides proxying of all requests to a backend server and requires just one line of code:
autoProxy.reverseProxy = httputil.NewSingleHostReverseProxy(url)
The next step is to define a request handler that intercepts requests and does a push of a docker image if needed, apart from that, it proxies all requests to the backend server using the reverse proxy we just created.
func (autoProxy *AutoRegistry) requestHandler() func(http.ResponseWriter, *http.Request) {
return func(w http.ResponseWriter, r *http.Request) {
autoProxy.pushImageIfNeeded(r, autoProxy.host+":"+autoProxy.port)
autoProxy.reverseProxy.ServeHTTP(w, r)
}
}
Here, pushImageIfNeeded checks whether we are dealing with a HEAD or GET request for a docker image and if so instructs docker to push the image. The essential part is getting the image details:
func ParseImage(path string) (image_name string, tag_name string, ok bool) {
r := regexp.MustCompile("^/?v2/(.*)/manifests/([^/]+)$")
matches := r.FindStringSubmatch(path)
if len(matches) == 3 {
return matches[1], matches[2], true
}
return "", "", false
}
which tries to parse the image and tag (or digest) from a request. The rest of the implementation (not shown) is querying the docker daemon and the registry and invoking a docker push (if needed).
The source code for this example is found here.
Deployment
Docker compose
The docker compose file can be found together with the source code.
k3d
For k3d, the cluster must be created with a registries.yaml as follows:
k3d cluster create dev --registry-config registries.yaml
where registries.yaml is as follows:
mirrors:
localhost:5000:
endpoint:
- http://host.k3d.internal:5000
The above makes sure that requests for images at localhost:5000 in kubernetes pods are mapped to host.k3d.internal:5000 which is local host on the host where the registry is running.
Final thoughts
I have been using this utility already for months now. It works both on windows and linux. It has turned out to be quite robust, and I haven’t seen any problems with it. The error handling is a bit simple (using panic which is against go conventions), but I have really not seen any panics yet.