Nonsense and other useful things

In April 2024 I successfully passed the CKS certification exam, but compared to the Certified Kubernetes Application Developer and Certified Kubernetes Administrator this was the toughest one yet. Not because the exam is particularly hard. The questions were all in closed form, I guess so that automatic grading is possible, but the main difficulties are that:

• there are many new topics such as runtime security with Falco, Seccomp, Apparmor, and gvisor, as well as security scanning tools such as trivy and kube-bench.
• the time pressure during the exam is high. I experienced that first hand by not being able to finish 1 of the questions. To go fast you need to prepare things very well. In particular, I setup my own single node kubernetes cluster using Vagrant so I could test out many topics using a standard kubeadm setup.
• the auditing during the exam was the worst experience yet. The proctor took more than 30 minutes to release the exam so I could start. Additionally, the proctor interrupted me twice, I think, for no good reason that caused a huge interruption of my flow.

Why I took the exam

Continue reading →

After the debacle with Rocky linux and Red Hat, I decided to move to a more future proof setup with my kubernetes cluster fully running on Debian. I chose Debian 12 at the time but unfortunately Nvidia did not yet support Debian 12 in their CUDA repository. Using the CUDA repository is the easiest way to install the Nvidia drivers and much better than running a script. See package manager installation on this Nvidia page. Therefore, I decided to use Debian 11 instead which was supported. That on its own gave some headaches in the setup and required some hacks to get it working. Also, looking at future migrations (considering e.g. replacing calico by cillium), it is a lot better to run the same OS on all kubenetes nodes.

Continue reading →

Upgrade for centos/rocky linux to new major versions is not supported. If you look for it there are procedures, but the changes between major versions can be so siginificant that this is not a good idea. Therefore, an approach I often use is to upgrade by installing the newer version next to the current version. Of course, I have also paritioned my old system to separate my home directory and other important directory into separate partitions so that the upgrade does not require copying user files.

This has some risks, since you might lose connectivity to the old linux version if the upgrade fails. Continue reading →

When I setup my kubernetes cluster using kubeadm some years ago, I decided to use a simple non-HA setup of kubernetes, because (1) it simplifies the setup and (2) the cluster will be running on a single server anyway. In fact I am using kubernetes mainly for deployment flexibility and not as much for high availability. Now I am running a mix of centos 8 stream and centos 7 nodes which will all be end of life in June 2024. With kubernetes, doing such an upgrade should be easy:

• add a new controller node on a newer OS and join it to the cluster
• remove the old controller
• replace worker nodes one by one

However, the first two steps will give problems since in a non-HA setup, the IP address of the API server is used by all components that connect to the API server. This will make it impossible to switch over to the new controller node in a transparent way.

Continue reading →

I am experimenting currently with argocd with the aim to have an (almost) fully automated bootstrapping of my kubernetes cluster at home. One of the first things to do when experimenting is to have a test environment. There are different deployment options for argocd to consider:

• deploy argocd in the cluster that it is managing
• deploy argocd in another cluster

Continue reading →

After setting up my kubernetes cluster at home back in June 2021, one of the first things I noticed was a lot more noise from the server. Apparently, it was just a lot of disk IO coming from kubernetes and in particular etcd. Therefore, I decided to fix this problem.
Continue reading →

I got a new laptop last week for work and needed to get it ready. Now new laptops usually mean new hardware and that can spell problems if you are installing linux. This is because the latest hardware may be used and linux usually needs some time to catch up with the latest hardware. This started out with the graphical installation not working. Luckily though, rocky linux 9 also supports a VNC install. So this issue was solved by connecting the laptop to wired ethernet and using VNC from another laptop to do the installation. This all went without problems. Next up was the NVIDIA driver for which I am using the RPM packages from the NVIDIA CUDA repository. These usually work fine, however, this time there was a mismatch where the driver was installed in the wrong directory; easily fixed though by fooling the NVIDIA RPMs by creating a symbolic link in /lib/modules: 5.14.0-162.12.1.el9_1.x86_64 -> 5.14.0-162.12.1.el9_1.0.2.x86_64/.

Finally, there was the sanity check to verify all devices such as webcam, microphone, speakers etc.. This is where all the problems started.
Continue reading →

This post describes how to monitor logs in kubernetes with grafana and loki. This covers the use case of logging for troubleshooting purposes. That is, it allows analysing human readable logs coming from multiple systems in one aggregated log. Human readable logs are required for troubleshooting and optimization. It is the bare minimum of logging that is required.
Continue reading →