After setting up my kubernetes cluster at home back in June 2021, one of the first things I noticed was a lot more noise from the server. Apparently, it was just a lot of disk IO coming from kubernetes and in particular etcd. Therefore, I decided to fix this problem.
\n<\/p>\n
Before we proceed it is important to note that my cluster is running on a single server, so high availability (apart from restarting failed containers) is not my aim. Of course, I will experiment with replicated storage such as longhorn<\/a> and perhaps ceph\/rook<\/a> in the future, but at the end of the day it is still a single server with a single kubernetes cluster and a single controller node. This means that running etcd from a ramdisk is an option. I am using a kubeadm<\/a> setup of kubernetes so etcd<\/em> is running as a pod inside my cluster.<\/p>\n Running etcd<\/em> from a ramdisk requires the following:<\/p>\n Systemd<\/a> is a replacement for init that has been in use for several years now. The good thing about systemd is that it allows modifying behavior. This fact is used by the pre startup hook and post stop hooks. These hooks are created by placing .conf files in the extension directory \/usr\/lib\/systemd\/system\/kubelet.service.d<\/em>.<\/p>\n The pre startup hook is as follows:<\/p>\n In this configuration, I have added a dependency on containerd<\/em>, since the prestartup hook requires the container runtime to be started. Also a pre-startup script is triggered and because the script is prefixed with ‘-‘, startup will fail when the restore fails.<\/p>\n The post startup hook is similar:<\/p>\n The scripts here contain all the intelligence for backing up and restoring etcd<\/em> data. In particular, the backup image requires a running etcd<\/em> and runs a backup command in a container. It preserves the last 10 backups taken and preserves one backup per day for the last 31 days. It also backs up the name of the docker image for etcd, since that may vary depending on the kubernetes version. That etcd<\/em> container is than used by the restore script so we are certain that backup and restore always use the same version of etcd<\/em> as the kubernetes cluster.<\/p>\n To mount the storage directory \/var\/lib\/etcd as a ramdisk requires adding a single line to \/etc\/fstab:<\/p>\n The final step is to run the periodic backup using a cron script placed in \/etc\/cron.d<\/em>:<\/p>\n Note that above there is also a defragmentation task because shortly after setting up monitoring with prometheus, I got messages about etcd<\/em> fragmentation.<\/p>\n Initially. the scripts were based on docker, but since that time, kubernetes no longer uses docker by default, and I have switched to containerd. To do this transparently, I added a docker script that simply delegates to containerd<\/em> using nerdctl<\/a><\/em>. In the same way, the backup solution can be adapted to other container runtimes.<\/p>\n See the full source code together with setup instructions here<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" After setting up my kubernetes cluster at home back in June 2021, one of the first things I noticed was a lot more noise from the server. Apparently, it was just a lot of disk IO coming from kubernetes and … Continue reading \n
[Unit]\r\nAfter=containerd.service\r\n\r\n[Service]\r\nExecStartPre=-\/opt\/wamblee\/etcd\/bin\/etcd-restore-to-tmpfs\r\n<\/pre>\n
[Service]\r\nExecStop=\/opt\/wamblee\/etcd\/bin\/etcd-cron\r\n<\/pre>\n
tmpfs \/var\/lib\/etcd tmpfs defaults,,noatime,size=2g 0 0 \r\n<\/pre>\n
*\/15 * * * * root \/opt\/wamblee\/etcd\/bin\/etcd-cron > \/var\/log\/wamblee-etcd-backup 2>&1\r\n30 0 * * * root \/opt\/wamblee\/etcd\/bin\/etcdctl defrag --cluster > \/var\/log\/wamblee-etcd-defrag 2>&1 \r\n<\/pre>\n