blog post<\/a>, but the procedure I will be using stays more close the standard kubeadm HA setup and achieves an end result that is more close to a HA setup since all configuration will be regenerated by kubeadm. For instance, we will be updating the certificates by adding the controlPlaneEndpoint flag to the kubadm-config instead of adding SANs. Before doing any of the procedures that follow, make sure to backup your \/etc\/kubernetes and \/var\/lib\/kubelet directories. The procedures all apply to kubernetes 1.26.4 with all nodes running Ubuntu 22.04.2 LTS. It was verified that after this step, a standard upgrade to kubernetes 1.27 was still possible.<\/p>\nStep 1: adding host entries for the control plane<\/h2>\n
In this step identify the IP address of the single control node and add a host entry on the controller and all worker nodes that points to this IP, e.g.:<\/p>\n
192.168.121.247 master\r\n<\/pre>\nStep 2: Regenerate certificates<\/h2>\n
Get ClusterConfiguration file:<\/p>\n
kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadm.yaml\r\n<\/pre>\nand add<\/p>\n
controlPlaneEndpoint: master:6443\r\n<\/pre>\nat top-level.
\nNow regenerate the certificates:<\/p>\n
rm -f \/etc\/kubernetes\/pki\/apiserver.* \r\nkubeadm init phase certs apiserver kubeadm --config kubeadm.yaml\r\n<\/pre>\nThe output of the above command should already show master<\/em> being added as one of the SANs.<\/p>\nNow verify the generated certificate:<\/p>\n
openssl x509 -in \/etc\/kubernetes\/pki\/apiserver.crt -text -noout\r\n<\/pre>\nYou should see DNS:master<\/em> appear as one of the SANs.<\/p>\nNext, restart the API server. This can be done by killing the API server manually, or by temporarily moving the kube-apiserver.yaml<\/em> from the \/etc\/kubernetes\/manifests<\/em> directory.
\nTo verify that it works, edit the .kube\/config<\/em> file in your home directory and modify the server URL to use master<\/em> instead of its IP. Then try some kubectl commands to check access.<\/p>\nFinally, upload the modified cluster configuration:<\/p>\n
kubeadm init phase upload-config --config kubeadm.yaml\r\n<\/pre>\nStep 3: edit config maps<\/h2>\n
Update the remaining ConfigMaps to update the server URL to use master<\/em> instead of the IP address.<\/p>\nkubectl edit cm -n kube-public cluster-info\r\nkubectl edit cm -n kube-system kube-proxy\r\n<\/pre>\nStep 4: Update scheduler, controller manager, and kubelet config files<\/h2>\n
Next update the configuration files for the other components:<\/p>\n
rm -f \/etc\/kubernetes\/*.conf\r\nkubeadm init phase kubeconfig all --config kubeadm.yaml\r\n<\/pre>\nIt is expected that after this, the scheduler and controller manager will still use the local api server instance. However, admin.conf<\/em> and kubelet.conf<\/em> should be using master<\/em> now instead of the IP address for the master. This is the same to what you get in a standard HA kubeadm cluster setup.<\/p>\nNow restart these components,<\/p>\n
systemctl daemon-reload\r\nsystemctl restart kubelet \r\nkubectl delete pod -n kube-system -l component=kube-scheduler\r\nkubectl delete pod -n kube-system -l component=kube-controller-manager\r\nkubectl delete pod -n kube-system -l k8s-app=kube-proxy\r\n<\/pre>\nand wait for all these to be running again.<\/p>\n
Next verify the etcd setup that that the peer URL is using the public IP of the controller node:<\/p>\n
root@master1:\/etc\/kubernetes\/pki\/etcd# export ETCDCTL_API=3\r\nroot@master1:\/etc\/kubernetes\/pki\/etcd# etcdctl --cacert ca.crt --cert server.crt --key server.key member list --write-out table\r\n+------------------+---------+---------+------------------------------+------------------------------+\r\n| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |\r\n+------------------+---------+---------+------------------------------+------------------------------+\r\n| 356b794c90ad4dca | started | master1 | https:\/\/192.168.121.247:2380 | https:\/\/192.168.121.247:2379 |\r\n+------------------+---------+---------+------------------------------+------------------------------+\r\n<\/pre>\nI have seen cases where joining a controller noded failed because the peer URL was using localhost and this will give problems later if a second node is joined since the other node will use the advertized peer URL by the etcd server. If the peer address is wrong, then use<\/p>\n
etcdctl member update MEMBERID --peer-urls=https:\/\/EXTERNAL_IP:2380\r\n<\/pre>\nto fix it.<\/p>\n
Step 5: update the kubelet on worker nodes<\/h2>\n
To update the kubelet on worker nodes edit \/etc\/kubernetes\/kubelet.conf<\/em> to use master<\/em>
\ninstead of the IP address of the API server, and after that restart the kubelet.<\/p>\nsystemctl daemon-reload \r\nsystemctl restart kubelet \r\n<\/pre>\nStep 6: setup new entries for all users in .kube\/config<\/h2>\n
As a result of this procedure the client certificate used to identify users will no longer work. Therefore, you must create new certificates for all users.<\/p>\n
Final thoughts<\/h2>\n
By investigating the differences between a non-HA and HA setup, it becomes easy to identify the precise differences between a HA and non-HA kubernetes setup. Based on this and using kubeadm tools as much as possible for updating configuration files it is possible to migrate a non-HA cluster to a HA cluster which can then be used to upgrade all nodes to newer OS versions. The final cluster obtained could still be updated to the next kubernetes version using the standard kubeadm upgrade procedure<\/a>.<\/p>\nOne major consequence is that you must regenerate all user certificates after upgrading the cluster to HA. This would normally be too much for a production setup, but there, you probably would have been using a cloud provider and would never run into the issue of having a non-HA cluster.<\/p>\n","protected":false},"excerpt":{"rendered":"
When I setup my kubernetes cluster using kubeadm some years ago, I decided to use a simple non-HA setup of kubernetes, because (1) it simplifies the setup and (2) the cluster will be running on a single server anyway. In … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[10],"tags":[],"_links":{"self":[{"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/posts\/2790"}],"collection":[{"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/comments?post=2790"}],"version-history":[{"count":57,"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/posts\/2790\/revisions"}],"predecessor-version":[{"id":2848,"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/posts\/2790\/revisions\/2848"}],"wp:attachment":[{"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/media?parent=2790"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/categories?post=2790"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/tags?post=2790"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}