As part of moving everything that is running in VMs on my server to kubernetes, the old wordpress installation had to be migrated to kubernetes as well. The website was previously running in a linux container based on systemd-nspawn, using a container which is basically running a full linux OS, including database.
The intention of the old setup was to move wordpress hosting away from an old server that also contained some private data. The intention was damage control by reducing the consequences of a hack of my wordpress website. At that time, my experiences running docker were not that positive regarding stability, so I chose to use standard linux containers with a simple interface on top called systemd-nspawn. Now, a few years on, the whole container ecosystem has matured, and now there are better ways to run containers such as kubernetes.
Setup
The new deployment consists of two containers and related volumes, configmaps, and secrets:
-
- mysql container: Using the official mysql image with a volume for the mysql data.
- wordpress container: Using the official wordpress image with a volume for the wordpress files.
For MySQL, the secret is used to define the root password through the MYSQL_ROOT_PASSWORD property, and the ConfigMap defines custom mysql configuration (e.g. buffer pool size). In my setup I the ConfigMap contains the following file which configures the buffer pool size for InnoDB:
[mysqld] innodb_buffer_pool_size = 256m
Both MySQL and wordpress require a single volume for storing data. Both are exposed using a service. The wordpress container has two additional config maps that are explained later. The mysql-brakkee-credentials secret contains the values of the required environment variables (WORDPRESS_DB_HOST, WORDPRESS_DB_NAME, WORDPRESS_DB_PASSWORD, WORDPRESS_DB_USER) to configure the database connection in wordpress.
Note that mysql is run in a separate namespace. This is because I have decided to host a single MySQL server with multiple database instances to reduce resource use and simplify the setup. Because of this, MySQL will be accessed from more than one namespace. As a result, it makes sens to deploy MySQL in a separate namespace.
mysql container
Before settling on using the official mysql image I looked at the percona operator for MySQL this operator provides custom resources for deployment of MySQL (in fact Xtradb with Percona improvements on top). The operator provides high availability, monitoring, backups and would be my first choice for any serious production deployment of MySQL on kubernetes.
Unfortunately, I could not get that to work with a simplified deployment with a single MySQL image and had to deal with hanging deletes of the operator and the operator simply not providing any output (kubectl describe). This was most likely caused by my simplified setup. However, in my setup at home the requirements are less critical, one of them being the time I can invest, and also, backups are less of an issue since I have those covered in a different way already using VM snapshots, see here, which is more than sufficient for what I use.
Since performance is critical for MySQL and NFS is not such a good choice for the data volume of MySQL, I used a host path volume. Since MySQL is sensitive when it comes to access rights, I created a separate user on the node that is hosting the volume:
useradd --shell /bin/false -M mysqlmain usermod -L mysqlmain id mysqlmain
This creates a new user without a login shell. The last command gives the new user’s user and group ids which are both 1001 in this case.
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql # A
spec:
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- weasel # B
claimRef:
name: data-mysql-0 # C
namespace: database
persistentVolumeReclaimPolicy: Retain # D
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce # E
hostPath:
path: "/data/mysql/main"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-mysql-0 # A
namespace: database
spec:
storageClassName: "" # F
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- # A: The name of the volume.
- # B: The nodeAffinity configuration links the volume to a specific node
- # C: The claimRef ties this PersistetVolume to the given PersistentVolumeClaim. The volume claim name is defined based on the volume name in the pod, the name of the mysql StatefulSet and the sequence number of the pod in the set. Since this is a non-replicated setup, there is only one pod which has number 0.
- # D: We never want to lose any data. In fact, we could have use a host path provisioner as well, but that can create issues when you delete your kubernetes cluster and set it up again. In particular, I want to manage storage myself since I want to know where my data is (no generated hash codes in directory names for instance), so I can setup a new cluster (or even a VM or docker setup) based on the data if I ever lose my cluster.
- # E: Note that ReadWriteOnce is not 100% safe since still multiple pods running on the same node could access the same data. For this purpose, the access mode ReadWriteOncePod should be used. However, this is not supported for this specific volume type since it is apparently not a CSI volume type.
- # F: An empty storage class name ensures that no provisioner is used. Storage is configured explicitly so that it is always clear where the data is and it is possible to survive kubernetes failures.
Below is the definition of the MySQL StatefulSet and Service:
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
namespace: exposure
labels:
app: mysql
spec:
serviceName: mysql
replicas: 1 # A
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
securityContext:
runAsUser: 1001 # B
runAsGroup: 1001 # B
containers:
- name: mysql
image: mysql:8.0.30
ports:
- containerPort: 3306
volumeMounts:
- name: data
mountPath: /var/lib/mysql
- name: config # C
mountPath: /etc/mysql/conf.d
envFrom:
- secretRef:
name: mysql-root # D
volumes:
- name: config
configMap:
name: mysql-config
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
name: mysql
namespace: exposure
spec:
type: ClusterIP
ports:
- port: 3306
targetPort: 3306
name: mysql
selector:
app: mysql
- # A: Only one replica since this is not a high available setup
- # B: Running as alternative user and group which must match the directory permissions used when creating the directory on the node
- # C: Additional configuration of mysql in the my.cnf format
- # D: The MySQL root password
wordpress container
For storage of the wordpress files NFS volumes are used like so
apiVersion: v1
kind: PersistentVolume
metadata:
name: wp-brakkee-data
labels:
app: wp-brakkee
spec:
claimRef:
name: wp-brakkee-data-wp-brakkee-0
namespace: brakkee-org
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: "/data/wordpress/brakkee"
server: 192.168.1.96
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wp-brakkee-data-wp-brakkee-0
namespace: brakkee-org
spec:
storageClassName: ""
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
NFS volumes allow flexibility of scheduling of the wordpress pod across nodes and performance is not really an issue. The setup is basically the same as for the MySQL container. In this case, a separate user with user 1002 and group 1002 are used. This differs from the user/group of the MySQL volumes so that the wordpress pod can never access MySQL data and provides a little add-on security. The setup is similar to the earlier MySQL volumes accept that node affinity is not required any more and that nfs is configured instead.
WordPress is then deployed as follows:
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: wp-brakkee
namespace: brakkee-org
spec:
serviceName: wp-brakkee
replicas: 1
selector:
matchLabels:
app: wp-brakkee-server
template:
metadata:
labels:
app: wp-brakkee-server
spec:
securityContext:
runAsUser: 1002 # A
runAsGroup: 1002 # A
containers:
- name: wp-brakkee
image: wordpress:5.5.3-php7.4-apache # B
ports:
- containerPort: 8080 # C
volumeMounts:
- name: wp-brakkee-data
mountPath: /var/www/html
- name: wp-brakkee-site-alias # D
mountPath: /etc/apache2/conf-enabled/sitealias.conf
subPath: sitealias.conf
- name: wp-brakkee-ports # E
mountPath: /etc/apache2/ports.conf
subPath: ports.conf
envFrom:
- secretRef: # F
name: mysql-brakkee-credentials
volumes:
- name: wp-brakkee-site-alias
configMap:
name: wp-brakkee-site-alias
- name: wp-brakkee-ports
configMap:
name: wp-brakkee-ports
volumeClaimTemplates:
- metadata:
name: wp-brakkee-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- # A: The alternative user and group under which wordpress runs. This is an unprivileged user so using this user only ports >= 1024 can be opened for listening.
- # B: The wordpress image based on Apache and PHP 7.4. I used an Apache image since I know apache really well and PHP 7.4 instead of PHP 8 since PHP 8 gave problems when upgrading. Apparently there are some compatibility issues (coming from PHP 5.5 of my previous wordpress install). I will save upgrade to PHP 8 to a later date.
- # C: Container port 8080 must be used since the default port 80 will not work for the unprivileged user.
- # D: This configures apache to allow hosting of the wordpress under the /site path. The configmap simply consists of a single line
Alias "/site" "/var/www/html"
that makes wordpress work under the /site URL. In my setup, there is another reverse proxy in front of wordpress so that only URLs below /site will be forwarded to wordpress. The approach listed here is easy since no modification of moves of files are required. It also allows the existing directory in the official wordpress container to be used as is. In my previous setup, I moved the wordpress files to a subdirectory of the /var/www/html directory, and that worked fine. However, if you do that with the official wordpress container, it will find at the next startup that wordpress is not initialized and will create new files in this directory. In fact, using the setup will allow wordpress to work both on / and on /site URLs.
- # E: This configures apache to listen on port 8080 instead of the default port 80. Again a single line
Listen 8080
- # F: This configures the credentials for connecting to MySQL.
Migration
It is always adventurous to upgrade wordpress since it never succeeds in one go. My first attempt was to try to copy the existing files and database backup from my older wordpress installation to a new installation .Problem here was that the previous installation was so old that no official images were available, so wordpress would somehow have to detect a version mismatch and upgrade. This went horribly wrong because of all kinds of obscure PHP errors, even when using a a wordpress and PHP version that were quite close to that of my previous installation.
Next step was to use a more black box migration. This means starting out with a clean wordpress install using the most recent apache based image of wordpress using PHP 8.
To make this work with wordpress behind a reverse proxy that does SSL termination, make sure that wp-config.php contains as one of the first lines:
if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https') $_SERVER['HTTPS']='on';
Then doing a migration using the ‘All in one WP migration‘ plugin. Immediately after installation, I set ‘WordPress Address (URL)” and “Site Address (URL)” to http://localhost:8080/site. This works because of the earlier Alias rule I configured. If you lose access to your wordpress installation after changing these URLs, you can always get access back by configuring this URL using wp-config.php by adding the old URL back:
define( 'WP_HOME', 'http://localhost:8080' ); define( 'WP_SITEURL', 'http://localhost:8080' );
Migration involves exporting wordpress on the old install to export a backup of files and database and importing it on the new install. The first thing you then encounter is upload limits. But these are easily fixed by adding the following to the .htaccess file in the root of the wordpress volume:
php_value upload_max_filesize 128M php_value post_max_size 128M php_value memory_limit 256M php_value max_execution_time 900 php_value max_input_time 900
After that, the import was started and seemed to work, but was hanging at the end. After a lot of experimentation, I decided to use a less recent image based on PHP 7.4 and this worked out of the box.
All this time, I was connecting to the wordpress pod using
kubectl port-forward wp-brakkee-0 8080
and using http://localhost:8080/site in my browser.
After everything is working, the last step is to configure kubernetes to forward https://brakkee.org/site URLs to the wordpress container. See my earlier post on how I am doing this, together with my post on certificate management.
This leads to another problem with a lot of resource URLs being broken. Apparently, wordpress uses full links to images and not relative paths. This problem was easily fixed by doing the whole import another time using the final URL. Of course, I could have also done this from the start, but it is good to verify at least while the site is not yet online that the import really works.
Network policies
As a final step, we need to secure the installation by adding network policies as described in a previous post. This involved allowing egress database traffic to all MySQL databases in the database namespace from the brakkee-org namespace, allowing egress access to wordpress, and allowing wordpress internet access but not LAN access and DNS access to do regular updates:
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: default-allow-nothing # A
namespace: brakkee-org
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: default-allow-access-to-databases # B
namespace: brakkee-org
spec:
podSelector: {}
egress:
- to:
- namespaceSelector:
matchLabels:
purpose: database
ports:
- port: 3306
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow-access-to-wp-brakkee-org # C
namespace: brakkee-org
spec:
podSelector:
matchLabels:
app: wp-brakkee-server
ingress:
- ports:
- port: 8080
from:
- podSelector:
matchLabels:
app: httpd-brakkee-org
namespaceSelector:
matchLabels:
purpose: exposure
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: external-internet-from-wordpress # D
namespace: brakkee-org
spec:
podSelector:
matchLabels:
app: wp-brakkee-server
egress:
- ports:
- port: 80
- port: 443
to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
- # A: Default allow-nothing rule to turn on network policies
- # B: Allow access to all MySQL databases in the databases namespace
- # C: Allow access to the wordpress container
- # D: Allow access to the internet and to DNS from the wordpress container
Next is to limit access to the MySQL database from only the current wordpress container. More access will be granted as more applications will use this database instance:
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: default-allow-nothing # A
namespace: database
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: default-allow-wp-brakkee-org # B
namespace: database
spec:
podSelector:
matchLabels:
app: mysql
ingress:
- from:
- podSelector:
matchLabels:
app: wp-brakkee-server
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: brakkee-org
ports:
- port: 3306
- # A: Allow nothing rule to enable network policies
- # B: Limit access to the database further by only allowing the wordpress container at this time.