Migrating a mailserver to k8s

It has been a long time since I setup a mail server. It started with my first mailserver on linux somewhere in 2000 using sendmail and University of Washington IMAP. This setup was assuming mail delivery to local system users. In other words, every e-mail had to correspond to a local system user. Getting it working was absolute hell, but it finally worked. Sendmail in particular seemed not to behave according to the documentation.

Then in 2006 my next setup was based on postfix and cyrus IMAP, decoupling mail boxes from system users. I bought a book about postfix and read it front to cover before starting. This was a much more pleasant experience. However, this was also not without fights in getting basic stuff to work. Over time, I added black listing, grey listing,  and spam detection to the setup. I also added a webmail user interface using squirrelmail later on. Finally, my ISP increased security and my mails often would get rejected. To fix this, my ISP required my to relay outgoing mail through their mail server and it turned out my postfix version was too low and could not handle it. I made a quick workaround for that by relaying outgoing mail to a newer postfix mail server running on another virtual server in my network. Problem solved, but it was getting painfully clear that the old setup was nearing its expiration date.

However, now it is time to say goodbye to this old setup. As part of my home project to migrate every workload from VMs to containers using kubernetes this is an ideal chance to get a new setup. Still a lot of respect for these older versions of cyrus and postfix for running for such a long time (16 years!) with basically zero maintenance. Would it be easier now, after all this time, to setup a new mail system? (spoiler alert: yes).

Architecture

Below a typical mail setup is shown.

To begin with there is an MTA (Mail Transfer Agent) with the responsibility to relay mail to remote users and to deliver mail to a mail storage system. The MTA. speaks SMTP (Simple Mail Transfer Protocol) to the outside world using an older variant on port 25 and a newer one on port 587. There is also another standard for SMTPS on port 465 but that is considered obsolete by most  The protocol on port 587 is named submission and is more secure than basic SMTP on port 25. The MTA delivers mail to a mail storage system using LMTP (Local Mail Transfer Protocol) which is a simplified version of SMTP geared towards delivering mail to mail storage systems. The MTA also provides integrations with spam detection (Spamassassin), black lists, grey lists, and virus scanning. In my pre-kubernetes setup, I am not using a virus scanner, but the new setup will include one.

Mail storage systems usually implement sieve, which is a mail filtering language for defining mail rules that support things like moving mail to other folders based on criteria, deletion, and out of office replies. For the remote management of these rules, the managesieve protocol is used.

The web mail system interfaces to the mail storage system using IMAP (Internet Message Access Protocol) or POP (Post Office Protocol). In my setup I am using IMAP exclusively since I want to be able to access mail on many different devices and see the same mails. Web mail uses the managesieve protocol to manage mail filtering rules on the mail storage system

The final system to consider is the mailing list provider. The MTA is configured to deliver mails destined for a hosted mailing list to the mailing list provider using LMTP. The provider in turn determines the recipients to send mail to and contacts the MTA again to send out these mails, using SMTP.  The mailing list provider also archives mails sent to the mailing list and usually provides a web interface as well.

The internal interfaces of the complete mail system are LMTP and managesieve. The external interfaces consist of IMAP, SMTP, and HTTP.

In general, mail systems are quite complex and are tightly integrated. Especially the integration between the MTA and the Mail Storage Provider is tight.

Deployment

These days, many pre-built containers are available and it is not necessary, as 16 years ago, to install everything from scratch and configure the integrations between components by hand. Instead, pre-built containers can be used which can be configured by specifying a small number of configuration files or environment variables. The picture below shows the new setup that shows the kubernetes pods and services that will be used.

In the above picture the pods are the UML node boxes and the services are the UML interfaces.

I have chosen the following components:

• docker-mailserver: This provides the postfix MTA and dovecot Mail Storage System pre-integrated. This is important since especially this integration is complex to setup from scratch. These days, dovecot appears more popular than cyrus so I am happy to go along with flow. Docker-mailserver is a pod with a single container that hosts both postfix and dovecot and provides additional facilities mentioned before such as virus scanning and spam filtering. Docker-mailserver uses supervisord, a sort of minimal init.d or systemd, to run all these processes in a single container. This goes against the grain, by running multiple processes in a single container, but for a tightly integrated system such as a mailserver this is a practical solution.
• mailman: Mailman will be used again for the Mailing List Provider. The mailman project provides pre-built containers which makes it easy to setup.
• roundcube: Roundcube will be replacing squirrelmail. Initially I was looking for pre-built docker containers for squirrelmail but could not find any mature and maintained ones. There are some private initiatives though but none of them seems to be mature. On the other hand, roundcube provides pre-built docker containers, a modern user interface, and prototyping against docker-mailserver showed that it was quite easy to get it to work, including use of the mail filters using the managesieve protocol.

The exposed services can be divided into two groups, external and internal. To simplify configuration we create the following services:

• IMAP and SMTP protocols: A load balancer service which is exposed directly to the internet based on certificates (see my earlier post).  Combining these in one service makes them share the same load balancer IP.
• HTTP, LMTP, and managesieve: one cluster IP service for every instance of the service. The HTTP service is exposed through an apache server in the kubernetes cluster and therefore does not need a load balancer or node port service.

Migration steps

To avoid a big bang migration, the core system containing postfix and dovecot is migrated first. Since LMTP is functionally a subset of SMTP, SMTP can be used by the MTA instead of LMTP to deliver mails to the Mailing List Provider. The difference is that instead of directly delivering mail to the Mailing List Provider using LMTP, the mail is delivered to the old mail server using SMTP which then delivers it to the Mailing List Provider using LMTP.

This allows migrating the MTA, keeping the Mailing List Provider the same in a first step to avoid a big bang migration. To do this, we configure separate rules for relaying mails destined for a mailing list to the old mail server that hosts mailman. That old mailserver is then configured to relay all mail through the new mail server. The web mail migration is simply postponed since I can live without a webmail interface for a couple of weeks without problems.

Configuration

To configure docker-mailserver, a number of settings must be done. These consist of:

• a general configuration file for docker-mailserver
• environment variables to turn features on and off
• a small number of customizations and fixes using a script and another postfix config fail containing only modified parameters.

The setup looks as follows:

The mail-var, mail-state, mail-logs, and mail-config are PersistentVolumeClaim objects that must be mounted in the container and may be empty when the container is first started. Initial permissions for these empty volumes may be 755, owned by root. In my deployment I am using hostpath volumes for this. See for instance my post on wordpress on how to tie these volumes to a specific node.

The config maps are generated using Kustomize based on a number of files:

kind: Kustomization
namespace: exposure

generatorOptions:
  disableNameSuffixHash: true               # A

configMapGenerator:
  - name: docker-mailserver-config
    envs:
      - docker-mailserver.env               # B
  - name: docker-mailserver-patch
    files:
      - user-patches.sh                     # C
      - postfix-main.cf
      - isp_password

• # A: Disable the randomly generated configmap hash by kustomize. This means the name will be exactly as specified.
• # B: ConfigMap containing the environment variables to set for the docker mailserver.
• # C: Several files to configure the mailserver
  • user-patches.sh: A script that will be run before the services start. This allows customization of the setup
  • postfix-main.cf: Custom postfix settings that will be added to /etc/postfix/main.cf
  • isp_password: This is a file containing the password the the relay server of my ISP

The secret finally is generated using automated certificate management. Note that only a single certificate is used. This is because I will be using only a single domain name to connect to SMTP and IMAP and I will use the same domain name for that regardless of the domain I am using for sending mail. It is possible to use different certificates but that makes the setup more complex and does not really add anything.

Container configuration

The docker-mailserver.env file contains all settings to configure docker-mailserver. I modified the following settings:

OVERRIDE_HOSTNAME=mail.example.com                # A
TZ=Europe/Amsterdam                               # B
ENABLE_CLAMAV=1
ENABLE_DNSBL=1
ENABLE_MANAGESIEVE=1
SSL_TYPE=manual                                   # C
SSL_CERT_PATH=/etc/certs/tls.crt
SSL_KEY_PATH=/etc/certs/tls.key
POSTFIX_MESSAGE_SIZE_LIMIT=102400000              # D
ENABLE_SPAMASSASSIN=1
ENABLE_SPAMASSASSIN_KAM=1
ENABLE_POSTGREY=1
SASLAUTHD_MECHANISMS=rimap                        # E

• # A: The domain name of the mailserver. Here I ran into the issue that if you define example.com as hostname then mails sent to user@example.com will result in postfix attempting to deliver the mail to the local system user named user. This is of course not what you want, so it is essential to define a hostname that is not used as a domain for your e-mail. This is easily solved using a subdomain.
• # B: Set the timezone. The advice of docker-mailserver to mount /etc/localtime into the container does not work, but the TZ variable worked out of the box.
• # C: Configuration of the SSL certificate. Here I am using manual and defining the location of the certificates. These certificates are mounted into the container using kubernetes Secret resources, see also my earlier post on certificates. There is a script in the docker container that checks for changes of these certificates and reloads services. So certificate updates will work with this setup.
• # D: I found out during migration of mail that some of my mails were larger than the default 10 MB so increased it to 100 MB.
• # E: This means that authentication is delegated to IMAP. This means that no separate users and passwords must be configured for SMTP. This is an advantage but also a disadvantage because it means that if the IMAP password is hacked, then the hacker will be able to use postfix as a relay server.

user-patches.sh

The user patches.sh script setups up various things:

• relay to the existing mailman configuration

  echo "
  list1               example.com
  list2               example.com
  " |
  while read user domain
  do
    if [[ -z "$domain" ]]
    then
      continue
    fi
    echo "$user@$domain relay:[oldmailmanhost]:25"
    for suffix in admin bounces confirm join leave owner request subscribe unsubscribe
    do
      echo "$user-$suffix@$domain relay:[oldmailmanhost]:25"
    done
  done >> /etc/postfix/transport_maps
  
  

  Note that there most also be a relayhost setting on the oldmailmanhost in /etc/postfix/main.cf to relay all mail back to the new mailserver.

• forwarding standard e-mails like info@DOMAIN, mailer-daemon@DOMAIN, and postmaster@DOMAIN to a single e-mail. These mails can be important so you want someone to receive them.

  echo "Setting up virtual_alias_maps"
  for domain in example.com example2.com
  do
    for user in mailer-daemon info postmaster
    do
      echo "$user@$domain me@example.com 
    done
  done > /etc/postfix/virtual
  
  

• copying various maps to postfix and calling postmap on them to create the corresponding .db files.

  echo "Copying postfix maps"
  cd /tmp/docker-mailserver
  copymaps="isp_password"
  allmaps="$copymaps transport_maps virtual"
  for file in $copymaps
  do
    echo "Copying table: $file"
    cp "$file" /etc/postfix
  done
  
  echo "Running postmap"
  for file in $allmaps
  do
    echo "Postmap: $file"
    postmap /etc/postfix/"$file"
  done
  
  

  Above, the isp_password file consists of a single line containing the hostname and username and
  password of my ISP for relaying e-mail:

  my.isp.com USER:PASSWORD
  

• fixing permissions of the /var/lib/clamav directory

  chmod 777 /var/lib/clamav
  

  This is a fix for a container issue that was also reported by other users.

postfix-main.cf

smtp_sasl_auth_enable = yes                                        # A
smtp_sasl_password_maps = hash:/etc/postfix/isp_password 
smtp_sasl_security_options = 
smtp_use_tls = yes 
smtp_tls_security_level = encrypt 
relayhost = [my.isp.com]:587 

mynetworks = 10.0.0.0/8, 192.168.178.0/24, 127.0.0.0/8, [::1]/128  # B

transport_maps = hash:/etc/postfix/transport_maps                  # C

• # A: Relaying outgoing mail through my ISP
• # B: allowing relaying from the standard local networks without a password
• # C: relaying of mailing list traffic to the old mail server.

Mail server deployment

All the above information is then combined into a single yaml file. I am using a StatefulSet here since different mail server instances should have different state and they should not be initialized or configured at the same time. Using a StatefulSet keeps the volumes for each instance separate. I am using only a single instance since I don’t need any more than that. As usual, the kubernetes yaml file can be derived from docker-compose files that are published by the maintainers.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mail
  namespace: exposure
spec:
  serviceName: mail
  replicas: 1
  selector:
    matchLabels:
      app: mail
  template:
    metadata:
      labels:
        app: mail
    spec:
      containers:
        - name: mailserver
          image: mailserver/docker-mailserver:11.1.0
          ports:
            - containerPort: 25                    # A
            - containerPort: 587                   # A
            - containerPort: 993                   # B 
            - containerPort: 4190                  # C
          envFrom:
            - configMapRef:                        # D 
                name: docker-mailserver-config
          volumeMounts:
            - name: mail-var
              mountPath: /var/mail
            - name: mail-state
              mountPath: /var/mail-state
            - name: mail-logs
              mountPath: /var/log/mail
            - name: mail-config
              mountPath: /tmp/docker-mailserver
            # Use subpath since the target dir is used for other files as well by the container
            # This adds only the single file.
            - name: mail-patch
              mountPath: /tmp/docker-mailserver/user-patches.sh
              subPath: user-patches.sh
              readOnly: true
            - name: mail-patch
              mountPath: /tmp/docker-mailserver/postfix-main.cf
              subPath: postfix-main.cf
              readOnly: true
            - name: mail-patch
              mountPath: /tmp/docker-mailserver/isp_password
              subPath: isp_password
              readOnly: true
            - name: certificates
              mountPath: /etc/certs
              readOnly: true
            # localtime
            #- name: mail-localtime
            #  subPath: localtime
            #  readOnly: true
            #  mountPath: /etc/localtime
          livenessProbe:                              # E 
            exec:
              command:
                - sh
                - -c
                - "ss --listening --tcp | grep -P 'LISTEN.+:smtp' || exit 1"
            initialDelaySeconds: 10
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 5
      volumes:
        - name: mail-patch
          configMap:
            name: docker-mailserver-patch
            defaultMode: 0555
        - name: certificates
          secret:
            secretName: example-com-tls-secret
  volumeClaimTemplates:
    - metadata:
        name: mail-var
      spec:
        volumeName: mail-var
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
    - metadata:
        name: mail-state
      spec:
        volumeName: mail-state
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
    - metadata:
        name: mail-config
      spec:
        volumeName: mail-config
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
    - metadata:
        name: mail-logs
      spec:
        volumeName: mail-logs
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi

• # A: SMTP ports
• # B: IMAPS ports
• # C: managesieve port
• # D: container configuration through environmnet variables.
• # E: the health check as copied from the docker-compose files of the docker-mailserver project.

The mailserver is exposed using the following service:

---
apiVersion: v1
kind: Service
metadata:
  name: mail
  namespace: exposure
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local                  # A
  loadBalancerIP: 192.168.178.244               # B 
  selector:
    app: mail
  ports:
  - name: smtp
    port: 25
  - name: submission
    port: 587
  - name: imaps
    port: 993
  - name: managesieve 
    port: 4190

 

Above, a LoadBalancing service is defined for the provided services. We use a fixed IP (# B) to support configuration on the router and externalTrafficPolicy of Local (#A). The latter is required so that postfix sees the original IP addresses of connections. Without this, black listing services won’t work. This approach with a single service is simple but exposes the managesieve port on the loadbalancer ip. However, access to the managesieve port can be restricted using network policies to just the pods that need it, see later.

Final configuration steps

The final configuration steps are to configure user accounts:

kubectl exec -it mail-0 -- setup email add user1@example.com
kubectl exec -it mail-0 -- setup email add user2@anotherdomain.com

and to configure DKIM

kubectl exec -it mail-0 -- setup config dkim

This last step will generate private and public keys for DKIM for each domain for which you created mail accounts in the earlier step. The tool will show which keys it has created, for instance:

> ./setup config dkim
[   INF   ]  Creating DKIM private key '/tmp/docker-mailserver/opendkim/keys/example.com/mail.private'

Then, kubectl exec into the mail container and examing the mail.txt file in the /tmp/docker-mailserver/opendkim/keys/example.com/ directory. This will show you what entry to add to DNS. The value to add are the quoted strings on a single line. To obtain this, select all the quoted strings from the output of cat mail.txt and remove newlines by running tr -d ‘\n’, pasting the quoted strings with newlines as input to the command. Then select all quoted strings from the output and paste them in the DNS value field.

 

Data migration

For data migration I used imapsync which is a tool that can synchronize mail folders on different mail storage systems using IMAP. Since it uses the IMAP protocol it doesn’t really matter what implementation is behind the mail storage system.

I used the docker image to avoid local installation problems. An example commandline is as follow:

docker run --network=host \
   gilleslamiral/imapsync /imapsync \
   --host1 "$host1" --user1 "${user1}" --password1 "$passwd1" \
   --notls1 -\
   --host2 "$host2" --user2 "${user2}" --password2 "$passwd" \
   --delete2

Some important flags are

• –network=host: Use host networking since I was actually using an SSH tunnel from localhost to my old mailserver to access it using IMAP (not IMAPS) and localhost within the container should resolve to the SSH tunnel.
• –notls1: This forces IMAP instead of IMAPS
• –delete2: Perform a full sync from server 1 to server 2, deleting messages on server 2 that no longer exist on server 1

Network policy

The final step is the addition of a new network policy to allow the mailserver to work:

---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: mail-ingress-egress
  namespace: exposure
spec:
  podSelector:
    matchLabels:
      app: mail
  ingress:
    - ports:
        - port: 25                      # A
        - port: 587
        - port: 993
        # - port: 4190
      from:
        - ipBlock:
            cidr: 0.0.0.0/0             # B
            except:
              - 1.2.3.4/24                   
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0             # C

• # A: incoming traffic on the defined ports for SMTP, and IMAP should be allowed. Access to managesieve (port 4190) will be granted in the future, but only to the webmail pod.
• # B: in combination with A allowing access from the internet with the exception of some evil IP addresses like 1.2.3.4/32 (just an example)
• # C: allowing full access to the internet for spam detection, virus definitions, black listing.

Next steps

The next step will be to deploy the webmail interface. After that, mailman will be migrated.

All in all this has been a lengthy write-up, but I think it would have been quite a bit longer without the use of the third-party containers and kubernetes. The setup was also a lot less nightmarish than my earlier mail setups.