These days, it is ill-advised to run a website (such as this one), over HTTP, even if there is no security risk at all. When hosting your website on HTTP, users will see a warning triangle in the address bar and many users will simply turn away. This is especially painful if the website is blogging on devops and security related things.
Currently, I am migrating everything I have locally to a more secure and future proof setup from VMs to kubernetes. The approach I decided to take is to start with the front-facing services and move everything from there step by step. Therefore, the first step is to move the reverse proxy from a VM to kubernetes. Since I don’t want to pay for certificates, I will be using Let’s Encrypt. Unfortunately, Let’s Encrypt only supports 90 day certificates so this means a lot of certificate renewals.
The ACME protocol
To support this Let’s Encrypt supports the ACME protocol, which allows for automatic certificate renewal. The ACME protocol automates the complete process. A certificate renewal process looks like this:
- create a new private key
- create a certificate signing request using the private key
- request a certificate from the CA (Certificate Authority)
- the CA asks for verification of ownership of the domain either using a HTTP challenge for a single domain or a DNS challenge for a wildcard domain. The HTTP challenges amounts to putting a certain file with the request content (the challenge) in the URL space of the domain. The DNS challenge typically asks for a TXT record to be created in the DNS for the domain with the requested content. If someone is able to do that then ownership is confirmed
- confirm to the HTTP or DNS challenge by creating the appropriate file or DNS record respectively
- notify the CA that the challenge was answered
- the CA checks the challenge
- the CA issues the certificate after the challenge is verified.
Automatic renewal: cert-manager
There are various tools for automatically certificates updates based on the ACME protocol, but most of them are for VMs and have options for directly updating apache of nginx configuration files. Imagine what kind of horror this could be if such a script would corrupt your special/home-grown configuration. On kubernetes things are a lot more clean using cert-manager.
Cert-manager introduces a number of concepts:
- Certificate: A custom resource describing a certificate. This defines the names (DNS alternative names) that will appear on the certificate as well as the name of the TLS secret that will be created and which is used by ingress.
- Issuer: The object that is responsible for using the ACME protocol to contact Let’s Encrypt. It uses a webhook for the specific DNS provider. The task of the web hook is to create and remove the challenge records as required by Let’s Encrypt. In my setup I am using the DNS challenge since I want to get wildcard certificates.
- Webhook: A custom webhook that is used by an issuer to comply to the challenges from the CA.
Installation
Install cert-manager by following the instructions on the website. In my case, I did the following:
helm repo add jetstack https://charts.jetstack.io
helm repo updatehelm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.8.0 \
--set installCRDs=true
After this, I used cmctl to verify the installation
OS=$(go env GOOS); ARCH=$(go env GOARCH); curl -sSL -o cmctl.tar.gz https://github.com/cert-manager/cert-manager/releases/download/v1.7.2/cmctl-$OS -$ARCH.tar.gz tar xvf cmctl.tar.gz cmctl check api
The cert-manager website contains more instructions to verify the installation using a self-signed certificate. It is recommended to follow these instructions to make sure that everything works.
Next up is the installation of the DNS madeeasy webhook.
helm repo add k8s-at-home https://k8s-at-home.com/charts/ helm repo updatehelm install dnschallenger k8s-at-home/dnsmadeeasy-webhook --namespace cert-manager -f values-dnsmadeeasychallenger.yaml
The values-dnsmadeeasychallenger.yaml configuration file is used to define the groupName which is used by the issuer to identify the webhook:
values-dnsmadeeaychallenger.yaml
# This name will need to be referenced in each Issuer's `webhook` stanza to # inform cert-manager of where to send ChallengePayload resources in order to # solve the DNS01 challenge. # This group name should be **unique**, hence using your own company's domain # here is recommended. groupName: dnsmadeeasy.challenger
Note that I installed the webhook in the same namespace as cert-manager since it is an integral part of certificate management and belongs in the same namespace.
Configuration
We need to define some essential information for everything to work:
- the DNS alternative names of the certificate. These are configured in the Certificate resource.
- the name of the TLS secret that will contain the generated certificates. This is configured in the Certificate resource.
- the API key and secret to be able to use the DnsMadeEasy API. This is a separate Secret resource.
Examples for brakkee.org are as follows:
brakkee-org-certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: wildcard-brakkee-org
namespace: exposure
spec:
dnsNames:
- "*.brakkee.org"
- "brakkee.org"
issuerRef:
name: dnsmadeeasy-issuer
kind: Issuer
secretName: brakkee-org-tls-secret
Note that I am defining the root domain brakkee.org as a separate name. This is because the wildcard comain *.brakkee.org only covers subdomains of brakkee.org but not the root domain brakkee.org.
dnsmadeeasy-apikey.yaml
apiVersion: v1 kind: Secret metadata: name: dnsmadeeasy-apikey namespace: cert-manager type: Opaque stringData: key: your_key_here secret: your_secret_here
Apart from this, we need to configure the issuer to use the webhook and to pass the DnsMadeEasy API key and secret on to the webhook to comply to the challenge:
dnsmadeeasy-issuer.yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: dnsmadeeasy-issuer
namespace: exposure
spec:
acme:
#server: https://acme-staging-v02.api.letsencrypt.org/directory
server: https://acme-v02.api.letsencrypt.org/directory
email: info@brakkee.org
privateKeySecretRef:
name: hosting-key-secret
solvers:
- dns01:
webhook:
groupName: dnsmadeeasy.challenger
solverName: dnsmadeeasy
config:
apiKeyRef:
name: dnsmadeeasy-apikey
key: key
apiSecretRef:
name: dnsmadeeasy-apikey
key: secret
In the above file, note the server attribute where you can use the staging URL for Let’s Encrypt if you want to test. This is useful because of the strict rate limits on the Let’s Encrypt API. Note the groupName which links back to the webhook we installed earlier. Also note the reference to the API key sedret to pass on configuration values to the web hook. The private key is stored in a separate secret (hosting-key-secret). This secret is created by the Issuer.
After applying the above resources, first a temporary secret will be created in the exposure namespace. After it is finished, you should have a new TLS secret brakkee-org-tls-secret in the exposure namespace that can be used in an ingress rule. During the process you can do a kubectl describe on the certificate resource to see the progress.
Note that I am creating all these resources in a separate exposure namespace. This is the namespace where I will have all SSL termination using ingress. Backend applications will typically be running in different namespaces. This will turn out to be important in a future post where I will go into setting up network policies to improve security.
Using the TLS secret in an ingress rule
Using the TLS secret in an ingress rule is straightforward. In my case I am forwarding all traffic for brakkee.org and *.brakkee.org to the same HTTPD backend service. For example:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-brakkee-org
namespace: exposure
spec:
tls:
- hosts:
- "*.brakkee.org"
- "brakkee.org"
secretName: brakkee-org-tls-secret
rules:
- host: "brakkee.org"
http: &proxy_rules
paths:
- path: /
pathType: Prefix
backend:
service:
name: httpd-reverse-proxy-brakkee-org
port:
number: 80
- host: "*.brakkee.org"
http: *proxy_rules
A nice trick here is to use a YAML anchor (&proxy_rules). This allows me to avoid duplicating the same rules for brakkee.org and *.brakkee.org. The backend service httpd-reverse-proxy-brakkee-org is not shown in this post but it is not too hard to adapt the example to use your own backend service. It is easy to add more domains with their own certificates simply by adding hosts entries to the tls section and by adding rules.
Certificate renewal
I have tested certificate renewal by using the spec.renewBefore field in the Certificate to force earlier renewal. For example:
apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: wildcard-brakkee-org namespace: exposure spec: renewBefore: 2136h ...
It is not possible to request a shorter duration of the certificate since Let’s Encrypt always generates 90 day certificates. Just choose renewBefore counting back from the 90 day expiry time of the certificate.
Force regeneration of the certificate
To force regeneration of the secret you can just delete the generated secret and it will get generated again. The same applies to the host key secret.
Rate limits
If you run into a rate limit at Let’s Encrypt, note that the rate limit applies to the list of requested DNS alternative names (*.brakkee.org, brakkee.org in my case). The current rate limit is 5 times per week for each unique combination of DNS alternative names. If you run into this limit, then temporarily add another DNS alternative name such as x.y.brakkee.org to the list. This name does not fall under *.brakkee.org. Using this it is easy to work around the limit. It is of course always better to use the staging URL of Let’s Encrypt before you start using tricks.
Costs
To use the DnsMadeEasy APIs I had to upgrade my membership to a business membership which costs 75 USD per year. But at least I am not manually updating certificates and not paying a similar amount per domain.