Nonsense and other useful things

These days, it is ill-advised to run a website (such as this one), over HTTP, even if there is no security risk at all. When hosting your website on HTTP, users will see a warning triangle in the address bar and many users will simply turn away. This is especially painful if the website is blogging on devops and security related things.

Currently, I am migrating everything I have locally to a more secure and future proof setup from VMs to kubernetes. The approach I decided to take is to start with the front-facing services and move everything from there step by step. Therefore, the first step is to move the reverse proxy from a VM to kubernetes. Since I don’t want to pay for certificates, I will be using Let’s Encrypt. Unfortunately, Let’s Encrypt only supports 90 day certificates so this means a lot of certificate renewals.

The ACME protocol

To support this Let’s Encrypt supports the ACME protocol, which allows for automatic certificate renewal. The ACME protocol automates the complete process. A certificate renewal process looks like this:

• create a new private key
• create a certificate signing request using the private key
• request a certificate from the CA (Certificate Authority)
• the CA asks for verification of ownership of the domain either using a HTTP challenge for a single domain or a DNS challenge for a wildcard domain. The HTTP challenges amounts to putting a certain file with the request content (the challenge) in the URL space of the domain. The DNS challenge typically asks for a TXT record to be created in the DNS for the domain with the requested content. If someone is able to do that then ownership is confirmed
• confirm to the HTTP or DNS challenge by creating the appropriate file or DNS record respectively
• notify the CA that the challenge was answered
• the CA checks the challenge
• the CA issues the certificate after the challenge is verified.

Automatic renewal: cert-manager

There are various tools for automatically certificates updates based on the ACME protocol, but most of them are for VMs and have options for directly updating apache of nginx configuration files. Imagine what kind of horrot this could be if such a script would corrupt your special/home-grown configuration. On kubernetes things are a lot more clean using cert-manager.

Cert-manager introduces a number of concepts:

• Certificate: A custom resource describing a certificate. This defines the names (DNS alternative names) that will appear on the certificate as well as the name of the TLS secret that will be created and which is used by ingress.
• Issuer: The object that is responsible for using the ACME protocol to contact Let’s Encrypt. It uses a webhook for the specific DNS provider. The task of the web hook is to create and remove the challenge records as required by Let’s Encrypt. In my setup I am using the DNS challenge since I want to get wildcard certificates.
• Webhook: A custom webhook that is used by an issuer to comply to the challenges from the CA.

Installation

Install cert-manager by following the instructions on the website. In my case, I did the following:

helm repo add jetstack https://charts.jetstack.io
helm repo updatehelm install \
  cert-manager jetstack/cert-manager \
    --namespace cert-manager \
    --create-namespace \
    --version v1.8.0 \
    --set installCRDs=true

After this, I used cmctl to verify the installation

OS=$(go env GOOS); ARCH=$(go env GOARCH); curl -sSL -o cmctl.tar.gz https://github.com/cert-manager/cert-manager/releases/download/v1.7.2/cmctl-$OS
-$ARCH.tar.gz
tar xvf cmctl.tar.gz
cmctl check api

The cert-manager website contains  more instructions to verify the installation using a self-signed certificate. It is recommended to follow these instructions to make sure that everything works.

Next up is the installation of the DNS madeeasy webhook.

helm repo add k8s-at-home https://k8s-at-home.com/charts/
helm repo updatehelm install dnschallenger k8s-at-home/dnsmadeeasy-webhook --namespace cert-manager -f values-dnsmadeeasychallenger.yaml 

The values-dnsmadeeasychallenger.yaml configuration file is used to define the groupName which is used by the issuer to identify the webhook:

values-dnsmadeeaychallenger.yaml

# This name will need to be referenced in each Issuer's `webhook` stanza to
# inform cert-manager of where to send ChallengePayload resources in order to
# solve the DNS01 challenge.
# This group name should be **unique**, hence using your own company's domain
# here is recommended.
groupName: dnsmadeeasy.challenger

Note that I installed the webhook in the same namespace as cert-manager since it is an integral part of certificate management and belongs in the same namespace.

Configuration

We need to define some essential information for everything to work:

• the DNS alternative names of the certificate. These are configured in the Certificate resource.
• the name of the TLS secret that will contain the generated certificates. This is configured in the Certificate resource.
• the API key and secret to be able to use the DnsMadeEasy API. This is a separate Secret resource.

Examples for brakkee.org are as follows:

brakkee-org-certificate.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-brakkee-org
  namespace: exposure
spec:
  dnsNames:
  - "*.brakkee.org"
  - "brakkee.org"
  issuerRef:
    name: dnsmadeeasy-issuer
    kind: Issuer
  secretName: brakkee-org-tls-secret

Note that I am defining the root domain brakkee.org as a separate name. This is because the wildcard comain *.brakkee.org only covers subdomains of brakkee.org but not the root domain brakkee.org.

dnsmadeeasy-apikey.yaml

apiVersion: v1
kind: Secret
metadata:
  name: dnsmadeeasy-apikey
  namespace: cert-manager
type: Opaque
stringData:
  key: your_key_here
  secret: your_secret_here

Apart from this, we need to configure the issuer to use the webhook and to pass the DnsMadeEasy API key and secret on to the webhook to comply to the challenge:

dnsmadeeasy-issuer.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: dnsmadeeasy-issuer
  namespace: exposure
spec:
  acme:
    #server: https://acme-staging-v02.api.letsencrypt.org/directory
    server: https://acme-v02.api.letsencrypt.org/directory
    email: info@brakkee.org
    privateKeySecretRef:
      name: hosting-key-secret
    solvers:
    - dns01:
        webhook:
          groupName: dnsmadeeasy.challenger
          solverName: dnsmadeeasy
          config:
            apiKeyRef:
              name: dnsmadeeasy-apikey
              key: key
            apiSecretRef:
              name: dnsmadeeasy-apikey
              key: secret

In the above file, note the server attribute where you can use the staging URL for Let’s Encrypt if you want to test. This is useful because of the strict rate limits on the Let’s Encrypt API. Note the groupName  which links back to the webhook we installed earlier. Also note the reference to the API key sedret to pass on configuration values to the web hook. The private key is stored in a separate secret (hosting-key-secret). This secret is created by the Issuer.

After applying the above resources, first a temporary secret will be created in the exposure namespace. After it is finished, you should have a new TLS secret brakkee-org-tls-secret in the exposure namespace that can be used in an ingress rule. During the process you can do a kubectl describe on the certificate resource to see the progress.

Note that I am creating all these resources in a separate exposure namespace. This is the namespace where I will have all SSL termination using ingress. Backend applications will typically be running in different namespaces. This will turn out to be important in a future post where I will go into setting up network policies to improve security.

Using the TLS secret in an ingress rule

Using the TLS secret in an ingress rule is straightforward. In my case I am forwarding all traffic for brakkee.org and *.brakkee.org to the same HTTPD backend service. For example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-brakkee-org
  namespace: exposure
spec: 
  tls: 
  - hosts:
    - "*.brakkee.org"
    - "brakkee.org"
    secretName: brakkee-org-tls-secret
  rules:
  - host: "brakkee.org"
    http: &proxy_rules
      paths:
      - path: /
        pathType: Prefix 
        backend:
          service:
            name: httpd-reverse-proxy-brakkee-org
            port: 
              number: 80
  - host: "*.brakkee.org"
    http: *proxy_rules

A nice trick here is to use a YAML anchor (&proxy_rules). This allows me to avoid duplicating the same rules for brakkee.org and *.brakkee.org. The backend service httpd-reverse-proxy-brakkee-org is not shown in this post but it is not too hard to adapt the example to use your own backend service. It is easy to add more domains with their own certificates simply by adding hosts entries to the tls section and by adding rules.

Certificate renewal

I have tested certificate renewal by using the spec.renewBefore field in the Certificate to force earlier renewal. For example:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-brakkee-org
  namespace: exposure
spec:
  renewBefore: 2136h
  ...

It is not possible to request a shorter duration of the certificate since Let’s Encrypt always generates 90 day certificates. Just choose renewBefore counting back from the 90 day expiry time of the certificate.

Force regeneration of the certificate

To force regeneration of the secret you can just delete the generated secret and it will get generated again. The same applies to the host key secret.

Rate limits

If you run into a rate limit at Let’s Encrypt, note that the rate limit applies to the list of requested DNS alternative names (*.brakkee.org, brakkee.org in my case). The current rate limit is 5 times per week for each unique combination of DNS alternative names. If you run into this limit, then temporarily add another DNS alternative name such as x.y.brakkee.org to the list. This name does not fall under *.brakkee.org. Using this it is easy to work around the limit. It is of course always better to use the staging URL of Let’s Encrypt before you start using tricks.

Costs

To use the DnsMadeEasy APIs I had to upgrade my membership to a business membership which costs 75 USD per year. But at least I am not manually updating certificates and not paying a similar amount per domain.

This post explains how to host services on Google Compute Engine, parts of this are applicable to regular (non-GKE) Kubernetes clusters as well. This post will cover:

• allowing multiple services to be deployed in different namespaces
• allowing multiple (sub)domains to be deployed
• making a service available through a public and fixed IP accessible using a host name
• making it accessible over HTTPS using a certificate
• HTTP redirect to HTTPS

The treatment of the subject is a bit high level with focus on what resources and fields are relevant and on the way different Kubernetes resources relate to each other. Also some snippets of YAML configuration are given. This should be sufficient to ‘roll your own’.

Deploying services in different namespaces

The basic architecture we start off with is as follows:

In this basic setup,  there is one Ingress object in a certain namespace and each Service that it exposes is in a separate namespace together with its deployment. There is a limitation in Ingress that does not allow it to be used when Services are in a different namespace than the Ingress resource. This limitation is expected to be lifted in a future version of Kubernetes using the Gateway API.

One workaround is to deploy a separate Ingress in every Service namespace. However, this leads to other problems since than separate IPs must be used for each service.
A better solution is to use a separate ingress namespace in which to deploy a reverse proxy (in the example httpd is used, but nginx is of course also possible) and from there to setup a reverse proxy using the cluster local DNS name for the service. Given that a service x is deployed in namespace y, it can be accessed using the DNS name x.y.svc.cluster.local.

On apache, this can be a configuration such as this:

ProxyPass /hello http://X.Y.svc.cluster.local/hello
ProxyPassReverse /hello http://X.Y.svc.cluster.local/jenkins


The architecture for two services resp. x in namespace y and x2 in namespace y2 then becomes:

This architecture is quite flexible. It allows SSL termination on a using a single Ingress rule and provides complete flexibility in allocating services to namespaces.

The apache deployment is configured using a config map that contains the httpd.conf file in a ConfigMap resource. The ConfigMap is mounted into the pod as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd
  namespace: exposure
  ...
spec:
  ...
  template:
    ...
    spec:
      containers:
        - name: httpd
          image: httpd:2.4
          ...
          volumeMounts:
            - name: httpd-conf
              mountPath: /usr/local/apache2/conf/httpd.conf
              subPath: httpd.conf
      volumes:
        - name: httpd-conf
          configMap: 
            name: httpd-config

Note the use of the subPath configuration in the deployment spec to mount only a single file from the httpd-conf volume, which keeps all other files in the /usr/local/apache2/conf/ directory intact. The config map can be created from an existing httpd.conf file as follows:

kubectl create configmap --namespace exposure \
  httpd-config --from-file=httpd.conf

The httpd.conf used is an adaptation of the file that comes standard with the container. Since we are terminating SSL using Ingress, the httpd config is without any SSL configuration or reference to certificate files.

Exposing the service externally

To expose the httpd service externally (and thereby all services for which it is a reverse proxy), we need to introduce the FrontendConfig resource. The FrontendConfig is a custom resource that is only available on GKE as far as I know and this is used to configure Ingress features.The SSL Policy is a GCP object that defines SSL features for the HTTPS connection.

Using this setup, we can configure:

• the certificate used
• HTTP to HTTPS redirect
• the external IP address

It is most easy to look at the Ingress configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpd-ingress
  annotations:
    #kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.global-static-ip-name: example-ip
    ingress.gcp.kubernetes.io/pre-shared-cert: example-com
    networking.gke.io/v1beta1.FrontendConfig: frontendconfig
  namespace: exposure
spec:
  # tls config not needed since a pre-shared certificate
  # is used. 
  #tls:
  #  - hosts:
  #      - example.com
  #    secretName: example-com-certificates

  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: httpd
                port:
                  number: 80

First of all, we see that all traffic is forwarded to httpd, which is ok since httpd distributes the traffic over backend services and as far as Ingress just routes to httpd. Next, we see several other features.

The annotation kubernetes.io/ingress.allow-http can be used to disable http but this is commented out since we allow http and will do a redirect to https instead using the FrontendConfig.

Next, the annotation kubernetes.io/ingress.global-static-ip-name defines the name of a previously created static IP address in GCP which is the static IP address under which the services are exposed.

The annotation ingress.gcp.kubernetes.io/pre-shared-cert defines the name of the pre-shared certificate that is used.

A pre-shared certificate is a certificate registered at GCP. Based on the domain name, GCP automatically chooses the certificate. This means that explicit configuration of the certificate using the tls section is not needed and this part is therefore commented out. It also allows the same certificate to be used by multiple GKE clusters and is thus lower maintenance than using the alternative setup by defining a secret per GKE cluster for the certificates.

To create the pre-shared certificate, concatenate the crt files of the certificate containing the certificate and certificate chain and put it into a single file as follows:

  cat example.com.crt example.com.2022.crt > full.crt

Next, create the pre-shared certificate from this file and the private key file:

gcloud compute ssl-certificates create example-com \
  --project example-project \
  --global \
  --certificate=full.crt \
  --private-key=example_2022.key

The final part is the networking.gke.io/v1beta1.FrontendConfig annotation which links the ingress resource to the frontend config.

The FrontendConfig, finally, is as follows:

apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
  name: frontendconfig
  namespace: exposure
spec:
  redirectToHttps:
    enabled: true
  sslPolicy: gke-ingress-ssl-policy

In this configuration we see that the redirect from http to https is configured. Also, there is reference to an SSL policy which defines SSL features. For instance:

gcloud compute ssl-policies create gke-ingress-ssl-policy \
    --profile MODERN \
    --min-tls-version 1.2