Hosting services on Google Kubernetes Engine

This post explains how to host services on Google Compute Engine, parts of this are applicable to regular (non-GKE) Kubernetes clusters as well. This post will cover:

  • allowing multiple services to be deployed in different namespaces
  • allowing multiple (sub)domains to be deployed
  • making a service available through a public and fixed IP accessible using a host name
  • making it accessible over HTTPS using a certificate
  • HTTP redirect to HTTPS

The treatment of the subject is a bit high level with focus on what resources and fields are relevant and on the way different Kubernetes resources relate to each other. Also some snippets of YAML configuration are given. This should be sufficient to ‘roll your own’.

Deploying services in different namespaces

The basic architecture we start off with is as follows:

PlantUML Syntax:<br />
hide circle<br />
class Ingress {<br />
namespace<br />
}<br />
class Service {<br />
namespace<br />
}<br />
class Deployment {<br />
namespace<br />
<p>Ingress –> “*” Service<br />
Service –> “1” Deployment</p>

In this basic setup,  there is one Ingress object in a certain namespace and each Service that it exposes is in a separate namespace together with its deployment. There is a limitation in Ingress that does not allow it to be used when Services are in a different namespace than the Ingress resource. This limitation is expected to be lifted in a future version of Kubernetes using the Gateway API.

One workaround is to deploy a separate Ingress in every Service namespace. However, this leads to other problems since than separate IPs must be used for each service.
A better solution is to use a separate ingress namespace in which to deploy a reverse proxy (in the example httpd is used, but nginx is of course also possible) and from there to setup a reverse proxy using the cluster local DNS name for the service. Given that a service x is deployed in namespace y, it can be accessed using the DNS name x.y.svc.cluster.local.

On apache, this can be a configuration such as this:

ProxyPass /hello http://X.Y.svc.cluster.local/hello
ProxyPassReverse /hello http://X.Y.svc.cluster.local/jenkins

The architecture for two services resp. x in namespace y and x2 in namespace y2 then becomes:

PlantUML Syntax:<br />
package “namespace: exposure” {<br />
object “expose:Ingress” as expose<br />
object “httpd:Service” as httpdservice<br />
object “httpd:Deployment” as httpddeploy<br />
object “httpd-config:ConfigMap” as httpdconfig<br />
}<br />
package “namespace: y” {<br />
object “x:Service” as xservice<br />
object “x:Deployment” as xdeploy<br />
}<br />
package “namespace: x2” {<br />
object “x2:Service” as x2service<br />
object “x2:Deployment” as x2deploy<br />
}<br />
expose –> httpdservice<br />
httpdservice -> httpddeploy<br />
httpddeploy “\nx.y.svc.cluster.local” –> xservice<br />
httpddeploy -> httpdconfig<br />
xservice -> xdeploy<br />
httpddeploy “\nx2.y2.svc.cluster.local” –> x2service<br />
x2service -> x2deploy</p>

This architecture is quite flexible. It allows SSL termination on a using a single Ingress rule and provides complete flexibility in allocating services to namespaces.

The apache deployment is configured using a config map that contains the httpd.conf file in a ConfigMap resource. The ConfigMap is mounted into the pod as follows:

apiVersion: apps/v1
kind: Deployment
  name: httpd
  namespace: exposure
        - name: httpd
          image: httpd:2.4
            - name: httpd-conf
              mountPath: /usr/local/apache2/conf/httpd.conf
              subPath: httpd.conf
        - name: httpd-conf
            name: httpd-config

Note the use of the subPath configuration in the deployment spec to mount only a single file from the httpd-conf volume, which keeps all other files in the /usr/local/apache2/conf/ directory intact. The config map can be created from an existing httpd.conf file as follows:

kubectl create configmap --namespace exposure \
  httpd-config --from-file=httpd.conf

The httpd.conf used is an adaptation of the file that comes standard with the container. Since we are terminating SSL using Ingress, the httpd config is without any SSL configuration or reference to certificate files.

Exposing the service externally

To expose the httpd service externally (and thereby all services for which it is a reverse proxy), we need to introduce the FrontendConfig resource. The FrontendConfig is a custom resource that is only available on GKE as far as I know and this is used to configure Ingress features.The SSL Policy is a GCP object that defines SSL features for the HTTPS connection.

PlantUML Syntax:<br />
package “Kubernetes” {<br />
object “expose:Ingress” as expose<br />
object “frontendconfig:FrontendConfig” as frontend<br />
}<br />
package “GCP” {<br />
object “gke-ingress-ssl-policy:SSLPolicy” as policy<br />
object “example-ip:IpAddress” as ipaddress<br />
object “example-com:PreSharedCertificate” as certificate<br />
<p>expose -right-> frontend<br />
frontend -down-> policy<br />
expose -down-> ipaddress<br />
expose -down-> certificate<br />

Using this setup, we can configure:

  • the certificate used
  • HTTP to HTTPS redirect
  • the external IP address

It is most easy to look at the Ingress configuration:

kind: Ingress
  name: httpd-ingress
  annotations: "false" example-ip example-com frontendconfig
  namespace: exposure
  # tls config not needed since a pre-shared certificate
  # is used. 
  #  - hosts:
  #      -
  #    secretName: example-com-certificates

    - host:
          - path: /
            pathType: Prefix
                name: httpd
                  number: 80

First of all, we see that all traffic is forwarded to httpd, which is ok since httpd distributes the traffic over backend services and as far as Ingress just routes to httpd. Next, we see several other features.

The annotation can be used to disable http but this is commented out since we allow http and will do a redirect to https instead using the FrontendConfig.

Next, the annotation defines the name of a previously created static IP address in GCP which is the static IP address under which the services are exposed.

The annotation defines the name of the pre-shared certificate that is used.

A pre-shared certificate is a certificate registered at GCP. Based on the domain name, GCP automatically chooses the certificate. This means that explicit configuration of the certificate using the tls section is not needed and this part is therefore commented out. It also allows the same certificate to be used by multiple GKE clusters and is thus lower maintenance than using the alternative setup by defining a secret per GKE cluster for the certificates.

To create the pre-shared certificate, concatenate the crt files of the certificate containing the certificate and certificate chain and put it into a single file as follows:

  cat > full.crt

Next, create the pre-shared certificate from this file and the private key file:

gcloud compute ssl-certificates create example-com \
  --project example-project \
  --global \
  --certificate=full.crt \

The final part is the annotation which links the ingress resource to the frontend config.

The FrontendConfig, finally, is as follows:

kind: FrontendConfig
  name: frontendconfig
  namespace: exposure
    enabled: true
  sslPolicy: gke-ingress-ssl-policy

In this configuration we see that the redirect from http to https is configured. Also, there is reference to an SSL policy which defines SSL features. For instance:

gcloud compute ssl-policies create gke-ingress-ssl-policy \
    --profile MODERN \
    --min-tls-version 1.2
This entry was posted in Devops/Linux. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *