Automatic TLS certificates with cert-manager and ingress-nginx // Hsn723

At the time of writing, cert-manager is still in alpha version and parts of this guide may no longer be valid. This guide assumes a kubernetes cluster using ingress-nginx, however steps involving the ingress can be easily adapted to other ingress solutions.

How to get an API key from your DNS provider is also not covered. At the very least you need an API key with read permission on your zone and edit permission on its DNS records. In this guide I use CloudFlare with the API key generated from the Edit Zone DNS template.

Setting up our project

I like to use Kustomize for organizing an applying manifests instead of running kubectl apply -f on individual files. It is also convenient for when you decide to use ArgoCD to automatically apply them. Here’s how our project will look like:

cert-manager/
    kustomization.yaml
    cert-manager.yaml
    cf-secret.yaml
    cf-issuer.yaml

grafana/
    kustomization.yaml
    ...
    ingress.yaml

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cert-manager.yaml
  - cf-secret.yaml
  - cf-issuer.yaml
namespace: cert-manager

cert-manager provides Helm charts for installation, however for learning purposes it’s a good idea to look at the manifests you’re going to apply and understand what they do. The contents of cert-manager.yaml are taken from the official releases with any hard-coded references to the kube-system namespace changed to cert-manager. As a matter of good practice, we’ll use a different namespace per project.

One more thing we changed from the default cert-manager.yaml is that we added --enable-certificate-owner-ref=true to the arguments for the cert-manager container in the cert-manager Deployment. What this does is cascade deletion to the automatically generated private key when we delete a Certificate resource. If you intend to always use the same private key you can omit this change.

The grafana project is only there to provide an example of using cert-manager to automatically provide TLS certificates to an Ingress.

Installing the DNS provider’s API key

Edit cf-secret.yaml, substituting <api token> with your API key.

 apiVersion: v1
 kind: Secret
 metadata:
   name: cf-api-key
 type: Opaque
 stringData:
   api-token: <api token>

Creating the CloudFlare issuer (test)

In order not to accidentally hit Let’s Encrypt’s rate limit, we’ll first use their staging issuer to verify everything is working as intended. Also, cert-manager provides two types of issuers: Issuer and ClusterIssuer. An Issuer is scoped to the namespace in which it is defined which is useful if you want to have a different issuer per project or in some multi-tenancy situations. A ClusterIssuer can issue certificates in any namespace. Here we will be using a ClusterIssuer, though the only difference definition-wise is the kind and where you place the YAML file.

Edit cf-issuer.yaml, substituting <email> with an email to provide to Let’s Encrypt (required):

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-test
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: <email>
    privateKeySecretRef:
      name: letsencrypt-test
    solvers:
    - dns01:
        cloudflare:
          email: <email>
          apiTokenSecretRef:
            name: cf-api-key
            key: api-token

Let’s Encrypt supports HTTP-01 and DNS-01 solvers. An HTTP-01 solver requires public access to a .well-known directory on the endpoint for which you are requesting a certificate and is therefore not suited for local development.

Install cert-manager with the test issuer

Apply the kustomizations in cert-manager:

kubectl apply -k ./cert-manager/

You should now be able to query for resources in the cert-manager namespace, for instance:

kubectl get pod,service,deployment,clusterissuer -n cert-manager

It may take some time, but the clusterissuer.cert-manager.io/letsencrypt-test resource should show up as Ready.

Issue a test certificate on an ingress

Next we’ll grab a certificate for the ingress on our grafana sample project. The ingress.yaml should look something like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grafana-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-test"
spec:
  tls:
  - hosts:
    - grafana.k8s.example.com
    secretName: grafana-tls
  rules:
  - http:
      paths:
      - backend:
          serviceName: grafana
          servicePort: grafana-port
    host: grafana.k8s.example.com

The cert-manager.io/cluster-issuer: "letsencrypt-test" annotation is where the magic happens. This tells cert-manager to use the letsencrypt-test ClusterIssuer to issue a certificate for our Ingress when it sees this annotation. The TLS hosts tells us for which FQDN host we want to issue the certificate for and secretName tells us the name of the Secret resource in which we want to store the private key for the certificate.

Then all we need to do is apply the kustomizations for our grafana project (the contents of the project being out of the scope of this guide):

kubectl apply -k ./grafana/

You should start seeing the certificate being issued. For instance, our grafana project uses the namespace monitoring so we’ll check for it using:

kubectl get certificate,secret,certificaterequest -n monitoring

You’ll see that a certificate.cert-manager.io, certificaterequest.cert-manager/io and secret resource has been generated a few seconds ago for grafana-tls. The Certificate might not show up as Ready yet, and this is normal as DNS-01 validation might take some time. You can get more detailed information using kubectl describe on the CertificateRequest.

At some point (1-2 minutes or so) the certificate should be ready, and you should be able to visit your test service over HTTPS. Your browser will show a warning saying the certificate can’t be trusted, which is normal as we are using the Let’s Encrypt staging environment. You can confirm that by checking the certificate. It should say something about it being signed by some Fake Let’s Encrypt CA or something along those lines.

Switchover to the production issuer and certificate

Now that we’ve verified that we can issue certificates as intended, we’ll switch out to Let’s Encrypt’s production issuer. We’ll edit cf-issuer.yaml and change the values to

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <email>
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - dns01:
        cloudflare:
          email: <email>
          apiTokenSecretRef:
            name: cf-api-key
            key: api-token

And simply re-apply the kustomization:

kubectl apply -k ./cert-manager/

We’ll also need to tell our ingress to use the new issuer (ingress.yaml):

-    cert-manager.io/cluster-issuer: "letsencrypt-test"
+    cert-manager.io/cluster-issuer: "letsencrypt"

And apply:

kubectl apply -k ./grafana/

At this point, no new certificates are created. The reason for that is that our test certificate exists and is still valid. We’ll simply delete it:

kubectl delete certificate grafana-tls

If in earlier steps we edited cert-manager.yaml and added the --enable-certificate-owner-ref=true argument, the private key is also deleted accordingly.

Now, our new certificate should have been automatically requested and is being processed:

kubectl get certificate,secret,certificaterequest -n monitoring

Once the certificate is Ready, we can visit our endpoint over HTTPS and it should not show any certificate error.

After we made sure everything is working, we can also delete our test issuer:

kubectl -n cert-manager delete clusterissuer letsencrypt-test

The cool thing about using the DNS-01 approach is that as long as we can write DNS records, it doesn’t matter if our ingress resources are publicly available or only available locally. For instance, you can have *.k8s.example.com point to 127.0.0.1 and still obtain a valid TLS certificate. Since .dev domains have HSTS enabled at the TLD level, by using this approach you will still be able to test locally with a valid certificate.

In a possible follow-up post, we’ll see how to use cert-manager to roll your own Certificate Authority to issue certificates used for mutual TLS (mTLS) between endpoints in different clusters. For instance to securely connect to a remote Prometheus instance in Grafana.