Automatic TLS certificates with cert-manager and ingress-nginx
At the time of writing, cert-manager
is still in alpha version and parts of this guide may no longer be valid. This guide assumes a kubernetes cluster using ingress-nginx
, however steps involving the ingress can be easily adapted to other ingress solutions.
How to get an API key from your DNS provider is also not covered. At the very least you need an API key with read permission on your zone and edit permission on its DNS records. In this guide I use CloudFlare with the API key generated from the Edit Zone DNS
template.
Setting up our project
I like to use Kustomize for organizing an applying manifests instead of running kubectl apply -f
on individual files. It is also convenient for when you decide to use ArgoCD to automatically apply them. Here’s how our project will look like:
cert-manager/
kustomization.yaml
cert-manager.yaml
cf-secret.yaml
cf-issuer.yaml
grafana/
kustomization.yaml
...
ingress.yaml
kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cert-manager.yaml
- cf-secret.yaml
- cf-issuer.yaml
namespace: cert-manager
cert-manager
provides Helm charts for installation, however for learning purposes it’s a good idea to look at the manifests you’re going to apply and understand what they do. The contents of cert-manager.yaml
are taken from the official releases with any hard-coded references to the kube-system
namespace changed to cert-manager
. As a matter of good practice, we’ll use a different namespace per project.
One more thing we changed from the default cert-manager.yaml
is that we added --enable-certificate-owner-ref=true
to the arguments for the cert-manager
container in the cert-manager
Deployment. What this does is cascade deletion to the automatically generated private key when we delete a Certificate resource. If you intend to always use the same private key you can omit this change.
The grafana
project is only there to provide an example of using cert-manager
to automatically provide TLS certificates to an Ingress.
Installing the DNS provider’s API key
Edit cf-secret.yaml
, substituting <api token>
with your API key.
apiVersion: v1
kind: Secret
metadata:
name: cf-api-key
type: Opaque
stringData:
api-token: <api token>
Creating the CloudFlare issuer (test)
In order not to accidentally hit Let’s Encrypt’s rate limit, we’ll first use their staging issuer to verify everything is working as intended. Also, cert-manager
provides two types of issuers: Issuer
and ClusterIssuer
. An Issuer
is scoped to the namespace in which it is defined which is useful if you want to have a different issuer per project or in some multi-tenancy situations. A ClusterIssuer
can issue certificates in any namespace. Here we will be using a ClusterIssuer
, though the only difference definition-wise is the kind
and where you place the YAML file.
Edit cf-issuer.yaml
, substituting <email>
with an email to provide to Let’s Encrypt (required):
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-test
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: <email>
privateKeySecretRef:
name: letsencrypt-test
solvers:
- dns01:
cloudflare:
email: <email>
apiTokenSecretRef:
name: cf-api-key
key: api-token
Let’s Encrypt supports HTTP-01
and DNS-01
solvers. An HTTP-01
solver requires public access to a .well-known
directory on the endpoint for which you are requesting a certificate and is therefore not suited for local development.
Install cert-manager with the test issuer
Apply the kustomizations in cert-manager
:
kubectl apply -k ./cert-manager/
You should now be able to query for resources in the cert-manager
namespace, for instance:
kubectl get pod,service,deployment,clusterissuer -n cert-manager
It may take some time, but the clusterissuer.cert-manager.io/letsencrypt-test
resource should show up as Ready.
Issue a test certificate on an ingress
Next we’ll grab a certificate for the ingress on our grafana
sample project. The ingress.yaml
should look something like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: grafana-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-test"
spec:
tls:
- hosts:
- grafana.k8s.example.com
secretName: grafana-tls
rules:
- http:
paths:
- backend:
serviceName: grafana
servicePort: grafana-port
host: grafana.k8s.example.com
The cert-manager.io/cluster-issuer: "letsencrypt-test"
annotation is where the magic happens. This tells cert-manager
to use the letsencrypt-test
ClusterIssuer to issue a certificate for our Ingress when it sees this annotation. The TLS hosts
tells us for which FQDN host we want to issue the certificate for and secretName
tells us the name of the Secret resource in which we want to store the private key for the certificate.
Then all we need to do is apply the kustomizations for our grafana
project (the contents of the project being out of the scope of this guide):
kubectl apply -k ./grafana/
You should start seeing the certificate being issued. For instance, our grafana
project uses the namespace monitoring
so we’ll check for it using:
kubectl get certificate,secret,certificaterequest -n monitoring
You’ll see that a certificate.cert-manager.io
, certificaterequest.cert-manager/io
and secret
resource has been generated a few seconds ago for grafana-tls
. The Certificate might not show up as Ready yet, and this is normal as DNS-01 validation might take some time. You can get more detailed information using kubectl describe
on the CertificateRequest.
At some point (1-2 minutes or so) the certificate should be ready, and you should be able to visit your test service over HTTPS. Your browser will show a warning saying the certificate can’t be trusted, which is normal as we are using the Let’s Encrypt staging environment. You can confirm that by checking the certificate. It should say something about it being signed by some Fake Let’s Encrypt CA or something along those lines.
Switchover to the production issuer and certificate
Now that we’ve verified that we can issue certificates as intended, we’ll switch out to Let’s Encrypt’s production issuer. We’ll edit cf-issuer.yaml
and change the values to
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: <email>
privateKeySecretRef:
name: letsencrypt
solvers:
- dns01:
cloudflare:
email: <email>
apiTokenSecretRef:
name: cf-api-key
key: api-token
And simply re-apply the kustomization:
kubectl apply -k ./cert-manager/
We’ll also need to tell our ingress to use the new issuer (ingress.yaml):
- cert-manager.io/cluster-issuer: "letsencrypt-test"
+ cert-manager.io/cluster-issuer: "letsencrypt"
And apply:
kubectl apply -k ./grafana/
At this point, no new certificates are created. The reason for that is that our test certificate exists and is still valid. We’ll simply delete it:
kubectl delete certificate grafana-tls
If in earlier steps we edited cert-manager.yaml
and added the --enable-certificate-owner-ref=true
argument, the private key is also deleted accordingly.
Now, our new certificate should have been automatically requested and is being processed:
kubectl get certificate,secret,certificaterequest -n monitoring
Once the certificate is Ready, we can visit our endpoint over HTTPS and it should not show any certificate error.
After we made sure everything is working, we can also delete our test issuer:
kubectl -n cert-manager delete clusterissuer letsencrypt-test
The cool thing about using the DNS-01 approach is that as long as we can write DNS records, it doesn’t matter if our ingress resources are publicly available or only available locally. For instance, you can have *.k8s.example.com
point to 127.0.0.1
and still obtain a valid TLS certificate. Since .dev
domains have HSTS enabled at the TLD level, by using this approach you will still be able to test locally with a valid certificate.
In a possible follow-up post, we’ll see how to use cert-manager
to roll your own Certificate Authority to issue certificates used for mutual TLS (mTLS) between endpoints in different clusters. For instance to securely connect to a remote Prometheus instance in Grafana.