Earlier this month I wrote about using cert-manager to automatically generate Let’s Encrypt certificates for Ingress resources in Kubernetes. Part of the setup involved adding a Secret containing the DNS provider’s API token, for use with the DNS-01 verification.

In real-world scenarios however, your YAML manifests may be stored in a git repository visible to the entire team or organization. In such cases, it is not desirable to have Secrets stored in plain text for everyone to see, especially when the contents of said Secret would enable anyone to add/remove DNS records on a production DNS zone.

One possible mitigation in the specific case of cert-manager is to use the delegated domains approach and create a separate, limited account in which the DNS records are only intended to be used for DNS-01 validation. The problems with this approach is that depending on organization policies, budget, limitations on delegation imposed by the DNS provider, etc. this may not always be feasible. Furthermore, this does not change anything to the fact that the credentials are stored in plain text.

One other possible approach that I am describing here is to use SealedSecrets to asymetrically encrypt secrets inside the repository.

Installing sealed-secrets

Installing sealed-secrets is pretty simple and you can simply follow the official instructions. In my case, I downloaded a copy of the controller.yaml file from the Releases page and as usual used Kustomize to apply it in case I want to apply any patches to the upstream manifest in the future.

We’ll also want to install the client-side command line tool, which can also easily be done following the instructions on the Releases page (v0.12.4 at the time of writing):

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.4/kubeseal-linux-amd64 -O kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Converting an existing Secret to a SealedSecret

If you set up cert-manager following my previous guide we can easily pick up from there and upgrade our Secret to a SealedSecret.

First, simply add the required annotation to cf-secret.yaml:

 apiVersion: v1
 kind: Secret
 metadata:
   name: cf-api-key
+  annotations:
+    sealedsecrets.bitnami.com/managed: "true"
 type: Opaque
 stringData:
   api-token: <api token>

If you are starting from scratch, you could also simply create a Secret like the above, without the annotation. Once we have our original Secret set-up, we’ll feed it to kubeseal to convert it:

kubeseal -o yaml -n cert-manager < cert-manager/cf-secret.yaml > cert-manager/cf-sealedsecret.yaml

Here, we are using the recommended default strict mode which tightly couples the secret with its name and namespace.

Applying the SealedSecret

Next we’ll modify our kustomization.yaml to include our newly created SealedSecret. For the time being, we’ll keep our original cf-secret.yaml so that our sealedsecrets.bitnami.com/managed: "true" annotation gets applied.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cert-manager.yaml
  - cf-secret.yaml
+  - cf-sealedsecret.yaml
  - cf-issuer.yaml
namespace: cert-manager

Then simply re-apply the kustomization:

kubectl apply -k ./cert-manager/

Once everything gets reconciliated, you’ll observe that API token in your original secret/cf-api-key has been encrypted:

kubectl -n cert-manager get sealedsecret cf-api-key -o json

You can now delete your original cf-secret.yaml and remove it from kustomization.yaml as it is no longer needed.

An important thing to note is that if you already committed and pushed cf-secret.yaml to a shared repository, converting it to a SealedSecret does not change anything to that fact and anyone looking at the git history can retrieve the original token. In such cases, instead of converting the Secret to a SealedSecret you should first revoke and recreate the API token and store it as a SealedSecret.

Lastly, anyone able to view sealed-secrets’s own Secret which contains the private key used to unseal SealedSecrets could be able to decrypt them. Unsealed Secrets are also available in plain text. It is therefore the responsibility of cluster administrators to properly configure RBAC rules around this accordingly.