Earlier this month I wrote about using
cert-manager to automatically generate Let’s Encrypt certificates for Ingress resources in Kubernetes. Part of the setup involved adding a Secret containing the DNS provider’s API token, for use with the DNS-01 verification.
In real-world scenarios however, your YAML manifests may be stored in a git repository visible to the entire team or organization. In such cases, it is not desirable to have Secrets stored in plain text for everyone to see, especially when the contents of said Secret would enable anyone to add/remove DNS records on a production DNS zone.
One possible mitigation in the specific case of
cert-manager is to use the delegated domains approach and create a separate, limited account in which the DNS records are only intended to be used for DNS-01 validation. The problems with this approach is that depending on organization policies, budget, limitations on delegation imposed by the DNS provider, etc. this may not always be feasible. Furthermore, this does not change anything to the fact that the credentials are stored in plain text.
One other possible approach that I am describing here is to use SealedSecrets to asymetrically encrypt secrets inside the repository.
sealed-secrets is pretty simple and you can simply follow the official instructions. In my case, I downloaded a copy of the
controller.yaml file from the Releases page and as usual used Kustomize to apply it in case I want to apply any patches to the upstream manifest in the future.
We’ll also want to install the client-side command line tool, which can also easily be done following the instructions on the Releases page (v0.12.4 at the time of writing):
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.4/kubeseal-linux-amd64 -O kubeseal sudo install -m 755 kubeseal /usr/local/bin/kubeseal
Converting an existing Secret to a SealedSecret
If you set up
cert-manager following my previous guide we can easily pick up from there and upgrade our Secret to a SealedSecret.
First, simply add the required annotation to
apiVersion: v1 kind: Secret metadata: name: cf-api-key + sealedsecrets.bitnami.com/managed: "true" type: Opaque stringData: api-token: <api token>
If you are starting from scratch, you could also simply create a Secret like the above, without the annotation. Once we have our original Secret set-up, we’ll feed it to
kubeseal to convert it:
kubeseal -o yaml -n cert-manager < cert-manager/cf-secret.yaml > cert-manager/cf-sealedsecret.yaml
Here, we are using the recommended default
strict mode which tightly couples the secret with its name and namespace.
Applying the SealedSecret
Next we’ll modify our
kustomization.yaml to include our newly created SealedSecret. For the time being, we’ll keep our original
cf-secret.yaml so that our
sealedsecrets.bitnami.com/managed: "true" annotation gets applied.
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - cert-manager.yaml - cf-secret.yaml + - cf-sealedsecret.yaml - cf-issuer.yaml namespace: cert-manager
Then simply re-apply the kustomization:
kubectl apply -k ./cert-manager/
Once everything gets reconciliated, you’ll observe that API token in your original
secret/cf-api-key has been encrypted:
kubectl -n cert-manager get sealedsecret cf-api-key -o json
You can now delete your original
cf-secret.yaml and remove it from
kustomization.yaml as it is no longer needed.
An important thing to note is that if you already committed and pushed
cf-secret.yaml to a shared repository, converting it to a SealedSecret does not change anything to that fact and anyone looking at the git history can retrieve the original token. In such cases, instead of converting the Secret to a SealedSecret you should first revoke and recreate the API token and store it as a SealedSecret.
Lastly, anyone able to view
sealed-secrets’s own Secret which contains the private key used to unseal SealedSecrets could be able to decrypt them. Unsealed Secrets are also available in plain text. It is therefore the responsibility of cluster administrators to properly configure RBAC rules around this accordingly.