Using SealedSecrets to protect sensitive keys
Earlier this month I wrote about using cert-manager
to automatically generate Let’s Encrypt certificates for Ingress resources in Kubernetes. Part of the setup involved adding a Secret containing the DNS provider’s API token, for use with the DNS-01 verification.
In real-world scenarios however, your YAML manifests may be stored in a git repository visible to the entire team or organization. In such cases, it is not desirable to have Secrets stored in plain text for everyone to see, especially when the contents of said Secret would enable anyone to add/remove DNS records on a production DNS zone.
One possible mitigation in the specific case of cert-manager
is to use the delegated domains approach and create a separate, limited account in which the DNS records are only intended to be used for DNS-01 validation. The problems with this approach is that depending on organization policies, budget, limitations on delegation imposed by the DNS provider, etc. this may not always be feasible. Furthermore, this does not change anything to the fact that the credentials are stored in plain text.
One other possible approach that I am describing here is to use SealedSecrets to asymetrically encrypt secrets inside the repository.
Installing sealed-secrets
Installing sealed-secrets
is pretty simple and you can simply follow the official instructions. In my case, I downloaded a copy of the controller.yaml
file from the Releases page and as usual used Kustomize to apply it in case I want to apply any patches to the upstream manifest in the future.
We’ll also want to install the client-side command line tool, which can also easily be done following the instructions on the Releases page (v0.12.4 at the time of writing):
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.4/kubeseal-linux-amd64 -O kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal
Converting an existing Secret to a SealedSecret
If you set up cert-manager
following my previous guide we can easily pick up from there and upgrade our Secret to a SealedSecret.
First, simply add the required annotation to cf-secret.yaml
:
apiVersion: v1
kind: Secret
metadata:
name: cf-api-key
+ annotations:
+ sealedsecrets.bitnami.com/managed: "true"
type: Opaque
stringData:
api-token: <api token>
If you are starting from scratch, you could also simply create a Secret like the above, without the annotation. Once we have our original Secret set-up, we’ll feed it to kubeseal
to convert it:
kubeseal -o yaml -n cert-manager < cert-manager/cf-secret.yaml > cert-manager/cf-sealedsecret.yaml
Here, we are using the recommended default strict
mode which tightly couples the secret with its name and namespace.
Applying the SealedSecret
Next we’ll modify our kustomization.yaml
to include our newly created SealedSecret. For the time being, we’ll keep our original cf-secret.yaml
so that our sealedsecrets.bitnami.com/managed: "true"
annotation gets applied.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cert-manager.yaml
- cf-secret.yaml
+ - cf-sealedsecret.yaml
- cf-issuer.yaml
namespace: cert-manager
Then simply re-apply the kustomization:
kubectl apply -k ./cert-manager/
Once everything gets reconciliated, you’ll observe that API token in your original secret/cf-api-key
has been encrypted:
kubectl -n cert-manager get sealedsecret cf-api-key -o json
You can now delete your original cf-secret.yaml
and remove it from kustomization.yaml
as it is no longer needed.
An important thing to note is that if you already committed and pushed cf-secret.yaml
to a shared repository, converting it to a SealedSecret does not change anything to that fact and anyone looking at the git history can retrieve the original token. In such cases, instead of converting the Secret to a SealedSecret you should first revoke and recreate the API token and store it as a SealedSecret.
Lastly, anyone able to view sealed-secrets
’s own Secret which contains the private key used to unseal SealedSecrets could be able to decrypt them. Unsealed Secrets are also available in plain text. It is therefore the responsibility of cluster administrators to properly configure RBAC rules around this accordingly.