A Kubernetes Secret looks like the answer to the question its name asks. It is not. Its values are base64, an encoding anyone can reverse, and by default the API server stores them unencrypted in etcd. Whoever can read the Secret, or a backup of etcd, reads the password.
In the teams I work with, the question came back as soon as the infrastructure and the configuration of the environments moved into code. Plain-text secrets in the repository were never an option. Promoting a release from one environment to the next means changing its secrets, especially in a DevSecOps approach. Once both are automated, a secrets manager is no longer optional.
This article walks through the review I presented this spring: eight tools, a verdict for each, the one I recommended, HashiCorp Vault, and the three ways it can deliver a secret to a pod. Choosing between them depends less on Vault than on where you want the secret to end up.
Build time and run time are two problems
A secret is not only a password. Any data that cannot be made public is one: a login and its password, an encryption key, a passphrase, a private key or a certificate, a token between two services, bank details. Between development and production, several people with different levels of access handle and change it.
At build time, the pipeline needs credentials of its own, and the CI tool keeps
them: protected and masked variables in GitLab, the Credentials plugin in
Jenkins. The non-sensitive configuration has four usual homes: Helm values per
environment (values-dev.yaml, values-prod.yaml), one application file per
environment, a ConfigMap, or a configuration server.
At run time, the application needs its secrets while it runs, and they have to come from a secrets management system. That is the problem the review addressed.
Eight tools, eight verdicts
The review started with what the existing tooling already offered:
- Ansible Vault encrypts secrets in a file for Ansible playbooks. Off topic: the applications were not deployed with Ansible.
- Helm Secrets, a Helm plugin, encrypts the values file kept in Git and decrypts it on the fly. Not enough: a light solution, tightly coupled to Helm.
- Barbican, the OpenStack service for secrets, certificates and encryption keys. Not recommended: complex integration, limited scalability, and very little adoption among organisations running OpenShift.
Then came the products of larger vendors and of companies that built a tool for their own needs:
- Conjur, CyberArk’s open-source offering to authenticate, control and audit access to tools, applications and containers. Too complex, very costly, and one more external dependency to maintain outside the cluster.
- EJSON, from Shopify, encrypts secrets in a JSON file stored in Git. Not enough: no access control, no rotation, and a lot of integration work.
- Knox, from Pinterest, stores and rotates the secrets other services use. Not mature enough: little feedback from production use, and a single release on GitHub.
- Red October, from Cloudflare, needs two keys held by two different people to decrypt a secret. No feedback from production use either, very limited scalability, a complicated integration and imprecise documentation.
The eighth, HashiCorp Vault, was the recommendation. It generates, stores and tightly controls access to tokens, passwords, certificates and encryption keys, through a UI, a CLI or an HTTP API, and it integrates with Kubernetes in three ways. A ninth candidate had ruled itself out: Square deprecated Keywhiz in September 2023 and points its users to Vault.
Three ways into the pod
All three modes rely on the same Vault, its policies and its audit log, and all three serve static and dynamic secrets. What differs is the component that writes the secret, and the place it lands.
The agent sidecar injector
A mutating admission webhook watches for pods that ask for injection. It adds an
init container, which fetches the secrets before the application starts, and a
Vault Agent sidecar, which keeps authenticating and renders the secrets again
for the life of the pod. Both write to an in-memory volume mounted at
/vault/secrets in every container. A pod opts in with annotations:
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/role: 'app'
vault.hashicorp.com/agent-inject-secret-database-config.txt: 'secret/data/app/database'
The application stays unaware of Vault: it reads a file. Vault Agent renders templates with Consul Template, so a secret can take the exact shape of the configuration file the application expects, and the agent renews leases and refreshes what it wrote. The cost is one more container in every pod, and some setup work on the operations side: the webhook, the Kubernetes auth method, a Vault role per service account.
The Secrets Store CSI driver
The Secrets Store CSI driver is a Kubernetes SIG project, and HashiCorp provides
the Vault provider for it. Both run as DaemonSets, one pod per node. A
SecretProviderClass, a namespaced resource, names the Vault role and the
secrets to fetch; the pod declares a CSI volume that points to it:
volumes:
- name: secrets-store
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: vault-database
During container creation, the driver asks the provider for the secrets and writes them as files in an ephemeral volume, and the pod does not start until they are there. It is quick to set up and needs no sidecar, but each workload carries more configuration in its manifests, and without Vault Agent behind it there is no templating: one file per key. The driver can also sync the files into a Kubernetes Secret, which brings back environment variables, and etcd with them. Rotation exists, as an alpha feature behind a flag, and an environment variable only changes when the pod restarts.
The volume is mounted from the node, which some platforms restrict. On OpenShift, the driver is not there by default: it comes as an operator still in Technology Preview.
The Vault Secrets Operator
The newest of the three: HashiCorp released it in beta in March 2023 and made it
generally available in June 2023. The operator runs as one Deployment and
watches custom resources. VaultConnection and VaultAuth say how to reach
Vault; VaultStaticSecret, VaultDynamicSecret and VaultPKISecret say what
to sync. It writes each secret into an ordinary Kubernetes Secret and keeps it
in step with Vault:
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: database-config
namespace: app
spec:
vaultAuthRef: app-auth
mount: secret
type: kv-v2
path: app/database
refreshAfter: 60s
destination:
name: database-config
create: true
rolloutRestartTargets:
- kind: Deployment
name: app
Applications read a plain Kubernetes Secret, as a volume or as environment variables, with no sidecar and no driver; when the value changes, the operator can restart the Deployments, StatefulSets or ReplicaSets that use it. Since version 0.5, released in February 2024, it can also reshape the secret data with templates, which the beta could not. The price is the one the two other modes avoid: the secret lives in etcd again. It is encrypted at rest only if the cluster has encryption at rest configured, and anyone allowed to read Secrets in the namespace reads it.
Four questions instead of a feature grid
A feature grid lists twenty lines on which the three modes mostly agree. Four questions separate them.
Where does the secret rest? In the pod’s memory with the injector, in a volume the node mounts with the CSI driver, in etcd with the operator. If etcd is not encrypted, the third answer reopens the problem the review set out to close.
Can the secret take the application’s format? Yes with the injector, through Vault Agent templates. Not with the CSI driver. Yes with the operator, since its February release.
Does it follow a rotation? The injector renews and re-renders, and the application has to reread its file. The CSI driver does it only with an alpha feature turned on. The operator updates the Secret and can restart the workload.
What runs in the cluster? A webhook and one sidecar per pod for the injector, a DaemonSet for the CSI driver, a single Deployment for the operator.
In short: the injector suits applications that read a configuration file and must follow a rotation; the operator suits workloads that already consume Kubernetes Secrets on a cluster whose etcd is encrypted; the CSI driver suits simple file secrets on platforms where it is supported.
Seven criteria before the tool
The review closed on the criteria a team should settle before naming any tool. They are the questions that eliminated the seven other candidates, and the ones that will judge Vault in turn.
- Integration. Do not connect the application to the secrets manager itself. Any tight coupling makes it rigid and hard to test. All three Vault modes respect this: the application reads a file or a variable, never the Vault API.
- Security. Fine-grained access control, key rotation, monitoring and revocation of access.
- Adaptability. The tool should serve Kubernetes and the existing virtual machines alike, so the same password is not copied into two places.
- Scalability. Can the tool be maintained, and does it handle a large volume of secrets without losing performance?
- Responsiveness. An active community and documentation that is complete, clear and easy to reach.
- Cost. The price of technical support, and whether a licence is needed. For Vault, the licence changed in August 2023: HashiCorp moved its future releases from the Mozilla Public License 2.0 to the Business Source License 1.1, which still lets an organisation use it, and forbids offering a product that competes with HashiCorp. At the end of 2023, OpenBao started under the Linux Foundation as a fork of the last MPL branch; it has not shipped a stable release yet. Both facts belong in the record the decision rests on.
- Know-how. What the teams already know, and the upskilling the choice requires, planned from the start.
Security sets the rule, operations runs it
A secrets manager also needs an owner for each of its rules. Security defines and maintains the policies: access and authorisations, how often each type of key rotates, how the unseal keys are distributed and stored, when the root token is revoked. Operations deploys the tool, runs and monitors it, automates access and maintains it. Both help choose it, and the rollout works only if they keep talking.
Choosing Vault settles one question and hands back three: where the secret may rest, which component writes it there, and who decides when it changes.