Getting Secret Management Right in Kubernetes - Cymatic Systems

kubernetes

security

devops

cloud-native

Getting Secret Management Right in Kubernetes

Declan Curran

May 14, 2026

One of the biggest challenges in migrating to Kubernetes is getting secret management right. The number of teams I've seen committing plaintext secrets to Git, or baking them into Docker images, is genuinely surprising. Most of the time it's simply because they don't have a better solution, or it's just what they've always done.

There are a few problems with this. From a security standpoint, you want to limit the blast radius if a secret is ever leaked. From a responsibility standpoint, access should be restricted to a small number of people, typically the infra admins actually provisioning databases and external services, not every developer on the team.

The subtler issue is separation of concerns: what you want in a repository is the application and maybe some defaults, not environment-specific config or secrets. An application should be reusable, deployable into any environment, with the environment supplying the secrets.

If you're running on Kubernetes, there are no excuses; there are plenty of mature solutions available. There are a few I've come across over the past few years - I'll get into them below.

What exactly is a Secret resource?

Kubernetes Secret objects are the underlying primitive that most secret management solutions build on, so it's worth understanding how they work. The tools I cover in this post largely exist to generate them from a secure source rather than having you define them directly.

The issue with managing Secret manifests by hand is that their values are only base64-encoded, not encrypted. Committing them to a repository, or embedding them in Helm charts, is essentially storing plaintext secrets in your codebase. The examples below are useful for understanding the format, but a hardcoded Secret manifest should never end up in version control.

Here's an example of a basic Kubernetes Secret:

apiVersion: v1<br>kind: Secret<br>metadata:<br>name: my-secret<br>namespace: default<br>type: Opaque<br>data:<br>username: dXNlcm5hbWU= # Base64 encoded "username"<br>password: cGFzc3dvcmQ= # Base64 encoded "password"

Secrets can be consumed by pods as mounted files or environment variables. Here's an example using the secret volume type:

volumeMounts:<br>- name: secret-volume<br>mountPath: "/etc/secret"<br>volumes:<br>- name: secret-volume<br>secret:<br>secretName: my-secret

The Tools Worth Knowing

Secret objects are fine; pushing them manually (without checking them into a git repo) is one way to manage secrets and it works. It's just not maintainable at scale, and it's not something a team can collaborate on cleanly. What you really want is a solution that automates the process, keeps secrets out of your repository directly, and can be managed consistently across environments. Several tools exist that will either encrypt secret manifests before they reach Git, or generate them at runtime from a secure external source.

Git-Crypt

Git-Crypt is not exactly new, but it works, and it has for a long time. It uses GPG keys to encrypt and decrypt files directly in a Git repository, not just Kubernetes secrets but anything. That means you can commit encrypted secret manifests alongside everything else, and they'll be decrypted locally when you unlock the repo with your key.

It doesn't integrate well with GitOps tooling like ArgoCD or FluxCD. It predates a lot of that ecosystem, but it has survived for a long time, and that counts for something.

The most common mistake I see is teams sharing a single GPG key across the whole organisation. I've seen a single key shared between around thirty people, at which point access control is effectively meaningless. Git-Crypt fully supports per-user keys, so this is a process problem rather than a tool limitation.

The other issue is key leakage. If a key is compromised, you can't simply revoke it and re-encrypt. Every commit in history that was encrypted with the old key is still accessible to anyone who has it. The only real remediation is rotating all of the affected secrets.

Example usage:

git-crypt unlock # Decrypt all files using your GPG key<br>git-crypt add-gpg-user USER_ID # Add a user with decryption access

Files to encrypt are specified in a .gitattributes file:

secretfile filter=git-crypt diff=git-crypt<br>*.key filter=git-crypt diff=git-crypt<br>secretdir/** filter=git-crypt diff=git-crypt

SOPS

If you do need to keep secrets in Git, SOPS is the modern way to do it. Like Git-Crypt, it stores secrets encrypted in the repository, so you can keep your existing workflow while moving to a much more secure approach.

SOPS supports age keys as a simpler alternative to GPG, but the real benefit comes when you integrate a cloud KMS provider. Hook it up to AWS KMS, GCP KMS, or Azure Key Vault, and SOPS will encrypt and decrypt using your existing identity (an IAM user or role, for example) rather than a...