Managing Secrets with SOPS

SOPS (Secrets OPerationS) encrypts secret values before they reach disk. You commit the encrypted YAML to Git; Flux decrypts it during reconciliation and applies the plaintext to the cluster. The plaintext never enters the repository.

Two rules govern this workflow:

1. Each cluster gets its own Age keypair. Never reuse a developer’s personal key across clusters.
2. Plaintext secrets never touch disk. Create files as empty scaffolding, then write through SOPS.

Prerequisites

Install sops and age. Both are declared in mise.toml and available after mise install.

Generate and store a cluster key

Run the init command to generate a keypair and store it in the cluster in one step:

kb k secrets init

This generates an Age keypair in memory, creates the sops-age secret in the flux-system namespace, and updates .sops.yaml with the new public key. It prints the private key to stdout — copy it to a password manager (e.g. KeePass) before the terminal clears. The private key is not written to any file.

If the sops-age secret already exists, the command exits without changes. Pass --force to replace the keypair and re-encrypt all SOPS files in the repository with the new key.

To store a key you generated manually (e.g. via age-keygen), pipe it straight into kubectl:

echo "AGE-SECRET-KEY-..." | kubectl create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey=/dev/stdin

Verify the cluster key matches what .sops.yaml expects:

kubectl --namespace flux-system get secret sops-age \
  -o jsonpath="{.data.age\.agekey}" | base64 -d | age-keygen -y

grep 'age:' .sops.yaml

The age-keygen -y flag reads a private key and prints the corresponding public key. The output should match one of the age: values in .sops.yaml.

How .sops.yaml routes secrets

The .sops.yaml file at the repo root maps path patterns to public keys:

creation_rules:
  # Developer-local secrets (e.g. GitHub tokens)
  - path_regex: '\.env\.json$'
    age: ssh-ed25519 AAAA...your-ssh-pubkey

  # Local cluster — infrastructure and dev/staging-local overlays
  - path_regex: '(infrastructure/.*secret.*\.yaml$|overlays/(dev|staging-local)/.*secret.*\.yaml$)'
    age: age1abc...local-cluster-public-key

  # Remote cluster — staging-remote and production overlays
  - path_regex: 'overlays/(staging-remote|production)/.*secret.*\.yaml$'
    age: age1xyz...remote-cluster-public-key

SOPS reads the first matching rule when encrypting a file. Secrets encrypted with the local key cannot be decrypted by the remote cluster, and vice versa — each cluster holds only its own private key.

Create an encrypted secret

The kb k secrets edit command handles the full create-or-edit lifecycle. If the file does not exist, it creates an empty encrypted file and opens it in $EDITOR. If the file already exists, it decrypts to a temp buffer, opens your editor, and re-encrypts on save.

kb k secrets edit infrastructure/base/monitoring/grafana-secret.yaml

The command fetches the Age key from the cluster at runtime — the key is passed via SOPS_AGE_KEY in the environment and never written to disk.

Paste the Kubernetes Secret manifest in your editor:

apiVersion: v1
kind: Secret
metadata:
  name: grafana-admin
  namespace: monitoring
type: Opaque
stringData:
  admin-user: admin
  admin-password: your-actual-password

SOPS encrypts only the values (admin-user, admin-password). The keys, metadata, and structure remain readable YAML. Save and close — the encrypted file is ready to commit.

To inspect a secret without editing, decrypt it to stdout:

kb k secrets decrypt infrastructure/base/monitoring/grafana-secret.yaml

Configure Flux decryption

Add a decryption block to the Flux Kustomization that manages the path containing your encrypted files:

clusters/local/infrastructure.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/base/monitoring
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age

The secretRef points to the sops-age secret from the init step. Flux uses it to decrypt any SOPS-encrypted files under path before applying them.

Commit workflow

Stage only the encrypted files — never stage a decrypted copy:

git add infrastructure/base/monitoring/grafana-secret.yaml
git add clusters/local/infrastructure.yaml
git commit -m "Add SOPS-encrypted Grafana credentials"
git push

Flux reconciles on the next interval. To apply immediately:

flux reconcile source git flux-system

Verify the secret landed in the cluster:

kubectl get secret grafana-admin -n monitoring
kubectl get secret grafana-admin -n monitoring \
  -o jsonpath='{.data.admin-password}' | base64 -d

Rotate a compromised key

1. Generate a new keypair: kb k secrets init --force
2. Copy the printed private key to your password manager
3. The --force flag replaces the cluster secret and re-encrypts all SOPS files in the repo
4. Commit and push the updated files

Only the affected cluster’s secrets need re-encryption. Other clusters use separate keys and are unaffected.