Secrets with SOPS and Flux

Secrets with SOPS and Flux

Flux decrypts SOPS-encrypted Kubernetes Secrets at reconciliation time. You commit encrypted YAML to Git; Flux applies the plaintext to the cluster. The plaintext never enters the repository.

Two rules:

1. Each cluster gets its own age keypair. Never reuse a developer’s personal key.
2. Plaintext secrets never touch disk. Create files as empty scaffolding, then edit through SOPS.

Prerequisites

Install sops and age (both are in mise.toml).

1) Generate a per-cluster age key

Each cluster gets a dedicated keypair. Do not derive it from a developer’s SSH key or reuse keys across clusters. If the local cluster is compromised, production secrets stay safe.

Generate the keypair:

age-keygen 2>&1

This prints the private key (the AGE-SECRET-KEY-... line) and the public key (the # public key: age1... comment). Copy both to a password manager (e.g. KeePass). Do not save the private key to a file on disk.

2) Store the private key in the cluster

Pipe the private key from your clipboard or password manager straight into kubectl:

echo "AGE-SECRET-KEY-..." | kubectl create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey=/dev/stdin

The private key now lives only in the cluster and in your password manager. Not on disk, not in Git.

Repeat for each cluster with that cluster’s own keypair.

Inspect whether the key exists:

kubectl get secrets -n flux-system
kubectl --namespace flux-system get secret sops-age

Verify the private key in the cluster matches the public key in .sops.yaml. Extract the private key from the cluster, derive its public key, and compare:

# Get the public key from the cluster's private key
kubectl --namespace flux-system get secret sops-age \
  -o jsonpath="{.data.age\.agekey}" | base64 -d | age-keygen -y

# Compare against what .sops.yaml expects
grep '^    age:' .sops.yaml

The age-keygen -y command reads a private key from stdin and prints the corresponding public key. If the output matches one of the age: values in .sops.yaml, the cluster can decrypt secrets matching that rule.

3) Configure .sops.yaml

The .sops.yaml file in the repo root tells SOPS which public key to use for which files. Path regexes route secrets to the correct cluster key:

.sops.yaml

creation_rules:
  # Developer-local secrets (e.g. GitHub tokens), uses personal SSH key
  - path_regex: '\.env\.json$'
    age: ssh-ed25519 AAAAC3NzaC1...your-ssh-pubkey

  # Local cluster (infrastructure + dev/staging-local app overlays)
  - path_regex: '(infrastructure/.*secret.*\.yaml$|overlays/(dev|staging-local)/.*secret.*\.yaml$)'
    age: age1abc...local-cluster-public-key

  # Remote cluster (staging-remote + prod app overlays)
  - path_regex: 'overlays/(staging-remote|production)/.*secret.*\.yaml$'
    age: age1xyz...remote-cluster-public-key

The infrastructure/ rule covers shared infrastructure secrets (like the Grafana password) that the local cluster decrypts. App-level secrets under overlays/ route to whichever cluster runs that environment.

Secrets encrypted with the local key cannot be decrypted by the remote cluster, and vice versa. Each cluster holds only its own private key.

4) Create and encrypt a secret (without plaintext on disk)

Create the file as an empty scaffold:

touch infrastructure/base/monitoring/grafana-secret.yaml

Edit it through SOPS, which encrypts on save:

sops infrastructure/base/monitoring/grafana-secret.yaml

SOPS opens $EDITOR with a blank file. Paste the content:

apiVersion: v1
kind: Secret
metadata:
  name: grafana-admin
  namespace: monitoring
type: Opaque
stringData:
  admin-user: admin
  admin-password: your-actual-password

Save and close. SOPS encrypts the values before writing to disk. The plaintext exists only in memory while the editor is open.

Alternatively, pipe through SOPS without an editor:

OUT=infrastructure/base/monitoring/grafana-secret.yaml
password=$(openssl rand -hex 32)
kubectl create secret generic grafana-admin \
  --namespace=monitoring \
  --from-literal=admin-user=admin \
  --from-literal=admin-password="${password}" \
  --dry-run=client \
  -o yaml \
| sops --encrypt --input-type yaml --output-type yaml \
  --filename-override "$OUT" \
  /dev/stdin > "$OUT"

Or the same approach from Python:

# /// script
# dependencies = []
# ///
import subprocess, secrets

password = secrets.token_hex(32)
output = "infrastructure/base/monitoring/grafana-secret.yaml"

kubectl = subprocess.run(
    ["kubectl", "create", "secret", "generic", "grafana-admin",
     "--namespace=monitoring",
     f"--from-literal=admin-user=admin",
     f"--from-literal=admin-password={password}",
     "--dry-run=client", "-o", "yaml"],
    capture_output=True, text=True, check=True,
)

sops = subprocess.run(
    ["sops", "--encrypt", "--input-type", "yaml", "--output-type", "yaml",
     "--filename-override", output, "/dev/stdin"],
    input=kubectl.stdout,
    capture_output=True, text=True, check=True,
)

with open(output, "w") as f:
    f.write(sops.stdout)

print(f"Wrote encrypted secret to {output}")
print(f"Password (save to password manager): {password}")

SOPS encrypts only the values, not the keys or metadata. The file remains valid YAML. You can inspect the structure without decrypting, but the secret values are opaque.

5) Enable decryption on the Flux Kustomization

Add a decryption block to the Flux Kustomization that manages the path containing encrypted secrets:

flux create kustomization infrastructure \
  --source=flux-system \
  --path=./infrastructure/base/monitoring \
  --prune \
  --interval=10m \
  --decryption-provider=sops \
  --decryption-secret=sops-age \
  --export

clusters/local/infrastructure.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/base/monitoring
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age

The secretRef points to the Secret from step 2. Flux uses it to decrypt any SOPS-encrypted files in that path.

6) Reference the secret from the HelmRelease

The kube-prometheus-stack chart accepts an existingSecret field for Grafana. Update the HelmRelease values:

# in infrastructure/base/monitoring/helmrelease.yaml
values:
  grafana:
    admin:
      existingSecret: grafana-admin
      userKey: admin-user
      passwordKey: admin-password

The chart reads the password from the Secret instead of generating one. Flux created that Secret by decrypting your SOPS file.

7) Commit and push

git add infrastructure/base/monitoring/grafana-secret.yaml
git add clusters/local/infrastructure.yaml
git add infrastructure/base/monitoring/helmrelease.yaml
git commit -m "Add SOPS-encrypted Grafana credentials"
git push

Flux reconciles, decrypts the secret, applies it, and the HelmRelease picks it up on its next upgrade cycle. See Flux for the full GitOps setup and Helm and Flux Integration for HelmRelease details.

Verify

First force the reconcile, rather than waiting:

flux reconcile source git flux-system

► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision main@sha1:5584ffd0e04f75874100b2c6d6ed5b69a1bd1e95

Check that the secret exists in the cluster:

kubectl get secret grafana-admin -n monitoring
kubectl get secret grafana-admin -n monitoring -o jsonpath='{.data.admin-password}' | base64 -d

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Use something like this in mise.toml

[tasks.edit]
description = "Edit a SOPS-encrypted file (fetches age key from cluster)"
usage = 'arg "<file>"'
run = """
SOPS_AGE_KEY=$(kubectl -n flux-system get secret sops-age -o jsonpath="{.data.age\\.agekey}" | base64 -d) \
  sops {{usage.file}}
"""

[tasks.decrypt]
description = "Decrypt a SOPS-encrypted file to stdout (fetches age key from cluster)"
usage = 'arg "<file>"'
run = """
SOPS_AGE_KEY=$(kubectl -n flux-system get secret sops-age -o jsonpath="{.data.age\\.agekey}" | base64 -d) \
  sops decrypt {{usage.file}}
"""

Decrypting and editing secrets locally

The private key lives in the cluster, not on your machine. To decrypt or edit locally, pull the key into an env var without writing to disk:

# Decrypt to stdout
SOPS_AGE_KEY=$(kubectl -n flux-system get secret sops-age -o jsonpath="{.data.age\.agekey}" | base64 -d) \
  sops decrypt infrastructure/base/monitoring/grafana-secret.yaml

# Edit in $EDITOR (re-encrypts on save)
SOPS_AGE_KEY=$(kubectl -n flux-system get secret sops-age -o jsonpath="{.data.age\.agekey}" | base64 -d) \
  sops infrastructure/base/monitoring/grafana-secret.yaml

The SOPS_AGE_KEY env var passes the private key in memory. It is never written to a file.

Commit and push after editing. Flux applies the updated secret on the next reconciliation.

Rotating the age key

If a cluster’s private key is compromised:

1. Generate a new age keypair (age-keygen)
2. Store the new private key in the cluster:

   kubectl delete secret sops-age -n flux-system
   echo "AGE-SECRET-KEY-..." | kubectl create secret generic sops-age \
     --namespace=flux-system \
     --from-file=age.agekey=/dev/stdin

3. Update .sops.yaml with the new public key
4. Re-encrypt all secrets for that cluster:

   find . -name '*.yaml' -exec grep -l 'sops:' {} \; | while read f; do
     sops --rotate --in-place "$f"
   done

5. Commit and push

Only the compromised cluster’s secrets need re-encryption. The other cluster’s secrets stay untouched because they use a different key.