Secrets

This page helps you create the Secrets Goiabada’s Kubernetes manifest reads, and keep them readable only by what needs them.

What each secret protects, and backing up the AES key, are the same on every platform: see Secrets.

Create the Secrets

The setup wizard writes them into goiabada-secrets.yaml, beside the manifest. That suits a first deployment on a cluster you control:

Back up the AES key from the file, before anything else: see Back up the AES key.
Apply the Secrets, then the manifest, so no pod starts with an older copy of the Secrets. Both files create the namespace:
Terminal window
```
kubectl apply -f goiabada-secrets.yaml -f goiabada-k8s.yaml
```
Apply this file only for a new deployment. Every run of the wizard writes newly generated secrets into it, and applied over a deployment that runs, they replace the keys its database is under. Nothing shows at first: a pod reads its Secrets only when it starts, so a changed Secret reaches no running pod. At the next restart, the auth server refuses to start with the new AES key, and the admin console with the new client secret, and the rollout waits with the earlier pods still serving. For a deployment that already runs, apply only goiabada-k8s.yaml, and change a Secret as Rotate secrets does, restart included.
Keep the file out of version control, or delete it once the Secrets are in the cluster. The wizard warns when it writes into a git working tree, and gives the line to add to its .gitignore. Applying the file again after a rotation puts the old keys back.

For production, keep the manifest and create the Secrets by another way instead. The manifest needs only the Secrets it reads to exist, however they got there.

Create the Secrets another way

From your terminal

kubectl create secret writes nothing to disk, and no value reaches the process list or your shell history: each one goes through a process substitution, a key straight from openssl and a password from a variable read without echo. In bash, once the manifest has created the namespace:

read -rsp 'Database password: ' DB_PASSWORD; echo
read -rsp 'Admin password: ' ADMIN_PASSWORD; echo
kubectl create secret generic goiabada-secrets -n goiabada \
  --from-file=db-password=<(printf %s "$DB_PASSWORD") \
  --from-file=admin-password=<(printf %s "$ADMIN_PASSWORD") \
  --from-file=auth-session-auth-key=<(openssl rand -hex 64 | tr -d '\n') \
  --from-file=auth-session-enc-key=<(openssl rand -hex 32 | tr -d '\n') \
  --from-file=admin-session-auth-key=<(openssl rand -hex 64 | tr -d '\n') \
  --from-file=admin-session-enc-key=<(openssl rand -hex 32 | tr -d '\n') \
  --from-file=oauth-client-secret=<(openssl rand -hex 32 | tr -d '\n')
kubectl create secret generic goiabada-encryption-key -n goiabada \
  --from-file=aes-encryption-key=<(openssl rand -hex 32 | tr -d '\n')
unset DB_PASSWORD ADMIN_PASSWORD

The values then live in the cluster alone: there’s no file to leak or commit, and none to restore from. Losing the namespace or the cluster loses them, so back up the AES key and keep the passwords wherever you keep others. Nothing is declarative, so a GitOps tool can’t recreate the Secrets, and rebuilding the cluster means creating them again with the same values: new session keys sign everybody out, a new OAuth client secret no longer matches the one in the database, and a new AES key can’t read it.

From a secret manager

The External Secrets Operator syncs Secrets from a secret manager, such as HashiCorp Vault or your cloud’s. You keep each value in the manager, and an ExternalSecret per Secret has the operator create it under the names and keys below, then refresh it:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: goiabada-encryption-key
  namespace: goiabada
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: my-secret-store                # yours
  target:
    name: goiabada-encryption-key
  data:
  - secretKey: aes-encryption-key
    remoteRef:
      key: goiabada/aes-encryption-key   # the entry in your secret manager

The second, goiabada-secrets, lists its seven keys the same way. The manager becomes the source of truth, with its own access control, audit log and backups; a manager kept apart from the database’s backups is a sound home for the AES key’s backup. The cost is an operator and its resources to run, a credential that lets it read the manager (workload identity where your cloud offers it), and a Kubernetes Secret that still exists, readable as below.

A value changed in the manager reaches the Secret at the next refresh, but a pod reads it only when it starts, so a pod restarted for any reason picks it up alone. The operator also writes the Secret back to what the manager holds, the keys its data lists and no other, so a kubectl patch of the Secret lasts until the next refresh. Change a session key or the AES key only as Rotate secrets says, with the changes in the manager as Rotating Secrets something else owns describes.

Sealed in your repository

SealedSecrets encrypts a Secret you can commit. kubeseal encrypts it with the public key of the controller running in your cluster, and only that controller can decrypt it, which it does by creating the Secret:

kubectl create secret generic goiabada-encryption-key -n goiabada --dry-run=client -o yaml \
  --from-file=aes-encryption-key=<(openssl rand -hex 32 | tr -d '\n') \
  | kubeseal --format yaml > goiabada-encryption-key.sealed.yaml

Seal goiabada-secrets the same way, with the keys of the first kubectl create secret above, and commit both files beside the manifest. A GitOps tool can then apply everything, and the repository holds no readable secret.

The controller’s private key now decrypts all of them: back it up as the SealedSecrets documentation describes, since without it the sealed files are useless. That backup doesn’t replace the AES key’s own, which must be readable without the cluster. A sealed Secret is bound to its name and namespace, so a renamed namespace means sealing again, and once unsealed it’s an ordinary Secret, readable as below. The controller writes the Secret from the sealed file, so a kubectl patch of the Secret doesn’t last, and a value changes by sealing it. Change a session key or the AES key only as Rotate secrets says, sealing what each step says as Rotating Secrets something else owns describes.

What the generated Secrets are

goiabada-secrets.yaml holds two Secrets: goiabada-encryption-key, holding the AES key alone, and goiabada-secrets, holding every other secret. The AES key has a Secret of its own so that a role can be allowed to read the others without it.

What they aren’t:

Encrypted. A Secret’s data is base64, which base64 -d reverses with no key. On disk, the file’s mode, 0600, is its only protection: anyone who reads the file, or a backup or a repository holding it, reads every secret.
Encrypted in the cluster, unless you make them so. The API server keeps Secrets in etcd, in the clear unless the cluster configures encryption at rest; many managed clusters do, so check yours. Anyone allowed to get, list or watch Secrets in the namespace reads them, and so does anyone who can create a pod there or kubectl exec into Goiabada’s, since the values are in the containers’ environment.
Rotated. Nothing changes them after the wizard writes them; Rotate secrets says how.
Backed up. Deleting the namespace deletes them, and the AES key needs a backup of its own.

The Secrets the manifest reads

The manifest reads two Secrets from its namespace, by these names and keys. Created by any route, the rows not marked optional are all it needs:

Secret	Key	Variable	Read by	Value
`goiabada-secrets`	`db-password`	`GOIABADA_DB_PASSWORD`	auth server	The database user’s password
`goiabada-secrets`	`admin-password`	`GOIABADA_ADMIN_PASSWORD`	auth server	The first administrator’s password, read only by the first start, which creates the account. Change it later in the admin console, not here
`goiabada-secrets`	`auth-session-auth-key`	`GOIABADA_AUTHSERVER_SESSION_AUTHENTICATION_KEY`	auth server	64 random bytes as 128 hex characters, `openssl rand -hex 64`
`goiabada-secrets`	`auth-session-enc-key`	`GOIABADA_AUTHSERVER_SESSION_ENCRYPTION_KEY`	auth server	32 random bytes as 64 hex characters, `openssl rand -hex 32`
`goiabada-secrets`	`admin-session-auth-key`	`GOIABADA_ADMINCONSOLE_SESSION_AUTHENTICATION_KEY`	admin console	64 random bytes as 128 hex characters, `openssl rand -hex 64`
`goiabada-secrets`	`admin-session-enc-key`	`GOIABADA_ADMINCONSOLE_SESSION_ENCRYPTION_KEY`	admin console	32 random bytes as 64 hex characters, `openssl rand -hex 32`
`goiabada-secrets`	`oauth-client-secret`	`GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRET`	both	The admin console’s OAuth client secret. The auth server reads it only at the first start, to create the admin console’s client with it; the admin console authenticates with it from then on, so it must match the client’s secret in the database, or the admin console refuses to start
`goiabada-encryption-key`	`aes-encryption-key`	`GOIABADA_AES_ENCRYPTION_KEY`	auth server	32 random bytes as 64 hex characters, `openssl rand -hex 32`. It encrypts the secrets the database holds
`goiabada-secrets`	`auth-session-auth-key-previous`	`GOIABADA_AUTHSERVER_SESSION_AUTHENTICATION_KEY_PREVIOUS`	auth server	Optional, held only while the session keys rotate: with the next key, the pair the auth server opens sessions with beside its current one
`goiabada-secrets`	`auth-session-enc-key-previous`	`GOIABADA_AUTHSERVER_SESSION_ENCRYPTION_KEY_PREVIOUS`	auth server	Optional, the other half of the auth server’s previous pair
`goiabada-secrets`	`admin-session-auth-key-previous`	`GOIABADA_ADMINCONSOLE_SESSION_AUTHENTICATION_KEY_PREVIOUS`	admin console	Optional, held only while the session keys rotate: with the next key, the admin console’s previous pair
`goiabada-secrets`	`admin-session-enc-key-previous`	`GOIABADA_ADMINCONSOLE_SESSION_ENCRYPTION_KEY_PREVIOUS`	admin console	Optional, the other half of the admin console’s previous pair
`goiabada-encryption-key`	`aes-encryption-key-previous`	`GOIABADA_AES_ENCRYPTION_KEY_PREVIOUS`	auth server	Optional, held only while the AES key rotates: the key the database is under, which the auth server re-encrypts from when it starts

Each Secret must exist before the pods that read it start. A pod that starts first waits in CreateContainerConfigError, and starts once they exist.

The manifest’s references to the five -previous keys are optional: a pod starts without them, and each server reads a missing one as no rotation in progress. Create them only as Rotate secrets says. A previous session pair is read whole or not at all, since a server refuses to start with one half of it, so add and remove its two keys together.

Who can read the AES key

RBAC grants and never denies, so the AES key’s Secret of its own protects it only from roles that name what they may read. This Role reads goiabada-secrets and not the key:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: goiabada-secrets-reader
  namespace: goiabada
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["goiabada-secrets"]
  verbs: ["get"]

A role that may list or watch Secrets in the namespace, or get them with no resourceNames, reads the key too, since a list returns every Secret with its data; so does one that may create pods in the namespace, or exec into the auth server’s. Goiabada’s own pods need no role at all: the kubelet reads the Secrets to set their environment, and the pods mount no service account token.

Rotating Secrets something else owns

When the External Secrets Operator or the SealedSecrets controller owns a Secret, its source, the secret manager or the sealed file, is what the Secret holds, and a step’s change goes there. Three rules apply to every step of Rotate secrets:

Put the whole state the step ends in into the source, every key the step says the Secret holds, and only those. Both write the Secret from their source, its keys and no other, so a key left out of the source is removed from the Secret, a -previous key included.
Wait until the Secret holds it before the step’s rollout or scale, since a pod reads whatever the Secret holds when it starts. Compare a digest of each key the step changed with the same digest of the value in your source, which shows the value without printing it:
Terminal window
```
digest() { kubectl get secret "$1" -n goiabada -o "jsonpath={.data.$2}" | base64 -d | sha256sum | cut -c1-16; }
digest goiabada-encryption-key aes-encryption-key-previous
```
Leave the source holding what the Secret holds when the rotation is done, so that a refresh, a reapply, a deleted Secret or a rebuilt cluster brings back the rotated keys and not the old ones.

With the External Secrets Operator, never change the value of an entry the ExternalSecret reads. The operator reads each line of its data from the manager on its own, not as one snapshot, so a refresh that runs while you change several entries can write the Secret with some changed and the rest not: halfway through the session keys’ swap, the new authentication key beside the old encryption key, a current pair that opens nothing while the previous pair is already the new one. A pod a crash or an eviction restarts then can’t open any session sealed under the old pair.

Instead, each new value goes into an entry of its own, named for the rotation it belongs to, such as goiabada/auth-session-auth-key-2026-10, and a step changes which entries the ExternalSecret maps, in one edit of it, applied once. A refresh reads every line under one version of the ExternalSecret, and the entries that version names don’t change while it reads them, so it writes the state before the edit or the state after it, each whole; a refresh already running when you apply the edit writes the state before it, and the next one the state after. Ask the operator to refresh at once rather than at its refreshInterval, then wait for the digests:

kubectl annotate externalsecret goiabada-encryption-key -n goiabada force-sync="$(date +%s)" --overwrite

For the session keys, adding the new pairs as the previous pairs creates four entries holding them, openssl rand -hex 64 for each authentication key and openssl rand -hex 32 for each encryption key, then adds the four lines mapping the -previous keys to them. The swap is one edit that maps the four current keys, auth-session-auth-key and the rest, to the new entries, and the four -previous keys to the entries the current keys mapped until then; no entry’s value changes. Step 3 removes the four -previous lines from data, restarts the pods once the Secret no longer holds those keys, then deletes the old pairs’ entries.
For the AES key, step 3 creates an entry holding a new key, openssl rand -hex 32, such as goiabada/aes-encryption-key-2026-10, then, in one edit, maps aes-encryption-key to it and adds a line mapping aes-encryption-key-previous to the entry aes-encryption-key mapped until then, the one holding the key the database is under. Once the Secret holds both, check that the digest of aes-encryption-key-previous is that of the key you backed up in step 1. Step 4 reads the new key out of the cluster. Step 6 removes the line from data, then restarts the auth server. The old key’s entry can stay as that key’s backup, for as long as you keep a database backup taken under it.

Removing a line from data before its entry, and deleting the entry only once the Secret no longer holds that key, keeps the operator from failing on an entry it can no longer read, which would leave the Secret as it was.

With SealedSecrets, each step’s state is sealed into the file and applied, by you or by your GitOps tool. kubeseal --merge-into adds or replaces keys in an existing sealed file, each key sealed on its own, and deleting a key’s line from the file’s spec.encryptedData removes that key. The values come from the Secret the controller unsealed, through process substitutions, so none reaches a file, an argument or the screen:

key() { kubectl get secret goiabada-secrets -n goiabada -o "jsonpath={.data.$1}" | base64 -d; }
seal() { kubectl create secret generic goiabada-secrets -n goiabada --dry-run=client -o json "$@" \
  | kubeseal --format yaml --merge-into goiabada-secrets.sealed.yaml; }

# Session keys: the new pairs as the previous pairs.
seal --from-file=auth-session-auth-key-previous=<(openssl rand -hex 64 | tr -d '\n') \
  --from-file=auth-session-enc-key-previous=<(openssl rand -hex 32 | tr -d '\n') \
  --from-file=admin-session-auth-key-previous=<(openssl rand -hex 64 | tr -d '\n') \
  --from-file=admin-session-enc-key-previous=<(openssl rand -hex 32 | tr -d '\n')

# Session keys: the pairs swapped, once the first file has been applied and unsealed.
seal --from-file=auth-session-auth-key=<(key auth-session-auth-key-previous) \
  --from-file=auth-session-enc-key=<(key auth-session-enc-key-previous) \
  --from-file=admin-session-auth-key=<(key admin-session-auth-key-previous) \
  --from-file=admin-session-enc-key=<(key admin-session-enc-key-previous) \
  --from-file=auth-session-auth-key-previous=<(key auth-session-auth-key) \
  --from-file=auth-session-enc-key-previous=<(key auth-session-enc-key) \
  --from-file=admin-session-auth-key-previous=<(key admin-session-auth-key) \
  --from-file=admin-session-enc-key-previous=<(key admin-session-enc-key)

Step 3 deletes the four -previous lines from spec.encryptedData, and restarts the pods once the controller has removed those keys from the Secret. For the AES key, step 3 seals the key the cluster holds now as the previous key, beside a new one, in one file:

kubectl create secret generic goiabada-encryption-key -n goiabada --dry-run=client -o json \
  --from-file=aes-encryption-key-previous=<(kubectl get secret goiabada-encryption-key -n goiabada \
    -o jsonpath='{.data.aes-encryption-key}' | base64 -d) \
  --from-file=aes-encryption-key=<(openssl rand -hex 32 | tr -d '\n') \
  | kubeseal --format yaml --merge-into goiabada-encryption-key.sealed.yaml

Before you apply it, check that the digest of aes-encryption-key in the cluster is that of the key you backed up in step 1: that’s the key the previous entry has to hold. Step 6 deletes the aes-encryption-key-previous line, then restarts the auth server. Commit each file as you apply it, so the repository’s file is always the one the controller last unsealed.

When a GitOps tool applies the manifest, the only Deployment field a rotation changes is the auth server’s replicas, during the AES key’s: the tool puts back the manifest’s count, which would start auth server pods while step 2 keeps them stopped. Suspend its automated sync for the AES key’s rotation (in Argo CD, turn off the Application’s automated sync; in Flux, flux suspend kustomization <name>) and resume it once step 5 has scaled back. The session keys’ rotation changes nothing the manifest sets: the -previous references are already in it, and kubectl rollout restart adds only an annotation to the pod template.

Next steps

Rotate secretsChange each secret without signing everybody out or losing the database.

Back up the AES keyWithout it, a restored database decrypts nothing.

SecurityWho can reach Goiabada's pods, and what they may do.