Skip to content
Goiabada

Rotate secrets

This page helps you replace each of Goiabada’s secrets without signing anybody out, or with the shortest interruption that secret allows.

Every rotation works the same way: a server reads its secrets only when it starts, so you change a secret where your setup keeps it, then restart the server that reads it. Each step has a tab per platform for its commands. Pick yours once, and every tab on the page follows.

Generate a new key with openssl rand -hex 64 for a session authentication key and openssl rand -hex 32 for the rest. Secrets says what each one protects.

The session keys

Section titled “The session keys”

Each server seals its browser sessions with its current session key pair, and opens them with that pair and then, when it’s set, a previous pair. So you make the new pair current and keep the old one as the previous pair for as long as a session sealed under it can live. Done this way, nobody is signed out.

  1. Make new pairs current, and the old pairs previous, then restart both servers.

    In docker-compose.override.yml, give each server’s current pair to its previous pair, and new keys to the current pair:

      goiabada-authserver:
        environment:
          # ...the other entries stay as they are
          - "GOIABADA_AUTHSERVER_SESSION_AUTHENTICATION_KEY=<new, openssl rand -hex 64>"
          - "GOIABADA_AUTHSERVER_SESSION_ENCRYPTION_KEY=<new, openssl rand -hex 32>"
          - "GOIABADA_AUTHSERVER_SESSION_AUTHENTICATION_KEY_PREVIOUS=<its value before>"
          - "GOIABADA_AUTHSERVER_SESSION_ENCRYPTION_KEY_PREVIOUS=<its value before>"
    

      goiabada-adminconsole:
        environment:
          # ...the other entries stay as they are
          - "GOIABADA_ADMINCONSOLE_SESSION_AUTHENTICATION_KEY=<new, openssl rand -hex 64>"
          - "GOIABADA_ADMINCONSOLE_SESSION_ENCRYPTION_KEY=<new, openssl rand -hex 32>"
          - "GOIABADA_ADMINCONSOLE_SESSION_AUTHENTICATION_KEY_PREVIOUS=<its value before>"
          - "GOIABADA_ADMINCONSOLE_SESSION_ENCRYPTION_KEY_PREVIOUS=<its value before>"

    Then restart:

    Terminal window
    docker compose up -d

    Compose stops a changed service’s container before it starts the new one, so two containers of one service never run together, and one restart does it.

  2. Wait for the maximum session lifetime since the new pairs became current. It’s on the admin console’s Settings → Sessions page, 24 hours unless you changed it, and it bounds both servers’ sessions. A session is sealed under the new pair the next time it’s saved, and none outlives that lifetime, so by then every signed-in session is under the new pair.

  3. Remove the previous pairs, and restart both servers, so that nothing opens sessions under the old pair any more.

    Delete the four _PREVIOUS lines from docker-compose.override.yml, then:

    Terminal window
    docker compose up -d

A sign-in still in progress, one not yet signed in, can still be under the old pair, so removing it may restart that sign-in, at the cost of the form being filled in. Replacing the keys in one step instead, with no previous pair, signs everybody out once.

A previous pair is read whole or not at all: a server refuses to start with one half of it set, so add and remove its two keys together.

The AES key

Section titled “The AES key”

The auth server encrypts and decrypts with GOIABADA_AES_ENCRYPTION_KEY alone. It reads GOIABADA_AES_ENCRYPTION_KEY_PREVIOUS only when it starts: if the database is still under that key, it re-encrypts all of it to the current one, in one transaction, before it listens.

From that commit on, an auth server still running with the old key decrypts nothing, and whatever it encrypts, a new client secret or an authenticator, is written under a key the database is no longer under and stays unreadable. So the key rotates with one auth server starting and no other running.

  1. Back up the database, and the current key, as Back up the AES key says.

  2. Make sure no other auth server runs while the new key is first used.

    Nothing to do: Compose stops the old container before it starts the new one.

  3. Make a new key current, and the old one previous.

    Under goiabada-authserver in docker-compose.override.yml, set GOIABADA_AES_ENCRYPTION_KEY to a new key, openssl rand -hex 32, and add the key it held before:

      goiabada-authserver:
        environment:
          # ...the other entries stay as they are
          - "GOIABADA_AES_ENCRYPTION_KEY=<new, openssl rand -hex 32>"
          - "GOIABADA_AES_ENCRYPTION_KEY_PREVIOUS=<its value before>"

  4. Back up the new key now, as Back up the AES key says: the next start puts the database under it.

  5. Start the auth server, and check that the re-encryption committed.

    Terminal window
    docker compose up -d --wait goiabada-authserver
    docker compose logs goiabada-authserver | grep 'rotated data-at-rest encryption'

    The record rotated data-at-rest encryption to the new GOIABADA_AES_ENCRYPTION_KEY says the re-encryption committed. From then on the database is under the new key alone, and GOIABADA_AES_ENCRYPTION_KEY must go on holding it: an auth server started with the old key there refuses to start, with the data encryption key does not decrypt the stored data, so the auth server cannot start.

    If the auth server stops instead with data-at-rest decrypts under neither GOIABADA_AES_ENCRYPTION_KEY nor GOIABADA_AES_ENCRYPTION_KEY_PREVIOUS, the previous key isn’t the one the database is under, and nothing was re-encrypted. The start migrates the database before it checks the keys, so a start that is also an upgrade has migrated its schema all the same.

  6. Remove the previous key, and restart the auth server, once you’ve checked that sign-ins and your clients work. Until then it’s harmless: at each start the auth server finds the database under the current key and re-encrypts nothing.

    Delete the GOIABADA_AES_ENCRYPTION_KEY_PREVIOUS line from docker-compose.override.yml, then:

    Terminal window
    docker compose up -d goiabada-authserver

Keep the old key as long as you keep a database backup taken before the rotation.

The database password

Section titled “The database password”

GOIABADA_DB_PASSWORD is the password of the database user the auth server signs in as, GOIABADA_DB_USERNAME. The database checks a password only when a connection opens, so a running auth server keeps the connections it holds through a change, and opens every new one with the password it started with. What a rotation costs depends on whether the database accepts two passwords for one user at once: MySQL 8.0.14 and later do, PostgreSQL and SQL Server don’t.

On MySQL

Section titled “On MySQL”

  1. Give the user its new password, keeping the current one as a second password that’s still accepted. openssl rand -hex 32 makes one.

    The image creates root twice: for connections from other containers (%), which is the auth server’s, and from inside its own (localhost). In docker compose exec mysql-server mysql -uroot -p:

    ALTER USER 'root'@'%' IDENTIFIED BY '<new password>' RETAIN CURRENT PASSWORD;
    ALTER USER 'root'@'localhost' IDENTIFIED BY '<new password>' RETAIN CURRENT PASSWORD;

  2. Store the new password where the auth server reads it, and restart the auth server. During the restart the old password and the new one are both accepted, so nothing fails.

    In docker-compose.override.yml, set GOIABADA_DB_PASSWORD and MYSQL_ROOT_PASSWORD to the new password, then restart the auth server alone:

    Terminal window
    docker compose up -d --no-deps goiabada-authserver

    Without --no-deps, Compose would also recreate the database’s container, whose variable changed, restarting the database under the auth server. That container takes the new value the next time it’s recreated, which changes nothing in the database.

  3. Discard the old password once the auth server is back.

    ALTER USER 'root'@'%' DISCARD OLD PASSWORD;
    ALTER USER 'root'@'localhost' DISCARD OLD PASSWORD;

On PostgreSQL and SQL Server

Section titled “On PostgreSQL and SQL Server”

These hold one password per user, which a change replaces at once, so the auth server has to restart right after it.

  1. Store the new password where the auth server reads it, without restarting anything. No running server reads it, and it’s now stored before the database depends on it. On PostgreSQL, openssl rand -hex 32 makes one; SQL Server’s password policy asks for three of upper case, lower case, digits and symbols, which a hex string lacks.

    In docker-compose.override.yml, set GOIABADA_DB_PASSWORD and POSTGRES_PASSWORD or MSSQL_SA_PASSWORD to the new password.

  2. Change the user’s password to the same value. On PostgreSQL, \password asks for the new password twice without echoing it, and sends it hashed, so it reaches neither the server log nor psql’s history.

    On PostgreSQL, run docker compose exec postgres-server psql -U postgres, then:

    \password postgres

    On SQL Server, in docker compose exec mssql-server /opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -C, which asks for the current password:

    ALTER LOGIN sa WITH PASSWORD = '<new password>';
    GO

  3. Restart the auth server at once.

    Terminal window
    docker compose up -d --no-deps goiabada-authserver

    Without --no-deps, Compose would also recreate the database’s container, whose variable changed, restarting the database under the auth server. On SQL Server that container’s healthcheck signs in with the value it holds, so it reports unhealthy until it’s recreated: recreate it with docker compose up -d when a short database restart suits.

Between steps 2 and 3, a running auth server keeps the connections it holds, but each one it opens is refused, and the request that needed it fails with a 500. It opens connections as its traffic grows, and replaces each after 5 idle minutes (GOIABADA_DB_CONN_MAX_IDLE_TIME) or 30 minutes of life (GOIABADA_DB_CONN_MAX_LIFETIME), so the shorter the gap, the fewer requests fail.

If the auth server then stops with Access denied, password authentication failed or Login failed in its log, the password you stored and the database’s disagree: set the database user’s password to the one you stored. On Kubernetes, read it back with kubectl get secret goiabada-secrets -n goiabada -o jsonpath='{.data.db-password}' | base64 -d.

The admin console’s client secret

Section titled “The admin console’s client secret”

GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRET is the secret of the admin console’s client, admin-console-client, which the auth server keeps, encrypted, in its database. The auth server reads its own copy only at its first start, to create that client, so it needs no restart. The admin console presents the secret on every sign-in, on every refresh of an administrator’s access token, and for the token it asks for to reach its own sessions. So a rotation changes the client’s secret in the database and the admin console’s copy, then restarts the admin console.

  1. In the admin console, open Clients, then admin-console-client, then its Authentication tab. Click Generate new secret, then Copy. Don’t save yet.

  2. Store the copied secret where the admin console reads it. Nothing reads it before a restart, and it’s now stored before the database depends on it.

    Paste it as GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRET under both services in docker-compose.override.yml. The auth server’s copy is read only to set up an empty database, so it can take the new one at its next restart.

  3. Save the page, and confirm both warnings: that you’re changing the client secret, then that this is a system-level client. The database now holds the new secret.

  4. Restart the admin console at once, then check by signing in from a private window.

    Terminal window
    docker compose up -d --no-deps goiabada-adminconsole

    Without --no-deps, Compose would also recreate the auth server, whose copy changed, stopping sign-in while it restarts.

Between steps 3 and 4, the auth server refuses the old secret, which the running admin console presents. A sign-in fails at its last step, an administrator whose access token falls due is signed out, and once the token the admin console holds for its sessions expires, within the access token lifetime, 5 minutes unless you changed it, every page behind the sign-in answers 500. Administrators signed in before step 3 stay signed in through the restart. The auth server and every other client work throughout.

If the admin console can no longer sign you in, because the database holds a secret its copy doesn’t, Locked out of the admin console sets a secret you know through the admin API. Have the client it needs before you rotate: once the admin console is locked out, it’s the only way back in.

The admin password

Section titled “The admin password”

GOIABADA_ADMIN_PASSWORD is read only by the first start, which creates the administrator. Change the administrator’s password in the admin console; changing the variable changes nothing.

Next steps

Section titled “Next steps”
SecretsWhat each secret protects, and backing up the AES key.
Locked out of the admin consoleGet back in after a secret or password changed.
Upgrade GoiabadaUpdate to a new release, and roll back.