Locked out of the admin console

This page helps you get back into the admin console when signing in to it stops working.

Find your case first:

• Signing in fails for every administrator with Sign-in failed or Something went wrong: the admin console’s client secret changed.
• The password you set in GOIABADA_ADMIN_PASSWORD is refused: you changed it after the first start.
• The only administrator can’t sign in, because they forgot their password, lost their authenticator, or their authenticator refuses every code: the last administrator can’t sign in.

The admin console’s client secret changed

The admin console signs administrators in as a client of the auth server, admin-console-client, and proves who it is with that client’s secret. It uses the same secret for the token it reaches its own sessions with, which the auth server stores. The auth server keeps the secret in its database. The admin console reads its copy from GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRET. When the two differ, the auth server refuses the admin console with invalid_client, and:

• a sign-in fails after the password, on Sign-in failed: “The admin console could not complete the sign-in with the auth server. This usually means the console’s client secret or the auth server’s address is wrong.”;
• an administrator already signed in is signed out once their access token falls due;
• once the token the admin console holds for its sessions expires, within the access token lifetime, 5 minutes unless you changed it, every page that needs a session answers 500 on Something went wrong, starting a sign-in included;
• an admin console that starts, or restarts, doesn’t start at all. It asks the auth server for that token before it listens, and stops when it’s refused: on Kubernetes its pod never becomes ready, so a rollout waits with the earlier pods still serving, and under Docker Compose its container keeps restarting.

The admin console’s log has the cause: the auth server's token endpoint answered 401 (invalid_client: Client authentication failed. Please review your client_secret.). When it refused to start, the record is the auth server does not accept the admin console's client secret, so the admin console cannot start, with that cause as its error.

The auth server and every other client keep working.

That happens when someone generated a new secret on the client’s Authentication tab and saved it without giving the admin console the same one, or when the admin console’s configuration lost its value. The auth server reads GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRET only on its first start, so changing the variable later never changes the database.

Fix it

If you know the secret the database holds, put it back in the admin console’s configuration and restart the admin console. Otherwise, set a secret you know through the Admin API, with a client of your own allowed authserver:manage.

It has to be manage. The admin console’s client is an administrator, and only a token with authserver:manage writes to one, so a manage-clients token is refused 403 MANAGE_SCOPE_REQUIRED. Holding manage makes your client an administrator too, so keep its secret with the care you give the admin console’s own.

Caution

Create that client before you need it: a client with Client credentials on, and the authserver resource’s manage permission granted on its Permissions tab. Once the admin console is locked out, it’s the only way back in.

1. Get a token for your client, and find the admin console client’s id:

   read -rsp 'API client secret: ' API_CLIENT_SECRET; echo
   TOKEN=$(printf %s "$API_CLIENT_SECRET" | curl -s https://auth.example.com/auth/token \
     -d grant_type=client_credentials -d client_id=<your API client> \
     --data-urlencode client_secret@- -d scope=authserver:manage | jq -r .access_token)
   unset API_CLIENT_SECRET
   CLIENT_ID=$(curl -s https://auth.example.com/api/v1/admin/clients -H "Authorization: Bearer $TOKEN" \
     | jq '.clients[] | select(.clientIdentifier == "admin-console-client") | .id')
   CLIENT_SECRET=$(openssl rand -hex 32)

2. Store the new secret where the admin console reads it, before the auth server holds it:

   Docker Compose

   Set GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRET to $CLIENT_SECRET under both services in docker-compose.override.yml.

   Native binaries

   Set GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRET to $CLIENT_SECRET in /etc/goiabada/goiabada.env, the env file both servers read.

   Kubernetes

   kubectl patch secret goiabada-secrets -n goiabada --type=merge --patch-file=/dev/stdin <<EOF
   {"stringData": {"oauth-client-secret": "$CLIENT_SECRET"}}
   EOF

3. Set the same secret on the client:

   curl -s -X PUT "https://auth.example.com/api/v1/admin/clients/$CLIENT_ID/authentication" \
     -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' --data-binary @- <<EOF
   {"isPublic": false, "clientSecret": "$CLIENT_SECRET"}
   EOF
   unset CLIENT_SECRET TOKEN

4. Restart the admin console:

   Docker Compose

   docker compose up -d --no-deps goiabada-adminconsole

   Native binaries

   sudo systemctl restart goiabada-adminconsole

   Kubernetes

   kubectl rollout restart deployment/goiabada-adminconsole -n goiabada

5. Sign in from a private window to check.

Neither secret reaches the process list. The Admin API accepts a client secret of 60 to 255 letters, digits, -, _ and ., which openssl rand -hex 32, 64 hex characters, is.

You changed GOIABADA_ADMIN_PASSWORD

The auth server reads GOIABADA_ADMIN_EMAIL and GOIABADA_ADMIN_PASSWORD only on its first start, when it creates the first administrator in an empty database. On every later start they do nothing, so the password you set there is refused at sign-in with “Authentication failed.”, and the old one still works.

Sign in with the password the administrator had, and change it under Account, Change password. If nobody knows it any more, see the next section.

The last administrator can’t sign in

The auth server never lets a change leave no enabled user holding authserver:manage: it refuses it with LAST_ADMINISTRATOR. That keeps the permission alive, but not a person able to use it. A user who set up two-factor authentication is always asked for a code when signing in to the admin console, and there are no backup codes. So when the only administrator forgets their password or loses their authenticator, they’re stuck. When nobody can let them back in, the auth server’s recover command can.

An authenticator that refuses every code, the right one included, was disabled after 100 wrong codes in a row, and the audit log records otp_authenticator_disabled. Whoever sent the codes knew the password, so set a new password first and then turn off two-factor authentication. In the admin console, one save does both in that order. With recover, it’s one run with both --password-stdin and --remove-otp. The administrator then signs in with the new password and sets up a new authenticator. See A user who lost their authenticator.

Another administrator can let them back in. Any user with authserver:manage can open Admin, Users, the user, and their Authentication tab, then type a new password in New password, or turn off Enabled under Two-factor authentication, and click Save. So the best fix comes before the problem: give authserver:manage to a second person.

A forgotten password can be reset by email, when SMTP is set up, the administrator’s email address is verified, and their mailbox still exists. A user cannot reset their password covers what stops it.

A client of your own with authserver:manage can do everything another administrator could, through the Admin API: set the administrator’s password, or turn off their two-factor authentication. It’s the same client the client secret fix needs.

Recover the user with the auth server

When none of these is left, the auth server’s recover subcommand restores sign-in for one named user. It needs the auth server’s database credentials, so you run it where the auth server runs: on its host, in its Compose service or in its pod. It does only what its flags ask. Whatever they ask, it ends all of the user’s sessions and revokes their refresh tokens, and the audit log records the recovery with the command as the actor.

Flag	What it does
--email <address>	Names the user to recover, by their stored address. Required.
--grant-manage	Grants the authserver:manage permission
--password-stdin	Sets a new password, read from the first line of the command’s input. It must meet your password policy.
--password <password>	Sets a new password given on the command line. Use --password-stdin instead, as below.
--remove-otp	Turns off the user’s two-factor authentication and removes their authenticator

One run can combine several flags. Type the new password at the first line, where nothing you type is shown. The second line hands it to the command’s input, and since printf is built into the shell, the password never appears in a command line:

Docker Compose

printf 'New password: ' && IFS= read -rs RECOVERY_PASSWORD && echo
printf '%s\n' "$RECOVERY_PASSWORD" | docker compose run --rm -T goiabada-authserver recover --email [email protected] --grant-manage --password-stdin
unset RECOVERY_PASSWORD

Native binaries

printf 'New password: ' && IFS= read -rs RECOVERY_PASSWORD && echo
printf '%s\n' "$RECOVERY_PASSWORD" | sudo -u goiabada sh -c 'cd /var/lib/goiabada && set -a && . /etc/goiabada/goiabada.env && set +a && exec /opt/goiabada/goiabada-authserver recover --email [email protected] --grant-manage --password-stdin'
unset RECOVERY_PASSWORD

Kubernetes

printf 'New password: ' && IFS= read -rs RECOVERY_PASSWORD && echo
printf '%s\n' "$RECOVERY_PASSWORD" | kubectl exec -i -n goiabada deployment/goiabada-authserver -- /app/goiabada-authserver recover --email [email protected] --grant-manage --password-stdin
unset RECOVERY_PASSWORD

When you don’t need a new password, leave out the first line, the pipe and --password-stdin.

Danger

Don’t give the password with --password on a server. A password on the command line is kept in the shell’s history and shown in the process list while the command runs. sudo writes its whole command line to the system log, and with an audit policy, the Kubernetes API server writes a kubectl exec command to its audit log. Keep --password for a script that has no other way to pass it, and when it was used, change the password under Account, Change password once the administrator is back in.

The recovery doesn’t reset the limits on wrong passwords. If the sign-in page then says “Too many attempts”, first stop whatever is still making attempts. Too many attempts or 429 says which limits a restart of the auth server resets and which only time does.

Next steps

AdministratorsWho is an administrator, and what only authserver:manage can do.

First sign-inThe first administrator and the admin console's sign-in.