Locked out of the admin console
This page helps you get back into the admin console when signing in to it stops working.
Find your case first:
- Signing in fails for every administrator with Sign-in failed or Something went wrong: the admin console’s client secret changed.
- The password you set in
GOIABADA_ADMIN_PASSWORDis refused: you changed it after the first start. - The only administrator can’t sign in, because they forgot their password, lost their authenticator, or their authenticator refuses every code: the last administrator can’t sign in.
- The sign-in form refuses everyone since the Turnstile challenge was turned on: the Turnstile challenge refuses every sign-in.
- The sign-in page or the admin console can’t be read or used since a brand changed: a brand made the pages unusable.
The admin console’s client secret changed
Section titled “The admin console’s client secret changed”The admin console signs administrators in as a client of the auth server, admin-console-client, and proves who it is with that client’s secret. It uses the same secret for the token it reaches its own sessions with, which the auth server stores. The auth server keeps the secret in its database. The admin console reads its copy from GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRET. When the two differ, the auth server refuses the admin console with invalid_client, and:
- a sign-in fails after the password, on Sign-in failed: “The admin console could not complete the sign-in with the auth server. This usually means the console’s client secret or the auth server’s address is wrong.”;
- an administrator already signed in is signed out once their access token falls due;
- once the token the admin console holds for its sessions expires, within the access token lifetime, 5 minutes unless you changed it, every page that needs a session answers
500on Something went wrong, starting a sign-in included; - an admin console that starts, or restarts, doesn’t start at all. It asks the auth server for that token before it listens, and stops when it’s refused: on Kubernetes its pod never becomes ready, so a rollout waits with the earlier pods still serving, and under Docker Compose its container keeps restarting.
The admin console’s log has the cause: the auth server's token endpoint answered 401 (invalid_client: Client authentication failed. Please review your client_secret.). When it refused to start, the record is the auth server does not accept the admin console's client secret, so the admin console cannot start, with that cause as its error.
The auth server and every other client keep working.
That happens when someone generated a new secret on the client’s Authentication tab and saved it without giving the admin console the same one, or when the admin console’s configuration lost its value. The auth server reads GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRET only on its first start, so changing the variable later never changes the database.
Fix it
Section titled “Fix it”If you know the secret the database holds, put it back in the admin console’s configuration and restart the admin console. Otherwise, set a secret you know through the Admin API, with a client of your own allowed authserver:manage.
It has to be manage. The admin console’s client is an administrator, and only a token with authserver:manage writes to one, so a manage-clients token is refused 403 MANAGE_SCOPE_REQUIRED. Holding manage makes your client an administrator too, so keep its secret with the care you give the admin console’s own.
-
Get a token for your client, and find the admin console client’s id:
Terminal window read -rsp 'API client secret: ' API_CLIENT_SECRET; echoTOKEN=$(printf %s "$API_CLIENT_SECRET" | curl -s https://auth.example.com/auth/token \-d grant_type=client_credentials -d client_id=<your API client> \--data-urlencode client_secret@- -d scope=authserver:manage | jq -r .access_token)unset API_CLIENT_SECRETCLIENT_ID=$(curl -s https://auth.example.com/api/v1/admin/clients -H "Authorization: Bearer $TOKEN" \| jq '.clients[] | select(.clientIdentifier == "admin-console-client") | .id')CLIENT_SECRET=$(openssl rand -hex 32) -
Store the new secret where the admin console reads it, before the auth server holds it:
Set
GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRETto$CLIENT_SECRETunder both services indocker-compose.override.yml.Set
GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRETto$CLIENT_SECRETin/etc/goiabada/goiabada.env, the env file both servers read.Terminal window kubectl patch secret goiabada-secrets -n goiabada --type=merge --patch-file=/dev/stdin <<EOF{"stringData": {"oauth-client-secret": "$CLIENT_SECRET"}}EOF -
Set the same secret on the client:
Terminal window curl -s -X PUT "https://auth.example.com/api/v1/admin/clients/$CLIENT_ID/authentication" \-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' --data-binary @- <<EOF{"isPublic": false, "clientSecret": "$CLIENT_SECRET"}EOFunset CLIENT_SECRET TOKEN -
Restart the admin console:
Terminal window docker compose up -d --no-deps goiabada-adminconsoleTerminal window sudo systemctl restart goiabada-adminconsoleTerminal window kubectl rollout restart deployment/goiabada-adminconsole -n goiabada -
Sign in from a private window to check.
Neither secret reaches the process list. The Admin API accepts a client secret of 60 to 255 letters, digits, -, _ and ., which openssl rand -hex 32, 64 hex characters, is.
You changed GOIABADA_ADMIN_PASSWORD
Section titled “You changed GOIABADA_ADMIN_PASSWORD”The auth server reads GOIABADA_ADMIN_EMAIL and GOIABADA_ADMIN_PASSWORD only on its first start, when it creates the first administrator in an empty database. On every later start they do nothing, so the password you set there is refused at sign-in with “Authentication failed.”, and the old one still works.
Sign in with the password the administrator had, and change it under Account, Change password. If nobody knows it any more, see the next section.
The last administrator can’t sign in
Section titled “The last administrator can’t sign in”The auth server never lets a change leave no enabled user holding authserver:manage: it refuses it with LAST_ADMINISTRATOR. That keeps the permission alive, but not a person able to use it. A user who set up two-factor authentication is always asked for a code when signing in to the admin console.
An administrator who lost their authenticator signs in with a recovery code, one of the ten they got when they set it up, through Use a recovery code instead on the page that asks for the code. They then turn two-factor authentication off under Account, Two-factor authentication, and set up a new authenticator. See Recovery codes.
So the only administrator is stuck when they forget their password, or lose their authenticator with no recovery code left. When nobody can let them back in, the auth server’s recover command can.
An authenticator that refuses every code, the right one included, was disabled after 100 wrong codes in a row, and the audit log records otp_authenticator_disabled. Recovery codes still sign in. Whoever sent the codes knew the password, so set a new password first and then turn off two-factor authentication. An administrator who signed in with a recovery code does both under Account. Otherwise, in the admin console, one save on the user’s Authentication tab does both in that order. With recover, it’s one run with both --password-stdin and --remove-otp. The administrator then signs in with the new password and sets up a new authenticator. See A user who lost their authenticator.
Another administrator can let them back in. Any user with authserver:manage can open Admin, Users, the user, and their Authentication tab, then type a new password in New password, or turn off Enabled under Two-factor authentication, and click Save. So the best fix comes before the problem: give authserver:manage to a second person.
A forgotten password can be reset by email, when SMTP is set up, the administrator’s email address is verified, and their mailbox still exists. A user cannot reset their password covers what stops it.
A client of your own with authserver:manage can do everything another administrator could, through the Admin API: set the administrator’s password, or turn off their two-factor authentication. It’s the same client the client secret fix needs.
Recover the user with the auth server
Section titled “Recover the user with the auth server”When none of these is left, the auth server’s recover subcommand restores sign-in for one named user. It needs the auth server’s database credentials, so you run it where the auth server runs: on its host, in its Compose service or in its pod. It does only what its flags ask. Whatever they ask, it ends all of the user’s sessions and revokes their refresh tokens, and the audit log records the recovery with the command as the actor.
| Flag | What it does |
|---|---|
--email <address> |
Names the user to recover, by their stored address. Required. |
--grant-manage |
Grants the authserver:manage permission |
--password-stdin |
Sets a new password, read from the first line of the command’s input. It must meet your password policy. |
--password <password> |
Sets a new password given on the command line. Use --password-stdin instead, as below. |
--remove-otp |
Turns off the user’s two-factor authentication and removes their authenticator |
--disable-turnstile |
Turns off the Turnstile challenge and keeps its keys. It names no user, so it needs no --email, and given alone it ends no sessions. |
One run can combine several flags. Type the new password at the first line, where nothing you type is shown. The second line hands it to the command’s input, and since printf is built into the shell, the password never appears in a command line:
printf 'New password: ' && IFS= read -rs RECOVERY_PASSWORD && echoprintf '%s\n' "$RECOVERY_PASSWORD" | docker compose run --rm -T goiabada-authserver recover --email [email protected] --grant-manage --password-stdinunset RECOVERY_PASSWORDprintf 'New password: ' && IFS= read -rs RECOVERY_PASSWORD && echoprintf '%s\n' "$RECOVERY_PASSWORD" | sudo -u goiabada sh -c 'cd /var/lib/goiabada && set -a && . /etc/goiabada/goiabada.env && set +a && exec /opt/goiabada/goiabada-authserver recover --email [email protected] --grant-manage --password-stdin'unset RECOVERY_PASSWORDprintf 'New password: ' && IFS= read -rs RECOVERY_PASSWORD && echoprintf '%s\n' "$RECOVERY_PASSWORD" | kubectl exec -i -n goiabada deployment/goiabada-authserver -- /app/goiabada-authserver recover --email [email protected] --grant-manage --password-stdinunset RECOVERY_PASSWORDWhen you don’t need a new password, leave out the first line, the pipe and --password-stdin.
The recovery doesn’t reset the limits on wrong passwords. If the sign-in page then says “Too many attempts”, first stop whatever is still making attempts. Too many attempts or 429 says which limits a restart of the auth server resets and which only time does.
The Turnstile challenge refuses every sign-in
Section titled “The Turnstile challenge refuses every sign-in”With the Turnstile challenge on, the sign-in form accepts a password only with a token Cloudflare verified. The auth server can’t check the site key or the widget’s hostname when you save them, so when either is wrong, every attempt is refused with “Complete the security check, then try again. The check needs JavaScript.” The widget may show an error, or not appear at all. It’s usually one of these:
- The site key is wrong, or belongs to another widget. Copy it again from the widget in the Cloudflare dashboard.
- The widget doesn’t list the auth server’s host, the host of
GOIABADA_AUTHSERVER_BASEURL. The widget then shows an error. - The widget lists another host, and the token is solved there. The auth server logs
refused a turnstile token solved on another hostnamewith thehostnamethe token names and theexpected_hostname.
The admin console signs in through the same form, so once you’re signed out, you can’t turn the challenge off there. Turn it off with the auth server’s recover command, where the auth server runs:
docker compose run --rm -T goiabada-authserver recover --disable-turnstilesudo -u goiabada sh -c 'cd /var/lib/goiabada && set -a && . /etc/goiabada/goiabada.env && set +a && exec /opt/goiabada/goiabada-authserver recover --disable-turnstile'kubectl exec -i -n goiabada deployment/goiabada-authserver -- /app/goiabada-authserver recover --disable-turnstileThe forms stop asking for the challenge at the next request, on every replica. Sign in, fix the widget in the Cloudflare dashboard or the site key under Admin, Turnstile, and turn the challenge on again. Before you sign out, check that a private window can sign in.
A brand made the pages unusable
Section titled “A brand made the pages unusable”A brand’s custom CSS, custom palette or uploaded font can leave the sign-in page and the admin console’s pages unreadable, or with controls you can’t reach. Open any admin console address with ?goiabada_safe_mode=true at the end, such as https://admin.example.com/admin/settings/branding?goiabada_safe_mode=true, sign in, and fix the brand. See Safe mode.