Skip to content

Locked out of the admin console

This page helps you get back into the admin console when signing in to it stops working.

Find your case first:

The admin console’s client secret changed

Section titled “The admin console’s client secret changed”

The admin console signs administrators in as a client of the auth server, admin-console-client, and proves who it is with that client’s secret. It uses the same secret for the token it reaches its own sessions with, which the auth server stores. The auth server keeps the secret in its database. The admin console reads its copy from GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRET. When the two differ, the auth server refuses the admin console with invalid_client, and:

  • a sign-in fails after the password, on Sign-in failed: “The admin console could not complete the sign-in with the auth server. This usually means the console’s client secret or the auth server’s address is wrong.”;
  • an administrator already signed in is signed out once their access token falls due;
  • once the token the admin console holds for its sessions expires, within the access token lifetime, 5 minutes unless you changed it, every page that needs a session answers 500 on Something went wrong, starting a sign-in included;
  • an admin console that starts, or restarts, doesn’t start at all. It asks the auth server for that token before it listens, and stops when it’s refused: on Kubernetes its pod never becomes ready, so a rollout waits with the earlier pods still serving, and under Docker Compose its container keeps restarting.

The admin console’s log has the cause: the auth server's token endpoint answered 401 (invalid_client: Client authentication failed. Please review your client_secret.). When it refused to start, the record is the auth server does not accept the admin console's client secret, so the admin console cannot start, with that cause as its error.

The auth server and every other client keep working.

That happens when someone generated a new secret on the client’s Authentication tab and saved it without giving the admin console the same one, or when the admin console’s configuration lost its value. The auth server reads GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRET only on its first start, so changing the variable later never changes the database.

If you know the secret the database holds, put it back in the admin console’s configuration and restart the admin console. Otherwise, set a secret you know through the Admin API, with a client of your own allowed authserver:manage.

It has to be manage. The admin console’s client is an administrator, and only a token with authserver:manage writes to one, so a manage-clients token is refused 403 MANAGE_SCOPE_REQUIRED. Holding manage makes your client an administrator too, so keep its secret with the care you give the admin console’s own.

  1. Get a token for your client, and find the admin console client’s id:

    Terminal window
    read -rsp 'API client secret: ' API_CLIENT_SECRET; echo
    TOKEN=$(printf %s "$API_CLIENT_SECRET" | curl -s https://auth.example.com/auth/token \
    -d grant_type=client_credentials -d client_id=<your API client> \
    --data-urlencode client_secret@- -d scope=authserver:manage | jq -r .access_token)
    unset API_CLIENT_SECRET
    CLIENT_ID=$(curl -s https://auth.example.com/api/v1/admin/clients -H "Authorization: Bearer $TOKEN" \
    | jq '.clients[] | select(.clientIdentifier == "admin-console-client") | .id')
    CLIENT_SECRET=$(openssl rand -hex 32)
  2. Store the new secret where the admin console reads it, before the auth server holds it:

    Set GOIABADA_ADMINCONSOLE_OAUTH_CLIENT_SECRET to $CLIENT_SECRET under both services in docker-compose.override.yml.

  3. Set the same secret on the client:

    Terminal window
    curl -s -X PUT "https://auth.example.com/api/v1/admin/clients/$CLIENT_ID/authentication" \
    -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' --data-binary @- <<EOF
    {"isPublic": false, "clientSecret": "$CLIENT_SECRET"}
    EOF
    unset CLIENT_SECRET TOKEN
  4. Restart the admin console:

    Terminal window
    docker compose up -d --no-deps goiabada-adminconsole
  5. Sign in from a private window to check.

Neither secret reaches the process list. The Admin API accepts a client secret of 60 to 255 letters, digits, -, _ and ., which openssl rand -hex 32, 64 hex characters, is.

The auth server reads GOIABADA_ADMIN_EMAIL and GOIABADA_ADMIN_PASSWORD only on its first start, when it creates the first administrator in an empty database. On every later start they do nothing, so the password you set there is refused at sign-in with “Authentication failed.”, and the old one still works.

Sign in with the password the administrator had, and change it under Account, Change password. If nobody knows it any more, see the next section.

The auth server never lets a change leave no enabled user holding authserver:manage: it refuses it with LAST_ADMINISTRATOR. That keeps the permission alive, but not a person able to use it. A user who set up two-factor authentication is always asked for a code when signing in to the admin console.

An administrator who lost their authenticator signs in with a recovery code, one of the ten they got when they set it up, through Use a recovery code instead on the page that asks for the code. They then turn two-factor authentication off under Account, Two-factor authentication, and set up a new authenticator. See Recovery codes.

So the only administrator is stuck when they forget their password, or lose their authenticator with no recovery code left. When nobody can let them back in, the auth server’s recover command can.

An authenticator that refuses every code, the right one included, was disabled after 100 wrong codes in a row, and the audit log records otp_authenticator_disabled. Recovery codes still sign in. Whoever sent the codes knew the password, so set a new password first and then turn off two-factor authentication. An administrator who signed in with a recovery code does both under Account. Otherwise, in the admin console, one save on the user’s Authentication tab does both in that order. With recover, it’s one run with both --password-stdin and --remove-otp. The administrator then signs in with the new password and sets up a new authenticator. See A user who lost their authenticator.

Another administrator can let them back in. Any user with authserver:manage can open Admin, Users, the user, and their Authentication tab, then type a new password in New password, or turn off Enabled under Two-factor authentication, and click Save. So the best fix comes before the problem: give authserver:manage to a second person.

A forgotten password can be reset by email, when SMTP is set up, the administrator’s email address is verified, and their mailbox still exists. A user cannot reset their password covers what stops it.

A client of your own with authserver:manage can do everything another administrator could, through the Admin API: set the administrator’s password, or turn off their two-factor authentication. It’s the same client the client secret fix needs.

When none of these is left, the auth server’s recover subcommand restores sign-in for one named user. It needs the auth server’s database credentials, so you run it where the auth server runs: on its host, in its Compose service or in its pod. It does only what its flags ask. Whatever they ask, it ends all of the user’s sessions and revokes their refresh tokens, and the audit log records the recovery with the command as the actor.

Flag What it does
--email <address> Names the user to recover, by their stored address. Required.
--grant-manage Grants the authserver:manage permission
--password-stdin Sets a new password, read from the first line of the command’s input. It must meet your password policy.
--password <password> Sets a new password given on the command line. Use --password-stdin instead, as below.
--remove-otp Turns off the user’s two-factor authentication and removes their authenticator
--disable-turnstile Turns off the Turnstile challenge and keeps its keys. It names no user, so it needs no --email, and given alone it ends no sessions.

One run can combine several flags. Type the new password at the first line, where nothing you type is shown. The second line hands it to the command’s input, and since printf is built into the shell, the password never appears in a command line:

Terminal window
printf 'New password: ' && IFS= read -rs RECOVERY_PASSWORD && echo
printf '%s\n' "$RECOVERY_PASSWORD" | docker compose run --rm -T goiabada-authserver recover --email [email protected] --grant-manage --password-stdin
unset RECOVERY_PASSWORD

When you don’t need a new password, leave out the first line, the pipe and --password-stdin.

The recovery doesn’t reset the limits on wrong passwords. If the sign-in page then says “Too many attempts”, first stop whatever is still making attempts. Too many attempts or 429 says which limits a restart of the auth server resets and which only time does.

The Turnstile challenge refuses every sign-in

Section titled “The Turnstile challenge refuses every sign-in”

With the Turnstile challenge on, the sign-in form accepts a password only with a token Cloudflare verified. The auth server can’t check the site key or the widget’s hostname when you save them, so when either is wrong, every attempt is refused with “Complete the security check, then try again. The check needs JavaScript.” The widget may show an error, or not appear at all. It’s usually one of these:

  • The site key is wrong, or belongs to another widget. Copy it again from the widget in the Cloudflare dashboard.
  • The widget doesn’t list the auth server’s host, the host of GOIABADA_AUTHSERVER_BASEURL. The widget then shows an error.
  • The widget lists another host, and the token is solved there. The auth server logs refused a turnstile token solved on another hostname with the hostname the token names and the expected_hostname.

The admin console signs in through the same form, so once you’re signed out, you can’t turn the challenge off there. Turn it off with the auth server’s recover command, where the auth server runs:

Terminal window
docker compose run --rm -T goiabada-authserver recover --disable-turnstile

The forms stop asking for the challenge at the next request, on every replica. Sign in, fix the widget in the Cloudflare dashboard or the site key under Admin, Turnstile, and turn the challenge on again. Before you sign out, check that a private window can sign in.

A brand’s custom CSS, custom palette or uploaded font can leave the sign-in page and the admin console’s pages unreadable, or with controls you can’t reach. Open any admin console address with ?goiabada_safe_mode=true at the end, such as https://admin.example.com/admin/settings/branding?goiabada_safe_mode=true, sign in, and fix the brand. See Safe mode.