Can't reach the auth server

This page helps you when the admin console can’t talk to the auth server.

What you see

Every admin console page answers 500 with a page titled “Can’t reach the auth server”:

The admin console couldn't load its settings from the auth server. Try again in a moment. The server log records the cause under the request ID below.

The admin console’s log has unable to fetch settings from the auth server, and its error says what the connection failed with: a name that doesn’t resolve, a connection refused, a timeout, or a certificate the admin console doesn’t trust. Or it starts authserver returned status, when something answered with an error instead of the settings. Or it has a redirect to and an address, when something answered with a redirect.

For up to 30 seconds after the auth server goes away, while the admin console still holds its settings, a page that needs a session answers Something went wrong instead; the home page still shows to a visitor who isn’t signed in.

The admin console’s own /health still answers healthy, because it doesn’t check the auth server. So a container or a pod looks fine while every page fails.

An admin console that starts while the auth server isn’t answering waits for it before it listens, so its pod doesn’t become ready meanwhile. Its log then has waiting for the auth server to issue the admin console's token every few seconds, with what each attempt met as its error. The causes below apply to the wait as well. A redirect doesn’t wait: the admin console stops with the auth server redirected the admin console's token request, so the admin console cannot start.

Why it happens

The admin console calls the auth server on every page, for its settings and for everything it shows. It calls it at GOIABADA_AUTHSERVER_INTERNALBASEURL, or at GOIABADA_AUTHSERVER_BASEURL when the internal URL is empty. Those are the admin console’s own variables: set on the auth server alone, they do nothing for the admin console.

The usual causes:

The auth server isn’t running yet, or stopped. Check its own /health.
The internal URL is wrong for where the admin console runs. Inside Docker Compose, localhost is the admin console’s own container, not the auth server’s. Use the auth server’s service name, such as http://goiabada-authserver:9090.
The internal URL is empty, and the public URL isn’t reachable from inside, such as a public hostname that only resolves outside your network, or a firewall that stops the admin console’s host from reaching it.
The internal URL is https, and its certificate isn’t trusted or doesn’t name the host in the URL.
On Kubernetes, the NetworkPolicies refuse a service mesh’s port. A mesh whose proxies deliver on a port of their own, such as Istio’s ambient mode or Linkerd, needs that port admitted beside the auth server’s: the error is a connection reset by peer, or authserver returned status 504 from the mesh’s proxy. See Who can reach Goiabada.
The internal URL reaches an address that redirects. The admin console doesn’t follow redirects, so its error says a redirect to and the address it was sent to. The usual one is the auth server’s http port when the auth server’s base URL is https and GOIABADA_AUTHSERVER_TRUST_PROXY_HEADERS is off: that listener only redirects to the base URL. Its /health still answers there, while every other path redirects. Use the auth server’s HTTPS listener, or turn trust on when a proxy that ends HTTPS fronts the auth server.
Something answers, but with an error, authserver returned status and the status: a proxy in between answering 404 or 502, or the auth server answering 500 because its database is down. The auth server’s /health doesn’t check the database, so it can answer healthy meanwhile: read the auth server’s log.

Fix it

Find the URL the admin console calls: its GOIABADA_AUTHSERVER_INTERNALBASEURL, or its GOIABADA_AUTHSERVER_BASEURL when that’s empty.
Run the admin console’s check-authserver command where the admin console runs: with Docker Compose, on Kubernetes or with native binaries. It asks that URL for the settings once, as the admin console does: it trusts the same certificate authorities and follows no redirect. It needs no session keys or client secret.

When the auth server answers with its settings, the check prints the URL and exits 0:
```
GET http://goiabada-authserver:9090/api/public/settings answered with the auth server's public settings
```
Otherwise it prints the error the admin console logs and exits 1. Match it against the causes above.
Set the admin console’s GOIABADA_AUTHSERVER_INTERNALBASEURL to an address the check reaches, and restart the admin console.

For an https internal URL, the admin console trusts the operating system’s certificate authorities. If your own authority signed the auth server’s certificate, set the admin console’s GOIABADA_ADMINCONSOLE_INTERNAL_CA_FILE to that authority’s PEM certificate, and restart the admin console. The check reads the same file.

Run the check with Docker Compose

From the directory holding docker-compose.yml, run it in the admin console’s container:

docker compose exec goiabada-adminconsole /app/goiabada-adminconsole check-authserver

When the admin console isn’t running, such as when it stopped or never started, run the check in a container of its own. It takes the service’s environment, network and mounts, and --no-deps leaves the other services as they are:

docker compose run --rm --no-deps goiabada-adminconsole check-authserver

To try another address before you change the file, give it to that container:

docker compose run --rm --no-deps -e GOIABADA_AUTHSERVER_INTERNALBASEURL=http://goiabada-authserver:9090 goiabada-adminconsole check-authserver

Run the check on Kubernetes

The commands use the wizard’s namespace, goiabada. While the admin console’s container is running, run the check in it. That includes an admin console still waiting for the auth server, which runs but doesn’t listen yet:

kubectl -n goiabada exec deploy/goiabada-adminconsole -c adminconsole -- /app/goiabada-adminconsole check-authserver

When the container isn’t running, find out what stopped it from the pod’s status and events, and from its current and previous logs:

kubectl -n goiabada get pods -l app=goiabada-adminconsole
kubectl -n goiabada describe pod <pod>
kubectl -n goiabada logs <pod> -c adminconsole
kubectl -n goiabada logs <pod> -c adminconsole --previous

The logs name what stopped it, such as a malformed setting, missing session keys or client secret, or the auth server refusing or redirecting the admin console’s client. Two kinds of stop are easy to confuse:

The startup probe restarted it. An admin console waiting for the auth server doesn’t listen, so after about 5 minutes its startup probe fails and Kubernetes restarts the container. The previous log ends with a run of waiting for the auth server to issue the admin console's token records, and then admin console stopped. Each record’s error is what its attempt met: a name that doesn’t resolve, a refused connection, a certificate, or a 5xx. That error is the cause to match above. kubectl describe pod shows the last state’s exit code as 0, and a failed startup probe in its events.
The auth server refused or redirected it. The previous log ends in the auth server does not accept the admin console's client secret, …, the auth server refused the admin console's token request, … or the auth server redirected the admin console's token request, …, and the exit code is 1. Waiting doesn’t help. A refused secret is covered in Locked out of the admin console, and a redirect is the redirect cause above.

Don’t run the check from a pod of your own. Where the cluster’s network plugin enforces NetworkPolicies, the wizard’s admits to the auth server’s port only pods labelled app: goiabada-adminconsole. A pod carrying that label joins the admin console’s Service, which then sends administrators to it.

Run the check with native binaries

Run it as the admin console’s service user, with the service’s environment file, here as Native binaries sets them up:

sudo -u goiabada sh -c 'cd /var/lib/goiabada && set -a && . /etc/goiabada/goiabada.env && set +a && exec /opt/goiabada/goiabada-adminconsole check-authserver'

When only signing in fails

Pages behind the sign-in need the browser to reach the auth server too. The admin console sends the browser to GOIABADA_AUTHSERVER_BASEURL to sign in, so that one must be the auth server’s public address, reachable from the administrator’s browser.

The sign-in can end on Sign-in failed after the password, when the browser reached the auth server but the admin console couldn’t finish the sign-in with it:

“The admin console could not complete the sign-in with the auth server. This usually means the console’s client secret or the auth server’s address is wrong.” Check the internal URL as above. If it answers, the client secret is the likelier cause: see Locked out of the admin console.
“The auth server’s answer could not be verified. This usually means the admin console and the auth server disagree about the issuer or the client.” The admin console couldn’t fetch the auth server’s signing keys from /certs at the internal URL, or the ID token didn’t name the issuer the auth server reports.

The connection to the auth server

The hop from the admin console to the auth server carries the admin console’s client secret and administrators’ tokens. With an http internal URL it’s plain HTTP, which is sound on a network you trust, such as a Docker Compose network or a cluster’s pod network. When the network isn’t trusted, use an https internal URL to the auth server’s own HTTPS listener, whose certificate names the host the admin console calls.

The admin console waits up to 10 seconds for each call to the auth server, up to 50 seconds for a test email, a new user’s setup email and an email verification code, and up to 5 minutes for a download of a user’s data.

Next steps

Environment variablesThe base URLs and the internal URL.

Locked out of the admin consoleWhen signing in to the admin console fails.