High availability

This page helps you run more than one pod of each Goiabada server, so a node going away doesn’t take sign-ins with it.

The setup wizard asks how many auth server pods to run, two unless you answer otherwise, and the manifest runs one admin console pod. At one pod, a node drain or a node failure stops that server until its replacement is ready.

Scale out

Check the database can take the connections. Every auth server pod opens a pool of its own, of at most GOIABADA_DB_MAX_OPEN_CONNS connections, 20 unless you set it, against the one database they share. The pods must fit under the database’s connection limit:
```
(replicas + surge) × GOIABADA_DB_MAX_OPEN_CONNS + headroom < max_connections
```
Connections explains each term, and the wizard’s completion message gives the sum for the count you answered. With the defaults, two replicas make 3 × 20 = 60 connections, and three make 4 × 20 = 80. A stock PostgreSQL allows 100, 3 of them reserved for superusers: that leaves 37 for the headroom at two replicas, room for one stopping pod, and 17 at three, room for none. A stock MySQL allows 151. Until the sum holds, raise the database’s max_connections, or lower GOIABADA_DB_MAX_OPEN_CONNS.

Lower it in a kustomization over the manifest, since the wizard writes the manifest again on every run. Save this as kustomization.yaml, beside goiabada-k8s.yaml, with the limit your sum needs:
```
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- goiabada-k8s.yaml
patches:
- target:
    kind: Deployment
    name: goiabada-authserver
  patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: goiabada-authserver
    spec:
      template:
        spec:
          containers:
          - name: authserver
            env:
            - name: GOIABADA_DB_MAX_OPEN_CONNS
              value: "10"
```
The container’s env overrides the ConfigMap, and changes the pod template, so applying it rolls the pods.
Set the auth server’s count. Answer the wizard’s replicas question, or write the manifest again over your answers file with the count you want, and apply it:
Terminal window
```
./goiabada-setup-linux-amd64 --answers=goiabada-setup.yaml --keep-secrets --authserver-replicas=3
kubectl apply -f goiabada-k8s.yaml
```
With a kustomization over the manifest, such as step 1’s, step 3’s or the Internal TLS guide’s, apply that instead, with kubectl apply -k .: applying the manifest alone drops its patches. kubectl scale changes the count too, until the next apply sets it back.
Scale the admin console, if you want more than one pod, after reading The admin console across replicas. The wizard writes one admin console pod every time, so keep your count in a kustomization over the manifest, adding its replicas to step 1’s if you saved it:
```
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- goiabada-k8s.yaml
replicas:
- name: goiabada-adminconsole
  count: 2
```
Apply it with kubectl apply -k ..
Decide on the rate limiter, whose per-pod limits multiply with the replicas: see Rate limiting across replicas.

Connections

Surge is the extra pod a rolling update runs beside the old ones. The generated Deployments set maxSurge: 1 and maxUnavailable: 0: a rollout starts one new pod, waits for it to be ready, and only then stops an old one, so three replicas briefly run four, and never fewer than three. A stopping pod holds its connections until it exits, up to a minute, and the rollout starts the next new pod as soon as one is ready, so several old pods can be stopping at once. Leave room for them in the headroom: at most replicas × GOIABADA_DB_MAX_OPEN_CONNS more, should every old pod still be stopping.
Headroom is every other connection the database must accept: the migrate subcommand, the maintenance connection a starting pod opens to create the database when GOIABADA_DB_CREATE is on, your own sessions, and any other application on the same server. The admin console opens none.
With a HorizontalPodAutoscaler, replicas is its maxReplicas, not today’s count, since it may scale out to it under load, at the worst moment.

A capped pool trades one failure for another, on purpose. When every connection of a pod is busy, its next request waits for one to be released, until the client or the gateway gives up, rather than the database refusing a new connection to every pod at once. Raising replicas without raising max_connections adds no capacity at the database: it moves the bottleneck from the pods’ queues to the database’s refusals. The pool’s settings are in Environment variables.

Disruptions and spread

Each Deployment has a PodDisruptionBudget with maxUnavailable: 1, so a voluntary disruption, such as a kubectl drain, a node pool upgrade or an autoscaler removing a node, evicts at most one of its pods at a time. At one replica that one is the only pod, and the server is down until its replacement is scheduled elsewhere and ready; the rolling update doesn’t help, because an eviction starts no surge pod first. Run at least two replicas to keep serving through a drain. The same budget then keeps all but one pod serving at any count, so it needs no change when you scale.

Each pod template also spreads its pods over nodes, with a topologySpreadConstraints on kubernetes.io/hostname, maxSkew: 1 and whenUnsatisfiable: ScheduleAnyway. At three replicas on three nodes it places one per node, so losing a node takes one pod rather than all of them. On fewer nodes than replicas it still schedules every pod rather than leave one pending.

Autoscaling

The manifest emits no HorizontalPodAutoscaler. You can add one; settle two things first:

The database’s connections, sized for maxReplicas plus the surge pod, as Connections says.
What to scale on. CPU is the auth server’s bottleneck under a burst of sign-ins (see Resource limits), so a CPU target relative to its 100m request is a reasonable signal. The admin console checks no passwords and rarely needs more than one or two replicas.

Rate limiting across replicas

The auth server’s limits counted by a user or an email always apply: wrong passwords for one account, 100 an hour, one-time codes, the Account pages’ password checks, email changes on the Account pages, email verification, and password-reset and registration mails to one address. The limits counted by an IP address are off unless you turn them on, with GOIABADA_AUTHSERVER_RATELIMITER_ENABLED. The wizard asks, and writes the answer into the auth server’s ConfigMap either way; --rate-limiter=true or =false answers without the prompt. Under the Envoy Gateway recipe, its default follows the gateway’s traffic policy, and under any other routing it’s off, since the wizard can’t know which address Goiabada sees:

Under Cluster, off. Goiabada sees a node’s address for every client, so a per-IP limit counts together every user the load balancer sends through one node: 30 password posts a minute and 20 forgot-password requests per 5 minutes per node, for example, which throttles sign-ins on a busy site. On, it also limits wrong passwords for one email from one network, and counts every client through one node as one network, so anyone who knows an email can block its password sign-ins for 15 minutes with 10 wrong passwords. Off, the limit of 100 wrong passwords an hour for one account still bounds guessing. Turn it on if a tighter bound matters more to you than that.
Under Local, on. Goiabada sees each client’s address, so each per-IP limit counts one client, behind a load balancer that passes connections through. One that proxies them shows Goiabada its own address for every client, and the per-IP limits then count every client together, as under Cluster. The gateway’s traffic policy has how to tell.

Not every limit holds across replicas:

Shared: the limits on guessing credentials, which count failed passwords (at sign-in and through the password grant), failed OTP codes, failed account password checks and failed email verification codes. They’re counted in the database every pod shares, so three replicas still allow one budget, and a rollout doesn’t reset it. If the database can’t answer such a check within 5 seconds, the request is answered with a server error rather than let through.
Per pod: every other limit: the per-IP limits, the limits on outgoing mail, and the limit on one account’s email changes. Behind the Service, three replicas give a client up to three times each budget, and a rollout resets them. For a limit across the whole deployment, use the gateway’s own rate limiting. /auth/authorize writes a database row for each request it accepts, so give it a budget at the gateway when the limiter is off, or when you want one budget across the replicas.

The budgets are in Rate limits. With the limiter on, the auth server also logs a warning at every start that it trusts one proxy hop with no list; behind Envoy alone that’s expected, as The startup warnings explains.

The admin console across replicas

With more than one admin console pod, two of them can refresh the same administrator’s access token at once, and each presents the same refresh token. The auth server refuses the second, or treats it as a replay, which revokes the session’s tokens and leaves a refresh_token_replay_detected entry in the audit log. Either way, the administrator may have to sign in again.

At one console replica there’s nothing to do. With more, keep each browser on one console pod with session affinity at the gateway, such as a consistent hash on the client’s address or on a cookie, which Envoy Gateway’s load balancing offers. A browser then moves to another pod only when the pods change, as in a rollout or a drain.

Resource limits

Both containers request 100m of CPU and 128Mi of memory, and are limited to 500m and 512Mi:

resources:
  requests:
    memory: "128Mi"
    cpu: "100m"
  limits:
    memory: "512Mi"
    cpu: "500m"

For the auth server, the CPU numbers come down to one operation. A sign-in costs one bcrypt password check at cost 10, about 35 ms of CPU on a modern core, which no limit slows when it runs alone. Under the 500m limit a pod checks about 14 passwords a second, and a burst queues behind that: measured, 16 sign-ins arriving at once took about 1.2 s to complete, and every other request on that pod waited with them. Cloud vCPUs are often slower than the core measured, and the 100m request is all the scheduler guarantees on a busy node. If your sign-in peak is higher, raise the CPU limit or add replicas.

The admin console checks no passwords and needs less; it carries the same numbers for simplicity.

Next steps

Probes and shutdownHow a pod starts, how long it may take, and how it stops.

DatabasePrepare the database every pod shares.

MonitoringWatch the pool waits and rate-limit refusals.