Security

This page helps you decide who in your cluster can reach Goiabada’s pods, and what those pods may do.

Lock it down

Restrict who reaches the pods with the NetworkPolicies, by answering yes when the setup wizard asks, or passing --network-policy. Check the two things in Who can reach Goiabada first.
Enforce the restricted Pod Security Standard, if you own the namespace outright and everything else running there meets it, cert-manager’s HTTP-01 solver pods included:
Terminal window
```
kubectl label namespace goiabada pod-security.kubernetes.io/enforce=restricted
```
Decide whether the hop from the admin console to the auth server needs encrypting, as Encrypt the hop to the auth server explains.
Limit who can read the Secrets, and the AES key above all: see Who can read the AES key.

Pod security

Every generated container runs with this securityContext, which is everything the restricted Pod Security Standard requires, and a read-only root file system besides:

securityContext:
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault

It sets no user or group ID, so each container runs as the user the images run as, uid and gid 10001. On OpenShift, the default restricted-v2 security context constraint (SCC), its admission policy for pods, admits the pods and assigns them IDs from the ranges OpenShift allocates to the namespace instead: a user ID for each container, and an fsGroup for the pod.

No writable volume is mounted, because neither server writes to its file system here.

A certificate or customization you mount from a Secret or ConfigMap is readable at the default mode, 0644. With a stricter defaultMode, such as 0440, the server reads the files through their group: add fsGroup: 10001 to the pod’s securityContext, in a kustomization over the manifest, and Kubernetes gives the mounted files that group. On OpenShift, leave fsGroup out: restricted-v2 assigns one, and refuses 10001.

Both pod specs also set:

serviceAccountName, an account of the server’s own, goiabada-authserver or goiabada-adminconsole, which the manifest declares, rather than the namespace’s default. Policies such as Kyverno’s that refuse pods running as default admit them, and a cloud workload identity or a policy can tell the two servers apart. No role is bound to either account, and image pull secrets the default account names don’t reach these pods.
automountServiceAccountToken: false, on the pods and on both accounts, because neither server calls the Kubernetes API, so a compromised pod gets no API credential.
enableServiceLinks: false, because Kubernetes would otherwise give each container variables for every Service in the namespace, and two of the generated Services are named so that those begin with GOIABADA_, such as GOIABADA_AUTHSERVER_PORT and GOIABADA_ADMINCONSOLE_SERVICE_HOST, which neither server reads.

The generated Namespace is labelled pod-security.kubernetes.io/warn: restricted and pod-security.kubernetes.io/audit: restricted. For any pod in it that breaks the standard, Goiabada’s own included, kubectl apply prints a warning and the API server writes an audit entry; nothing is refused. It doesn’t enforce, because the wizard accepts an existing namespace, whose other workloads’ pods would then be refused the next time they’re created. When you answer that the namespace already exists, neither file sets a label on it, and the manifest’s header has the command its owner can run to add these two.

Who can reach Goiabada

Without a NetworkPolicy, any pod in the cluster can call the goiabada-authserver and goiabada-adminconsole Services directly, around Envoy or whatever routes to them. Besides reaching the admin console from inside the cluster, such a pod chooses the address it’s rate limited and audited under, by sending its own X-Forwarded-For (see Client IP and proxy trust). The wizard asks whether to close that; the answer is no unless you say yes, or pass --network-policy.

On yes, the manifest carries one ingress-only NetworkPolicy per Deployment:

Pods	Admitted on	From
`goiabada-authserver`	TCP 9090	Your routing layer’s namespace, `envoy-gateway-system` for Envoy Gateway; the admin console’s pods
`goiabada-adminconsole`	TCP 9091	Your routing layer’s namespace, `envoy-gateway-system` for Envoy Gateway
`goiabada-authserver`	TCP 9190, with metrics exposed	The metrics scraper’s namespace, `monitoring` unless you name another
`goiabada-adminconsole`	TCP 9191, with metrics exposed	The metrics scraper’s namespace, `monitoring` unless you name another

The metrics ports are a second rule of each policy, so the scraper’s namespace reaches the metrics port and nothing else, and the routing layer and the admin console don’t reach it at all, unless they reach it through a service mesh, as below (see Scrape on Kubernetes).

The routing layer’s namespace is selected by its kubernetes.io/metadata.name label, which Kubernetes sets on every namespace. On yes, the wizard asks which namespace that is, or takes it from --proxy-namespace. Under the Envoy Gateway recipe the default is envoy-gateway-system, where Envoy Gateway runs its proxies; under any other routing there’s no default. The policies don’t affect the kubelet’s probes, which come from the pod’s own node (except under Istio’s ambient mode, as below), or cert-manager’s HTTP-01 solver pods, which carry other labels.

Under a service mesh, a pod’s proxy may receive meshed traffic on a port of its own instead of the server’s: 15008 under Istio’s ambient mode, 4143 under Linkerd. A policy admitting the server’s port alone then refuses the admin console’s calls to the auth server, and the routing layer’s too once its proxies join the mesh. The wizard asks those ports after the namespace, with --mesh-ports answering without the prompt, and each policy’s first rule admits them from the same sources. Between Istio’s sidecars, each server’s port is kept and none is needed, but a sidecar that ambient pods call needs 15008.

A mesh port carries a meshed caller to every port of the pod, so the policies then separate ports only for traffic outside the mesh: the admin console and a meshed routing layer reach the metrics ports too, and the mesh’s own authorization policies decide what they may call. A scraper that joins such a mesh arrives on the mesh port as well, which the scraper’s rule doesn’t admit, so its targets go down. Keep the scraper out of the mesh, or add the mesh port to the second rule of both policies. Under Istio’s ambient mode, the kubelet’s probes arrive from 169.254.7.127, which some network plugins check against the policies: Istio’s page on NetworkPolicy has the rule to add.

Before you answer yes, check two things:

Your network plugin enforces NetworkPolicy. Calico and Cilium do. Some clusters’ default plugin doesn’t, or does only once enforcement is turned on in the cluster’s settings; there the policies are accepted and do nothing.
Nothing else in the cluster calls the auth server through its Service. A resource server that fetches /certs or calls /userinfo at http://goiabada-authserver.goiabada:9090 is blocked once the policy applies: its connection times out or is refused, depending on the plugin. Admit its namespace by adding a peer to the first rule of the auth server’s policy, as its comment shows:
```
ingress:
- from:
  - namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: "envoy-gateway-system"
  - podSelector:
      matchLabels:
        app: goiabada-adminconsole
  - namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: my-resource-server
```
A workload that calls the auth server’s public URL goes through the routing layer and needs nothing.

The policies state no egress rules, so they restrict nothing the servers connect to. An egress rule would have to name every destination, and two can’t be named: the SMTP host and port are settings in the database, changed in the admin console, and the database host is often a DNS name, while a NetworkPolicy selects only pods, namespaces and IP blocks. A rule written for today’s addresses would cut email or the database when either moved.

To see what the policies admit:

kubectl describe networkpolicy -n goiabada

Encrypt the hop to the auth server

The admin console calls the auth server at GOIABADA_AUTHSERVER_INTERNALBASEURL, which the wizard sets to http://goiabada-authserver:9090: plain HTTP to the Service, inside the cluster. That hop carries the admin console’s client secret, its refresh token and administrators’ tokens. The gateway’s hop to each pod is plain HTTP too, and carries what users send, passwords included. Both are sound on a pod network only your cluster uses. The comment the manifest carries above the variable says so.

When the pod network isn’t one you trust, such as a cluster shared with workloads you don’t control, or nodes that talk across a network others can read, encrypt the hops. Any of these works, and none needs a change to Goiabada:

A network plugin that encrypts pod traffic, such as Calico’s or Cilium’s WireGuard mode. It encrypts every hop between nodes, the gateway’s included, and the manifest stays as it is.
A service mesh with mutual TLS between the pods, such as Istio or Linkerd. It covers the gateway’s hop too once the gateway’s proxies join the mesh. With the NetworkPolicies on, name the ports it delivers on (see Who can reach Goiabada). Check that its sidecars leave X-Forwarded-For as Envoy wrote it, or Goiabada reads the wrong client address: see A second proxy hop.
TLS at the auth server, for the admin console’s hop.
A BackendTLSPolicy, for the gateway’s hop to the auth server, once the auth server serves HTTPS.

If the admin console then can’t reach the auth server, see Can’t reach the auth server.

Check the database’s certificate

The wizard asks how the auth server should protect its connection to the database, and writes your answer into the goiabada-authserver-config ConfigMap as GOIABADA_DB_TLS_MODE. Where the database sits explains each mode.

For verify-ca and verify-full, the wizard also asks for a CA file: the authorities your database’s certificate chains to. Leave it empty to trust the system’s authorities, and the manifest gains nothing more.

With a CA file, the manifest carries its certificates. They’re not secret:

A ConfigMap, goiabada-db-ca, holds them as PEM under the key ca.pem. If the file you named also held a private key, the wizard leaves it out.
A read-only volume mounts that ConfigMap at /etc/goiabada/db-ca in the auth server’s pods.
GOIABADA_DB_TLS_CA_FILE in goiabada-authserver-config names the mounted file, /etc/goiabada/db-ca/ca.pem. The auth server then trusts those authorities in place of the system’s.

If a pod can’t verify the database’s certificate, it stops at start and crash-loops: see Database TLS connection fails.

Change the authorities

Write the manifest again with the new file, and apply it, or the kustomization over it with kubectl apply -k .. Applying rolls the auth server’s pods onto the new authorities:

./goiabada-setup-linux-amd64 --answers=goiabada-setup.yaml --keep-secrets --db-tls-ca-file=new-ca.pem
kubectl apply -f goiabada-k8s.yaml

Moving the database to a certificate from another authority? Put both authorities in the file, and apply it before the database switches. Once it has, do the same with the new authority alone.

Next steps

SecretsWho can read the Secrets, and other ways to create them.

Client IP and proxy trustHow Goiabada reads the address Envoy forwards.

Production checklistWhat to check before going live.