Metrics are not scraped

This page helps you when your scraper doesn’t collect Goiabada’s metrics.

What you see

One of these:

• Goiabada isn’t among the scraper’s targets, and no goiabada_ series exist.
• The targets are there, but down: Prometheus’s up is 0 for them, with a connection refused, a timeout or a 404 as the error.

Why it happens

Each server serves its metrics on a listener of its own, GET /metrics on port 9190 for the auth server and 9191 for the admin console, and the listener is off until you turn it on. Most failures are one of these:

• The listener is off. Every start writes a metrics listener configuration record with enabled, and only an enabled listener writes starting the metrics listener. The Docker Compose files and the native env file the setup wizard writes leave both off, and so does a Kubernetes manifest generated without metrics.
• The scraper calls the wrong port or path. Any path but /metrics answers 404, and the application’s own port, 9090 or 9091, serves no metrics.
• The listener only listens on 127.0.0.1, because GOIABADA_AUTHSERVER_LISTEN_HOST_METRICS or GOIABADA_ADMINCONSOLE_LISTEN_HOST_METRICS says so, and the scraper runs elsewhere.
• The scraper doesn’t read what the manifest gives it. kube-prometheus-stack ignores the prometheus.io/scrape pod annotations and scrapes only what a PodMonitor or ServiceMonitor names. The prometheus-community prometheus chart reads the annotations.
• No Prometheus selects the PodMonitor. A Prometheus the Prometheus Operator runs scrapes only the PodMonitors its podMonitorSelector matches, in the namespaces its podMonitorNamespaceSelector matches. kube-prometheus-stack, by default, only those labeled release: <its release name>.
• A NetworkPolicy refuses the scraper. With the generated NetworkPolicies, only the namespace you named for the scraper, monitoring unless you named another, reaches the metrics ports. A scraper in a service mesh that delivers on a port of its own, such as Istio’s ambient mode or Linkerd, arrives on that port instead, which they don’t admit: see Who can reach Goiabada.
• Istio’s sidecars moved the annotations to their own port, 15020, which the generated NetworkPolicies refuse: see Pod annotations.

Fix it

1. Check the listener answers, from beside the server. On Kubernetes:

   kubectl port-forward -n goiabada deploy/goiabada-authserver 9190:9190
   curl -s http://localhost:9190/metrics | grep goiabada_build_info

   With Docker Compose, from a container on the Compose network, curl -s http://goiabada-authserver:9190/metrics.

   If nothing answers, turn the listener on. On Kubernetes, write the manifest again with metrics on, and apply it. With Docker Compose or native binaries, set GOIABADA_AUTHSERVER_METRICS_ENABLED and GOIABADA_ADMINCONSOLE_METRICS_ENABLED to true, in the override or goiabada.env, and restart both servers.

2. If the listener answers but the scraper can’t reach it, check the listen host. Leave GOIABADA_AUTHSERVER_LISTEN_HOST_METRICS and GOIABADA_ADMINCONSOLE_LISTEN_HOST_METRICS unset, which listens on every address, unless the scraper runs on the same host.

3. On Kubernetes, match the manifest to what scrapes your cluster. For kube-prometheus-stack, or any Prometheus the Operator runs, use a PodMonitor and give it the labels your Prometheus selects by:

   kubectl get prometheus -A -o jsonpath='{..podMonitorSelector}'

   Write the manifest again with them, --podmonitor-labels, and apply it. For the prometheus-community chart, use the pod annotations. For Grafana Alloy or the OpenTelemetry Collector, keep the targets on the metrics container port and let the scraper’s service account list and watch pods in the goiabada namespace.

4. With the NetworkPolicies on, check the namespace your scraper’s pods actually run in:

   kubectl get pods -A | grep -i -e prometheus -e alloy -e collector

   If it isn’t the one the policies admit, write the manifest again with --metrics-namespace set to it, and apply it.

Caution

Don’t publish the metrics ports to fix a scrape. The listener has no authentication, so keep it off every Service, route, proxy and Compose ports: entry, and let only your scraper reach it.

The metrics listener

The metrics listener answers only GET /metrics, and isn’t part of the application’s routes. A scrape is neither logged nor counted in the request metrics, so the server’s own logs don’t show your scraper’s attempts. A port it can’t bind stops the server at start, as the other listeners’ do.

Alert on the scrape itself failing, Prometheus’s up at 0 for Goiabada’s targets, so a broken scrape doesn’t go unnoticed.

Next steps

MonitoringTurning the metrics on, scraping them and what to alert on.

Scrape on KubernetesWhat the setup wizard writes for each answer.