For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Learn how agentgateway exposes control plane and data plane metrics and how to set up monitoring with Prometheus and Grafana.

Metrics are numerical measurements collected at regular intervals that describe the state and behavior of a system. In a gateway environment, metrics tell you how many requests are being processed, how long they take, how many fail, how much memory the process uses, and whether the control plane is keeping up with configuration changes. Without metrics, you rely on guesswork to detect problems, diagnose slowdowns, or prove that an issue is resolved.

Agentgateway emits metrics automatically on two separate ports, one for the control plane and one for the data plane. No additional configuration is required to enable them. You can access either endpoint immediately after installation by port-forwarding to the respective pod and sending a request to the /metrics path.

Access metrics

You can inspect raw metrics directly from either endpoint without installing any monitoring tools.

1. View control plane metrics.

   1. Port-forward the control plane deployment.

      kubectl port-forward -n agentgateway-system deployment/agentgateway 9092:9092

   2. Query the metrics endpoint.

      curl http://localhost:9092/metrics

2. View data plane metrics.

   1. Port-forward the proxy deployment.

      kubectl port-forward -n agentgateway-system deploy/agentgateway-proxy 15020:15020
   2. Query the metrics endpoint.

      curl http://localhost:15020/metrics

Both endpoints return metrics in the OpenMetrics format. All agentgateway metrics use the agentgateway_ prefix. For the full list of available metrics, see Control plane metrics and Data plane metrics.

Scrape metrics for querying and visualization

Curling the metrics endpoint gives you a point-in-time snapshot, but it does not let you query metrics across time, set alerts, or see trends. To enable monitoring and alerting on metrics, you typically have an observability tool that scrapes the metrics endpoint on a regular interval and stores the data in a time series database that you can query and graph.

Agentgateway integrates well with the following tools:

• Prometheus: An open-source monitoring system that can scrape the /metrics endpoints on a regular interval. Metrics are stored as a time series, and you can use PromQL to query them. To scrape metrics from pods with Prometheus, you either add Prometheus-specific scraping annotations to the pod, or use the built-in PodMonitor and ServiceMonitor resources that the agentgateway Helm chart creates for you when you set monitoring.enabled to true.
• Grafana: An open-source visualization platform that connects to Prometheus as a data source and renders dashboards, graphs, and alerts from PromQL queries. Agentgateway ships a pre-built dashboard that covers requests, LLM traffic, MCP traffic, and connections.

The agentgateway Helm chart includes a monitoring section that creates the ServiceMonitor, PodMonitor, and Grafana dashboard ConfigMap for you so that you can use Prometheus and Grafana without any additional configuration to monitor the agentgateway control and data plane. These resources are discovered automatically when you use the OTel stack, which configures Prometheus and Grafana to pick them up with no extra setup.

Enable control and data plane scraping

Follow the OTel stack guide to install the observability tools, including Prometheus and Grafana, and set monitoring.enabled=true to create the ServiceMonitor, PodMonitor, and Grafana dashboard ConfigMap.

Scrape additional proxy pods

The agentgateway Helm chart creates a single PodMonitor resource in the release namespace when monitoring.enabled is set to true. By default, the PodMonitor resource only finds proxy pods that are deployed to the same release namespace. Because gateway proxies run in the namespace of the Gateway resource that provisions them, proxies in other namespaces are not scraped automatically. To scape proxy pods in other namespaces, you must use the namespaceSelector field.

Scrape proxies in all namespaces:

To scrape proxy pods in every namespace, set monitoring.proxy.namespaceSelector to any: true as shown in the following example.

cat <<EOF > monitoring-values.yaml
monitoring:
  enabled: true
  serviceMonitor:
    extraLabels:
      release: kube-prometheus-stack
  proxy:
    namespaceSelector:
      any: true
EOF

Limit the number of proxy namespaces:

To scrape proxy pods in specific namespaces only, use a matchNames list instead as shown in the following example.

cat <<EOF > monitoring-values.yaml
monitoring:
  enabled: true
  serviceMonitor:
    extraLabels:
      release: kube-prometheus-stack
  proxy:
    namespaceSelector:
      matchNames:
      - agentgateway-system
      - my-other-namespace
EOF

Scrape proxies for specific GatewayClasses:

By default, the PodMonitor selects proxy pods that belong to the agentgateway GatewayClass. If you use multiple GatewayClasses and want to scrape proxy pods for all of them, add each class name to monitoring.proxy.gatewayClassNames.

cat <<EOF > monitoring-values.yaml
monitoring:
  enabled: true
  serviceMonitor:
    extraLabels:
      release: kube-prometheus-stack
  proxy:
    gatewayClassNames:
    - agentgateway
    - my-other-gatewayclass
EOF

Grafana dashboard discovery across namespaces

When you install the kube-prometheus-stack Helm chart, Grafana is deployed as a pod with three containers: grafana, grafana-sc-datasources, and grafana-sc-dashboard. The grafana-sc-dashboard container is a sidecar from the kiwigrid/k8s-sidecar image that is bundled with the Grafana Helm chart. It watches the Kubernetes API for ConfigMaps that carry the grafana_dashboard: "1" label and automatically copies their JSON content into the Grafana pod’s dashboard directory, where Grafana’s file-based provisioner picks it up. No separate installation or Grafana restart is needed to see these dashboards.

By default, the sidecar only watches the namespace where Grafana is installed. If the ConfigMap for the agentgateway dashboard is in a different namespace, the sidecar does not discover it and the dashboard does not appear in Grafana.

To allow the sidecar to find ConfigMaps in all namespaces, set sidecar.dashboards.searchNamespace to ALL when installing or upgrading the kube-prometheus-stack chart.

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring \
  --reuse-values \
  --set grafana.sidecar.dashboards.searchNamespace=ALL

To restrict discovery to specific namespaces, pass a comma-separated list of namespace names instead of ALL:

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring \
  --reuse-values \
  --set grafana.sidecar.dashboards.searchNamespace="monitoring,agentgateway-system"

Learn more