For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Learn how the AgentgatewayModel API provides a model-centric way to serve LLMs in Kubernetes.

Learn how the AgentgatewayModel API provides a model-centric way to serve LLMs in Kubernetes.

About

Clients talk to an LLM gateway in terms of models. They send a request with "model": "gpt-5-mini" in the body and expect the gateway to figure out the rest.

Historically in Kubernetes, you assembled that experience yourself from a listener, an HTTPRoute that matched on the request body, an AgentgatewayBackend for each provider, and an AgentgatewayPolicy for the AI behavior. Every LLM deployment needed the same scaffolding, and that scaffolding stayed visible in your configuration.

The AgentgatewayModel API removes the scaffolding. Each resource declares one client-facing model and attaches to a parent, such as a Gateway listener. Agentgateway derives the routing, so you configure only what is specific to your setup.

Every model that attaches to the same parent is aggregated into a single model table, called a model router. From that table, agentgateway serves the following behavior.

• Model extraction from the request body.
• The standard LLM API paths, such as /v1/chat/completions.
• Model discovery on /v1/models.
• Per-model provider routing.
• OpenAI-compatible error responses for unknown models.

The parent that you choose decides which model router a model joins, and where that router is served. For more information, see Parent types.

Model-centric vs. route-centric configuration

Both approaches are supported. Use this table to choose.

Parent types

Each AgentgatewayModel lists one or more parents in spec.parentRefs. The parent kind decides which model router the model joins, and which paths serve that router.

A Gateway parent is the default choice. Use an HTTPRoute parent when one listener needs more than one independent set of models, or when a group of models needs its own policies. For more information, see Path-scoped models on an HTTPRoute.

Models on different routers are isolated from each other. A request to one router’s paths can select only the models on that router, and /v1/models on that router lists only those models.

Listener opt-in

A listener serves LLM traffic only when it explicitly allows the route kinds that its models attach through, in allowedRoutes.kinds. Allow the AgentgatewayModel kind for models that attach directly to the listener, and the HTTPRoute kind for models that attach through a route.

The following example allows models to attach directly to the listener.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: agentgateway-proxy
  namespace: agentgateway-system
spec:
  gatewayClassName: agentgateway
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
      # Enables the listener's built-in LLM paths
      kinds:
      - group: agentgateway.dev
        kind: AgentgatewayModel

Adding the route kind turns on the listener’s built-in LLM paths. At first the listener has zero models, so every request returns a model_not_found error. Models become available as they attach.

A listener can allow both AgentgatewayModel and HTTPRoute, so LLM endpoints and ordinary HTTP routes coexist on the same port. When you set allowedRoutes.kinds, the listener accepts only the kinds that you list, so include every kind that you need. Models that attach through an HTTPRoute parent need only the HTTPRoute kind, because the route is what attaches to the listener.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: agentgateway-proxy
  namespace: agentgateway-system
spec:
  gatewayClassName: agentgateway
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
      kinds:
      - group: agentgateway.dev
        kind: AgentgatewayModel
      - group: gateway.networking.k8s.io
        kind: HTTPRoute

Path-scoped models on an HTTPRoute

A Gateway parent gives a listener one model router at the listener root. That setup is enough for a single set of models, but not when one listener must serve several independent sets. To create additional routers on the same listener, declare each one with an HTTPRoute and attach models to the route instead of the Gateway.

Use an HTTPRoute parent for the following cases.

• Serve the LLM paths under a path prefix, such as /tenant-a/v1/chat/completions.
• Give one listener several model tables, so that /v1/models returns a different list per path.
• Apply an AgentgatewayPolicy to a group of models rather than to one model or to the whole listener.

Route requirements

An HTTPRoute is a valid parent for an AgentgatewayModel only when it meets the following requirements. A route that does not opt in with an AgentgatewayModel backend stays an ordinary route, and a route that opts in but breaks one of the other requirements is rejected.

Requirements are checked per model. When a model’s parent reference fails one of them, the model reports Accepted: False with the reason in the condition message. To check, run kubectl get agentgatewaymodel <name> -n <namespace> -o yaml and read status.parents.

Example

The following HTTPRoute declares a model router under /tenant-a, and the model attaches to that route’s models rule. The AgentgatewayPolicy targets the same rule, so it applies to every model on that router and to nothing else on the listener.

The listener that the route attaches to must allow the HTTPRoute kind, as shown in Listener opt-in.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: tenant-a-models
  namespace: agentgateway-system
spec:
  parentRefs:
  - name: agentgateway-proxy
    sectionName: http
  rules:
  # Name the rule so that models and policies can select it
  - name: models
    matches:
    - path:
        type: PathPrefix
        value: /tenant-a
    # The wildcard model backend is what turns this rule into a model router
    backendRefs:
    - group: agentgateway.dev
      kind: AgentgatewayModel
      name: "*"
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayModel
metadata:
  name: gpt-5-mini
  namespace: agentgateway-system
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: tenant-a-models
    sectionName: models
  provider: OpenAI
  policies:
    auth:
      secretRef:
        name: openai-secret
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: tenant-a-auth
  namespace: agentgateway-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: tenant-a-models
    sectionName: models
  traffic:
    basicAuthentication:
      secretRef:
        name: tenant-a-users

With this configuration, clients reach the router under the route’s prefix.

• GET /tenant-a/v1/models lists gpt-5-mini and no other models.
• POST /tenant-a/v1/chat/completions with "model": "gpt-5-mini" serves the request.
• Both paths require the basic authentication credentials that the policy sets.

Agentgateway strips the /tenant-a prefix before it forwards the request, so the provider receives the standard /v1/chat/completions path.

Router scoping constraints

• A virtual model and the concrete models it selects must share one router. A weighted or conditional virtual model resolves its targets by model name inside its own router’s table. Attach the virtual model and its targets to the same parent; otherwise, the request fails with virtual_model_not_resolved.
• A root-path model route cannot share a listener with directly attached models. If a model-serving rule matches /, has no matches at all, or has a match with no path, then no AgentgatewayModel can attach directly to the same listener through a Gateway parent. Both would claim the same paths, so the model on the route is rejected with a conflict message. Give the route a distinct prefix, or move the directly attached models onto routes.
• The rule name is part of the router identity. Renaming a route rule moves its models to a new router. If you omit the rule name, the rule’s index in the list identifies the router instead, so reordering rules has the same effect.

Concrete and virtual models

Every model is an AgentgatewayModel resource. A model either names a particular language model the provider that serves it (“concrete” model), or routes across other models (“virtual” model). The two fields are mutually exclusive.

The following diagram shows a client requesting the balanced virtual model, which splits traffic 80/20 across two internal concrete models. Each concrete model forwards to its provider.

    flowchart LR
  C[Client requests 'balanced'] --> V["Virtual model: balanced<br/>(weighted 80/20)"]
  V -->|80| F["Concrete model: fast<br/>Internal"]
  V -->|20| P["Concrete model: premium<br/>Internal"]
  F --> FP[Provider]
  P --> PP[Provider]
  style V fill:#7734be,color:#fff
  

Concrete models

A model that names its provider is a concrete model. It is an AgentgatewayModel that sets spec.provider, so the gateway knows which provider serves the request.

The following example shows the pieces that a concrete model configures. Only parentRefs and provider are required.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayModel
metadata:
  name: gpt-5-mini
  namespace: agentgateway-system
spec:
  # The parent that serves this model, here a Gateway listener
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
    sectionName: http
  # The model name that clients request; defaults to metadata.name
  match:
    model: gpt-5-mini
  # Whether clients can request this model directly
  visibility: Public
  # The provider that serves this model
  provider: OpenAI
  # Optional: override the provider address
  baseURL: https://api.openai.com
  # Optional: policies that apply to this model only
  policies:
    auth:
      secretRef:
        name: openai-secret

Review the following table to understand this configuration.

Model matching

Use spec.match.model to control which model name in a request selects this resource. When you omit spec.match, the model matches metadata.name exactly.

For example, a model named gpt-5-mini with no spec.match serves requests that send "model": "gpt-5-mini" in the body. To publish a different name, set spec.match.model explicitly.

A match must use one of the following forms. Wildcards in any other position are rejected.

A wildcard model usually pairs with a model transformation, because the provider expects its own model name rather than the client-facing one. For an example, see Serve a model.

Model aliasing

A model resource is an alias by construction. The name that clients request is metadata.name or match.model, and the name that the provider receives comes from a model transformation. To publish fast as an alias for gpt-3.5-turbo, create a model named fast that rewrites the field.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayModel
metadata:
  name: fast
  namespace: agentgateway-system
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
    sectionName: http
  provider: OpenAI
  policies:
    transformations:
    - field: model
      expression: '"gpt-3.5-turbo"'

Each alias is a separate resource, so it can carry its own credentials, authorization rules, and guardrails. For the AgentgatewayBackend equivalent, see Model aliasing.

Visibility

Use spec.visibility to control whether clients can request a model directly.

Set Internal on concrete models that exist only as virtual model targets.

Model policies

Concrete models accept an inline spec.policies block that supports the following policies. Each policy applies only to the model that declares it.

Virtual models cannot set spec.policies, because a virtual model has no provider of its own to authenticate to, transform for, or health check. Configure these policies on the concrete models that the virtual model targets.

Providers

Set spec.provider to one of the supported provider names, such as OpenAI. For a full list of providers, see the model provider API docs.

Some providers require a matching settings field.

• Azure requires spec.azure.
• VertexAI requires spec.vertexai.
• Bedrock requires spec.bedrock.
• Custom requires spec.custom.
• Ollama requires spec.baseURL.

Use spec.custom.backendRef to serve a model from a Kubernetes backend, such as an InferencePool.

Use spec.baseURL to override the provider address and base path prefix. It must be an absolute http or https URL with a host, and it cannot target localhost, loopback, or link-local addresses. Query parameters, fragments, and user info are not supported.

Virtual models

A model that routes across other models is a virtual model. It is an AgentgatewayModel that sets spec.virtualModel instead of spec.provider, so it has no provider of its own. It publishes one client-facing name and selects a concrete model to serve each request.

The following example splits traffic across two concrete models by weight.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayModel
metadata:
  name: balanced
  namespace: agentgateway-system
spec:
  # The Gateway listener that serves this model
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
    sectionName: http
  # Routing strategy across other AgentgatewayModel resources
  virtualModel:
    weighted:
      targets:
      - modelRef:
          name: internal-fast
        weight: 80
      - modelRef:
          name: internal-premium
        weight: 20

Routing strategies

Each virtual model uses exactly one strategy.

For examples of each strategy, see Virtual models.

Constraints

• Targets must be AgentgatewayModel resources in the same namespace. Cross-namespace references are not supported.
• Virtual models must be Public. The restriction stops virtual models from targeting each other, which could otherwise create routing loops.
• Virtual models cannot set spec.policies. Configure policies on the concrete target models instead.

Verify that a model attached

Each AgentgatewayModel reports one entry in status.parents per parent reference, with an Accepted condition for the attachment and a ResolvedRefs condition for the references in the spec, such as virtual model targets and Secrets.

kubectl get agentgatewaymodel gpt-5-mini -n agentgateway-system -o yaml

Accepted: False means the parent did not accept the model, and the condition message states why. For an HTTPRoute parent, the message names the requirement that the route or the rule broke. For more information, see Route requirements.

You can also confirm attachment from the data plane by listing the models on the router’s /v1/models path, or by reading the attachedRoutes count on the Gateway listener.

Known limitations

• The API is experimental and turned off by default.
• An AgentgatewayPolicy cannot target an AgentgatewayModel directly. To scope a policy to a group of models, target the HTTPRoute rule that the models attach to. For more information, see Path-scoped models on an HTTPRoute.