Azure

Configuration and setup for Azure AI services provider

Configure Azure as an LLM provider in agentgateway.

Azure supports two endpoint types:

Azure OpenAI (OpenAI): Connect to Azure OpenAI Service deployments at {resourceName}.openai.azure.com.
Azure AI Foundry (Foundry): Connect to Azure AI Foundry project endpoints at {resourceName}.services.ai.azure.com.

Before you begin

Install and set up an agentgateway proxy.

Authentication

You can authenticate to Azure with an API key or with implicit Entra ID authentication through DefaultAzureCredential. On Kubernetes, implicit authentication can obtain a token from managed identity or workload identity. It does not require a Kubernetes secret or policies.auth.

You can also configure explicit managed identity or workload identity authentication in the AgentgatewayBackend. Managed identity and workload identity use different credential sources. For either source, the proxy must be able to reach an Azure managed identity endpoint.

Use Microsoft Entra Workload ID on AKS

To authenticate with Microsoft Entra Workload ID, prepare the AKS cluster and the proxy service account before you create the AgentgatewayBackend.

Enable workload identity on the AKS cluster.
Create a user-assigned managed identity. Then, grant that identity the least-privilege role that the Azure AI resource requires. For example, the Azure AI User role grants access to Azure AI Foundry.
Create a federated credential that trusts the service account that the proxy uses. By default, the deployer names this service account after the Gateway. For the default gateway from the quickstart, use the following subject. The subject must match the namespace and the service account name exactly.
```
system:serviceaccount:agentgateway-system:agentgateway-proxy
```

Create an AgentgatewayParameters resource that annotates the service account with the managed identity client ID, and that labels the proxy pod for the Azure workload identity webhook. If the gateway already refers to an AgentgatewayParameters resource, add these overlays to that resource instead.

kubectl apply --server-side -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayParameters
metadata:
  name: azure-workload-identity
  namespace: agentgateway-system
spec:
  serviceAccount:
    metadata:
      annotations:
        azure.workload.identity/client-id: <managed-identity-client-id>
  deployment:
    spec:
      template:
        metadata:
          labels:
            azure.workload.identity/use: "true"
EOF

Attach the AgentgatewayParameters resource to the gateway.

kubectl patch gateway agentgateway-proxy \
  --namespace agentgateway-system \
  --type merge \
  --patch '{"spec":{"infrastructure":{"parametersRef":{"group":"agentgateway.dev","kind":"AgentgatewayParameters","name":"azure-workload-identity"}}}}'

Set up access to Azure

Retrieve the resource name and, if applicable, the project name from the Azure AI Foundry portal or the Azure portal. For example:
- For an Azure OpenAI endpoint like https://{my-resource}.openai.azure.com, the resource name is my-resource.
- For an Azure AI Foundry endpoint like https://{my-resource}.services.ai.azure.com and path /api/projects/{my-project}, the resource name is my-resource and the project name is my-project. If the resource name and the project name are the same, you can leave the projectName field empty.
Store the API key to access your model deployment in an environment variable. If you are using implicit Entra ID authentication (such as managed identity or workload identity), you can skip this step.
```
export AZURE_API_KEY=<insert your model deployment key>
```

Create a Kubernetes secret to store your API key. If you are using implicit Entra ID authentication, skip this step.

kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: azure-secret
  namespace: agentgateway-system
type: Opaque
stringData:
  Authorization: $AZURE_API_KEY
EOF

Create an AgentgatewayBackend resource to configure the Azure LLM provider.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: azure
  namespace: agentgateway-system
spec:
  ai:
    provider:
      azure:
        resourceName: my-resource
        resourceType: OpenAI
        model: gpt-4.1-mini
  policies:
    auth:
      secretRef:
        name: azure-secret
EOF

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: azure
  namespace: agentgateway-system
spec:
  ai:
    provider:
      azure:
        resourceName: my-resource
        resourceType: Foundry
        projectName: my-project
        model: gpt-4.1-mini
  policies:
    auth:
      secretRef:
        name: azure-secret
EOF

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: azure
  namespace: agentgateway-system
spec:
  ai:
    provider:
      azure:
        resourceName: my-resource
        resourceType: OpenAI
        model: gpt-4.1-mini
EOF

To use explicit managed identity or workload identity authentication instead of an API key, apply one of the following AgentgatewayBackend configurations.

To use workload identity, first prepare the AKS cluster as described in Use Microsoft Entra Workload ID on AKS. Then, select workload identity in the AgentgatewayBackend.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: azure
  namespace: agentgateway-system
spec:
  ai:
    provider:
      azure:
        resourceName: my-resource
        resourceType: OpenAI
        model: gpt-4.1-mini
  policies:
    auth:
      azure:
        workloadIdentity: {}
EOF

To use the identity of the Azure resource that runs the proxy, leave managedIdentity empty.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: azure
  namespace: agentgateway-system
spec:
  ai:
    provider:
      azure:
        resourceName: my-resource
        resourceType: OpenAI
        model: gpt-4.1-mini
  policies:
    auth:
      azure:
        managedIdentity: {}
EOF

To use an identity that you manage separately from the Azure resource, set the client ID of that identity. You can set the object ID or the resource ID instead.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: azure
  namespace: agentgateway-system
spec:
  ai:
    provider:
      azure:
        resourceName: my-resource
        resourceType: OpenAI
        model: gpt-4.1-mini
  policies:
    auth:
      azure:
        managedIdentity:
          clientId: <managed-identity-client-id>
EOF

Review the following table to understand this configuration. For more information, see the API reference.

Setting	Description
`ai.provider.azure`	Define the Azure provider.
`azure.resourceName`	The Azure resource name used to construct the endpoint hostname.
`azure.resourceType`	The endpoint type: `OpenAI` for Azure OpenAI Service, or `Foundry` for Azure AI Foundry.
`azure.model`	The model to use for requests, such as `gpt-4.1-mini`.
`azure.projectName`	The Foundry project name. Required when `resourceType` is `Foundry`.
`azure.apiVersion`	Optional API version override. Defaults to `v1`. For legacy deployments, use a dated version like `2025-01-01-preview`.
`policies.auth.azure.workloadIdentity`	Use Azure workload identity. Leave the object empty. The proxy uses the federated token and the Azure environment variables that the Azure workload identity webhook projects into the proxy pod.
`policies.auth.azure.managedIdentity`	Use an Azure managed identity. Leave the object empty to use the system-assigned identity. To use a user-assigned identity, set one of `clientId`, `objectId`, or `resourceId`.

Create an HTTPRoute resource that routes incoming traffic to the AgentgatewayBackend. The following example sets up a route. Note that agentgateway automatically rewrites the endpoint to the appropriate chat completion endpoint of the LLM provider for you, based on the LLM provider that you set up in the AgentgatewayBackend resource.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: azure
  namespace: agentgateway-system
spec:
  parentRefs:
    - name: agentgateway-proxy
      namespace: agentgateway-system
  rules:
  - backendRefs:
    - name: azure
      namespace: agentgateway-system
      group: agentgateway.dev
      kind: AgentgatewayBackend
EOF

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: azure
  namespace: agentgateway-system
spec:
  parentRefs:
    - name: agentgateway-proxy
      namespace: agentgateway-system
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /azure
    backendRefs:
    - name: azure
      namespace: agentgateway-system
      group: agentgateway.dev
      kind: AgentgatewayBackend
EOF

Send a request to the LLM provider API along the route that you previously created. Verify that the request succeeds and that you get back a response from the chat completion API.

Cloud Provider LoadBalancer:

curl "$INGRESS_GW_ADDRESS/v1/chat/completions" -H content-type:application/json  -d '{
   "model": "gpt-4.1-mini",
   "messages": [
     {
       "role": "system",
       "content": "You are a helpful assistant."
     },
     {
       "role": "user",
       "content": "Write a short haiku about cloud computing."
     }
   ]
 }' | jq

Localhost:

curl "localhost:8080/v1/chat/completions" -H content-type:application/json  -d '{
   "model": "gpt-4.1-mini",
   "messages": [
     {
       "role": "system",
       "content": "You are a helpful assistant."
     },
     {
       "role": "user",
       "content": "Write a short haiku about cloud computing."
     }
   ]
 }' | jq

Cloud Provider LoadBalancer:

curl "$INGRESS_GW_ADDRESS/azure" -H content-type:application/json  -d '{
   "model": "gpt-4.1-mini",
   "messages": [
     {
       "role": "system",
       "content": "You are a helpful assistant."
     },
     {
       "role": "user",
       "content": "Write a short haiku about cloud computing."
     }
   ]
 }' | jq

Localhost:

curl "localhost:8080/azure" -H content-type:application/json  -d '{
   "model": "gpt-4.1-mini",
   "messages": [
     {
       "role": "system",
       "content": "You are a helpful assistant."
     },
     {
       "role": "user",
       "content": "Write a short haiku about cloud computing."
     }
   ]
 }' | jq

Example output:

{
  "id": "chatcmpl-9A8B7C6D5E4F3G2H1",
  "object": "chat.completion",
  "created": 1727967462,
  "model": "gpt-4.1-mini",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Floating servers bright,\nData streams through endless sky,\nClouds hold all we need."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 28,
    "completion_tokens": 19,
    "total_tokens": 47
  }
}

Use Claude models on Azure AI Foundry

Azure AI Foundry hosts Anthropic Claude models at native Anthropic endpoints. When you set resourceType: Foundry and a model name that starts with claude-, agentgateway automatically routes requests to the Anthropic-native path (/anthropic/v1/messages) instead of the OpenAI-compatible path, and injects the required anthropic-version header. No extra configuration is needed beyond specifying a Claude model name.

Note

For more information about Claude models on Azure AI Foundry, see the Microsoft documentation.

Create a Kubernetes secret to store your Azure AI Foundry API key.

export AZURE_API_KEY=<insert your Azure AI Foundry API key>

kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: azure-claude-secret
  namespace: agentgateway-system
type: Opaque
stringData:
  Authorization: $AZURE_API_KEY
EOF

Create an AgentgatewayBackend resource that uses the azure provider with resourceType: Foundry and a Claude model name.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: azure-claude
  namespace: agentgateway-system
spec:
  ai:
    provider:
      azure:
        resourceName: my-foundry-resource
        resourceType: Foundry
        projectName: my-project
        model: claude-3-5-haiku-20241022
  policies:
    auth:
      secretRef:
        name: azure-claude-secret
EOF

Review the following table to understand this configuration.

Setting	Description
`azure.resourceName`	The Azure AI Foundry resource name used to construct the endpoint hostname.
`azure.resourceType`	Set to `Foundry` to use Azure AI Foundry endpoints.
`azure.projectName`	The Foundry project name.
`azure.model`	The Claude model to use, for example `claude-3-5-haiku-20241022`. The model name must start with `claude-` to trigger routing to the Anthropic-native endpoint.
`policies.auth.secretRef`	References the secret that holds the Azure AI Foundry API key. The key is automatically sent in the `Authorization` header.

Create an HTTPRoute resource that routes incoming traffic to the AgentgatewayBackend.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: azure-claude
  namespace: agentgateway-system
spec:
  parentRefs:
    - name: agentgateway-proxy
      namespace: agentgateway-system
  rules:
  - backendRefs:
    - name: azure-claude
      namespace: agentgateway-system
      group: agentgateway.dev
      kind: AgentgatewayBackend
    matches:
       - path:
           type: PathPrefix
           value: /azure-claude #Path  example
EOF

Send a request to verify the setup.

Cloud Provider LoadBalancer:

curl "$INGRESS_GW_ADDRESS/azure-claude" -H content-type:application/json -d '{
  "max_tokens": 256,
  "messages": [
    {
      "role": "user",
      "content": "Hello!"
    }
  ]
}' | jq

Localhost:

curl "localhost:8080/azure-claude" -H content-type:application/json -d '{
  "max_tokens": 256,
  "messages": [
    {
      "role": "user",
      "content": "Hello!"
    }
  ]
}' | jq

Next steps

Multiple endpointsSet up other API endpoints such as embeddings or models. Prompt guardsSet up prompt guards for your LLM traffic. LLM observabilityView metrics and logs for LLM traffic.

Azure

Before you begin

Authentication

Use Microsoft Entra Workload ID on AKS

Set up access to Azure

Use Claude models on Azure AI Foundry

Next steps

What could be improved?