LLM

Route requests to supported LLM providers with the agentgateway binary.

Verified

Configure the agentgateway binary to route chat completion requests to an LLM provider.

Before you begin

Install the agentgateway binary.
Download and install the agentgateway binary. Alternatively, you can manually download the binary from the agentgateway releases page.
To install the latest release:
curl -sL https://agentgateway.dev/install | bash
Example output:
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 8878 100 8878 0 0 68998 0 --:--:-- --:--:-- --:--:-- 69359 Downloading https://github.com/agentgateway/agentgateway/releases/download/v1.5.0/agentgateway-darwin-arm64 Verifying checksum... Done. Preparing to install agentgateway into /usr/local/bin Password: agentgateway installed into /usr/local/bin/agentgateway
To install a specific version, pass the --version flag. Use any release tag from the agentgateway releases page, such as v1.5.0. The version must start with v (the script adds the v if you omit it).
curl -sL https://agentgateway.dev/install | bash -s -- --version v1.5.0
A nightly build has no release tag and is not listed on the releases page. Instead, each nightly build is a run of the nightly GitHub Actions workflow, and you download the binary from that run’s artifacts.
1. Go to the nightly builds in GitHub Actions and click the run that you want to install from.
2. Copy the run ID from the end of that run’s URL, such as 24873456345 in https://github.com/agentgateway/agentgateway/actions/runs/24873456345.
3. Using the gh CLI, download the binary artifact for your OS. The following example uses macOS. For other operating systems, replace release-binary-mac with release-binary-linux, release-binary-linux-arm, or release-binary-windows.
  gh run download 24873456345 -R agentgateway/agentgateway -n release-binary-mac
4. Make the binary file executable and move it to your binary location, such as in the following example.
  chmod +x agentgateway sudo mv agentgateway /usr/local/bin/agentgateway
5. Verify that you have the nightly build. The version string of a nightly build is 0.0.0-alpha.<commit>, not a release tag.
  agentgateway --version
  Example output:
  { "version": "0.0.0-alpha.813d7d0", "git_revision": "813d7d0ab4757db7c8ed5a639bc63c0bb20ac116", "rust_version": "1.95.0", "build_profile": "release", "build_target": "aarch64-apple-darwin" }

Get credentials for the provider that you want to use. The steps below cover API keys, cloud credentials, GitHub Copilot, custom providers, and local Ollama models.

Steps

Route to an LLM provider through agentgateway.

Step 1: Set up provider credentials

Set up credentials for the provider that you want to use. For production credential options, see the provider reference.

Note

Each step in this guide has a tab for every provider that the UI supports. Not all of the tabs fit on the screen at once, so scroll the tab bar to the right to reach the rest. The tabs are in the same order as the Provider dropdown list in the UI.

export OPENAI_API_KEY='<your-api-key>'

Step 2: Start agentgateway

You add the model from the UI in the next steps, so you can start agentgateway without a config file. When you run agentgateway without specifying a config, it bootstraps a basic config at ~/.config/agentgateway/config.yaml and uses it automatically.

agentgateway

Example output:

info  app  serving UI at http://localhost:4000/ui

Step 3: Enable LLM

Open the agentgateway UI.
On the first run, the Welcome to Agentgateway wizard opens. Click Enable LLM, and then click Continue.

The Gateway Overview home page opens, with rows for LLM, MCP, and Traffic.

Step 4: Add a model

In the LLM section of the navigation menu, click Models, and then click Add model. Follow the tab for your provider.

For Incoming model match, enter gpt-3.5-turbo.
From Provider, select OpenAI.
For Provider API key, select Env var and enter OPENAI_API_KEY.
Click Save model.

Step 5: Send a chat completion request

From another terminal, send a request to the chat completions endpoint.

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: OpenAI through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "claude-haiku-4-5",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Anthropic through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gemini-2.5-flash",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Gemini through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gemini-2.5-flash",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Vertex AI through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "amazon.nova-lite-v1:0",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Amazon Bedrock through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-4.1",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Azure through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-4.1",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: GitHub Copilot through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "my-model",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Custom through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "openai/gpt-oss-120b",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Baseten through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-oss-120b",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Cerebras through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "command-a-03-2025",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Cohere through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-32B",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: DeepInfra through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "deepseek-v4-flash",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: DeepSeek through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "accounts/fireworks/models/gpt-oss-120b",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Fireworks AI through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "llama-3.1-8b-instant",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Groq through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-32B",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Hugging Face through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "mistral-small-latest",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Mistral AI through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "llama3.2",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Ollama through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "anthropic/claude-haiku-4.5",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: OpenRouter through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "meta-llama/Llama-3.3-70B-Instruct-Turbo",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: Together AI through agentgateway works"
      }
    ]
  }'

curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "grok-4.3",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: xAI through agentgateway works"
      }
    ]
  }'

You can send the same request from the built-in playground.

Open the LLM playground.
From Model, select gpt-3.5-turbo.
In User message, enter Reply with exactly: OpenAI through agentgateway works.
Click Send.