Provider Routing

Never worry about provider outages again. The AI Gateway automatically routes your requests to the best available provider, with instant failover when things go wrong.

The Problem

The Solution

Provider routing gives you access to models across multiple providers through a single OpenAI-compatible API. When you hit rate limits or a provider has issues, the gateway automatically retries with exponential backoff and can fall back to alternative models you configure. All without setup or code changes.

Using Provider Routing

Zero configuration required for OpenAI models. Just request a model:

const response = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Hello!" }]
});

That's it. The gateway automatically:

• Routes to the correct provider based on the model prefix
• Retries transient failures with exponential backoff
• Falls back to alternative models if you configure a fallback chain

Your request succeeds even when providers fail.

How It Works

The gateway routes requests based on model name prefixes. Models without a prefix are routed to OpenAI. Models with a provider prefix (e.g., groq/llama-3.1-70b) are routed to the corresponding provider.

Routing Priority:

1. Your provider keys (BYOK) if configured
2. CoreValue's managed keys (credits) - automatic fallback at 0% markup

Retries: The gateway automatically retries transient failures (429, 500, 502, 503, 504, 529) with exponential backoff and jitter. Retry behavior is configured via environment variables on the gateway, not per-request headers.

Failover: Use fallback chains (Cova-Fallbacks header) to fail over to different models when the primary model returns a retryable error.

Advanced: Customizing Routing

The default routing handles most use cases. Customize only if you need specific control:

Route to a Specific Provider

Force requests to a specific provider by prefixing the model name with the provider prefix:

model: "groq/llama-3.1-70b"  // Route through Groq

When to use: You want to use a specific provider for cost, latency, or feature reasons.

What happens: The gateway strips the prefix and forwards the request to that provider's base URL. The provider is determined by the prefix — see the Provider Prefixes table below.

Bring Your Own Keys (BYOK)

Add your provider API keys in Provider Settings:

What happens: Your keys are always tried first, then CoreValue's managed keys as fallback. This gives you control over provider accounts while maintaining reliability.

Benefits: Use provider credits, meet compliance requirements, or maintain direct provider relationships while still getting automatic failover.

Retry Triggers

The gateway automatically retries when encountering these transient errors:

Retries use exponential backoff with jitter and respect the provider's Retry-After header. The X-Cova-Retry-Count response header indicates how many retries were attempted.

Provider Prefixes

The gateway recognizes 18 model prefixes. When a request model starts with one of these prefixes, the prefix is stripped and the request is forwarded to the corresponding provider's base URL.

Fallback Chains

In addition to provider routing (fail over across providers for the same model), you can configure fallback chains to fail over to different models using the Cova-Fallbacks header.

Real World Examples

Scenario: Provider Outage

Your production app uses a Groq model. Groq goes down at 3am.

// Your code doesn't change — the gateway retries automatically
const response = await client.chat.completions.create({
  model: "groq/llama-3.1-70b",
  messages: [{ role: "user", content: "Process this customer request" }]
});

What happens: The gateway retries with exponential backoff. If you configured a fallback chain, it falls back to the next model. Your app stays online, customers never notice.

Scenario: Using Fallback Chains

You want to try a fast Groq model first, then fall back to GPT-4o if Groq fails:

import { buildHeaders } from "@cova/gateway";

const response = await client.chat.completions.create({
  model: "groq/llama-3.1-70b",
  messages: messages,
  extraHeaders: buildHeaders({
    fallbacks: ["gpt-4o", "anthropic/claude-sonnet-4"],
  }),
});

What happens: Tries groq/llama-3.1-70b first, then falls back to gpt-4o, then anthropic/claude-sonnet-4 if each preceding model returns a retryable error.

Scenario: Using BYOK with Credits Fallback

You have your own OpenAI API key but want credits as a safety net.

// Just request the model — BYOK is tried first, then PTB credits
const response = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Analyze this data" }]
});

What happens: The gateway tries your BYOK key first. If your key fails or hits rate limits, it falls back to CoreValue-managed credits (PTB) at 0% markup.

Next Steps