Keep AI working when a provider fails
Retries, fallback models, fallback to another provider, and downgrading instead of refusing: what CoreValue does on its own and what you set up.
Providers have outages, rate limits and bad minutes. This guide covers what the gateway does about them automatically, and the three things you can add so a failure does not reach your users.
What happens automatically: retries
Every request that fails with status 429, 500, 502, 503, 504 or 529,
or with a network error, is retried up to three times. The wait between tries
grows from half a second to at most eight seconds, and a provider's
Retry-After is honoured up to that limit. The gateway stops retrying a request
after 60 seconds of waiting.
Nothing to set up. A response that needed retries carries X-Cova-Retry-Count.
1. Fall back to another model
If a model still fails after its retries, the gateway can try the next model in a list. Send the list with each request:
client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}],
extra_headers={"Cova-Fallbacks": '["gpt-4o-mini"]'},
)The next model is tried when the current one fails with 401, 402, 403,
429, 500, 502, 503, 504 or 529. A list holds up to five models. The
Cova-Model response header tells you which model answered.
To set this once instead of in code, create a chain on Gateway → Routing
(Fallback Chains). A chain starts with the model it protects and lists what
to try after it; for each step, On Codes says which statuses move on to the
next (empty means any error). A Cova-Fallbacks header on a request takes
precedence over a saved chain.
2. Fall back to another provider
A fallback model goes to the same provider unless you name the provider. To
cross providers, use the base URL https://gateway.corevalue.dev/gateway/v1
and prefixed model names, with a provider key added for each provider:
client = OpenAI(
base_url="https://gateway.corevalue.dev/gateway/v1",
api_key=os.environ["COVA_API_KEY"],
)
client.chat.completions.create(
model="groq/llama-3.3-70b-versatile",
messages=[{"role": "user", "content": "Hello!"}],
extra_headers={
"Cova-Fallbacks": '["anthropic/claude-sonnet-4-5-20250929", '
'"together/meta-llama/Llama-3.3-70B-Instruct-Turbo"]'
},
)If Groq is down, the request is answered by Anthropic, then Together, in the OpenAI format your code expects. See Other providers for every prefix.
3. Downgrade instead of refusing
A budget that runs out does not have to stop your product. Set an organisation
or team budget's action to Downgrade model and add a mapping for each model
you use, for example gpt-4o → gpt-4o-mini. Over budget, requests are
rewritten to the cheaper model and served; Cova-Model shows the model used.
A request for a model with no mapping is refused, so map every model. See Budget configuration.
Repeated requests: caching
For requests that repeat exactly, such as the same prompt sent many times,
Cova-Cache-Enabled: true serves later copies from CoreValue's cache without
calling the provider. A cache hit carries Cova-Cache: HIT. It does not help
with new requests during an outage. See Caching.
Not available yet
Adding a second key for the same provider does not spread traffic across keys or move to the other key when one is rate-limited; the gateway uses the first key. Use a fallback model on another provider instead.