Token Limit Exception Handlers

Automatically handle requests that exceed a model's context window using truncate, middle-out, or fallback strategies.

Legacy / Unsupported. The Cova-Token-Limit-Exception-Handler and Cova-Model-Override request headers were supported by the legacy Cloudflare Worker proxy. They are not read by the current AI Gateway (gateway/src/) and sending them has no effect. This page is retained for historical reference only. See the Header Directory for the list of unsupported headers.

When prompts get large, requests can exceed the model's maximum context window. CoreValue can automatically apply strategies to keep your request within limits or switch to a fallback model — without changing your app code.

What This Does

Estimates tokens for your request based on model and content
Accounts for reserved output tokens (e.g., max_tokens, max_output_tokens)
Applies a chosen strategy only when the estimated input exceeds the allowed context

CoreValue uses provider-aware heuristics to estimate tokens and a best-effort approach across different request shapes.

Strategies

Truncate (truncate): Normalize and trim message content to reduce token count.
Middle-out (middle-out): Preserve the beginning and end of messages while trimming middle content to fit the limit.
Fallback (fallback): Switch to an alternate model when the request is too large. Provide multiple candidates in the request body model field as a comma-separated list (first is primary, second is fallback).

For fallback, CoreValue picks the second candidate if needed. When under the limit, CoreValue normalizes the model to the primary. If your body lacks model, set Cova-Model-Override.

Quick Start

Add the Cova-Token-Limit-Exception-Handler header to enable a strategy.

import { OpenAI } from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.corevalue.dev/v1",
  apiKey: process.env.COVA_API_KEY,
});

// Middle-out strategy
await client.chat.completions.create(
  {
    model: "gpt-4o", // or "gpt-4o, gpt-4o-mini" for fallback
    messages: [
      { role: "user", content: "A very long prompt ..." }
    ],
    max_tokens: 256
  },
  {
    headers: {
      "Cova-Token-Limit-Exception-Handler": "middle-out"
    }
  }
);

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://gateway.corevalue.dev/v1",
    api_key=os.getenv("COVA_API_KEY"),
)

# Fallback strategy with model candidates
resp = client.chat.completions.create(
    model="gpt-4o, gpt-4o-mini",
    messages=[{"role": "user", "content": "A very long prompt ..."}],
    max_tokens=256,
    extra_headers={
        "Cova-Token-Limit-Exception-Handler": "fallback",
    }
)

curl --request POST \
     --url https://gateway.corevalue.dev/v1/chat/completions \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $COVA_API_KEY" \
     -H "Cova-Token-Limit-Exception-Handler: truncate" \
     --data '{
       "model": "gpt-4o",
       "messages": [{"role": "user", "content": "A very long prompt ..."}],
       "max_tokens": 256
     }'

Configuration

Enable and control via headers:

Cova-Token-Limit-Exception-Handlerstringrequired

One of: truncate, middle-out, fallback.

Cova-Model-Overridestring

Optional. Used for token estimation and model selection when the request body doesn't include a model or you need to override it.

Fallback Model Selection

Provide candidates in the body: model: "primary, fallback"
CoreValue chooses the fallback when input exceeds the allowed context
When under the limit, CoreValue normalizes the model to the primary

Notes

Token estimation is heuristic and provider-aware; behavior is best-effort across request shapes.
Allowed context accounts for requested completion tokens (e.g., max_tokens).
Changes are applied before the provider call; your logged request reflects the applied strategy.