DocumentationGuides
AI GatewayConcepts

Prompt Caching

Cache frequently-used context across LLM providers for reduced costs and faster responses

Prompt caching allows you to cache frequently-used context (system prompts, examples, documents) and reuse it across multiple requests at significantly reduced costs.

Why Prompt Caching

Reduce Token Costs

Cached prompts are processed at significantly reduced rates by providers (up to 90% savings)

Faster Processing

Providers skip re-processing cached prompt segments for faster response times

Automatic Optimization

Works out-of-the-box with OpenAI compatible AI Gateway across all providers

OpenAI and Compatible Providers

Automatic caching for prompts over 1024 tokens. Use the prompt_cache_key parameter for better cache hit control.

Compatible providers: OpenAI, Grok, Groq, Deepseek, Moonshot AI

Quick Start

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.corevalue.dev/v1",
  apiKey: process.env.COVA_API_KEY,
});

const response = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [
    {
      role: "system",
      content: "Very long system prompt that will be automatically cached..." // 1024+ tokens
    },
    {
      role: "user", 
      content: "What is machine learning?"
    }
  ],
  prompt_cache_key: `doc-analysis-${documentId}` // Optional: control caching keys
});

Pricing

OpenAI charges standard rates for cache writes and offers significant discounts for cache reads. Exact pricing varies by model.

CoreValue Model Registry

View supported models and their caching capabilities

OpenAI Prompt Caching Documentation

Official OpenAI prompt caching guide

Anthropic (Claude)

Anthropic provides advanced caching with cache control breakpoints (up to 4 per request) and TTL control.

Using OpenAI SDK with CoreValue Types

The @cova/gateway SDK extends OpenAI types to support Anthropic's cache control through the OpenAI-compatible interface:

npm install @cova/gateway
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.corevalue.dev/v1",
  apiKey: process.env.COVA_API_KEY,
});

const response = await client.chat.completions.create({
  model: "anthropic/claude-3.5-haiku",
  messages: [
    {
      role: "system",
      content: "You are a helpful assistant...",
      cache_control: {
        type: "ephemeral",
        ttl: "1h"
      }
    },
    {
        role: "assistant",
        content: "Example assistant message.",
        cache_control: { type: "ephemeral" }
    },
    {
      role: "user",
      content: [
        {
          type: "text",
          text: "This content will be cached.",
          cache_control: {
            type: "ephemeral",
            ttl: "5m"
          }
        },
        {
          type: "image_url",
          image_url: {
            url: "https://example.com/image.jpg",
            detail: "low"
          },
          cache_control: { type: "ephemeral" }
        }
      ]
    }
  ],
  temperature: 0.7
});

Cache Key Mapping

Anthropic uses user_id as a cache key on their servers. When using the OpenAI-compatible AI Gateway, these parameters automatically map to Anthropic's user_id:

  • prompt_cache_key
  • safety_identifier
  • user
const response = await client.chat.completions.create({
  model: "anthropic/claude-3.5-haiku",
  messages: [/* your messages */],
  prompt_cache_key: "doc-analysis-v1", // Maps to Anthropic's user_id for cache keying
  cache_control: {
    type: "ephemeral", 
    ttl: "1h"
  }
});

Current Limitation: Anthropic cache control is currently enabled for caching messages only. Support for caching tools is coming soon.

Pricing Structure

Anthropic uses a simple multiplier-based pricing model for prompt caching.

OperationMultiplierExample (Claude Sonnet @ $3/MTok)
Cache Read0.1×$0.30/MTok
Cache Write (5 min)1.25×$3.75/MTok
Cache Write (1 hour)2.0×$6.00/MTok

Key Points

  • TTL Options: 5 minutes or 1 hour
  • Providers: Available on Anthropic API, Vertex AI, and AWS Bedrock
  • Limitation: Vertex AI and Bedrock only support 5-minute caching
  • Minimum: 1024 tokens for most models

Calculation Example

Base input price: $3/MTok
5-min cache write: $3 × 1.25 = $3.75/MTok
1-hour cache write: $3 × 2.0 = $6.00/MTok
Cache read: $3 × 0.1 = $0.30/MTok

Learn More

Anthropic Prompt Caching Documentation

Google Gemini

Google uses a multiplier plus storage cost model for context caching.

Pricing Structure

OperationMultiplierStorage Cost
Cache Read0.25×N/A
Cache Write1.0×+ Storage fee

Storage Rates:

  • Gemini 2.5 Pro: $4.50/MTok/hour
  • Gemini 2.5 Flash: $1.00/MTok/hour
  • Gemini 2.5 Flash-Lite: $1.00/MTok/hour

Key Points

  • TTL: 5 minutes only
  • Cache Types: Implicit (automatic) and Explicit (manual)
  • Minimum: 1024 tokens (Flash), 2048 tokens (Pro)
  • Discount: 75% off input costs for cache reads

Calculation Example

For Gemini 2.5 Pro (≤200K tokens):

Base input price: $1.25/MTok
Storage rate: $4.50/MTok/hour

Cache write (5 min):
- Input cost: $1.25 × 1.0 = $1.25
- Storage cost: $4.50 × (5/60) = $0.375
- Total: $1.625/MTok

Cache read: $1.25 × 0.25 = $0.31/MTok

Tiered Pricing

Gemini 2.5 Pro has different rates for larger contexts:

Context SizeInput PriceCache ReadCache Write (5 min)
≤200K tokens$1.25/MTok$0.31/MTok$1.625/MTok
>200K tokens$2.50/MTok$0.625/MTok$2.875/MTok

Google Gemini SDK (Native)

Use Google's official @google/genai SDK with Cova AI Gateway — no Google API key required, just your Cova key.

Reasoning

Enable reasoning through a unified API on CoreValue's AI Gateway

On this page

Why Prompt CachingOpenAI and Compatible ProvidersQuick StartPricingAnthropic (Claude)Using OpenAI SDK with CoreValue TypesCache Key MappingPricing StructureKey PointsCalculation ExampleGoogle GeminiPricing StructureKey PointsCalculation ExampleTiered Pricing