LiteLLM Integration

Use Cova AI Gateway with LiteLLM to get top tier observability for your LLM requests.

Introduction

LiteLLM is a unified interface for calling LLM APIs.

Integration Steps

Log into CoreValue or create an account. Once you have an account, you can generate an API key here.

COVA_API_KEY=sk-cova-XXXXXXXXXXXXXXXX

Install required dependencies

pip install litellm python-dotenv

Point LiteLLM at the Cova AI Gateway by setting api_base to the gateway URL and using the openai/ prefix for models:

import os
from litellm import completion
from dotenv import load_dotenv

load_dotenv()

# Route through Cova AI Gateway
response = completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    api_base="https://gateway.corevalue.dev/v1",
    api_key=os.getenv("COVA_API_KEY")
)

print(response.choices[0].message.content)

You can find all supported models at corevalue.dev/models.

All your LiteLLM requests are now visible in your CoreValue dashboard.

With the above setup, any calls to any LiteLLM request will automatically be logged and monitored by CoreValue. Review them in your CoreValue dashboard.

While you're here, why not give us a star on GitHub? It helps us a lot!

Complete Working Examples

Basic Completion

import os
from litellm import completion
from dotenv import load_dotenv

load_dotenv()

# Simple completion
response = completion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Tell me a fun fact about space"}],
    api_base="https://gateway.corevalue.dev/v1",
    api_key=os.getenv("COVA_API_KEY")
)

print(response.choices[0].message.content)

Streaming Responses

import os
from litellm import completion
from dotenv import load_dotenv

load_dotenv()

# Streaming example
response = completion(
    model="openai/anthropic/claude-sonnet-4",
    messages=[{"role": "user", "content": "Write a short story about a robot learning to paint"}],
    stream=True,
    api_base="https://gateway.corevalue.dev/v1",
    api_key=os.getenv("COVA_API_KEY")
)

print("🤖 Assistant (streaming):")
for chunk in response:
    if hasattr(chunk.choices[0].delta, 'content') and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print("\n")

Custom Properties and Session Tracking

Add metadata to track and filter your requests:

import os
from litellm import completion
from dotenv import load_dotenv

load_dotenv()

response = completion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "What's the weather like?"}],
    api_base="https://gateway.corevalue.dev/v1",
    api_key=os.getenv("COVA_API_KEY"),
    extra_headers={
        "Cova-User-Id": "user-789",
        "Cova-Property-Environment": "production",
        "Cova-Property-App-Version": "2.1.0",
        "Cova-Property-Feature": "weather-query"
    }
)

print(response.choices[0].message.content)

Provider Selection and Fallback

Cova's AI Gateway supports automatic failover between providers. Use the provider/model format to select a specific provider:

import os
from litellm import completion
from dotenv import load_dotenv

load_dotenv()

# Automatic routing (cheapest provider)
response = completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    api_base="https://gateway.corevalue.dev/v1",
    api_key=os.getenv("COVA_API_KEY")
)

# Manual provider selection (anthropic/)
response = completion(
    model="openai/anthropic/claude-sonnet-4",
    messages=[{"role": "user", "content": "Hello!"}],
    api_base="https://gateway.corevalue.dev/v1",
    api_key=os.getenv("COVA_API_KEY")
)

For fallback chains, use the Cova-Fallbacks header:

response = completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    api_base="https://gateway.corevalue.dev/v1",
    api_key=os.getenv("COVA_API_KEY"),
    extra_headers={
        "Cova-Fallbacks": '["anthropic/claude-sonnet-4"]'
    }
)

Advanced Features

Caching

Enable caching to reduce costs and latency for repeated requests:

import os
from litellm import completion
from dotenv import load_dotenv

load_dotenv()

# Enable caching for this request
response = completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    api_base="https://gateway.corevalue.dev/v1",
    api_key=os.getenv("COVA_API_KEY"),
    extra_headers={
        "Cova-Cache-Enabled": "true"
    }
)

print(response.choices[0].message.content)

# Subsequent identical requests will be served from cache
response2 = completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    api_base="https://gateway.corevalue.dev/v1",
    api_key=os.getenv("COVA_API_KEY"),
    extra_headers={
        "Cova-Cache-Enabled": "true"
    }
)

print(response2.choices[0].message.content)

Rate Limiting

Apply rate limiting policies to control request rates:

import os
from litellm import completion
from dotenv import load_dotenv

load_dotenv()

response = completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
    api_base="https://gateway.corevalue.dev/v1",
    api_key=os.getenv("COVA_API_KEY"),
    extra_headers={
        "Cova-RateLimit-Policy": "10000;w=3600"
    }
)

print(response.choices[0].message.content)