Streaming

When you make a streaming request through the CoreValue AI Gateway, the response cannot carry the Cova-Request-Cost header: headers are sent before the first chunk, and the cost is only known once the stream ends. The gateway reports it in one of two places instead.

• SSE streams (text/event-stream): the gateway appends a final Server-Sent Events frame containing per-call FinOps metadata — the actual cost, the provider that served the request, your remaining credit balance, and whether cross-provider translation was applied. The /meta endpoint is the fallback when this frame is missing.
• Every other stream format — Bedrock's application/vnd.amazon.eventstream and Google's streaming JSON: no frame is appended, because appending text would corrupt a binary event stream or a JSON document. Read the cost from the /meta endpoint, using the Cova-Id response header.

Final SSE Frame Format

For SSE streams only. After the upstream stream closes (including after the OpenAI-format data: [DONE] frame), the gateway appends one final SSE event:

data: {"cova":{"requestId":"...","provider":"...","model":"...","status":"success","requestCost":0.0015,"creditsRemaining":48.77,"gatewayMode":"translated","translationWarning":"logprobs"}}

The frame is a single SSE data: line terminated by \n\n (two newlines). The JSON payload is nested under a "cova" key so the SDK can detect it by checking for "cova" in lastChunk.

CovaGatewayMeta Fields

The cova object in the final frame contains the following fields:

Using the SDK

The @cova/gateway SDK's streamChat() wraps the OpenAI streaming call and yields chunks unchanged. After [DONE], it yields one final { cova: CovaGatewayMeta } value with the actual cost.

import OpenAI from "openai";
import { streamChat } from "@cova/gateway";

const client = new OpenAI({
  apiKey: process.env.COVA_API_KEY,
  baseURL: "https://gateway.corevalue.dev/v1",
});

const stream = streamChat(client, {
  model: "groq/llama-3.3-70b-versatile",
  messages: [{ role: "user", content: "Hello!" }],
  stream: true,
});

for await (const chunk of stream) {
  if ("cova" in chunk) {
    // Final frame — per-call FinOps metadata
    console.log(chunk.cova.requestCost);
    console.log(chunk.cova.provider);
    console.log(chunk.cova.creditsRemaining);
  } else {
    // Normal streaming chunk — process as usual
    process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  }
}

from openai import OpenAI
from cova_gateway import stream_chat

client = OpenAI(
    api_key=os.environ["COVA_API_KEY"],
    base_url="https://gateway.corevalue.dev/v1",
)

stream = stream_chat(client, model="groq/llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)

for chunk in stream:
    if "cova" in chunk:
        # Final frame — per-call FinOps metadata
        print(chunk["cova"]["requestCost"])
        print(chunk["cova"]["provider"])
    else:
        print(chunk.choices[0].delta.content or "", end="")

Upfront Cost Estimate (M4)

Before the stream even begins, the gateway may set the Cova-Request-Cost-Estimate response header (M4 milestone 1.5a). This is the upfront worst-case cost estimate — the escrow reservation amount from the wallet check.

Fallback: /meta Endpoint

For Bedrock and Google streaming JSON responses this is the only source of the cost. For SSE streams, use it when the final frame is missing or the requestCost field is absent. Call it after the stream completes, with the Cova-Id response header as the request id:

curl https://gateway.corevalue.dev/gateway/v1/meta/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer $COVA_API_KEY"

See the Meta Endpoint page for full details on response codes and the CovaGatewayMeta shape.

Related