Streaming FinOps

When you make a streaming request through the CoreValue AI Gateway, the gateway appends a final Server-Sent Events (SSE) frame containing per-call FinOps metadata — the actual cost, the provider that served the request, your remaining credit balance, and whether cross-provider translation was applied.

This is the primary delivery path for cost metadata in streaming responses. The /meta endpoint is the fallback when this frame is missing.

Final SSE Frame Format

After the upstream stream closes (including after the OpenAI-format data: [DONE] frame), the gateway appends one final SSE event:

data: {"cova":{"requestId":"...","provider":"...","model":"...","status":"success","requestCost":0.0015,"creditsRemaining":48.77,"gatewayMode":"translated","translationWarning":"logprobs"}}

The frame is a single SSE data: line terminated by \n\n (two newlines). The JSON payload is nested under a "cova" key so the SDK can detect it by checking for "cova" in lastChunk.

CovaGatewayMeta Fields

The cova object in the final frame contains the following fields:

Using the SDK

The @cova/gateway SDK's streamChat() wraps the OpenAI streaming call and yields chunks unchanged. After [DONE], it yields one final { cova: CovaGatewayMeta } value with the actual cost.

import OpenAI from "openai";
import { streamChat } from "@cova/gateway";

const client = new OpenAI({
  apiKey: process.env.COVA_API_KEY,
  baseURL: "https://gateway.corevalue.dev/v1",
});

const stream = streamChat(client, {
  model: "groq/llama-3.1-70b",
  messages: [{ role: "user", content: "Hello!" }],
  stream: true,
});

for await (const chunk of stream) {
  if ("cova" in chunk) {
    // Final frame — per-call FinOps metadata
    console.log(chunk.cova.requestCost);
    console.log(chunk.cova.provider);
    console.log(chunk.cova.creditsRemaining);
  } else {
    // Normal streaming chunk — process as usual
    process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  }
}

from openai import OpenAI
from cova_gateway import stream_chat

client = OpenAI(
    api_key=os.environ["COVA_API_KEY"],
    base_url="https://gateway.corevalue.dev/v1",
)

stream = stream_chat(client, model="groq/llama-3.1-70b",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)

for chunk in stream:
    if "cova" in chunk:
        # Final frame — per-call FinOps metadata
        print(chunk["cova"]["requestCost"])
        print(chunk["cova"]["provider"])
    else:
        print(chunk.choices[0].delta.content or "", end="")

Upfront Cost Estimate (M4)

Before the stream even begins, the gateway may set the Cova-Request-Cost-Estimate response header (M4 milestone 1.5a). This is the upfront worst-case cost estimate — the escrow reservation amount from the wallet check.

Fallback: /meta Endpoint

When the final SSE frame is missing or the requestCost field is absent, use the /meta endpoint to retrieve the metadata after the stream completes:

curl https://gateway.corevalue.dev/gateway/v1/meta/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer $COVA_API_KEY"

See the Meta Endpoint page for full details on response codes and the CovaGatewayMeta shape.

Related