DocumentationGuides
GuidesIntegrations

Logging OpenAI Batch API Requests with CoreValue

Learn how to track and monitor OpenAI Batch API requests by posting completed results to the CoreValue gateway log endpoint.

Feature gating: This guide uses Users and Custom Properties, which are non-free features (Growth tier or higher). They may be disabled in your deployment. Requires: Growth tier or higher.

The OpenAI Batch API allows you to process large volumes of requests asynchronously at 50% cheaper costs than synchronous requests. However, tracking these batch requests for observability can be challenging since they don't go through the standard real-time proxy flow.

This guide shows you how to post completed batch results to the CoreValue gateway's log endpoint so you get full visibility into costs, performance, and request patterns. See Zero-Friction BaseURL Swap for the standard real-time flow.

Why Track Batch Requests?

Batch processing offers significant cost savings, but without proper tracking, you lose visibility into:

  • Cost analysis: Understanding the true cost of your batch operations
  • Performance monitoring: Tracking completion times and success rates
  • Request patterns: Analyzing which prompts and models perform best
  • Error tracking: Identifying failed requests and common issues
  • Usage analytics: Understanding your batch processing patterns over time

By posting batch results to the CoreValue gateway log endpoint, you get all the observability benefits of real-time requests for your batch operations.

Prerequisites

Before getting started, you'll need:

Installation

First, install the required packages:

npm install openai dotenv
# or
yarn add openai dotenv
# or
pnpm add openai dotenv

Not using TypeScript? The gateway log endpoint is usable in any language via HTTP requests. See the Python Quickstart for Python examples, or use curl directly.

Environment Setup

Create a .env file in your project root:

OPENAI_API_KEY=your_openai_api_key_here
COVA_API_KEY=your_corevalue_api_key_here

Complete Implementation

Here's a complete example that demonstrates the entire batch workflow with CoreValue logging:

import OpenAI from "openai";
import fs from "fs";
import dotenv from "dotenv";

dotenv.config();

const COVA_API_KEY = process.env.COVA_API_KEY!;
const GATEWAY_LOG_URL = "https://gateway.corevalue.dev/oai/v1/log";

/**
 * Post a single request/response pair to the CoreValue gateway log endpoint.
 * This is the same endpoint the gateway uses internally for real-time requests,
 * so batch logs show up alongside your real-time traffic in the dashboard.
 */
async function logToCoreValue(
  requestBody: Record<string, unknown>,
  responseBody: Record<string, unknown>,
  additionalHeaders: Record<string, string> = {},
) {
  const headers: Record<string, string> = {
    Authorization: `Bearer ${COVA_API_KEY}`,
    "Content-Type": "application/json",
    ...additionalHeaders,
  };

  const payload = {
    request: requestBody,
    response: responseBody,
  };

  const res = await fetch(GATEWAY_LOG_URL, {
    method: "POST",
    headers,
    body: JSON.stringify(payload),
  });

  if (!res.ok) {
    throw new Error(`CoreValue log failed: ${res.status} ${await res.text()}`);
  }
}

// Initialize OpenAI client
const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY!,
});

function createBatchFile(filename: string = "data.jsonl") {
  const batchRequests = [
    {
      custom_id: "req-1",
      method: "POST",
      url: "/v1/chat/completions",
      body: {
        model: "gpt-4o-mini",
        messages: [
          {
            role: "user",
            content:
              "Write a professional email to schedule a meeting with a client about quarterly business review",
          },
        ],
        max_tokens: 300,
      },
    },
    {
      custom_id: "req-2",
      method: "POST",
      url: "/v1/chat/completions",
      body: {
        model: "gpt-4o-mini",
        messages: [
          {
            role: "user",
            content:
              "Explain the benefits of cloud computing for small businesses in simple terms",
          },
        ],
        max_tokens: 250,
      },
    },
    {
      custom_id: "req-3",
      method: "POST",
      url: "/v1/chat/completions",
      body: {
        model: "gpt-4o-mini",
        messages: [
          {
            role: "user",
            content:
              "Create a Python function that calculates compound interest with proper error handling",
          },
        ],
        max_tokens: 400,
      },
    },
  ];

  const jsonlContent = batchRequests
    .map((req) => JSON.stringify(req))
    .join("\n");
  fs.writeFileSync(filename, jsonlContent);
  console.log(`Created batch file: ${filename}`);
  return filename;
}

async function uploadFile(filename: string) {
  console.log("Uploading file...");

  try {
    const file = await openai.files.create({
      file: fs.createReadStream(filename),
      purpose: "batch",
    });

    console.log(`File uploaded: ${file.id}`);
    return file.id;
  } catch (error) {
    console.error("Error uploading file:", error);
    throw error;
  }
}

async function createBatch(fileId: string) {
  console.log("Creating batch...");

  try {
    const batch = await openai.batches.create({
      input_file_id: fileId,
      endpoint: "/v1/chat/completions",
      completion_window: "24h",
    });

    console.log(`Batch created: ${batch.id}`);
    console.log(`Status: ${batch.status}`);
    return batch;
  } catch (error) {
    console.error("Error creating batch:", error);
    throw error;
  }
}

async function waitForCompletion(batchId: string) {
  console.log("Waiting for batch completion...");

  while (true) {
    try {
      const batch = await openai.batches.retrieve(batchId);
      console.log(`Status: ${batch.status}`);

      if (batch.status === "completed") {
        console.log("Batch completed!");
        return batch;
      } else if (
        batch.status === "failed" ||
        batch.status === "expired" ||
        batch.status === "cancelled"
      ) {
        throw new Error(`Batch failed with status: ${batch.status}`);
      }

      console.log("Waiting 5 seconds...");
      await new Promise((resolve) => setTimeout(resolve, 5000));
    } catch (error) {
      console.error("Error checking batch status:", error);
      throw error;
    }
  }
}

async function retrieveAndLogResults(batch: any) {
  if (!batch.output_file_id || !batch.input_file_id) {
    throw new Error("No output or input file available");
  }

  console.log("Retrieving batch results...");

  try {
    // Get original requests
    const inputFileContent = await openai.files.content(batch.input_file_id);
    const inputContent = await inputFileContent.text();
    const originalRequests = inputContent
      .trim()
      .split("\n")
      .map((line) => JSON.parse(line));

    // Get batch results
    const outputFileContent = await openai.files.content(batch.output_file_id);
    const outputContent = await outputFileContent.text();
    const results = outputContent
      .trim()
      .split("\n")
      .map((line) => JSON.parse(line));

    console.log(`Found ${results.length} results`);

    // Create mapping of custom_id to original request
    const requestMap = new Map();
    originalRequests.forEach((req) => {
      requestMap.set(req.custom_id, req.body);
    });

    // Log each result to CoreValue
    for (const result of results) {
      const { custom_id, response } = result;

      if (response && response.body) {
        console.log(`\nLogging ${custom_id}...`);

        const originalRequest = requestMap.get(custom_id);

        if (originalRequest) {
          // Modify model name to distinguish batch requests
          const modifiedRequest = {
            ...originalRequest,
            model: originalRequest.model + "-batch",
          };

          const modifiedResponse = {
            ...response.body,
            model: response.body.model + "-batch",
          };

          // Log to CoreValue with additional metadata
          await logToCoreValue(modifiedRequest, modifiedResponse, {
            "Cova-User-Id": "batch-demo",
            "Cova-Property-CustomId": custom_id,
            "Cova-Property-BatchId": batch.id,
            "Cova-Property-ProcessingType": "batch",
            "Cova-Property-Provider": "openai",
          });

          const responseText =
            response.body.choices?.[0]?.message?.content || "No response";
          console.log(`${custom_id}: "${responseText.substring(0, 100)}..."`);
        } else {
          console.log(`Could not find original request for ${custom_id}`);
        }
      }
    }

    console.log(
      `\nSuccessfully logged all ${results.length} requests to CoreValue!`,
    );
    return results;
  } catch (error) {
    console.error("Error retrieving results:", error);
    throw error;
  }
}

async function main() {
  console.log("OpenAI Batch API with CoreValue Logging\n");

  // Validate environment variables
  if (!process.env.COVA_API_KEY) {
    console.error("Please set COVA_API_KEY environment variable");
    return;
  }

  if (!process.env.OPENAI_API_KEY) {
    console.error("Please set OPENAI_API_KEY environment variable");
    return;
  }

  try {
    // Complete batch workflow
    const filename = createBatchFile();
    const fileId = await uploadFile(filename);
    const batch = await createBatch(fileId);
    const completedBatch = await waitForCompletion(batch.id);
    await retrieveAndLogResults(completedBatch);

    // Cleanup
    if (fs.existsSync(filename)) {
      fs.unlinkSync(filename);
      console.log(`Cleaned up ${filename}`);
    }
  } catch (error) {
    console.error("Error:", error);
  }
}

if (require.main === module) {
  main();
}

Key Implementation Details

1. Gateway Log Endpoint

The logToCoreValue helper posts request/response pairs to the gateway's /oai/v1/log endpoint with your Cova-* headers:

const res = await fetch("https://gateway.corevalue.dev/oai/v1/log", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${COVA_API_KEY}`,
    "Content-Type": "application/json",
    "Cova-User-Id": "batch-demo",
    "Cova-Property-BatchId": batch.id,
  },
  body: JSON.stringify({ request: requestBody, response: responseBody }),
});

2. Batch Request Processing

The workflow follows OpenAI's standard batch process:

  1. Create batch file: Format requests as JSONL
  2. Upload file: Send to OpenAI's file storage
  3. Create batch: Submit for processing
  4. Wait for completion: Poll until finished
  5. Retrieve results: Download and process outputs

3. CoreValue Logging Strategy

Each batch result is logged individually to CoreValue with:

  • Original request data: Preserves the initial request structure
  • Batch response data: Includes the actual LLM response
  • Custom metadata: Adds batch-specific tracking properties
await logToCoreValue(modifiedRequest, modifiedResponse, {
  "Cova-User-Id": "batch-demo",
  "Cova-Property-CustomId": custom_id,
  "Cova-Property-BatchId": batch.id,
  "Cova-Property-ProcessingType": "batch",
});

4. Model Name Modification

The example modifies model names to distinguish batch requests:

const modifiedRequest = {
  ...originalRequest,
  model: originalRequest.model + "-batch",
};

This helps you filter and analyze batch vs. real-time requests in CoreValue's dashboard.

Advanced Features

Custom Properties for Analytics

Add custom properties to track additional metadata:

"Cova-Property-Department": "marketing",
"Cova-Property-CampaignId": "q4-2024",
"Cova-Property-Priority": "high"

Error Handling and Retry Logic

Implement robust error handling for production use:

async function logWithRetry(
  request: any,
  response: any,
  headers: Record<string, string>,
  maxRetries = 3,
) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      await logToCoreValue(request, response, headers);
      return;
    } catch (error) {
      console.log(`Logging attempt ${attempt} failed:`, error);
      if (attempt === maxRetries) throw error;
      await new Promise((resolve) => setTimeout(resolve, 1000 * attempt));
    }
  }
}

Batch Status Tracking

Track the entire batch lifecycle in CoreValue:

// Log batch creation
await logToCoreValue(
  { batch_id: batch.id, operation: "batch_created" },
  { status: "in_progress", file_id: fileId },
  {
    "Cova-Property-BatchId": batch.id,
    "Cova-Property-Operation": "batch_lifecycle",
  },
);

Monitoring and Analytics

Once logged, you can use CoreValue's dashboard to:

  • Analyze costs: Compare batch vs. real-time request costs
  • Monitor performance: Track batch completion times and success rates
  • Filter by properties: Use custom properties to segment analysis
  • Set up alerts: Get notified of batch failures or cost spikes
  • Export data: Download detailed analytics for further analysis

Best Practices

  1. Use descriptive custom_ids: Make them meaningful for debugging
  2. Add relevant properties: Include metadata that helps with analysis
  3. Handle errors gracefully: Implement retry logic for logging failures
  4. Monitor batch status: Track the entire lifecycle, not just results
  5. Clean up files: Remove temporary files after processing
  6. Validate environment: Check API keys before starting batch operations

Learn More

With this setup, you now have comprehensive observability for your OpenAI Batch API requests, enabling better cost management, performance monitoring, and request analytics at scale.

Integrating CoreValue with GitHub Actions

Automate the monitoring and caching of LLM calls in your CI pipelines with CoreValue.

Overview

Prompt engineering is the strategic crafting of prompts to guide Large Language Models to produce accurate and desired outputs.

On this page

Why Track Batch Requests?PrerequisitesInstallationEnvironment SetupComplete ImplementationKey Implementation Details1. Gateway Log Endpoint2. Batch Request Processing3. CoreValue Logging Strategy4. Model Name ModificationAdvanced FeaturesCustom Properties for AnalyticsError Handling and Retry LogicBatch Status TrackingMonitoring and AnalyticsBest PracticesLearn More