Request Pipeline
The 12-step gateway request pipeline
Every request that enters the CoreValue AI Gateway passes through the same 12-step pipeline. The pipeline is implemented in gateway/src/server.ts and is responsible for format detection, authentication, key resolution, rate limiting, caching, and the async post-processing that records usage and deducts credits.
The gateway is the sole LLM request path post-M3. The legacy Cloudflare Worker proxy has been decommissioned — all traffic flows through the Hono gateway on Cloud Run.
The 12 Steps
Detect the incoming request format (OpenAI, Anthropic, Google, or gateway-prefixed), extract the requested model, determine whether streaming is enabled, and normalize the request path.
Handled by: request adaptation layer in server.ts.
Look up the provider handler for the normalized provider. If the model begins with a known prefix (e.g. groq/, anthropic/), the prefix is stripped and the matching provider handler is selected. Requests that don't match a known provider prefix are rejected (SSRF prevention).
Handled by: MODEL_PREFIX_MAP in gateway/src/providers/gateway.ts.
Validate the CoreValue API key (Authorization: Bearer $COVA_API_KEY). Unauthenticated requests are rejected before any further processing.
Handled by: auth middleware.
Read the Cova-* request headers: Cova-User-Id, Cova-Prompt-Id, Cova-Property-{Name}, Cova-RateLimit-Policy, and Cova-Fallbacks. Invalid values are sanitized or silently dropped — the request continues with safe defaults.
Handled by: covaHeaders middleware (gateway/src/middleware/covaHeaders.ts).
Apply the global sliding-window rate limit. This protects the gateway as a whole before any org-level or per-key limits are evaluated.
Handled by: global rate-limit middleware.
Select the upstream provider API key to use. Your own keys (BYOK) are tried first when configured; otherwise the gateway falls back to CoreValue-managed PTB (pay-the-bill) keys.
Handled by: key resolution logic in server.ts.
This step only runs when a PTB (pay-the-bill) managed key will serve the request. BYOK requests skip it.
Verify the organization wallet has enough credit balance to cover the estimated cost of the request. If the balance is insufficient, the request is rejected before it is forwarded upstream.
Handled by: wallet/escrow check in server.ts.
Apply the org-level bucket rate limit (from DB config or the Cova-RateLimit-Policy header). Then determine the final streaming flag and model, run the disallow-list check (W4), and apply the budget gate (W2) which may downgrade the model if the estimated cost exceeds the org's budget.
Handled by: bucket rate-limit middleware + budget gate.
If Cova-Cache-Enabled (or Cova-Cache-Read) is set, compute the cache key and look up a cached response. On a hit, the cached response is returned immediately with Cova-Cache: HIT and the request skips the remaining forward steps.
Handled by: cache middleware (gateway/src/middleware/cache.ts).
Send the (possibly translated) request to the upstream provider. The gateway sets Cova-Id on the forwarded request for log correlation and streams the response back to the client.
Handled by: provider handler + streaming layer (gateway/src/lib/streaming.ts).
Return the response to the client with all Cova-* response headers (Cova-Id, Cova-Provider, Cova-Model, Cova-Status, Cova-Request-Cost, etc.). For streaming requests, a final SSE cova frame is appended after [DONE].
Handled by: buildResponseHeaders() in streaming.ts.
Credit deduction in this step only runs for PTB (pay-the-bill) managed keys. BYOK requests skip credit deduction.
After the response is returned, the gateway performs async post-processing in the background:
- S3 upload — store the full request/response payload
- Credit deduction (PTB only) — debit the org wallet for the actual cost
- Bucket rate-limit cost recording — record cents-based usage for bucket limits
- Cache save — store the response if
Cova-Cache-Save/Cova-Cache-Enabledwas set - Log to API server — emit the analytics record to Core
Handled by: async post-processing pipeline in server.ts.
PTB-only Steps
Two steps in the pipeline only execute when a CoreValue-managed PTB key serves the request (i.e. you are using credits rather than your own provider keys):
| Step | PTB-only behavior |
|---|---|
| 7. Wallet check | Verifies the org wallet has sufficient balance before forwarding. BYOK requests skip this entirely. |
| 12. Credit deduction | Debits the actual cost from the org wallet after the request completes. BYOK requests are not charged. |
When Cova-Request-Cost-Estimate is present on the response, it reflects the upfront escrow reservation amount from the wallet check (step 7) — it is a UX hint, not the final charge. The actual charge is Cova-Request-Cost (non-streaming) or the requestCost field in the final SSE frame (streaming).