Real-Time Access Control and Credit System for High-Scale LLM Products

openai · Tech · 2026 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: High-scale LLM product access control and billing infrastructure

OpenAI needed Codex and Sora users to keep working after hitting rate limits while preserving fair access, system performance, real-time correctness, and auditable credit billing.

Constraints:

• Hard rate limits protected capacity but created frustrating interruptions for engaged users.
• Pure usage-based billing from the first token would discourage early exploration.
• Access decisions had to be made synchronously inside interactive product requests.
• Credit consumption needed to be provably correct, idempotent, auditable, and reconcilable.

Architecture

OpenAI built an in-house real-time access engine that models access as a decision waterfall across rate limits, credits, promotions, and entitlements. A distributed usage and balance system checks per-user/per-feature rate windows and available credits synchronously, returns one definitive access outcome, and settles credit debits asynchronously through idempotent streaming processors and atomic balance-update records.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Codex and Sora requests"] --> B["Real-time access engine"]
  B --> C["Rate-limit windows"]
  B --> D["Credit balances"]
  B --> E["Product usage events"]
  E --> F["Monetization events"]
  F --> G["Streaming async processor"]
  G --> H["Balance updates"]
  H --> D
  H --> I["Audit and reconciliation"]

Components

Data flow

A Codex or Sora request enters a single evaluation path. The access engine checks rate-limit windows, verifies credit availability when needed, and returns a real-time decision. Product usage events drive monetization events, which drive balance updates through a streaming async processor. Stable idempotency keys, serialized balance updates, and atomic database transactions keep retries, replays, and concurrent requests from corrupting credit balances.

Why this architecture

OpenAI chose a unified in-house waterfall because third-party metering was not tightly coupled enough for immediate product decisions or transparent user outcomes. Splitting synchronous access decisions from asynchronous audited debits protected user momentum while preserving billing correctness.

Models used: Codex · Sora

Scale

• Codex and Sora adoption quickly exceeded original usage expectations.
  “both Codex and Sora have seen rapid adoption, with usage quickly pushing beyond what we originally expected.”article, stated
• The system tracks usage at per-user and per-feature granularity.
  “Tracks per‑user, per‑feature usage”article, stated
• The system performs synchronous access decisions but near-real-time asynchronous balance updates.
  “asynchronous (but still near-real-time) balance updates instead of synchronous updates”article, stated
• The access foundation is intended to extend beyond Codex and Sora to more products over time.
  “The same foundation can extend to more products over time; Codex and Sora are just the beginning.”article, stated

Alternatives considered (roads not taken)

Raise rate limits

Rejected because: Higher limits would weaken demand smoothing and fairness controls and risk capacity exhaustion.

Use pure asynchronous usage billing

Rejected because: Delayed billing would create lag, overages, reconciliation issues, and visible user-facing failures.

Use third-party usage billing and metering platforms

Rejected because: Existing platforms did not satisfy immediate decisioning and integrated transparency requirements.

Strict real-time balance enforcement

Rejected because: OpenAI accepted brief overshoots and refunds to prioritize provable correctness, trust, and user momentum.

Patterns

cost-aware-routing (operations)

The access waterfall routes a request from included rate-limit capacity to credit-backed continuation when limits are exhausted.

safety-policy-enforcement (governance)

A unified access engine enforces rate limits, credits, promotions, and entitlements through one decision path.

observability-tracing (operations)

Usage, monetization, and balance datasets form an audit trail that supports replay, reconciliation, and explanations.

durable-execution (reliability)

Stable idempotency keys and serialized balance updates prevent retries, worker restarts, and concurrent requests from double-debiting credits.

Tradeoffs

Split synchronous access decisions from asynchronous credit debits.

Chose Real-time decisioning with near-real-time balance updates over Fully synchronous credit-balance mutation on the request path.

Rationale: This preserves interactive responsiveness while maintaining an auditable debit trail, accepting small balance-delay overshoots with refunds.

Build usage and balance infrastructure in-house.

Chose Integrated in-house access engine over Third-party billing and metering platforms.

Rationale: The product needed immediate correctness and outcome transparency that separate metering tools could not provide.

Use a waterfall rather than a binary access gate.

Chose Layered decision stack across limits, credits, promotions, and entitlements over Simple allowed/blocked gate.

Rationale: The waterfall lets users keep working when one access layer is exhausted while still preserving fairness and system constraints.

Gotchas (what bit them)

Traditional access models force a poor choice between hard stops and billing friction.

Root cause: Rate limits protect capacity but frustrate engaged users, while usage billing from the first token discourages exploration.

Resolution: OpenAI built a hybrid real-time waterfall that can transition from rate limits to credits within the same request.

Third-party billing platforms were not sufficient for interactive LLM-product decisions.

Root cause: They were designed for invoicing/reporting rather than immediate allow-or-block decisions tied to product state.

Resolution: OpenAI built the decision waterfall in-house for real-time correctness, timing, and observability control.

Retries and concurrent requests can double-charge users unless billing events are idempotent and serialized.

Root cause: Distributed usage systems see retries, replays, worker restarts, and simultaneous requests against the same balance.

Resolution: Every event carries an idempotency key, balance updates are serialized per account, and balance changes are written atomically with audit records.

Evaluation

The article frames evaluation around operational correctness and user trust rather than model-quality metrics: real-time access decisions, auditable datasets, idempotent debits, reconciliation, and predictable user-facing outcomes.

Metrics: Real-time decision availability · No double debit on retries or restarts · Replayable and reconcilable usage/billing layers · Transparency into why requests were allowed or blocked · User ability to continue work after rate limits when credits are available

Outcomes

• Codex and Sora access can continue beyond rate limits through credits without a visible mode switch.
• The system produces one definitive access outcome per request.
• Usage, monetization, and balance records can be audited, replayed, and reconciled.
• The access-control foundation can extend to additional products.

The system supports rapidly adopted OpenAI products, per-user/per-feature usage windows, real-time request decisions, near-real-time async credit updates, and account-serialized balance records.