Scaling a High-Traffic LLM Chat Application to 30,000 Messages Per Second

character.ai · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: High-scale consumer LLM chat serving

Character.ai needed to scale a consumer LLM chat product from hundreds to tens of thousands of messages per second while controlling GPU cost, latency, prompt complexity, cache correctness, and database reliability.

Constraints:

• Every generated message pulls substantial chat history through the serving hot path.
• Serving 30,000 messages per second creates major GPU cost pressure.
• Latency improvements can be bought with over-provisioning, but that is expensive.
• Long response generation keeps connections open for 12-25 seconds.
• Prompt construction and cache indexing bugs can pass simple message-in/message-out tests while damaging quality.

Architecture

Character.ai built and served its own foundation models, allowing upstream model architecture decisions such as multi-query attention. The serving hot path relies on GPU attention-cache reuse with a 95% cache-hit ratio, prompt templates managed through Prompt Poet's YAML/Jinja layer with smart truncation priorities, and a migration from monolith toward services and sharded databases. The design optimizes for the tight cost-latency-quality tradeoff of very high-volume conversational generation.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["User chat request"] --> B["Prompt Poet templating"]
  B --> C["Model serving layer"]
  C --> D["GPU attention cache"]
  C --> E["Streaming response connection"]
  E --> A
  C --> F["Database services"]
  G["Experiment monitoring"] --> B

Components

Data flow

A user message enters the chat hot path with prior character definitions and conversation history. Prompt Poet constructs a structured prompt and truncates lower-priority sections when context overflows. The serving layer reuses cached computations for prior context, computes only the new user-message suffix where possible, runs the model, streams a response over a long-lived connection, and writes/reads chat state through database services that had to evolve from monolithic access toward sharded service boundaries.

Why this architecture

Character.ai's ownership of pretraining and post-training let it optimize model architecture for serving economics. Multi-query attention reduced GPU cache pressure, cache reuse made 30,000 messages per second financially survivable, and structured prompt templating reduced bugs when evolving multi-character chat prompts.

Models used: Character.ai foundation models · Multi-query attention model variants

Scale

• Character.ai scaled from about 300 generations per second to over 30,000 messages per second.
  “went from I think like 300 uh Generations per second to this summer we went we over 30,000 mes per second”transcript, stated
• The growth represented roughly two orders of magnitude over about 18 months.
  “so we went through like two orders of magnitude in about 18 months”transcript, stated
• The generation hot path moved roughly seven or eight gigabytes per second over the network.
  “our network does seven or 8 G gab per second just on just on this hot path um for Generation”transcript, stated
• P50 full-message response time was about 12.5 seconds, down from about 25 seconds.
  “the total response time is I think p50 is like 12.5 seconds today meaning like that's the time it takes to generate an entire message”transcript, stated
• The GPU cache strategy delivered a 95% cache-hit ratio.
  “so we have like a 95% cash AIT ratio which allows us to serve this this GPS um at a decent cost”transcript, stated

Alternatives considered (roads not taken)

Over-provision GPUs to reduce latency

Rejected because: It lowers latency but dramatically increases serving cost.

Use a fixed model architecture supplied externally

Rejected because: Owning the model stack let Character.ai choose multi-query attention upstream for serving economics.

Continue prompt concatenation and string slicing across components

Rejected because: It made significant prompt changes nearly impossible without bugs.

Patterns

semantic-caching (serving)

Character.ai reused GPU-layer computations for unchanged prompt context across successive conversation turns.

latency-optimization (serving)

The system optimized latency through model architecture, cache reuse, and serving-path improvements rather than only buying more GPUs.

cost-aware-routing (economics)

Serving architecture choices were driven by the cost of GPU inference at 30,000 messages per second.

prompt-versioning (prompting)

Prompt Poet made prompt construction structured, experimentable, and safer to modify at scale.

context-compaction (context)

Structured YAML prompt sections carried truncation priorities so context overflow removed less important content first without destroying cache reuse.

Tradeoffs

Latency versus GPU spend

Chose architecture and cache optimization over only over-provisioning GPUs.

Rationale: Over-provisioning would improve latency but at unacceptable cost for 30,000 messages per second.

Model ownership

Chose build foundation models from day zero over serving a fixed external architecture.

Rationale: Owning the pretraining stack allowed serving-oriented model architecture decisions such as multi-query attention.

Database incident response

Chose fix-forward with partial vacuum over rollback to a previous day's backup.

Rationale: Rollback would have lost a day of user data; fix-forward avoided that at the cost of a 30-40 minute outage.

Gotchas (what bit them)

Cache-indexing bugs can preserve syntax while destroying quality.

Root cause: A serving rewrite missed indexing computation results for user messages, so the 95% cached context no longer represented the real prompt.

Resolution: The team traced the issue to cache indexing and fixed it with a small code change.

Prompt construction across services became fragile.

Root cause: Prompt strings were concatenated and sliced across multiple components and databases.

Resolution: Build Prompt Poet with YAML/Jinja templates, runtime variables, control flow, and truncation priorities.

The database layer became the Achilles heel at scale.

Root cause: High query volume and write characteristics triggered a low-level PostgreSQL transaction-ID bug.

Resolution: The team evaluated rollback versus fix-forward, ran a partial vacuum, and took the site down briefly rather than losing a day of data.

Evaluation

Character.ai evaluated changes through production engagement and performance experiments, watching latency, cost, engagement, time spent, messages sent, A/B treatment quality, and cache behavior.

Metrics: 300 generations per second to over 30,000 messages per second · Roughly two orders of magnitude growth in about 18 months · Seven or eight GB/s on the generation hot path · 95% cache-hit ratio · P50 full response around 12.5 seconds, down from around 25 seconds · Prompt changes evaluated for performance, engagement, time spent, and messages sent

Outcomes

• Character.ai scaled from roughly 300 generations per second to over 30,000 messages per second.
• Multi-query attention gave a 5x reduction in GPU cache requirements.
• A 95% cache-hit ratio made the service economically viable.
• P50 full-response latency improved from about 25 seconds to about 12.5 seconds.
• Prompt Poet made prompt changes and experiments safer at scale.
• The team recovered from a severe PostgreSQL transaction-ID incident without losing a day of user data.

The system handled over 30,000 messages per second, seven or eight GB/s on the generation path, long-lived 12.5-second P50 responses, and a 95% cache-hit ratio.