Scaling LLM and ML Models to 300M Monthly Requests with Self-Hosting

storygraph · Media & Entertainment · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: consumer recommendation and LLM infrastructure

StoryGraph needed to serve ML and LLM-powered book features to millions of users with a tiny team, while keeping cloud/API costs and user privacy under control.

Constraints:

• The app operated at hundreds of millions of monthly requests with one AI/DevOps engineer.
• Third-party LLM APIs would have been too expensive and raised privacy concerns.
• January 1 traffic spikes repeatedly exposed bottlenecks in servers, queues, databases, and load balancing.

Architecture

StoryGraph self-hosted its AI/ML stack on dedicated servers, including specialized recommendation, similarity, vision, and LLM models. The architecture evolved from naive file/watch-folder and load-balancer approaches into Redis queues for ML work, dedicated servers, distributed database infrastructure, HAProxy and PgBouncer on web servers, and self-hosted observability/logging systems.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["StoryGraph application"] --> B["Redis ML job queue"]
  A --> C["Self-hosted LLM service"]
  A --> D["Self-hosted specialized ML models"]
  B --> D
  C --> E["Dedicated model servers"]
  D --> E
  A --> F["Local HAProxy layer"]
  F --> G["PgBouncer connection pool"]
  G --> H["Yugabyte database cluster"]

Components

Data flow

User requests for recommendations, similar users, book previews, and content checks enter the application, queue or route to self-hosted ML/LLM services, run on dedicated servers, and return results without sending private user data to external LLM APIs. Database traffic is distributed through local HAProxy and PgBouncer layers to the Yugabyte/Postgres cluster.

Why this architecture

StoryGraph self-hosted because cost, privacy, and control outweighed managed-service convenience, especially for a freemium app making heavy LLM calls.

Models used: self-hosted large language model · book mood classifier · recommendation models · similar-user models · book-cover vision model

Scale

• StoryGraph had crossed 6 million monthly users.
  “crossed 6 million monthly users and hundreds of millions of requests per”transcript, stated
• The case targets 300 million monthly requests.
  “if you want to get to figure out how to get to 300 million requests as a oneperson AI devops team”transcript, stated
• The LLM features produce about 1 million model requests on an average day.
  “that's about 1 million requests on an average day or about 10 L Lish model requests a second”transcript, stated
• Using GPT-4 for current LLM use would have cost about $770,000 per month.
  “it would have cost us $770,000 a month to implement chat gbt 4 in all the places where we use large language models today”transcript, stated
• Before January 1, StoryGraph preemptively scaled to 10 web servers and 60 databases.
  “preemptively scaled up the servers 10 web servers and 60 databases I didn't want to draw all 600 lines here”transcript, stated

Alternatives considered (roads not taken)

Use a watch folder for ML jobs

Rejected because: Multiple servers could pick up the same file and fight over work.

Route recommendation requests directly through load-balanced model servers

Rejected because: Long-running requests quickly exhausted available servers.

Use ChatGPT-4 APIs for LLM features

Rejected because: It would have compromised privacy and cost an estimated $770,000 per month.

Patterns

privacy-preserving-processing (privacy)

StoryGraph self-hosted LLM and ML features to avoid sending sensitive user data to third-party LLM APIs.

cost-aware-routing (production-operations)

The team optimized architecture around infrastructure and API economics, choosing self-hosting where managed services were too expensive.

batch-offline-processing (workflow)

Redis queues coordinate long-running recommendation and ML jobs across many workers.

latency-optimization (production-operations)

The January 1 debugging work traced slow requests to thread/session distribution rather than database CPU.

Tradeoffs

LLM hosting

Chose self-hosted LLM and ML infrastructure over third-party ChatGPT/API calls for heavy features.

Rationale: Self-hosting preserved privacy and avoided extremely high per-request API costs.

Job coordination

Chose Redis queue workers over watch folders or direct load balancing to long-running workers.

Rationale: A queue safely orders work across many servers and avoids duplicate processing or server exhaustion.

Operational style

Chose self-managed dedicated infrastructure over managed cloud services for every subsystem.

Rationale: The team accepted operational complexity because dedicated/self-hosted systems made the economics work.

Gotchas (what bit them)

Naive distributed file claiming broke down for multiple ML workers.

Root cause: Multiple servers needed a safe way to coordinate jobs without fighting over files.

Resolution: Use a proper Redis queue rather than watch folders and timestamp fights.

Direct load balancing to long-running ML servers failed under modest concurrency.

Root cause: Recommendation requests took 30-60 seconds, exhausting the small server pool.

Resolution: Move ML jobs behind a queue so workers process jobs in order.

Session stickiness overloaded only a few web servers.

Root cause: An HAProxy cookie pinned users to the same servers, creating uneven session distribution.

Resolution: Remove the cookie setting so users distribute across servers.

Bottlenecks appeared at each new traffic threshold.

Root cause: Capacity dashboards for one layer hid limits elsewhere in the stack.

Resolution: Iteratively inspect queues, proxies, connection pools, server threads, and session behavior.

Evaluation

StoryGraph evaluated by production traffic survival, cost reduction, privacy preservation, queue behavior, server/database load, request latency, and whether users could access ML-powered features during January traffic spikes.

Metrics: monthly requests · monthly users · daily LLM requests · requests per second · server sessions · database CPU · API cost estimate

Outcomes

• StoryGraph scaled to 300 million monthly requests.
• It self-hosted about a dozen ML models and a heavily used LLM feature.
• Self-hosting avoided an estimated $770,000 per month GPT-4 cost.
• The team identified and fixed HAProxy session pinning during the January 1 scaling incident.

The case is unusually rich in operational failure detail, including several discarded architectures before the working self-hosted design.