Scaling LLM Inference to Serve 400M+ Monthly Search Queries

perplexity · Tech · 2024 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI search inference serving

Perplexity needed to serve hundreds of millions of monthly search queries with multiple LLMs while balancing latency-sensitive user experience against inference cost.

Constraints:

Each search query triggers multiple AI inference requests.
More than 20 AI models must run simultaneously under strict cost and interactivity SLAs.
Different model sizes and use cases require different batching, scheduling, and parallelism choices.

Architecture

Perplexity deployed models on NVIDIA H100 GPU pods managed by Triton Inference Server and TensorRT-LLM inside Kubernetes. Smaller classifier models detect user intent, route tasks to specific model pods, and an in-house front-end scheduler routes traffic based on load and usage while Triton batches requests and reports GPU utilization. For latency-sensitive models, the team uses tensor parallelism across four or eight GPUs; for other settings, data or pipeline parallelism can maximize throughput.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Search query workflow"] --> B["Intent classifier models"]
  B --> C["Front-end scheduler"]
  C --> D["Kubernetes cluster"]
  D --> E["H100 GPU pods"]
  C --> F["Triton Inference Server"]
  F --> E
  E --> G["TensorRT-LLM"]
  G --> H["Llama model deployments"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Search query workflow`	User-facing Perplexity search flow that triggers multiple inference requests.	`Intent classifier models`
`Intent classifier models`	Smaller models that detect task type and select routing destinations.	`Search query workflow`, `Front-end scheduler`
`Front-end scheduler`	In-house scheduler that routes traffic to pods based on load, usage, and SLAs.	`Intent classifier models`, `Kubernetes cluster`, `Triton Inference Server`
`Kubernetes cluster`	Hosts GPU pods and scales deployments for fluctuating traffic.	`Front-end scheduler`, `H100 GPU pods`
`H100 GPU pods`	One-or-more H100 GPU deployments serving model workloads.	`Kubernetes cluster`, `Triton Inference Server`, `TensorRT-LLM`
`Triton Inference Server`	Inference middleware that batches requests, serves backends, and exports GPU metrics.	`Front-end scheduler`, `H100 GPU pods`, `TensorRT-LLM`
`TensorRT-LLM`	Optimized LLM runtime used for Llama-based serving.	`H100 GPU pods`, `Triton Inference Server`, `Llama model deployments`
`Llama model deployments`	8B, 70B, 405B and other user-facing language models.	`TensorRT-LLM`

Data flow

A search query produces multiple inference requests. Smaller classifier models determine user intent, route tasks such as completion to H100 GPU pods, Triton/TensorRT-LLM serve optimized Llama-family and other models, the Kubernetes front-end scheduler balances load against SLA targets, and inference metrics feed scheduling and scaling decisions.

3 supporting quotes

“Each query represents multiple AI inference requests.”article, stated

“User tasks detected by the classifiers, like text completion, are then routed to specific models deployed on GPU pods.”article, stated

“Triton Inference Server is a critical component of Perplexity’s deployment architecture. It serves optimized models across various backends”article, stated

Why this architecture

The architecture lets Perplexity route heterogeneous requests to model deployments that meet each use case's cost and latency targets, while using GPU metrics, batching, and parallelism to keep utilization high.

3 supporting quotes

“To define the right SLAs for the company’s diverse use cases, Perplexity’s inference team conducts comprehensive A/B testing”article, stated

“Their goal is to maximize GPU utilization while consistently meeting the target SLA for each specific use case.”article, stated

“By improving batching while meeting target SLAs, inference serving cost is optimized.”article, stated

Models used: classifier models · Llama 3.1 8B · Llama 3.1 70B · Llama 3.1 405B · embedding models under 1B parameters · more than 20 AI models

Scale

Perplexity handles more than 435 million queries each month.
“an AI-powered search engine that handles more than 435 million queries each month.”article, stated
The inference team serves over 20 AI models simultaneously.
“the inference team at Perplexity serves over 20 AI models simultaneously.”article, stated
A benchmarked load-balancing chart used a moderate request rate of 16 QPS.
“during an internal Perplexity benchmark at a moderate request rate of 16 QPS”article, stated
Tensor parallelism across four H100 GPUs reduced relative cost per million tokens by up to 3x for latency-sensitive requests.
“Sharding the Llama 8B model using tensor parallelism across four NVIDIA Hopper GPUs reduces Perplexity’s relative cost per million token by up to 3x”article, stated
Perplexity estimated approximately $1 million in annual savings for the Related-Questions feature.
“the team estimated that they were able to save approximately $1 million annually by serving models that power their Related-Questions feature”article, stated

Alternatives considered (roads not taken)

Use third-party LLM provider APIs

Rejected because: Perplexity hosts models when doing so meets strict SLAs at lower cost.

“Perplexity inference team’s decision to host models depends on their ability to serve these models at a lower cost while still meeting their strict SLAs”article, stated

Use low tensor parallelism for latency-sensitive large models

Rejected because: The team found four- and eight-GPU tensor parallelism lowered serving cost within a fixed GPU budget for strict SLAs.

“the team opted to increase tensor parallelism to four and eight GPUs, which they found yields lower serving costs for very latency-sensitive requests”article, stated

One scheduling policy for all traffic

Rejected because: Scheduling algorithms affect worst-percentile inter-token latency, so Perplexity continuously optimizes request routing.

“The scheduling algorithm used by the front-end scheduler can affect inter-token latency, particularly in improving the worst percentile of performance”article, stated

Patterns

`model-routing` (model-operations)

Classifier models route each user task to the appropriate model pod.

“To match each user request with the appropriate model, the company relies on smaller classifier models that help determine user intent. User tasks detected by the classifiers, like text completion, are then routed to specific models deployed on GPU pods.”article, stated

`latency-optimization` (production-operations)

Perplexity tunes scheduling, batching, parallelism, and model placement to meet interactivity SLAs.

“The scheduling algorithm used by the front-end scheduler can affect inter-token latency, particularly in improving the worst percentile of performance (Figure 2).”article, stated

`cost-aware-routing` (production-operations)

Hosting and parallelism choices are made by comparing SLA compliance and cost against third-party APIs.

“their ability to serve these models at a lower cost while still meeting their strict SLAs, compared to using third-party LLM provider APIs.”article, stated

`eval-driven-development` (evaluation)

A/B tests determine SLA configurations and user-experience tradeoffs for model serving.

“Perplexity’s inference team conducts comprehensive A/B testing, evaluating different configurations and their impact on user experience.”article, stated

Tradeoffs

Serving strategy

Chose host optimized models on H100/Triton/TensorRT-LLM over defaulting to third-party LLM APIs.

Rationale: Self-hosted/cloud-hosted optimized deployments can lower cost while meeting strict SLAs.

“This post details some of the deployment best practices and TCO savings based on their hands-on experience.”article, stated

Parallelism choice

Chose higher tensor parallelism for latency-sensitive requests over throughput-oriented data or pipeline parallelism for all cases.

Rationale: For strict interactivity SLAs, higher tensor parallelism reduced serving cost within fixed GPU budget.

“Due to the strict SLAs, the team opted to increase tensor parallelism to four and eight GPUs, which they found yields lower serving costs for very latency-sensitive requests within a fixed GPU budget.”article, stated

Future serving architecture

Chose disaggregated serving exploration over coupling prefill and decode on the same GPU type.

Rationale: Separating prefill and decode can improve throughput and let each phase use different hardware.

“disaggregating serving, a groundbreaking technique that separates the prefill and decode inference phases of an LLM workflow onto separate NVIDIA GPUs.”article, stated

Gotchas (what bit them)

Inference requests vary by latency sensitivity.

Root cause: Some models are hidden inside workflows while user-facing models directly affect experience.

Resolution: Use low batch sizes for small real-time retrieval models and deeper performance analysis for user-facing Llama models.

“For smaller models, such as embedding models under 1 billion parameters used in real-time retrieval, the focus is on achieving the lowest possible latency.”article, stated

Load-balancing policy can dominate tail latency.

Root cause: Different scheduling algorithms affect worst-percentile inter-token latency.

Resolution: Continuously optimize scheduler behavior and account for sequence-length variation.

“The team at Perplexity constantly looks for new scheduler optimizations, including how to better account for sequence length variations across requests.”article, stated

Throughput-maximizing parallelism may not minimize cost for latency-sensitive requests.

Root cause: Strict SLAs change the economics of tensor, data, and pipeline parallelism.

Resolution: Use tensor parallelism of four or eight GPUs for latency-sensitive models while using data/pipeline parallelism where throughput matters more.

“Data or pipeline parallelism was useful for maximizing throughput in less latency-sensitive settings.”article, stated

Evaluation

Perplexity evaluates serving configurations with A/B testing, SLA compliance, GPU utilization, inter-token latency, time to first token, tokens per second per user, cost per million queries/tokens, and cost versus third-party APIs.

Metrics: inter-token latency · time to first token · tokens per second per user · cost per million queries · cost per million tokens · GPU utilization · SLA compliance

“For user-facing models such as Llama 8B, 70B, and 405B, which have a greater impact on user experience and deployment costs, the team conducts a deeper performance analysis and evaluates key metrics such as time to first token, tokens per second per user, and cost per million queries.”article, stated

“It serves optimized models across various backends, batches incoming user requests, and provides GPU utilization metrics to the scheduler. This supports scaling up or down the number of deployments and GPUs based on the amount of the inference requests.”article, stated

Outcomes

Perplexity serves more than 435 million queries each month.
The platform serves over 20 AI models simultaneously.
Tensor parallelism reduced relative cost per million tokens by up to 3x for a latency-sensitive Llama 8B case.
Perplexity estimated approximately $1 million annual savings for Related Questions by serving models itself.

This source is vendor-authored by NVIDIA, so it is strong on stack details and savings claims but naturally emphasizes NVIDIA components.

“The Perplexity team now uses TensorRT-LLM in combination with proprietary LLM runtimes built with optimized CUDA kernels”article, stated

“approximately $1 million annually by serving models that power their Related-Questions feature on cloud-hosted NVIDIA GPUs.”article, stated