Building a Production-Grade LLM Orchestration System for Conversational Search

perplexity · Tech · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Conversational search with live retrieval and citations

Perplexity needed to turn LLMs into a fast, reliable conversational answer engine that could search live information, cite sources, ask clarifying questions, personalize behavior, use external tools, and avoid becoming a slow or hallucination-prone LLM wrapper.

Constraints:

• LLMs alone have knowledge cutoffs and cannot provide live sourced answers
• combining many tools and LLMs increases latency and creates many possible failure points
• early product latency was roughly five to seven seconds per query
• search queries can be ambiguous and need clarifying questions or reformulation
• quality must be re-evaluated whenever prompts or search indexes change
• tooling abstractions could hide latency optimizations needed for a search product

Architecture

Perplexity built a conversational search orchestration layer that combines LLMs with search indexes, vector databases, Wolfram Alpha, rendering libraries, database plugins, citations, personalization prompts, query classification, query reformulation, summarization, and follow-up generation. Different LLMs handle classification, reformulation into subqueries, final answer synthesis, and related-question generation, while in-house engineering controls latency across the full stack rather than relying on high-level LLM tooling libraries.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  search-user-interface[Search user interface] --> orchestration-layer[Orchestration layer]
  search-user-interface --> ai-profile[AI profile]
  ai-profile --> orchestration-layer
  orchestration-layer --> query-classification-llm[Query classification LLM]
  query-classification-llm --> query-reformulation-llm[Query reformulation LLM]
  query-reformulation-llm --> search-indexes[Search indexes]
  orchestration-layer --> tool-integrations[Tool integrations]
  search-indexes --> summarization-llm[Summarization LLM]
  tool-integrations --> summarization-llm
  summarization-llm --> citation-renderer[Citation renderer]
  citation-renderer --> search-user-interface
  orchestration-layer --> follow-up-generation-llm[Follow-up generation LLM]
  follow-up-generation-llm --> search-user-interface
  orchestration-layer --> quality-eval-suite[Quality eval suite]
  product-stack[Product stack] --> search-user-interface
  product-stack --> search-indexes
  product-stack --> orchestration-layer

Components

Data flow

A user asks a question in Perplexity or Copilot. If needed, the system asks clarifying questions or applies profile instructions, classifies the query, reformulates it into multiple subqueries, searches live indexes and tool sources, retrieves multiple results, uses LLMs to summarize and answer with citations, and generates contextual follow-up questions. Evals run when prompts, model choices, or search indexes change.

Why this architecture

Perplexity's thesis was that production search requires LLM reasoning plus live tools, not a standalone model. The system decomposes work into specialized LLMs and tool calls, uses retrieval to avoid knowledge cutoffs, keeps citations central for trust, and controls the entire engineering stack so latency can be optimized across orchestration, retrieval, rendering, and model calls.

Models used: GPT-3.5 Turbo · GPT-4 · Claude · Claude Instant · MPT fine-tunes · custom embeddings · query classification LLM · query reformulation LLM · follow-up generation LLM · summarization LLM

Scale

• Early Perplexity queries took about five to six seconds.
  “product we are our product had like five to six second latency per query and in fact like one of our investor friends uh Dan Daniel gross he used to test the product at the beginning and he even joked saying like you you guys should”transcript, stated
• The speaker also described the early experience as six to seven seconds.
  “it's that slow like it used to take six to seven seconds and now it's like incredibly fast like uh the first thing anybody comes and tells me about perplexity is like how do you guys make it so fast and I think that's the”transcript, stated
• A query can be reformulated into three or four follow-up subqueries.
  “these questions right and then uh that gets reformulated into like three or four follow-up queries and and then a search runs for all these follow-up queries and pulls up all these results like and then uh the co-pilot looks at like 16 results which would not have”transcript, stated
• Copilot can inspect about 16 results for one expanded query.
  “like and then uh the co-pilot looks at like 16 results which would not have been possible with just like one single query right um so that expands the diversity of pages of the co-pilot sees um and creates a very high recall content for the llm now that pulls up”transcript, stated
• The orchestration system was described as serving tens or hundreds of thousands of users at scale.
  “orchestrate the whole thing together into one functional working piece that just works seamlessly reliably at scale for like tens of hundreds of thousands of users”transcript, stated

Alternatives considered (roads not taken)

Use high-level tool libraries such as LangChain or Dust

Rejected because: Perplexity wanted end-to-end control to capture latency optimizations those libraries would hide.

Rely on a pure LLM without tools

Rejected because: A pure LLM lacks live search/index knowledge and cannot create the same tool-backed experiences.

Use one search query only

Rejected because: Reformulating into several subqueries increases result diversity, recall, and context quality for the answer LLM.

Use OpenAI embeddings by default

Rejected because: Perplexity trained its own embeddings and did not see OpenAI holding the same lead in embeddings as in LLMs.

Choose foundation models by preference rather than evals

Rejected because: The speaker says model selection in the orchestration process should be driven by eval results.

Patterns

rag-retrieval (conversational-search)

Perplexity combines LLM reasoning with live search indexes so answers can use current retrieved information and citations.

tool-use-orchestration (llm-orchestration)

The system orchestrates LLMs with search engines, Wolfram, rendering libraries, databases, and other tools instead of asking a model to do everything alone.

query-rewriting (search-recall)

User questions are reformulated into several subqueries so Copilot retrieves a more diverse and useful source set.

model-routing (model-selection)

Perplexity uses different models for summarization, preprocessing, classification, and reasoning, with model choices governed by evals.

eval-driven-development (quality)

Perplexity runs AI evals whenever prompts or search indexes change and uses dogfooding plus user complaints to improve quality.

latency-optimization (production-performance)

Perplexity controls the stack in-house and avoids high-level LLM tool libraries so it can optimize the multi-tool, multi-LLM path for search latency.

Tradeoffs

Build orchestration in-house

Chose end-to-end engineering controlled by Perplexity over LangChain, Dust, and similar high-level tooling libraries.

Rationale: Perplexity needed latency optimizations across all layers of the stack.

Use retrieval and tools rather than pure parametric knowledge

Chose RAG with search indexes, databases, Wolfram, and citations over LLM-only answers from model parameters.

Rationale: Search and tools provide live knowledge and robustness while LLMs provide reasoning and natural-language synthesis.

Split the search task into specialized LLM workers

Chose classification, reformulation, summarization, and follow-up generation LLMs over one model doing every step monolithically.

Rationale: Different parts of conversational search need different specialized behavior and can be orchestrated as workers.

Use user-controlled prompt profiles for personalization

Chose AI profile/global instructions over traditional topic/domain preference picking only.

Rationale: Prompt profiles reduce repeated prompt engineering and let users guide the AI in natural language.

Train custom embeddings

Chose Perplexity-trained embeddings over OpenAI embeddings.

Rationale: Perplexity judged OpenAI's embedding advantage weaker than its LLM advantage.

Gotchas (what bit them)

Many-tool orchestration can make latency unacceptable.

Root cause: Every tool and LLM step adds work, and Perplexity's first product versions took five to seven seconds per query.

Resolution: The team controlled the full stack in-house and optimized end-to-end rather than relying on high-level tooling libraries.

Chained tools increase hallucination and breakage risk.

Root cause: When many components are chained together, a single wrong part can break the whole answer.

Resolution: Perplexity relies on human feedback, prompt engineering updates, prompt version control, and AI/model-quality benchmarks.

A conversational search product is more than an LLM wrapper.

Root cause: Search, tool integrations, rendering, database plugins, query reformulation, follow-up generation, and model orchestration are separate software layers.

Resolution: Perplexity built an orchestration suite around OpenAI/Claude models and its own backend/search stack.

Ambiguous queries need user agency and clarification.

Root cause: Users often ask underspecified search questions, so the model needs help choosing the right part of the web.

Resolution: Copilot asks clarifying questions and the profile system lets users preload persistent preferences and instructions.

Model migrations and model choice create engineering work.

Root cause: Changing model providers or using different models by task requires benchmarks, prompt migrations, and eval-driven decisions.

Resolution: Perplexity runs rigorous evals and chooses models by task, using GPT/Claude for strong reasoning and smaller models for narrower work.

Evaluation

Perplexity evaluates answer quality with AI evals, benchmarks after prompt/search-index changes, dogfooding, production feedback, user complaints, and task-based model selection.

Metrics: answer quality after prompt changes · answer quality after search-index changes · model-quality benchmarks · foundation-model selection by eval results · latency per query · user complaints and dogfooding findings

Outcomes

• Perplexity shipped a conversational search engine that combines LLMs, live search, tools, and citations.
• The product improved from roughly six-to-seven-second early latency to a speed users noticed.
• Copilot can ask clarifying questions, reformulate queries into multiple subqueries, retrieve many results, and generate follow-up questions.
• Perplexity uses evals whenever prompts or search indexes change.
• The system uses OpenAI, Claude, custom embeddings, Elasticsearch, Quadrant, Kubernetes, React/Next.js, SwiftUI, and native Android.
• The team reported organic traction from prominent users and media coverage, while selling Pro subscriptions with Copilot and GPT-4 usage.

The talk describes early five-to-seven-second query latency, query reformulation into three or four subqueries, Copilot considering around 16 results for one question, and orchestration intended to work reliably at scale for tens or hundreds of thousands of users.