Scaling an AI-Powered Search and Research Assistant from Prototype to Production

perplexity_ai · Tech · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI-powered search and research assistant

Perplexity needed to turn LLM-backed search from an internal Slack bot into a production research assistant that could answer live questions, cite sources, iterate quickly, and control cost and latency.

Constraints:

• Pure language-model answers are not trustworthy for current or data-grounded questions without live retrieval.
• Search assistant orchestration must connect indices, LLMs, conversational rendering, citations, and multimodal/file workflows reliably at inference time.
• Consumer search experiences are sensitive to latency, especially on phones and weak networks.
• GPT-4-level quality was expensive and slower, while cheaper models needed tuning to match user experience.
• The market included large incumbents, so iteration speed and product quality mattered more than one launch.

Architecture

Perplexity began as an internal Slack bot for company-building and enterprise-search questions, then added Bing search and summarization to answer live questions. The product evolved into a search/research assistant that orchestrates live search indices, LLMs, citation ranking, conversational context, generative UI clarification, fine-tuned GPT-3.5, file upload, and custom Llama inference infrastructure.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["User research interface"] --> B["Search orchestration layer"]
  A --> F["Generative UI clarifier"]
  B --> C["Live search index"]
  C --> G["Citation ranking layer"]
  G --> B
  B --> D["LLM answer generator"]
  D --> E["Fine-tuned GPT-3.5 Copilot model"]
  E --> F
  D --> H["Custom Llama inference stack"]

Components

Data flow

User questions enter the assistant, which retrieves live search results or file context, ranks and selects sources, asks clarifying questions for Copilot when needed, sends grounded context to an LLM or fine-tuned model, renders an answer with citations, and feeds user/product observations back into iteration on models, ranking, and inference latency.

Why this architecture

Perplexity's central claim is that useful AI search requires orchestration: models must be grounded in real data, citations must match answers, UI must adapt to research tasks, and inference must be fast enough for consumer use.

Models used: GPT-3.5 · fine-tuned GPT-3.5 · GPT-4 · Llama · Claude 2

Scale

• The company incorporated on August 3 and had an internal Slack bot by September 27.
  “we incorporated on August 3rd uh and then like on September 27th we had this slack bot”transcript, stated
• The speaker says Perplexity launched a week after ChatGPT's late-November launch.
  “end of November Chad GPT launches considered the iPhone moment for AI a week after that we launched perplexity”transcript, stated
• The fine-tuned GPT-3.5 Copilot comparison found users unable to distinguish models 80% of the time.
  “80 of the time the user is unable to say like which model is which”transcript, stated
• Early launch advice targeted getting 100 people to use the product.
  “go get 100 people you know like 100 people to use your product”transcript, stated

Alternatives considered (roads not taken)

Pure chatbot without live search

Rejected because: Answers cannot be trusted for live or data-grounded questions without a search index.

Fully autonomous AutoGPT-style agent

Rejected because: The team believed autonomy was not ready and designed Copilot to work with the user through clarifying questions.

Use GPT-4 for Copilot by default

Rejected because: Fine-tuned GPT-3.5 gave much better speed and user experience while approaching GPT-4 perceived quality.

Rely only on provider-hosted models

Rejected because: The team wanted custom inference control and launched self-served Llama models with best-in-class latency claims.

Patterns

rag-retrieval (retrieval)

Perplexity grounds LLM answers in live search indexes and files rather than relying on model memory.

tool-use-orchestration (agent-architecture)

The product orchestrates search, LLMs, citations, file upload, generative UI, and custom inference infrastructure.

fine-tuning-adaptation (model-ops)

Perplexity fine-tuned GPT-3.5 to make Copilot faster and cheaper while keeping quality close to GPT-4.

latency-optimization (optimization)

The team optimized model choice and custom Llama serving around throughput, latency, and time to first token.

human-in-the-loop-gate (agent-architecture)

Copilot asks clarifying questions and works with users instead of attempting full autonomy.

Tradeoffs

Copilot interaction model

Chose clarifying-question generative UI over fully autonomous AutoGPT-style behavior.

Rationale: The team judged full autonomy immature and chose a companion workflow that works with the user.

Copilot model optimization

Chose fine-tuned GPT-3.5 over default GPT-4 for every Copilot request.

Rationale: Fine-tuned GPT-3.5 improved speed and directness while users usually could not tell it apart from GPT-4.

Product strategy versus incumbents

Chose keep iterating on platform, flywheel, and infrastructure over assuming a large incumbent launch ended the market.

Rationale: The team believed better product iteration would compound despite Microsoft/Bing pressure.

Gotchas (what bit them)

LLMs alone cannot answer live factual questions reliably.

Root cause: A language model needs real data and current search context.

Resolution: Add Bing/live search and summarization, then build orchestration around search plus LLMs.

Latency made early answers feel broken.

Root cause: Initial answer generation took seven or eight seconds.

Resolution: Iterate on product and inference until the experience became almost instantaneous.

API/plugin demos can fail at inference time.

Root cause: Announcing connections to APIs or data providers is easier than orchestrating the correct call reliably.

Resolution: Treat orchestration and citation quality as core product engineering work.

Evaluation

Perplexity compared user preference for fine-tuned models, monitored speed/throughput/latency, and referenced external evaluations on perceived utility and citation F1.

Metrics: user indistinguishability between fine-tuned GPT-3.5 and GPT-4 · throughput · latency · time to first token · perceived utility · citation F1

Outcomes

• Perplexity moved from internal Slack bot to public web search in a few months.
• Fine-tuned GPT-3.5 made Copilot faster while most users could not distinguish it from GPT-4.
• The company built custom Llama serving for latency and throughput.
• External evaluation framed citation F1 and perceived utility as important search-assistant metrics.

The case emphasizes rapid company/product timelines, high daily search-request volume, and latency-sensitive consumer use on mobile and weak networks.