Building Robust Enterprise Search with LLMs and Traditional IR

glean · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise search and employee knowledge discovery

Glean needed to help employees find internal information across many SaaS applications, permissions, people, documents, mentions, and company-specific jargon while avoiding a technology-first trap where LLMs, embeddings, or vector search are marketed as solutions without solving the underlying search and retention problem.

Constraints:

• Enterprise knowledge lives across many SaaS apps with API and permissions boundaries.
• Users search for project names, acronyms, people, documents, presentations, and internal resources rather than only natural-language questions.
• Search quality depends on ranking, query understanding, document understanding, synonyms, personalization, and permissions, not embeddings alone.
• A pure search box may not be compelling enough to retain users in an enterprise workflow.
• LLMs struggle with fresh information unless retrieval/search supplies current context.
• Latency and infrastructure cost matter for production search and LLM features.

Architecture

Glean builds an enterprise search and employee portal around a hybrid retrieval stack. SaaS APIs provide documents, messages, mentions, and permissions. Core information retrieval, synonym handling, query understanding, acronym expansion, document understanding, and vector search generate candidates. A ranking and personalization layer prioritizes results using role, department, team relationships, co-authorship, organization structure, and interaction history. The product expands beyond search into an employee portal with Go links, trending documents, personalized feed, collections, and aggregated mentions from Slack, Jira, GitHub, email, and other apps. LLM features are treated as user-tested additions that still depend on the search backend for freshness and retrieval-augmented generation.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  saas-connectors[SaaS connectors] --> permission-model[Permission model]
  saas-connectors --> enterprise-index[Enterprise index]
  permission-model --> enterprise-index
  query-understanding[Query understanding] --> synonym-expansion[Synonym expansion]
  synonym-expansion --> core-ir-retrieval[Core IR retrieval]
  enterprise-index --> core-ir-retrieval
  enterprise-index --> vector-retrieval[Vector retrieval]
  enterprise-index --> document-understanding[Document understanding]
  core-ir-retrieval --> ranking-layer[Ranking layer]
  vector-retrieval --> ranking-layer
  document-understanding --> ranking-layer
  permission-model --> ranking-layer
  personalization-layer[Personalization layer] --> ranking-layer
  ranking-layer --> employee-portal[Employee portal]
  vector-retrieval --> rag-answer-layer[RAG answer layer]
  rag-answer-layer --> employee-portal

Components

Data flow

SaaS connectors index enterprise documents, messages, and permission metadata into a search corpus. User queries pass through query understanding and synonym/acronym expansion, then retrieve candidates through traditional IR and vector search. Document understanding and personalization signals such as department, team relationships, co-authorship, and interactions feed the ranking layer. Results surface in search and in a broader employee portal with Go links, feeds, trending documents, collections, and aggregated mentions. When LLM answers are used, search still supplies fresh context for retrieval-augmented generation rather than expecting the model to know current enterprise data.

Why this architecture

Glean avoids an embeddings-only or LLM-first architecture because search quality comes from sustained ranking rigor and subproblem tuning. Hybrid retrieval keeps mature IR signals and adds vector search where it helps. Personalization is necessary because enterprise relevance depends on role, team, authorship, and interactions. The employee portal exists because search alone does not keep users returning. RAG is used as an LLM integration pattern because enterprise answers require fresh indexed context.

Models used: Vector embedding models for semantic search · LLMs for experimental chat and retrieval-augmented answers · GPT-4 as an example synthetic data generator for specialized NLP tasks · Smaller distilled or LoRA-adapted models for task-specific production use

Scale

• Glean started in 2019 and reached a $1 billion valuation in 2022.
  “Glean is an enterprise search company founded in 2019 by a team of former Google engineers”summary, stated

  “The company reached unicorn status with a Series C led by Sequoia at a $1 billion valuation in 2022.”summary, stated
• The source names Databricks, Canva, Confluent, Duolingo, Samsara, and Fortune 50 customers or logos.
  “new logos across Databricks, Canva, Confluent, Duolingo, Samsara, and more in the Fortune 50”article, stated
• The enterprise search problem spans companies using roughly 10 to 100 apps.
  “every company uses, you know, 10 to a hundred apps.”article, stated
• The product can aggregate hundreds of Slack pings plus Jira, email, and GitHub mentions.
  “it's like all of the hundred Slack pings that you have, plus the Jira pings, plus the, the, the email, all of that in one place”article, stated
• The interview frames internal search value around roughly 10 to 20 searches per day.
  “you're gonna get see about 10 to 20 searches that's gonna save about this much time, uh, a day.”article, stated

Alternatives considered (roads not taken)

Rely on AI-powered vector search as the core differentiator

Rejected because: Glean treats vector search as useful but insufficient; user experience depends on ranking rigor, IR, synonyms, query understanding, and personalization.

Ship only a search engine

Rejected because: A search box alone is easy for employees to forget, so the product became a portal with feeds, mentions, Go links, and proactive surfaces.

Train LLMs to answer fresh enterprise questions directly

Rejected because: Models cannot be retrained quickly or cheaply enough for fresh enterprise data; search-backed retrieval remains necessary.

Build document Q&A before solving indexing and user workflow

Rejected because: The interview describes document-Q&A startups discovering they need all-document indexing and enterprise search as the real product.

Patterns

hybrid-search (retrieval)

Glean combines traditional IR, synonym/query understanding, acronym expansion, document understanding, and vector search instead of relying on embeddings alone.

retrieval-ranking (retrieval)

The system depends on slow, rigorous ranking-tuning work and personalization rather than a single advanced model.

query-rewriting (retrieval)

Query understanding, synonyms, and acronym expansion are treated as independent retrieval subproblems that improve internal search.

rag-retrieval (generation)

LLM answers depend on retrieval in the backend so current enterprise context can be provided to the model.

synthetic-data-generation (data)

The interview highlights using GPT-4 to generate synthetic NER training data as a practical production NLP pattern.

model-distillation (modeling)

Large-model synthetic data can be distilled into smaller, faster, cheaper task-specific models.

latency-optimization (performance)

The interview treats search and chat latency as a direct engagement risk, so LLM features must optimize response time.

Tradeoffs

Use hybrid retrieval

Chose Traditional IR plus vector search and ranking over Embeddings or LLMs as the only retrieval mechanism.

Rationale: Mature search needs many tuned subcomponents; vector search is useful but not sufficient.

Become an employee portal

Chose Search plus feed, mentions, Go links, and proactive surfaces over A standalone enterprise search box.

Rationale: The portal creates repeated daily value and retention that search alone may not.

Experiment with LLM chat based on user response

Chose Launch what users like best over Shipping chat because it is fashionable.

Rationale: Glean positions LLM features as user-tested product additions, not technology-first mandates.

Use search-backed generation for freshness

Chose RAG over retraining for current information over Expecting an LLM to memorize fresh enterprise data.

Rationale: Enterprise information changes too quickly for constant retraining, so fresh context must be retrieved at query time.

Gotchas (what bit them)

AI buzzwords can obscure whether search is actually better.

Root cause: Users may see one or two good queries and one bad query and still be unable to evaluate whether the engine is genuinely smart.

Resolution: Invest in painstaking ranking tuning and honest evaluation rather than technology marketing alone.

Pure search has weak retention as an enterprise product.

Root cause: Employees may use a new search tool once and forget it unless it becomes part of their daily portal workflow.

Resolution: Add portal features such as feed, mentions, trending docs, collections, and Go links.

Fresh enterprise information cannot be solved by retraining LLMs fast enough.

Root cause: LLMs cannot constantly index new sources and serve current answers cost-efficiently.

Resolution: Use search-backed retrieval or RAG to provide fresh context at answer time.

Document-Q&A startups often rediscover enterprise search the hard way.

Root cause: Users do not naturally ask every question to a fixed document set, and answering across all documents requires indexing, permissions, and search infrastructure.

Resolution: Start from the user's information-retrieval problem and build indexing before layering LLM answers on top.

Final model-training cost understates true experimentation cost.

Root cause: Architecture exploration, hyperparameter tuning, failed runs, and debugging can cost much more than the final successful run.

Resolution: Account for total experimentation cost, not only the final training step.

Evaluation

The case emphasizes long-term relevance tuning and user-product evaluation rather than one benchmark. Glean evaluates search by the real user experience of ranking, personalization, retention, and whether portal features draw employees back. The broader interview also calls out latency-engagement tradeoffs, freshness, cost transparency, and the need to test whether LLM features solve real problems rather than merely demonstrate technology.

Metrics: Search result relevance and ranking quality · User retention from employee portal usage · Personalization fit by role, team, authorship, and interactions · Freshness of indexed enterprise data · Latency versus engagement · Infrastructure and model-training cost · Customer productivity argument such as 10 to 20 searches per day

Outcomes

• Glean reached a $1 billion valuation in 2022.
• The source names customers or logos including Databricks, Canva, Confluent, Duolingo, Samsara, and Fortune 50 companies.
• The product evolved from enterprise search into a broader employee portal.
• The architecture treats LLMs and vector search as components inside a larger retrieval and ranking system.
• The interview does not report precision, recall, NDCG, latency, daily active users, or query-volume metrics.

Concrete scale includes company founding in 2019, a 2022 $1b valuation, companies using 10 to 100 SaaS apps, all mentions from Slack/Jira/GitHub/email in one place, and a productivity framing around 10 to 20 searches per employee per day. Search-quality metrics are not disclosed.