Fine-tuning Custom Embedding Models for Enterprise Search

glean · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Tech

Glean needed enterprise search embeddings that understand each customer's internal dialect, permissions, and heterogeneous enterprise data sources.

Constraints:

• enterprise data differs across customers and applications
• search quality underpins RAG answer quality
• customer data requires strong security and governance
• models need to adapt as company language changes

Architecture

Glean's enterprise AI stack starts by normalizing customer content from Google Drive, GitHub, Jira, Confluence, Slack, meetings, and other applications into a unified per-customer data model. Search sits underneath the assistant and agent layers, so Glean adapts a strong general embedding model into a company-specific language model through continued pre-training, then converts it into a search embedding model with task-specific pairs. Training signals include document title/body pairs, anchor links, co-access behavior, public embedding datasets, synthetic QA pairs for sparse customers, and production query-click or proxy feedback. Focused model-evaluation suites check specific query behaviors before improvements flow back into semantic search, RAG answers, and agents.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  enterprise-connectors["enterprise-connectors"] --> unified-data-model["unified-data-model"]
  unified-data-model --> continued-pretraining["continued-pretraining"]
  unified-data-model --> pair-generation-datasets["pair-generation-datasets"]
  pair-generation-datasets --> synthetic-qa-generator["synthetic-qa-generator"]
  continued-pretraining --> customer-specific-embedding-model["customer-specific-embedding-model"]
  pair-generation-datasets --> customer-specific-embedding-model
  customer-specific-embedding-model --> glean-search["glean-search"]
  glean-search --> glean-assistant["glean-assistant"]
  glean-search --> glean-agents-layer["glean-agents-layer"]
  glean-search --> query-click-feedback["query-click-feedback"]
  query-click-feedback --> customer-specific-embedding-model
  customer-specific-embedding-model --> automated-eval-suite["automated-eval-suite"]

Components

Data flow

Customer content is pulled from enterprise applications into a unified data model. Glean starts with a strong general embedding model, continues pre-training on company data to learn the customer's dialect, and then trains the model for search using positive pairs from title/body data, document anchors, co-access sessions, public datasets, synthetic QA, and eventually query-click or citation-click feedback. The resulting customer-specific embedding model powers Glean Search, which in turn supplies context to the assistant and agent layers. Model changes are checked with focused evaluations for query behaviors such as paraphrase understanding before being treated as search-quality improvements.

Why this architecture

Glean makes search the foundation because poor retrieval causes hallucinations and employee confusion in RAG systems. Customer-specific models are favored over generic large models because enterprises have private dialects, project names, heterogeneous applications, and security boundaries. The two-stage training path separates domain adaptation from search-task optimization: masked language modeling teaches company language, but pair-based embedding training is needed for retrieval. Evaluation is decomposed into focused model unit tests because end-to-end enterprise RAG has many moving parts and hundreds of customer-specific models.

Models used: custom embedding models · BERT-based language models · LLMs for synthetic question generation

Scale

• Glean builds a custom embedding model for each customer.
  “glean builds custom embedding models for each one of our customers because each one of our customers is very different and unique”transcript, stated
• The approach produced a reported 20% search-system improvement after six months of learning from feedback and company changes.
  “after 6 months you can see like a 20% improvement from our overall search system just by gaining like by learning from user feedback adapting to the nuances of the company”transcript, stated
• The evaluation problem scales to hundreds of customer-specific models.
  “if you have hundreds of customers you have hundred the models you need to think about managing and thinking about how you can do a robust evaluation for all those different models”transcript, stated
• Synthetic QA generation is reserved for small corpuses or low-activity customers where real user signals are sparse.
  “synthetic data especially for really small corpuses that might not have a ton of documents that you to be able to leverage”transcript, implied

Alternatives considered (roads not taken)

Using only large general-purpose LLMs or generic embedding models

Rejected because: Generic models do not reliably capture customer-specific language, project names, application context, and private enterprise dialects.

Stopping after continued pre-training with masked language modeling

Rejected because: Masked language modeling adapts company language, but search needs task-specific pair training to make embeddings perform well.

Treating every Slack message as its own training document

Rejected because: Individual messages are often too short and context-poor; Glean models threads or short-time-window conversations as documents instead.

Evaluating only the whole RAG system end to end

Rejected because: The full RAG stack has too many moving parts, so Glean also unit-tests focused embedding-model behaviors to isolate improvements.

Patterns

fine-tuning-adaptation (model-operations)

Glean builds a separate custom embedding model for each customer to capture company-specific language.

data-quality-loop (feedback)

User feedback and new company documents are treated as ongoing signals for improving the embedding model.

rag-retrieval (retrieval)

Search quality is treated as the foundation for enterprise RAG because poor retrieval leads to hallucinations and confusion.

Tradeoffs

Build smaller customer-specific embedding models instead of relying only on generic large models.

Chose fine-tuned per-customer embedding models over large general LLMs or generic embedding models.

Rationale: Company-specific language and privacy constraints make generic models less reliable for enterprise search.

Train search embeddings with task-specific pairs after domain pre-training.

Chose pair-based embedding fine-tuning over masked-language-modeling adaptation alone.

Rationale: Masked language modeling teaches company language, but does not directly optimize retrieval quality.

Use focused model unit tests alongside end-to-end RAG evaluation.

Chose component-level embedding evaluations over only holistic end-to-end RAG evaluation.

Rationale: The RAG system has many moving parts; focused evaluations make it easier to isolate which customer models or query patterns need work.

Gotchas (what bit them)

Generic embedding models miss customer-specific language.

Root cause: Each enterprise has its own internal terminology, project names, and data structures.

Resolution: Glean performs continued pre-training on company data before training search embeddings.

Evaluation

Glean uses user feedback and search behavior such as query-click pairs as signals for embedding-model quality.

Metrics: search quality · query-click pairs · user feedback

Outcomes

• The transcript reports about a 20% search-system improvement after six months of learning from feedback and company changes.

The approach implies operating custom models across many customers.