Implementing Question-Answering Over Sales Conversations with Deal Me at Gong

gong · Tech · 2023 · source Agentic: no · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Natural-language question answering over B2B sales conversation and deal data

Gong needed a way for sales managers and representatives to ask free-form questions over complex deal histories that include calls, emails, CRM metadata, pricing changes, stakeholders, and long sales cycles. The system had to answer quickly, avoid hallucinating unsupported deal facts, cite source material, manage long contexts and cost, and decide when a purpose-built Gong feature was better than an LLM answer.

Constraints:

• a single enterprise deal can contain hundreds of emails and thousands of call transcripts
• answers needed to synthesize calls, emails, CRM metadata, notes, and summaries
• the team targeted approximately 5 seconds for end-to-end responses
• hallucinations are especially problematic in sales decisions
• customers demanded source traceability for answers
• Gong served approximately 4,000 customer companies with thousands of users each, making LLM cost a major concern
• model versions and prompt behavior changed over time

Architecture

Gong built Ask Me as a natural-language Q&A layer over unified deal intelligence data. The architecture first normalizes calls, emails, CRM metadata, meeting notes, and summaries into a unified API. At query time, the system uses prompt engineering, metadata injection, query rewriting, HyDE-style retrieval, and source-by-source querying rather than stuffing an entire deal history into one prompt. Parallel source results are merged by an aggregation chain into a final answer, then validated against source data and decorated with citations that link back to call or email locations. Usage analysis later added a routing layer: questions that match existing specialized features are directed there, while novel questions receive the full LLM treatment.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Sales_user["Sales user"] --> Ask_Me_UI["Ask Me UI"]
  Ask_Me_UI --> Query_router["Query router"]
  Query_router --> Specialized_Gong_features["Specialized Gong features"]
  Query_router --> Prompt_engineering_layer["Prompt engineering layer"]
  Prompt_engineering_layer --> Unified_deal_data_API["Unified deal data API"]
  Unified_deal_data_API --> Call_transcript_store["Call transcript store"]
  Unified_deal_data_API --> Email_store["Email store"]
  Unified_deal_data_API --> CRM_metadata_store["CRM metadata store"]
  Unified_deal_data_API --> Meeting_notes_and_summaries["Meeting notes and summaries"]
  Prompt_engineering_layer --> HyDE_retrieval_layer["HyDE retrieval layer"]
  HyDE_retrieval_layer --> Source_query_workers["Source query workers"]
  Prompt_engineering_layer --> Source_query_workers
  Call_transcript_store --> Source_query_workers
  Email_store --> Source_query_workers
  CRM_metadata_store --> Source_query_workers
  Source_query_workers --> Aggregation_chain["Aggregation chain"]
  Aggregation_chain --> Validation_layer["Validation layer"]
  Validation_layer --> Citation_resolver["Citation resolver"]
  Call_transcript_store --> Citation_resolver
  Email_store --> Citation_resolver
  Citation_resolver --> Ask_Me_UI
  Prompt_engineering_layer --> QA_and_evaluation_pipeline["QA and evaluation pipeline"]
  Validation_layer --> QA_and_evaluation_pipeline
  Cost_and_latency_controls["Cost and latency controls"] --> Query_router
  Cost_and_latency_controls --> Source_query_workers

Components

Data flow

A sales user asks a question about a deal. The system enriches the query with deal context, user role, and deal stage, then queries selected sources such as conversations, emails, and CRM metadata at conversation or email-batch granularity. HyDE and query rewriting improve retrieval, individual source answers are collected, and a synthesis step produces the final response. A validation layer checks claims against source data, citations link claims to exact transcripts or emails, and routing logic may send common questions to specialized Gong features instead of the free-form LLM path.

Why this architecture

The architecture avoids a single huge prompt over an entire deal history. Querying at conversation or email-batch granularity improves accuracy and keeps context manageable, parallel aggregation helps meet the 5-second target, validation and citations build trust, and routing common questions to specialized features reduces cost while improving UX.

• Dramatically reduces costs • Provides better user experience for common queries"

— summary, stated

Models used: large language models for deal Q&A · HyDE hypothetical answer/question generation · fine-tuned models in specialized Gong features · GPT-3.5 · general LLMs for synthesis and validation

Scale

• A single B2B deal can span dozens of people and many months.
  “Involve dozens of people from both the selling and buying organizations • Span many months of sales activity”summary, stated
• A single deal can generate hundreds of emails and thousands of call transcripts.
  “Generate hundreds of emails and thousands of call transcripts • Include complex pricing negotiations with frequent changes”summary, stated
• The end-to-end response target was approximately 5 seconds.
  “The team set a target of approximately 5 seconds for end-to-end response time.”summary, stated
• The product had roughly 4,000 customer companies, each with thousands of potential users.
  “With approximately 4,000 customer companies, each containing thousands of users who might make multiple queries per day”summary, stated
• Query analysis found approximately 70% of user questions matched existing specialized Gong features.
  “The team discovered that approximately 70% of questions users asked were for information that Gong already provided through specialized, purpose-built features”summary, stated
• Only the remaining 30% of truly novel questions needed the full LLM path.
  “Only the 30% of truly novel queries get the full LLM-based treatment”summary, stated

Alternatives considered (roads not taken)

Stuff entire deal histories into a single prompt

Rejected because: Long contexts caused models to lose information and hurt accuracy, so the team queried at conversation or email-batch granularity.

Free-form Q&A for all user questions

Rejected because: Most questions matched existing purpose-built features with better accuracy, UX, and cost profile.

Return unsupported answers without traceability

Rejected because: Customers needed to trust answers, so v2 added source citations down to transcript and audio locations.

Sequentially query every possible source

Rejected because: The 5-second latency target forced the team to choose sources carefully and parallelize processing.

"This constraint was described as "very limiting" and required careful optimization of:

• Which sources to query (can't query everything)"

Assume prompts remain stable across model versions

Rejected because: Model version changes changed prompt behavior, so Gong needed QA, testing, and adaptation infrastructure.

• Comprehensive QA and testing infrastructure • Continuous evaluation pipelines"

— summary, stated

Patterns

rag-retrieval (knowledge-architecture)

Ask Me retrieves and synthesizes deal evidence from calls, emails, CRM metadata, notes, and summaries rather than answering from the model alone.

"Data Unification Pipeline

The first major technical challenge was unifying all deal data into a format that could be fed to an LLM."

query-rewriting (retrieval)

The prompt layer can reformulate the user's question with deal context before retrieval and answer generation.

prompt-chaining (workflow-orchestration)

The system queries individual sources, aggregates intermediate answers, and synthesizes a final response through a multi-step chain.

guardrails-io-filtering (quality)

A post-generation validation layer verifies claims against source data before answers are returned.

cost-aware-routing (operations)

Gong routes common questions to cheaper specialized features and reserves LLM inference for novel queries.

latency-optimization (operations)

The team optimized source selection, parallelism, and model choice to meet a roughly 5-second response target.

eval-driven-development (quality)

Model version drift forced comprehensive QA, continuous evaluation, and rapid prompt/model adaptation.

"This necessitated building:

• Comprehensive QA and testing infrastructure • Continuous evaluation pipelines"

retrieval-ranking (retrieval)

HyDE improves matching between questions and relevant content by generating hypothetical answers or questions for embedding search.

Tradeoffs

Context granularity

Chose conversation-level or email-batch querying over entire deal histories in one prompt.

Rationale: Smaller source units improved accuracy and kept context manageable while still supporting synthesis through aggregation.

Hybrid product routing

Chose route common queries to purpose-built features over send every question through Ask Me's full LLM pipeline.

Rationale: Specialized features often had higher accuracy, better UX, and lower cost than general LLM answers.

Latency target

Chose approximately 5-second answers with careful source and parallelism choices over querying all possible sources for maximum completeness.

Rationale: The response-time constraint was limiting, so source selection and parallel processing were necessary.

Source-backed explainability

Chose citations to exact call/email evidence over opaque natural-language answers.

Rationale: Users needed to trust and verify answers, so the product had to expose supporting source locations.

Ship and learn from real query patterns

Chose launch quickly and analyze actual usage over fully anticipating customer needs before deployment.

Rationale: Actual user questions revealed a large gap between free-form Q&A expectations and better existing product flows.

Gotchas (what bit them)

A single prompt over all deal data was too large and inaccurate.

Root cause: Deal histories can contain hundreds of emails, thousands of call transcripts, and metadata; long-context models lost information in the middle.

Resolution: Query at approximately the conversation or email-batch level and aggregate the results.

Hallucinated deal facts could mislead sales users.

Root cause: The model might fabricate entities or events, such as a person who never joined the deal.

Resolution: Add a validation layer that checks claims against source data before returning responses.

Trust required citations, not just fluent answers.

Root cause: Customers needed to verify where claims came from in calls or emails.

Resolution: v2 added source tracing to call/email locations, transcript segments, and audio clips.

LLM cost became a product architecture constraint.

Root cause: Thousands of users across thousands of customer companies could issue multiple queries per day.

Resolution: Route common questions to existing specialized features and optimize model/source selection for the remaining novel questions.

Provider model versions changed prompt behavior.

Root cause: Prompts that worked on one GPT-3.5 version could fail or behave differently months later.

Resolution: Build comprehensive QA, continuous evaluation, and fast adaptation paths for new model versions.

A general LLM UI exposed product gaps rather than simply solving them.

Root cause: User queries showed that many needs were already better served by specialized features, but users did not always know those features existed.

Resolution: Use query analysis to route users and reveal where the product's existing capabilities should be surfaced.

Evaluation

Gong's evaluation approach combined hallucination validation against source data, citation traceability, QA and testing infrastructure for model version changes, continuous evaluation pipelines, and usage analysis of real user queries. The summary also reports a strategic evaluation result: 70% of user questions were better served by specialized features, reshaping the product.

Metrics: 5-second end-to-end latency target · claim validation against source data · citation/source traceability · query match rate to specialized features: approximately 70% · novel query share requiring LLM treatment: approximately 30% · model version QA and continuous evaluation

• Comprehensive QA and testing infrastructure • Continuous evaluation pipelines"

Outcomes

• Gong built Ask Me as a natural-language Q&A feature over deal data.
• The system added v2 citation support after customer demand for traceability.
• Usage analysis showed approximately 70% of questions matched existing specialized features.
• The product strategy shifted to intelligent routing for common questions and full LLM treatment for the remaining 30% of novel queries.
• The case demonstrates that a general LLM interface can become a discovery mechanism for product gaps and routing opportunities.

Because this extraction is summary-only, the architecture should be weighted below transcript- or article-backed cases. Still, the summary is detailed enough to support a reconstructed architecture and several high-value system patterns: query routing, validation, citations, cost-aware routing, and context-size management.