Building an Internal AI-Powered Customer Reference Discovery Platform

databricks · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Internal customer-reference discovery for sales and marketing

Databricks needed sales and marketing teams to find credible, relevant customer proof points across thousands of customer stories scattered through YouTube talks, databricks.com case studies, internal slides, LinkedIn articles, and Medium posts. Manual search and tribal knowledge caused overused references, missed industries or use cases, and uncertainty about whether a found story had credible outcomes.

Constraints:

• customer stories lived across a dozen platforms with no unified search
• users needed to evaluate story quality, not just find text matches
• references could be public or internal-only and needed access tags
• sales and marketing users needed non-technical access through an app
• simple keyword lookups needed low latency while questions needed reasoning and sources
• production use required logs, user identity, conversation history, quality assurance, and follow-up
• development needed reproducible CI/CD and local-to-production authentication
• teams wanted future API and MCP access so Reffy could be embedded into other workflows

Architecture

Databricks built Reffy as a full-stack agentic RAG application on the Databricks platform. Lakeflow Jobs orchestrate ETL notebooks that ingest public and internal customer-story sources into a Bronze Delta Lake table governed by Unity Catalog. AI Functions prompt Gemini 2.5 to apply a 31-point quality score, extract metadata, and tag stories as public or internal-only, producing a Silver table. A Gold table filters low-quality stories and builds a retrieval summary column, then synchronizes it to Databricks AI Search. A DSPy tool-calling agent uses hybrid keyword/semantic search, Databricks re-ranking, MLflow logging, and Model Serving/Agent Framework deployment. A React/FastAPI Databricks App uses Lakebase for chat history, logs, and identities, while AI/BI dashboards monitor usage, latency, themes, and content gaps.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Source_story_repositories["Source story repositories"] --> Lakeflow_ETL_notebooks["Lakeflow ETL notebooks"]
  Google_Apps_Scripts["Google Apps Scripts"] --> Lakeflow_ETL_notebooks
  Lakeflow_ETL_notebooks --> Bronze_Delta_table["Bronze Delta table"]
  Bronze_Delta_table --> Unity_Catalog["Unity Catalog"]
  Bronze_Delta_table --> AI_Functions_quality_scorer["AI Functions quality scorer"]
  AI_Functions_quality_scorer --> Silver_quality_table["Silver quality table"]
  Silver_quality_table --> Unity_Catalog
  Silver_quality_table --> Gold_retrieval_table["Gold retrieval table"]
  Gold_retrieval_table --> Databricks_AI_Search_index["Databricks AI Search index"]
  Databricks_AI_Search_index --> Databricks_re_ranker["Databricks re-ranker"]
  Databricks_AI_Search_index --> DSPy_customer_stories_agent["DSPy customer stories agent"]
  Databricks_re_ranker --> DSPy_customer_stories_agent
  DSPy_customer_stories_agent --> MLflow["MLflow"]
  MLflow --> Model_Serving_Agent_Framework["Model Serving Agent Framework"]
  Model_Serving_Agent_Framework --> React_FastAPI_Databricks_App["React FastAPI Databricks App"]
  Databricks_Python_SDK["Databricks Python SDK"] --> React_FastAPI_Databricks_App
  React_FastAPI_Databricks_App --> Lakebase_conversation_store["Lakebase conversation store"]
  Lakebase_conversation_store --> AI_BI_monitoring_dashboard["AI BI monitoring dashboard"]
  Databricks_Asset_Bundles_CI_CD["Databricks Asset Bundles CI CD"] --> React_FastAPI_Databricks_App
  Databricks_Asset_Bundles_CI_CD --> Model_Serving_Agent_Framework

Components

Data flow

Lakeflow Jobs run notebooks that scrape YouTube transcripts, LinkedIn/Medium articles, public databricks.com customer stories, and Google Apps Script exports from internal slides/docs into a Bronze Delta table under Unity Catalog. AI Functions call Gemini 2.5 to score stories against a 31-point rubric, extract metadata, quotes, and shareability tags, and write a Silver table. The Gold layer filters low-scoring stories and concatenates key fields into a summary column that is synchronized to an AI Search index. A DSPy agent queries the index with hybrid keyword/semantic retrieval and a re-ranker, is logged in MLflow, and is served through Model Serving. React/FastAPI surfaces the app; Lakebase stores conversations, logs, and user identities; Lakeflow plus AI Functions summarize logs into dashboard metrics and gap analysis.

Why this architecture

Reffy uses a full data-engineering-plus-agent stack because the core problem is not just answer generation: stories first have to be collected, governed, scored, tagged, filtered, summarized for retrieval, indexed, searched, cited, monitored, and embedded into an app that sales and marketing can use. The hybrid keyword/LLM interaction avoids using expensive reasoning for simple lookup, while Lakebase logs and AI/BI dashboards turn usage into a feedback loop for story curation.

Models used: Gemini 2.5 · DSPy tool-calling agent · Databricks AI Search · Databricks re-ranker · Databricks Model Serving Agent Framework · AI Functions

Scale

• Reffy indexes over 2,400 Databricks customer references.
  “discover and analyze over 2,400 Databricks customer references, delivering personalized responses, cross-story analysis, quotes, and more.”article, stated
• More than 1,800 Databricks sales and marketing users used Reffy in its first two months.
  “In its first two months, over 1,800 people in Databricks sales & marketing have run upward of 7,500 queries on Reffy.”article, stated
• Users ran more than 7,500 queries in the first two months.
  “In its first two months, over 1,800 people in Databricks sales & marketing have run upward of 7,500 queries on Reffy.”article, stated
• The source corpus includes thousands of YouTube talks plus case studies, slides, LinkedIn articles, and Medium posts.
  “Databricks has thousands of YouTube talks, case studies on databricks.com, internal slides, LinkedIn articles, Medium posts.”article, stated
• The quality classifier applies a 31-point scoring system.
  “we classify the text by applying a rigorous 31-point scoring system (developed by our Value team) to each story”article, stated
• Pure keyword searches return top results in under two seconds.
  “if you just enter a few keywords, Reffy will return top results in less than two seconds.”article, stated

Alternatives considered (roads not taken)

Continue Slack asks, remembered folders, and manual search

Rejected because: Manual reference discovery led to good-enough choices, overused references, and missed stories.

Search all stories without quality scoring

Rejected because: Databricks needed to know whether stories had credible outcomes and why Databricks mattered, so low-quality stories were scored and filtered.

Use one interaction mode for every query

Rejected because: Reffy separates fast keyword lookup from longer-form LLM answers with reasoning and sources.

Deploy the serving endpoint on GPU infrastructure

Rejected because: Most processing happens on the model provider side, so a small CPU instance is enough and reduces infrastructure cost.

Keep Reffy only as a standalone application

Rejected because: Databricks plans API and MCP access so teams can use customer intelligence inside existing workflows and tools.

Patterns

rag-retrieval (retrieval)

Reffy builds a retrieval-ready Gold table, syncs it to AI Search, and lets a DSPy agent retrieve customer stories with hybrid keyword and semantic search plus re-ranking.

tool-use-orchestration (architecture)

The application orchestrates Lakeflow, Unity Catalog, AI Functions, AI Search, re-ranking, MLflow, Model Serving, React/FastAPI, Lakebase, Databricks Apps, and GitHub Actions.

eval-driven-development (evaluation)

A 31-point rubric and Gemini 2.5 AI Function score each story for business challenge, solution, outcome credibility, and Databricks-specific value before retrieval.

observability-tracing (operations)

Lakebase logs are processed into dashboards showing DAU, response times, recent inputs/responses, themes, gap analysis, and story-quality distributions.

structured-output-contract (data)

AI Functions extract structured fields such as country, industry, products, competition, quotes, and public/internal tags into the Silver table.

model-routing (interaction)

Reffy routes user intent between pure keyword retrieval for fast lookup and longer-form LLM reasoning with sources for questions.

Tradeoffs

Build a medallion data pipeline before the agent layer.

Chose Bronze/Silver/Gold story processing with scoring and retrieval summaries over pointing an agent directly at scattered source documents.

Rationale: The agent needed high-quality, filtered, metadata-rich, retrieval-optimized story records to answer well.

Use AI Functions and Gemini 2.5 to score story quality.

Chose LLM-based quality and metadata extraction over treating every customer story as equally useful.

Rationale: The quality score lets Reffy filter weak stories and extract structured fields that improve retrieval, analytics, and shareability handling.

Support both keyword lookup and longer-form agent answers.

Chose dual-mode hybrid search and LLM reasoning over one answer style for every query.

Rationale: Fast keyword lookup serves simple searches in under two seconds, while question answering can spend more time producing reasoned, cited responses.

Serve on small CPU instances.

Chose CPU Model Serving endpoint over GPU infrastructure for the serving layer.

Rationale: Gemini handles most processing through the provider, so the serving layer mainly orchestrates retrieval and requests.

Gotchas (what bit them)

A reference library can fail even when the perfect story exists.

Root cause: Stories were distributed across many platforms with no unified search, so users relied on Slack, half-remembered folders, and who they knew.

Resolution: Reffy consolidated sources into governed tables and made them searchable through a RAG-based agent.

Retrieval quality depends on story quality, not just embedding similarity.

Root cause: A found story might have vague claims or weak outcomes even if it matches the query.

Resolution: Databricks scores each story with a 31-point rubric, filters low-quality stories, and extracts quality metadata before indexing.

The glamorous agent layer can distract teams from the data-engineering work that makes the agent useful.

Root cause: Multiple teams had solved pieces of the problem while gravitating toward the AI layer.

Resolution: The team centered ETL, story scoring, and retrieval-oriented data structuring as core architecture, not support work.

A conversational app needs operational state and logs, not just a stateless model endpoint.

Root cause: Users return to conversations, quality assurance needs logs, and product teams need to understand usage and gaps.

Resolution: Reffy persists conversation history, logs, and user identities in Lakebase and processes those logs into monitoring dashboards.

Notebook-to-serving iteration can slow agent development if every prompt or tool change requires redeploying.

Root cause: Agent development needs fast experiments with prompts, retrieval, and tool behavior.

Resolution: The team used DSPy so agents could be tested iteratively in Databricks notebooks before Model Serving deployment.

Evaluation

Databricks evaluates Reffy through the upstream 31-point story-quality scoring system, hybrid retrieval with re-ranking, MLflow logging, usage telemetry in Lakebase, AI/BI dashboards for DAU and response time, recent input/response review, AI-generated themes and gap analysis, and static quality analysis across industries and content types. The article provides adoption and operational metrics, but does not describe a formal retrieval-relevance benchmark or user-satisfaction score.

Metrics: 31-point story quality score · Daily Active Users · average response times · recent inputs and responses · themes and gap analysis · story quality across industries · story quality across content types · over 1,800 users in first two months · upward of 7,500 queries in first two months · keyword results in less than two seconds

Outcomes

• Reffy made over 2,400 customer references discoverable and analyzable.
• More than 1,800 Databricks sales and marketing users ran more than 7,500 queries in the first two months.
• Databricks reports more relevant and consistent storytelling, faster campaign execution, and customer proof used at scale.
• The system revealed high user interest in Agent Bricks and Lakebase stories, guiding future reference curation.
• The team plans to expose Reffy through an API and MCP server so customer intelligence can reach other workflows and tools.

Reffy launched in December 2025, was described in March 2026, and served more than 1,800 sales and marketing users with upward of 7,500 queries in its first two months. It indexes over 2,400 customer references drawn from thousands of YouTube talks, website case studies, internal slides, LinkedIn, and Medium, with keyword results under two seconds.