Building a Customer Support AI Assistant: From PoC to Production

elastic · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer support generative AI

Elastic wanted to test whether generative AI could improve support workflows by producing case summaries and draft initial replies inside the support team's existing Salesforce case-management process.

Constraints:

the proof of concept had to fit Elastic's existing Google Cloud Platform and Salesforce Service Cloud setup
support answers needed product-specific knowledge that a general LLM did not already have
the team needed low-friction user feedback from support engineers working live cases
the initial rollout had to be small enough for subjective feedback review rather than automated evaluation

Architecture

Elastic built a Salesforce-integrated proof of concept. A function accepted a Salesforce case ID, retrieved the case text, sent it to Vertex AI with an engineered prompt, and posted the generated summary or draft reply back into Salesforce Chatter. Support engineers used Chatter likes and threaded comments as the feedback loop. The results then informed the next architecture: a unified chat assistant backed by Elasticsearch/RAG rather than a simple prompt over case text.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  salesforce-service-cloud[Salesforce Service Cloud] --> case-retrieval-function[Case retrieval function]
  case-retrieval-function --> vertex-ai-prompt-runner[Vertex AI prompt runner]
  vertex-ai-prompt-runner --> salesforce-chatter-feedback[Salesforce Chatter feedback]
  salesforce-chatter-feedback --> feedback-review-process[Feedback review process]
  feedback-review-process --> support-ai-chat-assistant-roadmap[Support AI Chat Assistant roadmap]
  support-ai-chat-assistant-roadmap --> elasticsearch-retrieval-layer[Elasticsearch retrieval layer]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Salesforce Service Cloud`	Existing support case-management system and source of case details.	`Case retrieval function`, `Salesforce Chatter feedback`
`Case retrieval function`	Takes a Salesforce case ID, retrieves case text, and prepares prompt input.	`Salesforce Service Cloud`, `Vertex AI prompt runner`
`Vertex AI prompt runner`	LLM execution layer used for the initial summaries and draft replies.	`Case retrieval function`, `Salesforce Chatter feedback`
`Salesforce Chatter feedback`	Delivery and feedback surface where support users rated or commented on generated content.	`Salesforce Service Cloud`, `Vertex AI prompt runner`, `Feedback review process`
`Feedback review process`	Human review loop for analyzing likes, threaded comments, and subjective feedback.	`Salesforce Chatter feedback`, `Support AI Chat Assistant roadmap`
`Support AI Chat Assistant roadmap`	Approved next-stage architecture direction with unified chat and Elasticsearch-backed RAG.	`Feedback review process`, `Elasticsearch retrieval layer`
`Elasticsearch retrieval layer`	Planned retrieval layer for product documentation and internal knowledge base context.	`Support AI Chat Assistant roadmap`

Data flow

A support case ID triggered retrieval of Salesforce case details. The retrieved text was combined with a prompt and sent to Vertex AI. The generated case summary or draft reply was posted back to the case as a Salesforce Chatter post, where support users could like or comment on the output. The team reviewed that feedback and used the low positive-sentiment rate to justify adding Elasticsearch-backed retrieval and a unified chat interface.

3 supporting quotes

“1. The function accepted the Salesforce unique case ID as input and retrieved the case details as text. 2. The retrieved text would then be automatically sent to Vertex AI combined with the following engineered prompt:”article, stated

“The retrieved text would then be automatically sent to Vertex AI combined with the following engineered prompt:”article, stated

“3. The AI-generated response was posted to the case via a Salesforce Chatter Post. That was basically it! The lone exception was for long-running cases, where we had to break down the text into summaries of summaries. Once we landed on a design, we had this up and running in a week.”article, stated

Why this architecture

Elastic chose a Salesforce and Vertex AI proof of concept because the required infrastructure was already approved and close to the users. Chatter provided a fast feedback surface without building a separate UI. The architecture was intentionally temporary: once feedback showed that general LLM knowledge was insufficient, the team moved toward a RAG-backed chat assistant.

3 supporting quotes

“This existing setup made it straightforward to integrate our initial proof of concept with Vertex AI, which was already enabled internally and compliant with our security and privacy policies.”article, stated

“This was a critical step in the process and reduced friction in the feedback loop as users could work cases and provide their feedback in the same operational system.”article, stated

“we knew the next evolution would need to include Elasticsearch for improved response accuracy via a retrieval augmented generation architecture.”article, stated

Models used: Vertex AI LLM

Scale

The proof of concept ran for 44 days.
“## Objectively evaluating results and making decisions Days Open 44 Generated Content 940 Feedback 217”article, stated
The assistant generated 940 content pieces during the trial.
“## Objectively evaluating results and making decisions Days Open 44 Generated Content 940 Feedback 217”article, stated
Only about 16 percent of feedback was positive.
“The initial user feedback yielded a ~16% positive response rate, which was lower than expected. Reviewing subjective feedback revealed that the LLM lacked in-depth knowledge of our products, which hindered its ability to address technical support queries.”article, stated

Alternatives considered (roads not taken)

Sophisticated automated LLM accuracy evaluation during the PoC

Rejected because: The manageable data population let the team review every comment manually, so they postponed heavier evaluation methods.

“Still, we intentionally avoided that for the proof of concept as our data population was manageable, and we wanted to review every comment.”article, stated

Keeping separate AI functions instead of a unified chat interface

Rejected because: Elastic concluded that consolidating functions into one chat interface would curate inputs more consistently and support better workflows.

“we decided that the optimal experience would be to consolidate these and all other potential functions into a unified chat interface.”article, stated

Prompt-only support generation without retrieval

Rejected because: The model lacked access to product documentation and internal knowledge base articles, so the next architecture needed Elasticsearch-backed retrieval.

“the LLM lacked in-depth knowledge of our products, which hindered its ability to address technical support queries.”article, stated

Patterns

`eval-driven-development` (evaluation)

Elastic used a bounded PoC with explicit user feedback to decide whether the assistant was useful enough and what architecture had to change before production.

“Armed with this decision, we decided to build a scalable proof of concept that would allow us to operationalize these workflows for a subset of our users while including a feedback mechanism to rate and improve quality.”article, stated

`human-in-the-loop-gate` (workflow)

Support users reviewed generated text in the same case-management workflow, providing likes and comments before the team trusted the output broadly.

“we could use standard Chatter features for "likes" to identify positive sentiment and threaded responses to capture subjective feedback.”article, stated

`rag-retrieval` (retrieval)

The first PoC exposed that product-specific support work required retrieval over documentation and internal knowledge, leading Elastic toward a RAG assistant.

“lacked access to key data sources like our product documentation and internal knowledge base articles.”article, stated

`prompt-chaining` (orchestration)

The PoC separated support use cases into engineered prompts for case summaries and draft replies before considering a consolidated assistant.

“The retrieved text would then be automatically sent to Vertex AI combined with the following engineered prompt:”article, stated

Tradeoffs

Start with existing Salesforce and Vertex AI infrastructure

Chose a fast PoC integrated with approved internal systems over building a new support assistant stack first.

Rationale: Existing GCP, Service Cloud, and Vertex AI access let the team learn quickly before committing to a broader product.

“Using Vertex AI as our LLM for this proof of concept was an easy decision.”article, stated

Use Salesforce Chatter as the feedback mechanism

Chose likes and threaded replies in the operational support tool over a separate evaluation application.

Rationale: Keeping feedback in Salesforce reduced friction for support users who were already working cases there.

“users could work cases and provide their feedback in the same operational system.”article, stated

Treat the PoC result as learning rather than production readiness

Chose RAG-backed redesign over scaling the prompt-only Vertex AI workflow.

Rationale: The low positive feedback and product-knowledge gap made retrieval a prerequisite for the next system.

“The initial user feedback yielded a ~16% positive response rate, which was lower than expected.”article, stated

Gotchas (what bit them)

Generic LLM knowledge did not satisfy technical support needs.

Root cause: The model did not have Elastic product documentation or internal knowledge-base context.

Resolution: Move the next version toward Elasticsearch-backed retrieval and a support chat assistant.

“Reviewing subjective feedback revealed that the LLM lacked in-depth knowledge of our products, which hindered its ability to address technical support queries.”article, stated

Positive sentiment was much lower than expected.

Root cause: The assistant handled generic summaries better than product-specific replies.

Resolution: Use the low positive response rate as decision evidence instead of scaling the PoC unchanged.

“The model performed much better with generic summaries and responses that didn't require specific product knowledge.”article, stated

The first architecture could not evaluate accuracy at scale.

Root cause: It relied on manageable manual feedback rather than retrieval-grounded automated accuracy evaluation.

Resolution: Plan a RAG architecture that would improve response precision and support larger-scale evaluation.

“This should allow us to evaluate accuracy at scale and significantly improve the precision of our responses.”article, stated

Evaluation

Elastic evaluated the PoC through Salesforce Chatter likes and threaded comments, then manually reviewed all feedback because the trial population was manageable.

Metrics: 940 AI-generated content pieces · ~16% positive response rate · 44-day proof-of-concept window

"Days Open

Generated Content

940"