Building a Production-Grade GenAI Customer Support Assistant with Comprehensive Observability

elastic · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Observability for a production GenAI customer support assistant

Elastic needed to launch and operate a RAG-powered Support Assistant with enough observability to prove adoption, detect regressions, debug LLM and RAG failures, manage abuse, and respond quickly when customer-facing chatbot behavior degraded.

Constraints:

• monitoring workloads needed to stay separate from production and staging data clusters
• the team needed request traces, logs, Azure OpenAI metrics, synthetics, infrastructure monitoring, dashboards, and alerts in one operational view
• streaming LLM responses required measuring time to first token and total completion duration
• client-side first-generation timeouts were initially invisible to server-side APM
• the assistant needed to decline out-of-scope and billing questions consistently enough to monitor rejections
• RAG context could create payloads larger than default server request limits

Architecture

Elastic built an observability stack around the Support Assistant using a dedicated Elastic Cloud monitoring cluster, Elastic Node APM in the Node API, Filebeat logs, a console wrapper that appends APM trace IDs, Elastic Synthetics monitors for the app and upstream dependencies, Stack Monitoring from data clusters, and Azure OpenAI integration metrics shipped through Elastic Agent on a GCP VM. Kibana dashboards track status and milestones, APM custom spans measure first-token and completion durations, alerting thresholds are derived from dashboard metrics and baselines, prompt-level decline responses are monitored, and abuse controls use endpoint-specific rate limits plus feature flags.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  support-assistant[Support Assistant] --> support-portal-frontend[Support Portal frontend]
  support-portal-frontend --> node-api-service[Node API service]
  node-api-service --> rag-search[RAG search]
  node-api-service --> azure-openai-service[Azure OpenAI service]
  node-api-service --> elastic-node-apm-client[Elastic Node APM client]
  node-api-service --> filebeat-and-log-wrapper[Filebeat and log wrapper]
  node-api-service --> feature-flag-system[Feature flag system]
  rag-search --> application-data-clusters[Application data clusters]
  elastic-node-apm-client --> monitoring-elastic-cloud-cluster[Monitoring Elastic Cloud cluster]
  filebeat-and-log-wrapper --> monitoring-elastic-cloud-cluster
  elastic-synthetics-monitors[Elastic Synthetics monitors] --> monitoring-elastic-cloud-cluster
  application-data-clusters --> stack-monitoring[Stack Monitoring]
  stack-monitoring --> monitoring-elastic-cloud-cluster
  azure-openai-service --> azure-openai-integration[Azure OpenAI integration]
  azure-openai-integration --> monitoring-elastic-cloud-cluster
  monitoring-elastic-cloud-cluster --> kibana-dashboards[Kibana dashboards]
  kibana-dashboards --> alerting-rules[Alerting rules]

Components

Data flow

Support Assistant traffic flows through the Support Portal frontend and Node API to RAG search and Azure OpenAI chat completion. The Node APM client captures transactions and custom spans, Filebeat ships correlated logs, synthetics monitors check the app and upstream services, Stack Monitoring observes data clusters, and Azure OpenAI integration sends model-service telemetry. Kibana dashboards and alerts then expose usage, latency, errors, capacity, milestones, declined requests, rate-limit hits, and payload-size failures.

Why this architecture

Elastic treated GenAI observability as a launch prerequisite rather than an afterthought. A separate monitoring cluster protected production data workloads, trace/log correlation made debugging feasible, custom spans exposed streaming-specific latency, dashboards showed usage and value, alerts converted dashboard metrics into action, and prompt/endpoint instrumentation captured LLM-specific risks such as declined requests, timeouts, abuse, and RAG payload expansion.

Models used: GPT-4 · GPT-4o · Azure OpenAI LLM service

Scale

• A bug caused one endpoint to exceed 100 transactions per minute before the fix reduced it to about 1 TPM.
  “We saw in the APM data that the throughput for one of our endpoints was well over 100 transactions per minute, which was unreasonably large for the size of our user base.”article, stated

  “We could confirm the fix when we saw the throughput reduce to a much more reasonable 1 TPM. It's also how I know we served up our 100th chat completion 21 hours post launch (you love to see it folks).”article, stated
• The assistant reached its 100th chat completion 21 hours after launch.
  “It's also how I know we served up our 100th chat completion 21 hours post launch (you love to see it folks). This post will discuss the observability needs for a successful launch, and then some unique observability considerations for a chatbot use case such as our Support Assistant.”article, stated
• The first-generation timeout threshold was 10 seconds.
  “The other thing we did to try to give our assistant a more responsive experience was to enforce a 10 second timeout on getting the first chunk of generated text back. Being able to see trends in this type of error is critical for us to be able to know if our service is reliable, or overloaded.”article, stated
• The customer rate limit was set to 20 chat completions in one hour.
  “The team agreed on imposing a limit of 20 chat completions in a one hour window. This is as many chats for our customer users in an hour as our heaviest internal users send in a whole day, while limiting any malicious users to ~11% of that theoretical max based on latency for a full completion.”article, stated
• The top internal user had sent more than 70 chats in one day during the observed week.
  “I looked at the previous week's data, and saw that our heaviest internal users had sent 10-20 chat messages per day on average with the top user sending over 70 in a single day. I also had latency metrics to tell me that the average completion time was 20 seconds.”article, stated
• The theoretical one-tab maximum was about 180 chats in one hour based on 20-second average completion time.
  “That means the theoretical max chats for a single user in an hour where they use a single tab is 180 chats in an hour. The team agreed on imposing a limit of 20 chat completions in a one hour window.”article, stated

Alternatives considered (roads not taken)

Wait until late development to add APM

Rejected because: Early APM captures development errors and sample data needed for dashboard validation and alert baselines.

Keep first-generation timeout detection in browser code

Rejected because: The server could not reliably observe client aborts, so timeout/error capture needed to move into the API layer.

Use a separate third-party service to decide whether to respond

Rejected because: Elastic initially chose prompt instructions plus a standardized decline response, while leaving a separate LLM/service as a future hardening option.

Rely on the general application-wide rate limit

Rejected because: The chat completion endpoint needed a much lower limit tuned to LLM cost, latency, customer behavior, and abuse risk.

Keep sending full RAG result content through the browser payload

Rejected because: Large RAG context caused HTTP 413 errors; the long-term plan was to send limited metadata and fetch content server-side by ID.

Patterns

observability-tracing (llmops-operations)

Elastic correlated APM transactions, custom spans, logs, synthetics, stack monitoring, Azure OpenAI metrics, dashboards, and alerts to operate the Support Assistant.

rag-retrieval (support-assistant)

The Support Assistant uses RAG search context in chat completion payloads for Elastic product support.

guardrails-io-filtering (support-scope)

The prompt declines unrelated or unsupported topics such as non-support requests and billing, and standardized declines are monitored as observable events.

latency-optimization (streaming-ux)

Elastic uses streaming responses, first-token custom spans, a 10-second first-generation timeout, and server-side abort handling to preserve perceived responsiveness.

safety-policy-enforcement (abuse-control)

The assistant is gated to logged-in support subscribers, uses a lower endpoint-specific rate limit, tracks HTTP 429s, and can block access through feature flags.

Tradeoffs

Use a dedicated monitoring cluster

Chose separate Elastic Cloud cluster for observability data over mixing monitoring data with production and staging data clusters.

Rationale: Separating monitoring from application data reduces interference and keeps observability data centralized for analysis.

Add prompt instructions before user input for declines

Chose prompt-level hardening with standardized decline response over separate third-party or dedicated LLM moderation call in the first implementation.

Rationale: Prompt-level scope control was simpler to launch, while the team kept service-based classification as a future hardening option.

Set a lower assistant-specific rate limit

Chose 20 chat completions per one-hour window over the generous general application rate limit.

Rationale: The assistant endpoint had higher cost and abuse risk, but the limit still needed to avoid penalizing enthusiastic legitimate users.

Use boosted payload size as a short-term RAG fix

Chose increase accepted payload size over blocking launches on a deeper orchestration refactor.

Rationale: Increasing the limit eliminated HTTP 413s quickly, but the team recognized server-side fetch-by-ID as the long-term design.

Page only on actionable alerts

Chose different notification thresholds and channels based on urgency/actionability over paging for every dashboard anomaly.

Rationale: Unactionable alerts train teams to ignore alerting noise.

Gotchas (what bit them)

Client-side first-token timeouts were invisible to server APM.

Root cause: The timeout ran in browser code, and the API server had no practical built-in way to know the client aborted.

Resolution: Elastic moved the timeout and AbortController into the API layer, captured the error with APM, and sent an Error event before closing the stream.

A generic error stream close left the client stuck in loading state.

Root cause: Closing the server connection without sending a proper generated event gave the browser no signal to update UI state.

Resolution: The server sends an Error event before closing the streaming response.

Declined responses were hard to monitor until standardized.

Root cause: The LLM declined inconsistently, making it difficult to count and investigate rejections.

Resolution: Elastic added a standard decline response and logged matches with captureError.

RAG context exceeded default payload limits.

Root cause: The browser sent full RAG result content plus the user's input to the chat completion endpoint.

Resolution: Elastic increased accepted payload size as a short-term fix and planned a server-side fetch-by-ID refactor.

A data-loading bug looked like unexpectedly high throughput.

Root cause: The system loaded the same data repeatedly from the server.

Resolution: APM endpoint throughput made the anomaly visible and confirmed the fix when TPM dropped.

Evaluation

Elastic evaluated the Support Assistant operationally through status dashboards, APM traces, custom spans, alert thresholds, milestone dashboards, synthetic checks, declined-request tracking, HTTP status monitoring, rate-limit metrics, and production bug/incident signals.

Metrics: chat completions · unique users · returning users · errors and HTTP status codes · RAG search latency · chat completion latency · time to first generated token · total completion duration · HTTP 429 rate-limit events · HTTP 413 payload-size errors · declined-request matches

Outcomes

• Elastic caught and fixed a repeated data-loading bug after APM showed endpoint throughput over 100 TPM.
• The observability stack confirmed the fix when throughput fell to about 1 TPM.
• The milestones dashboard showed the 100th chat completion 21 hours after launch.
• Moving first-generation timeout handling into the API made timeout errors observable and recoverable in the UI.
• Monitoring found HTTP 413 responses from oversized RAG payloads, and increasing the accepted payload size stopped those transactions after release.
• No user hit the 20-completion hourly rate limit in the first couple of weeks after launch.

The observability setup tracks launch-to-now usage, all-time/7-day/30-day milestone windows, RAG and completion latency, 10-second first-token timeouts, 20-completion hourly rate limits, HTTP status codes, users with most errors, and Azure OpenAI service logs/metrics.