UI/UX Design Considerations for Production GenAI Chatbots

elastic · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production GenAI chatbot UX

Elastic needed a customer support chatbot UI that made long LLM/RAG latency, streaming stalls, and multi-source context control tolerable for users.

Constraints:

• Total end-to-end latency was 5.1-11.5 seconds.
• First streamed byte could take 3-6 seconds.
• Streaming connections could stall after a successful 200 OK.
• Users needed control over which data sources inform responses.
• The interface had to work within Elastic's component library.

Architecture

Elastic built a custom chat interface with Elastic UI components, branded animated loading states, streaming support, an AbortController-based killswitch after 10 seconds of stream inactivity, and a prepended context menu that lets users choose data sources such as case history and knowledge base for RAG responses.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  custom_chat_interface["Custom chat interface"] --> context_menu["Context menu"]
  custom_chat_interface --> loading_animation["Loading animation"]
  custom_chat_interface --> streaming_request_handler["Streaming request handler"]
  context_menu --> rag_search_service["RAG search service"]
  rag_search_service --> llm_response_stream["LLM response stream"]
  llm_response_stream --> streaming_request_handler
  streaming_request_handler --> killswitch_timeout_controller["Killswitch timeout controller"]

Components

Data flow

A user submits a support question with selected context sources. The client sends the request to the server, RAG search runs against Elasticsearch, the server calls the LLM, streamed bytes return to the client, the loading animation keeps the user engaged, and a killswitch aborts streams that go silent for more than 10 seconds.

Why this architecture

Elastic treated UX as part of production LLMOps because latency, stalled streams, and opaque context sources can destroy trust even when retrieval and generation are technically working.

Models used: LLM · ELSER · Elastic Support Assistant

Scale

• Initial request latency was 100-500ms.
  “Initial request (client to server): 100-500ms”summary, stated
• RAG search took 1-2.5 seconds.
  “RAG search (server to Elasticsearch cluster): 1-2.5 seconds”summary, stated
• LLM call latency was 1-2.5 seconds.
  “Call to LLM (server to LLM): 1-2.5 seconds”summary, stated
• First streamed byte took 3-6 seconds.
  “First streamed byte (LLM to server to client): 3-6 seconds”summary, stated
• Total end-to-end latency was 5.1-11.5 seconds.
  “Total end-to-end latency: 5.1-11.5 seconds”summary, stated
• The killswitch triggers after 10 seconds of stream inactivity.
  “if no data was received for 10 seconds during an active stream”summary, stated

Alternatives considered (roads not taken)

Non-streaming full-response HTTP

Rejected because: The initial endpoint felt like it took forever before returning an answer.

Generic spinner

Rejected because: The team built a branded loading animation to keep users engaged.

Wait indefinitely on silent streams

Rejected because: A 10-second silence predicted failure or more than one minute of delay.

Patterns

fallback-degradation (stream failure handling)

The killswitch aborts streams that go silent for 10 seconds and returns an error instead of hanging.

human-in-the-loop-gate (context selection)

The UI lets users choose which data sources feed the RAG response.

latency-optimization (chat UX)

Streaming and loading states address long response latency.

Tradeoffs

Implement streaming plus timeout control.

Chose Streaming handler with AbortController killswitch over Plain full-response requests or indefinite streaming.

Rationale: Users need earlier feedback and bounded failure behavior.

Let users control RAG context sources.

Chose Prepended context menu over Always using all available data sources.

Rationale: User-controlled context improves transparency and relevance for multi-source support questions.

Gotchas (what bit them)

LLM streaming can hang after a successful 200 OK.

Root cause: The connection may succeed while generation stalls or fails silently.

Resolution: Abort the stream after 10 seconds of inactivity.

Users lose patience before LLMs return.

Root cause: End-to-end response latency can exceed 5-11 seconds.

Resolution: Use streaming and custom animated loading states to communicate progress.

Evaluation

Elastic evaluated the UI through latency measurements and observed streaming failure behavior during production chatbot development.

Metrics: 100-500ms initial client-server request. · 1-2.5s RAG search. · 1-2.5s LLM call. · 3-6s first streamed byte. · 5.1-11.5s total latency. · 10s stream-inactivity timeout.

Outcomes

• Elastic built a custom chat interface for a production GenAI support chatbot.
• Streaming and branded loading states improved perceived responsiveness.
• The killswitch pattern bounded silent stream failures.
• The context menu gave users control over RAG data sources.

The case reports detailed latency ranges and a 10-second stream-timeout threshold.