Implementing LLM Observability for Natural Language Querying Interface

honeycomb · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM observability for natural-language querying in an observability product

Honeycomb shipped a natural-language querying interface that solved many initial use cases, but the remaining long tail of paying-customer failures required a way to improve prompts, RAG behavior, parsing, and validation without regressing behaviors that already worked.

Constraints:

• the initial release solved about 80% of use cases but left a critical 20% long tail
• LLM prompt changes can introduce regressions in behaviors the team did not know were working
• traditional unit tests, IDE debugging, and regression tests are difficult or impossible for some LLM workflows
• the RAG pipeline had roughly 40 steps whose decisions could affect model behavior
• the system includes Honeycomb, OpenAI, and Honeycomb's query engine as separate distributed components
• the team needed to inspect full user experience traces, not just LLM latency and errors
• the feature needed to improve daily while preserving the user's ability to query observability data

Architecture

Honeycomb treated its natural-language query feature as a distributed system and instrumented it with unusually deep observability. Each request captures user/team context, every prompt sent to OpenAI, decisions across an approximately 40-step RAG pipeline, full LLM responses, parsed data structures, validation failures, and user feedback. Those details are connected into an end-to-end distributed trace that includes the application, OpenAI, the RAG pipeline, and the query engine. Engineers then filter failures, inspect traces, make prompt or fine-tuning changes, deploy daily, and compare success/failure rates over the following 24 hours to form a continuous improvement flywheel.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Natural_language_query_UI["Natural language query UI"] --> RAG_pipeline["RAG pipeline"]
  Natural_language_query_UI --> Honeycomb_application_trace["Honeycomb application trace"]
  RAG_pipeline --> OpenAI_API["OpenAI API"]
  RAG_pipeline --> Parsed_response_pipeline["Parsed response pipeline"]
  OpenAI_API --> Parsed_response_pipeline
  Parsed_response_pipeline --> Query_engine["Query engine"]
  Parsed_response_pipeline --> Failure_analysis_views["Failure analysis views"]
  Honeycomb_application_trace --> RAG_pipeline
  Honeycomb_application_trace --> OpenAI_API
  Honeycomb_application_trace --> Query_engine
  Honeycomb_application_trace --> Failure_analysis_views
  User_feedback_signal["User feedback signal"] --> Honeycomb_application_trace
  Failure_analysis_views --> Prompt_engineering_loop["Prompt engineering loop"]
  Prompt_engineering_loop --> RAG_pipeline
  Prompt_engineering_loop --> Daily_deployment_review["Daily deployment review"]
  Daily_deployment_review --> Honeycomb_application_trace
  OpenTelemetry_ecosystem["OpenTelemetry ecosystem"] --> Honeycomb_application_trace

Components

Data flow

A user types a natural-language request and clicks get query. Honeycomb records user and team context, runs the request through a RAG pipeline with roughly 40 decision steps, sends full prompts to OpenAI, parses and validates the LLM response, and executes the resulting query against Honeycomb's query engine when valid. Every step is captured as distributed tracing spans, including failures such as invalid JSON or no query output. Engineers group failures by input and model response, open traces to inspect root cause, change prompts or model behavior, deploy the fix, and observe the next 24 hours of real user behavior.

Why this architecture

Honeycomb chose distributed tracing because the LLM feature is a multi-component distributed system, and prompt/RAG changes can alter behavior in ways traditional tests do not catch. Capturing every decision and failure mode gives engineers the data needed to isolate long-tail failures, make targeted prompt or fine-tuning changes, and verify whether real user outcomes improved.

Models used: OpenAI LLM API

Scale

• The first release solved about 80% of use cases and left a 20% long-tail failure set.
  “we solved about 80% of our use cases in the first release which I think was pretty awesome”transcript, stated

  “there's a 20% that was really really a massive long tale of things and every time we would we would fix one thing another thing could potentially break”transcript, stated
• The RAG pipeline has about 40 decision steps per request.
  “every single decision that we make in our rag pipeline which is about 40 or so steps”transcript, stated
• The LLM portion of a trace is 48 spans long.
  “the slice of that Trace that deals with the llm and deals with this stuff is 48 spans in length”transcript, stated
• One collapsed RAG section alone can contain more than 20 spans.
  “there's a collaps one thing that's over 20 spans in length there's a whole lot of decisions that we're making before we ever make a request to open Ai”transcript, stated
• The team deploys daily and reviews the previous 24 hours of data.
  “you repeat that again and again you deploy daily you look at what you did over the previous day”transcript, stated

  “you look at the past 24 hours after you've deployed it and you say okay did our rate of success or failure for the entire user experience go up or down”transcript, stated
• OpenTelemetry supported 11 languages at the time of the talk.
  “you can use open Telemetry which is supported for already 11 uh um languages right now”transcript, stated

Alternatives considered (roads not taken)

Treat initial beta launch as production-ready

Rejected because: Honeycomb found that a useful first release still left important customer edge cases and required long-term iteration.

Rely on conventional unit tests, IDE debugging, and regression tests

Rejected because: The speaker says those tools are exceptionally difficult or impossible for some LLM work.

Monitor only latency, token usage, and errors

Rejected because: Honeycomb needed the full prompt, RAG decisions, LLM responses, parsing, validation, and user feedback to isolate failures.

Trace only the LLM API call

Rejected because: Honeycomb needed the trace connected to the full application and user experience.

Make prompt changes by intuition alone

Rejected because: Prompt changes can cause regressions, so Honeycomb uses observed failures and 24-hour outcome checks to guide changes.

Patterns

observability-tracing (operations)

Honeycomb instruments the full LLM user experience as distributed traces with 48 spans covering application, RAG, OpenAI, parsing, validation, query execution, and failures.

eval-driven-development (quality)

The team uses observed production failures, prompt changes, daily deploys, and 24-hour success/failure comparisons as the empirical loop for making the LLM feature reliable.

rag-retrieval (knowledge-architecture)

The feature uses a RAG pipeline with many decision steps to supply context before sending the request to OpenAI.

structured-output-contract (quality)

Honeycomb parses LLM responses into data structures, validates them, and records failures such as invalid JSON.

data-quality-loop (quality)

The system captures user feedback and real-world behavior as data for prompt engineering, fine-tuning, and evaluation improvements.

Tradeoffs

Deep request capture

Chose capturing full prompts, responses, RAG decisions, validation outcomes, and user feedback over capturing only standard latency, token, and error telemetry.

Rationale: The full context is needed to isolate specific failure modes and feed prompt engineering, fine-tuning, and evaluations.

Distributed tracing

Chose trace the full user request across Honeycomb, OpenAI, and query execution over observing the OpenAI call in isolation.

Rationale: The user experience spans multiple systems, so the trace has to connect the entire application path.

Production-behavior iteration loop

Chose daily deploys followed by 24-hour success/failure observation over one-time prelaunch testing or intuition-led prompt tuning.

Rationale: Real user behavior reveals whether prompt changes improve or regress the full end-to-end experience.

User-facing natural-language query interface

Chose let users type what they want and generate the query over requiring new users and prospects to learn Honeycomb's query UI first.

Rationale: The sales team could onboard prospects by letting them use the text box instead of teaching the interface.

Gotchas (what bit them)

A good first LLM launch is not the same as production reliability.

Root cause: Users expect more after launch, and long-tail customer edge cases appear once real usage begins.

Resolution: Instrument the feature deeply and plan for continuous post-launch iteration.

Prompt edits can create invisible regressions.

Root cause: Subtle prompt changes can massively alter LLM behavior, including behavior the team did not realize was working.

Resolution: Use trace data and real success/failure rates to validate changes after deployment.

RAG pipelines multiply sources of LLM behavior change.

Root cause: Every additional context source or RAG decision becomes another variable affecting the model response.

Resolution: Capture every RAG pipeline decision and inspect the trace before making prompt or fine-tuning fixes.

Latency and error metrics alone miss the reasons an LLM feature failed the user.

Root cause: Failures can arise from prompts, RAG decisions, model responses, parsing, validation, or query execution.

Resolution: Capture full prompts, responses, parsed structures, validation outcomes, and every possible failure mode.

LLM features need distributed-system debugging.

Root cause: The product application, OpenAI, and the query engine are separate systems connected by the user request path.

Resolution: Build a distributed trace connected to the full user experience rather than only the LLM call.

Evaluation

Honeycomb evaluates the LLM feature through production observability rather than only offline tests. Engineers group failure events, inspect traces, identify invalid responses or missing queries, deploy prompt/model changes, and compare full user-experience success or failure over the next 24 hours. The same captured data supports prompt engineering, fine-tuning, and formal evaluations.

Metrics: about 80% of use cases solved in first release · 20% long-tail failure set important to paying customers · about 40 RAG pipeline steps · 48 spans in the LLM trace slice · more than 20 spans in one collapsed RAG section · daily deploy cycle · 24-hour post-deploy success/failure comparison · OpenTelemetry support for 11 languages

Outcomes

• Honeycomb launched a natural-language querying interface around May 3rd.
• The first release solved about 80% of use cases.
• Deep observability helped the team iterate on the remaining paying-customer long tail.
• The feature contributed to higher product retention, especially for new teams.
• The feature contributed to higher conversion rates to paid tiers.
• The sales team could let prospects type desired queries instead of teaching the UI first.
• Honeycomb expects OpenTelemetry AI/vector instrumentation conventions and libraries to make this easier over time.

The case is a strong production LLMOps example because it ties architecture, operations, and business outcomes together: the LLM feature is monitored at the request-trace level, not as a black-box API call, and each improvement is checked against real behavior.