The Hidden Complexities of Building Production LLM Features: Lessons from Honeycomb's Query Assistant

honeycomb · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Observability query assistance

Honeycomb wanted a production natural-language Query Assistant that could translate plain English into structured observability queries while dealing with large customer schemas, latency, security, and compliance constraints.

Constraints:

• Some customers had schemas with more than 5,000 fields.
• LLM latency ranged from 2 to 15+ seconds.
• Chained LLM calls multiplied latency and compounded error rates.
• The product had to avoid prompt-injection and compliance risks.
• The UI needed to fit Honeycomb's existing query workflow rather than become a generic chat interface.

Architecture

Honeycomb built Query Assistant as a focused text box and button in its standard query interface. It assembles prompt context from user input, Honeycomb query structure, domain knowledge, customer schemas, examples, and existing query state; sends that prompt to an LLM; parses and validates the response; then corrects what it can before executing through Honeycomb's query engine.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  honeycomb_query_ui["Honeycomb query UI"] --> prompt_assembly_service["Prompt assembly service"]
  schema_trimming_logic["Schema trimming logic"] --> prompt_assembly_service
  few_shot_examples["Few-shot examples"] --> prompt_assembly_service
  prompt_assembly_service --> llm_query_generator["LLM query generator"]
  llm_query_generator --> output_parser_and_validator["Output parser and validator"]
  output_parser_and_validator --> honeycomb_query_engine["Honeycomb query engine"]
  honeycomb_query_engine --> honeycomb_query_ui

Components

Data flow

A user enters a natural-language observability question. Query Assistant builds a prompt containing query grammar, instrumentation knowledge, trimmed schema context, few-shot examples, and current query context. The LLM returns a structured query, which is parsed, validated, corrected where possible, and then executed in the normal Honeycomb interface.

Why this architecture

Honeycomb treated the LLM as one feature engine inside a constrained product workflow, because chat UI, chaining, and huge untrimmed schemas were too slow or unreliable for production query generation.

Models used: gpt-3.5-turbo · Claude 100k · GPT-4

Scale

• Some customer schemas exceeded 5,000 unique fields.
  “Some Honeycomb customers have schemas with over 5,000 unique fields, far exceeding what can fit in the context window”summary, stated
• LLM latency ranged from 2 to more than 15 seconds.
  “latency ranged from 2 to 15+ seconds depending on various factors including model choice, input complexity, schema size, and prompt instructions.”summary, stated
• The feature shipped within about a month.
  “ultimately shipping the feature within a month while acknowledging that LLMs are engines for features rather than complete products.”summary, stated

Alternatives considered (roads not taken)

Chat-first interface

Rejected because: Honeycomb considered chat the wrong interaction model for this query-building task.

Chained LLM calls

Rejected because: Chaining multiplied latency and reduced end-to-end accuracy.

Use Claude 100k with full schemas

Rejected because: It was slower and more hallucination-prone than smaller relevant schema subsets.

Patterns

context-engineering (context management)

Honeycomb curated prompt context by trimming schemas, adding query structure, and providing domain examples rather than dumping full customer schemas into the prompt.

structured-output-contract (output validation)

The system parsed and validated structured Honeycomb query output before execution.

safety-policy-enforcement (security)

Security controls addressed prompt injection and compliance risks around production LLM queries.

Tradeoffs

Embed query assistance into the existing UI.

Chose A minimal textbox and button in the standard Honeycomb interface over A separate chat experience.

Rationale: The task was query construction, and the existing interface already supported review and execution.

Avoid multi-call chains.

Chose A focused prompt plus validation over Chained LLM calls for iterative query construction.

Rationale: Chaining increased latency and reduced compounded accuracy without tangible quality gains.

Gotchas (what bit them)

Large schemas can exceed LLM context windows.

Root cause: Some observability datasets had thousands of fields.

Resolution: Trim schemas to recently active fields and use relevant subsets instead of full schemas.

Chaining LLM calls can collapse reliability.

Root cause: Each step adds latency and compounds probability of error.

Resolution: Avoid LangChain-style chained calls when they do not improve query generation.

Large-context models are not automatically better.

Root cause: Full schemas slowed Claude 100k and increased hallucinations.

Resolution: Prefer smaller relevant schema subsets when possible.

Evaluation

The case emphasizes production validation and correction rather than reporting a formal benchmark; Honeycomb validated generated query structure before execution and learned from latency/context constraints during development.

Metrics: Schemas with over 5,000 fields. · LLM latency from 2 to 15+ seconds. · Illustrative compounded accuracy drop from 90% per step to 59% after five steps.

Outcomes

• Honeycomb shipped Query Assistant within about a month.
• The system handled query generation through schema trimming, prompt context, parsing, and validation.
• The team documented practical limits around context windows, latency, chaining, and security.

The case cites customer schemas over 5,000 fields and LLM latency from 2 to more than 15 seconds.