Building and Scaling an LLM-Powered Query Assistant in Production

honeycomb · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Observability query onboarding

Honeycomb needed new users to translate natural-language questions and mental models of service behavior into Honeycomb query objects, because difficulty learning the query interface slowed activation, retention, and the path to paid usage.

Constraints:

• users bring unpredictable natural-language inputs and unique dataset schemas
• LLM behavior cannot be unit-tested or debugged like deterministic software
• the feature needed to improve activation without replacing manual querying
• LLM costs and provider rate limits had to remain low enough for a core product feature
• latency and timeout behavior depended partly on OpenAI-hosted model performance
• discoverability had to balance helping new users without disrupting experienced Query Builder users

Architecture

Honeycomb built Query Assistant as a natural-language-to-Honeycomb-query feature. The system uses GPT-3.5-turbo for query object generation and text-embedding-ada-002 plus Redis-stored dataset schema embeddings for context. The team chose GPT-3.5 over GPT-4 after prompt engineering for cost and latency, aggressively reduced prompt size under the original 90k tokens/minute rate limit, captured user inputs, LLM outputs, errors, and metadata, and used production data plus SLOs to iterate. Query Assistant emits intentionally expressive queries, incorporates existing-query context for modifications, and added team-defined Suggested Queries after customer feedback exposed schema-specific shortcomings.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  query-assistant-ui[Query Assistant UI] --> prompt-builder[Prompt builder]
  prompt-builder --> schema-embedding-store[Schema embedding store]
  embedding-model[Embedding model] --> schema-embedding-store
  prompt-builder --> gpt-3-5-query-generator[GPT-3.5 query generator]
  gpt-3-5-query-generator --> honeycomb-query-runner[Honeycomb query runner]
  honeycomb-query-runner --> query-assistant-ui
  query-assistant-ui --> feedback-widget[Feedback widget]
  feedback-widget --> production-analysis-pipeline[Production analysis pipeline]
  production-analysis-pipeline --> slo-monitoring[SLO monitoring]
  slo-monitoring --> iteration-process[Iteration process]
  production-analysis-pipeline --> iteration-process
  iteration-process --> prompt-builder
  suggested-queries[Suggested Queries] --> prompt-builder

Components

Data flow

A user types a natural-language request into Query Assistant, possibly with an existing Honeycomb query already in context. The product builds a prompt with the user's dataset context and schema embeddings from Redis, sends it to GPT-3.5-turbo, and receives a Honeycomb Query object rather than verbose prose. The generated query runs in Honeycomb, user feedback marks results helpful or not, and Honeycomb captures the input, output, errors, and metadata for later analysis. Production observations, SLOs, and customer feedback then inform prompt updates, Suggested Query integration, onboarding/discoverability changes, and daily iteration when needed.

Why this architecture

Honeycomb used an LLM because the hard problem was translating each user's unique natural language and schema context into a Honeycomb Query object, not matching a fixed command. The team chose to ship quickly because nondeterministic LLM features only reveal real input/output behavior in production. GPT-3.5-turbo was kept for production because prompt engineering made it reliable enough at far lower cost than GPT-4, while embeddings and Suggested Queries gave the prompt schema-specific context. SLOs replaced deterministic regression tests as the guardrail for behavior changes.

Models used: GPT-3.5-turbo · text-embedding-ada-002 · GPT-4

Scale

• Honeycomb released Query Assistant to all customers about one month after starting the LLM capability.
  “About a month later, we released Query Assistant to all customers as an experimental feature.”article, stated
• During rapid iteration the team sometimes shipped every day.
  “After a period of rapid iteration, sometimes shipping every day, we were able to bring Query Assistant out of experimentation”article, stated
• Week-six manual query retention was 26.5% for teams using Query Assistant versus 4.5% for non-users.

  "For new teams that used Query Assistant, by week 6, 26.5% still ran manual queries

• For new teams that did not use Query Assistant, by week 6, 4.5% still ran manual queries"

— article, stated

• 33% of active teams using Query Assistant created complex queries versus 15.7% of non-users.

  "* 33% of teams that used Query Assistant created a complex query

• 15.7% of teams that did not use Query Assistant created a complex query"

— article, stated

• 11% of teams using Query Assistant created a Board versus 3.6% of non-users.

  "* 11% of teams that used Query Assistant created a Board

• 3.6% of teams that did not use Query Assistant created a Board"

— article, stated

• OpenAI API costs averaged about $30 per month, and each Redis node cost about $100 for vectors.
  “On average, our monthly OpenAI bill is… $30. Even if we switched to GPT-4—which we won’t, due to GPT-3.5 performing very well—it would still cost us less than $1k per month.”article, stated

  “Finally, we store embedding vectors for dataset schemas in a Redis cluster in AWS. Each node costs us about $100 to store vectors.”article, stated
• At launch, average GPT-3.5 latency was about five seconds and P99 was 30 seconds or higher.
  “When we went live with Query Assistant in May, latency wasn’t great. The average request to GPT-3.5-turbo took about five seconds, but our P99 was measured in 30 seconds or higher.”article, stated

Alternatives considered (roads not taken)

Perfect the feature before launch with traditional tests

Rejected because: Honeycomb could not predict user inputs, LLM responses, or user-specific contextual data, so production learning was necessary.

Traditional unit/regression tests as the main LLM quality guard

Rejected because: LLMs are nondeterministic black boxes, so Honeycomb used captured production data and SLOs instead.

Use GPT-4 in production

Rejected because: GPT-3.5 performed well enough after prompt work and was much cheaper.

Optimize only for experienced users by hiding Query Assistant below the Query Builder

Rejected because: Free-tier adoption underperformed because many users did not notice the feature, so Honeycomb adjusted onboarding and nurture emails.

Patterns

structured-output-contract (interface)

Query Assistant outputs a Honeycomb Query object rather than free-form prose, making the LLM response directly executable in the product.

eval-driven-development (quality)

Honeycomb used production interaction capture, systematic analysis, SLOs, and activation metrics as the quality loop for nondeterministic LLM behavior.

observability-tracing (operations)

The team treated user input, model response, errors, and metadata as production observability signals for the LLM feature.

rag-retrieval (context)

Query Assistant uses embeddings for dataset schemas and pulls in Suggested Queries to guide the model for custom field names.

cost-aware-routing (model-operations)

Honeycomb engineered prompts so GPT-3.5-turbo could handle the production task instead of relying on GPT-4.

data-quality-loop (product-feedback)

Detailed customer feedback, especially from Intercom, directly shaped model context through team-defined Suggested Queries.

Tradeoffs

Development strategy

Chose ship-to-learn with production data over traditional pre-launch perfection.

Rationale: The team could not predict user inputs or LLM outputs, so the only reliable feedback came from real interactions.

Quality guardrail

Chose SLO-based monitoring on real production behavior over deterministic regression tests.

Rationale: SLOs can detect behavior drift in nondeterministic outputs where fixed unit tests are inadequate.

Model choice

Chose GPT-3.5-turbo over GPT-4 for production query generation.

Rationale: Prompt engineering made GPT-3.5 reliable enough while keeping monthly costs tiny.

Query Assistant output style

Chose often emit more complex queries over only simplest possible queries.

Rationale: Complex WHERE and GROUP BY clauses taught users that the Honeycomb interface was flexible and helped them graduate to manual querying.

UI placement

Chose place Query Assistant below Query Builder over moving the query UI around for maximum AI discoverability.

Rationale: The team wanted discoverability for new users while staying out of existing users' way, though later data showed this hurt awareness.

Gotchas (what bit them)

LLM inputs and outputs could not be predicted up front.

Root cause: Every user's natural language and contextual dataset differed, and the model's response to that combination was nondeterministic.

Resolution: Ship early, capture real interactions, and iterate from production data instead of waiting for full pre-launch certainty.

Traditional regression tests did not fit the LLM feature.

Root cause: LLMs are nondeterministic black boxes rather than deterministic functions.

Resolution: Use SLOs over real-world data to monitor whether changes fix problems without regressing working behavior.

Discoverability undercut free-tier adoption.

Root cause: Placing Query Assistant below the Query Builder kept it out of the way but made many users miss it.

Resolution: Change onboarding and nurture email campaigns to explain Query Assistant as a way to start querying.

Provider latency made early production requests timeout.

Root cause: At launch, GPT-3.5 average latency was about five seconds and P99 was 30 seconds or more.

Resolution: Rely on provider infrastructure improvements and continue monitoring latency; by October average duration halved and P99 fell several times.

Custom field names caused model misses for customer schemas.

Root cause: Schemas can encode domain-specific names that the base prompt does not naturally understand.

Resolution: Pull team-defined Suggested Queries into the context so the model has schema-local guidance.

Evaluation

Honeycomb evaluated Query Assistant through production interaction capture, SLOs, pricing-tier adoption, manual-query retention, complex-query creation, Board and Trigger creation, Query Assistant retention, helpfulness feedback, customer feedback, and operating-cost/latency monitoring.

Metrics: active teams using Query Assistant by pricing plan · manual querying retention over several weeks · complex query creation · Board creation · Trigger creation · Query Assistant retention · helpfulness feedback · OpenAI API cost · average latency · P99 latency

• Manual querying retention over several weeks
• Percentage of teams who ran a manual query and then created a “complex” query"

Outcomes

• Query Assistant moved from experiment to core product offering
• teams using Query Assistant showed much higher week-six manual query retention
• teams using Query Assistant created more complex queries and Boards than non-users
• users tended to graduate from Query Assistant into manual querying
• OpenAI monthly cost averaged about $30
• unexpected inputs such as DSL expressions and trace IDs produced runnable queries
• Trigger creation impact was weak and free-tier adoption was below hopes

The feature was released to all customers about a month after concept, then iterated from production data; operational cost was only a few hundred dollars per month including Redis.