Natural Language Query Interface with Production LLM Integration
honeycomb · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)
Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.
Problem
Domain: Observability query authoring
Honeycomb needed average developers and product managers to use its observability query interface successfully, not only SREs and platform engineers.
Constraints:
- Existing query tooling caused drop-off among less specialized users.
- The feature needed to ship into production quickly.
- Generated output had to be executable JSON query specifications.
- Prompt behavior needed iteration from real production data.
Architecture
Honeycomb used GPT-3.5 to translate natural-language questions into JSON query specifications, then iterated prompts from production usage to improve success and activation metrics.
Diagram confidence: reconstructed (reconstructed from prose).
flowchart LR natural-language-query-assistant["Natural Language Query Assistant"] gpt-3-5-query-translator["GPT-3.5 Query Translator"] honeycomb-query-engine["Honeycomb Query Engine"] activation-metrics["Activation Metrics"] natural-language-query-assistant --> gpt-3-5-query-translator gpt-3-5-query-translator --> honeycomb-query-engine honeycomb-query-engine --> activation-metrics activation-metrics --> gpt-3-5-query-translator
Components
| Component | Role | Interacts with |
|---|---|---|
Natural Language Query Assistant |
Accepts user questions about observability data. | GPT-3.5 Query Translator |
GPT-3.5 Query Translator |
Converts natural language into JSON query specifications. | Honeycomb Query Engine |
Honeycomb Query Engine |
Executes generated query specifications against observability data. | Activation Metrics |
Activation Metrics |
Measures query success, complex-query creation, and board usage for prompt iteration. | GPT-3.5 Query Translator |
Data flow
Users enter natural-language observability questions, GPT-3.5 converts them into Honeycomb query JSON, the platform executes those specifications, and production outcomes feed prompt iteration.
Why this architecture
Natural-language query generation lowered the activation barrier for non-SRE users while retaining Honeycomb's executable JSON query model.
Models used: GPT-3.5 API
Scale
- Honeycomb shipped the feature in one month.“shipped a production feature in one month”summary, stated
- Success rate improved from 75% to 94%.“the success rate improved from 75% to 94%”summary, stated
- Teams using the assistant were more than twice as likely to create complex queries.“more than twice as likely to create complex queries”summary, stated
- Teams using the assistant were nearly three times more likely to add queries to boards.“nearly three times more likely to add queries to boards”summary, stated
Alternatives considered (roads not taken)
Leave users to learn the existing query interface
Rejected because: Less specialized users struggled and dropped off.
One-shot prompt without production iteration
Rejected because: Production data was used to improve success from 75% to 94%.
Patterns
query-rewriting (query-generation)
The assistant rewrites natural language into Honeycomb JSON query specifications.
eval-driven-development (quality)
Prompt engineering used production data and activation metrics to raise success rate.
Tradeoffs
Ship with GPT-3.5
Chose Production GPT-3.5 API integration over Waiting for a custom model or full query-language redesign.
Rationale: The team could ship in one month and iterate from production behavior.
Generate JSON query specs
Chose Structured Honeycomb query output over Free-form natural-language answers.
Rationale: Executable specs let the assistant integrate directly with the existing observability platform.
Gotchas (what bit them)
Expert query interfaces can block broader adoption even when powerful.
Root cause: SREs and platform engineers were comfortable, but average developers and product managers were not.
Resolution: Add natural-language generation of executable query specs.
Production usage reveals prompt failures not seen before launch.
Root cause: Real users ask diverse and messy observability questions.
Resolution: Iterate prompts based on production data and success metrics.
Evaluation
Honeycomb measured success rate and activation metrics tied to paying-customer behavior.
Metrics: Query success rate · Complex-query creation · Queries added to boards
Outcomes
- The feature shipped in one month.
- Success rate improved from 75% to 94%.
- Assistant-using teams were more than twice as likely to create complex queries.
- Assistant-using teams were nearly three times more likely to add queries to boards.
The case reports activation lift and production success rates rather than total query volume.