Converting Natural Language to Structured GraphQL Queries Using LLMs

cato_networks · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: SASE event search

Cato Networks needed to make its SASE Events page usable for customers who did not know the product glossary or manual filter syntax. The page can contain millions of events over a selected time range, and users needed natural-language search that still produced valid, API-compatible filters.

Constraints:

• The Events page can cover potentially millions of events.
• Manual filtering requires product glossary knowledge.
• Model outputs cannot be trusted without validation.
• The product API expects structured GraphQL-compatible queries.
• Feature release depended on an error rate below 0.05.

Architecture

Cato built a natural-language search service on Amazon EKS that prompts Claude 3.5 Sonnet through Amazon Bedrock to emit JSON conforming to an IETF JSON schema, validates the response, applies failure policies, and deterministically translates valid structured output into GraphQL API requests for the Events page.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  management-console["Management console"] --> natural-language-search-service["Natural language search service"]
  natural-language-search-service --> amazon-bedrock["Amazon Bedrock"]
  amazon-bedrock --> claude-35-sonnet["Claude 35 Sonnet"]
  natural-language-search-service --> json-schema-validator["JSON schema validator"]
  json-schema-validator --> natural-language-search-service
  natural-language-search-service --> graphql-server["GraphQL server"]
  benchmark-suite["Benchmark suite"] --> amazon-bedrock
  benchmark-suite --> claude-35-sonnet

Components

Data flow

A user enters free text in the management console. The Natural Language Search service builds a prompt with the JSON schema and product-specific instructions, sends it to Claude 3.5 Sonnet on Bedrock, validates the JSON response against the same schema, handles violations with warnings or failures, converts valid filters into GraphQL, and queries the GraphQL server.

Why this architecture

Cato deliberately put a schema-valid JSON layer between natural language and GraphQL. That made model output easy to validate and allowed deterministic API translation while Bedrock made model benchmarking and swapping practical.

Models used: anthropic.claude-3-5-sonnet-20241022-v2:0 · Foundation models available through Amazon Bedrock

Scale

• The Events page can contain potentially millions of events over a selected time range.
  “With potentially millions of events over a selected time range, the goal is to refine these events using various filters”article, stated
• Cato validated the approach with hundreds of text queries and expected JSON outputs.
  “we’ve created a benchmark with hundreds of text queries and their corresponding expected JSON outputs.”article, stated
• Feature release required an error rate below 0.05.
  “releasing the feature was contingent on achieving an error rate below 0.05”article, stated

Alternatives considered (roads not taken)

Manual event filtering

Rejected because: It is time-consuming and requires deep product glossary knowledge.

Have the model generate GraphQL directly

Rejected because: Direct GraphQL generation would be harder to validate safely than JSON schema output.

Use prompt engineering forever for complex schemas

Rejected because: Including the full schema in every prompt can create a high token count, so fine-tuning may reduce cost later.

Patterns

structured-output-contract (output)

The prompt requires JSON output validated against an IETF JSON schema before API translation.

eval-driven-development (evaluation)

Cato benchmarked models with expected JSON outputs and release-gated on an error-rate criterion.

model-routing (model-selection)

Bedrock let Cato transition between models and choose the one meeting accuracy, latency, and cost needs.

Tradeoffs

Use JSON schema as the intermediate representation

Chose validated JSON then deterministic GraphQL over direct GraphQL from the model.

Rationale: The schema can validate model output and identify exact violations before hitting the product API.

Use prompt engineering rather than fine-tuning initially

Chose schema-rich prompt through Bedrock over custom model fine-tuning.

Rationale: The prompt solution met the error criterion, though the authors note fine-tuning could reduce cost for complex schemas.

Gotchas (what bit them)

Schema violations can break product functionality.

Root cause: Model behavior is non-deterministic and may generate unsupported keys or invalid filter formats.

Resolution: Validate JSON against the schema and apply policies by failure type.

Translation failures create poor user experience.

Root cause: Invalid JSON or invalid schema output cannot become a useful event query.

Resolution: Gate release on low error rate and return warnings/failures based on validation results.

Evaluation

Cato benchmarked foundation models on hundreds of text queries with expected JSON outputs, classifying responses as success, partial, or error.

Metrics: Valid JSON · Valid by schema · Full filter match · Partial filter match · Invalid JSON/schema error rate · Latency · Cost

Outcomes

• Query time dropped from minutes of manual filtering to near-instant results.
• Customers gave positive feedback.
• Multi-language input made the Events page more accessible.
• Account admins reported near-zero time to value.

The feature targets Events pages with potentially millions of events and was benchmarked against hundreds of expected-query examples.