AI-Powered Customer Segmentation with Natural Language Interface

klaviyo · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Marketing customer segmentation

Klaviyo needed to let less experienced marketers express customer segments in plain language and help expert users automate repetitive complex segment-building work.

Constraints:

Generated segment definitions must conform to Klaviyo's JSON schema
LLM outputs are stochastic and can have multiple valid answers
The feature is customer-facing, increasing validation and maintenance risk

Architecture

Klaviyo's Segments AI converts natural language segment requests into Klaviyo segment JSON through many small OpenAI-powered subtasks rather than one monolithic prompt. The system uses prompt chaining, mostly parallel/separable subtasks, few-shot examples embedded in system instructions, specialized chatbot-agent subtasks, final composition into the segment schema, and a validation suite combining LLM evaluators with hand-designed regression cases.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  natural-language-segment-request["natural-language-segment-request"]
  prompt-chain-orchestrator["prompt-chain-orchestrator"]
  openai-llm-api["openai-llm-api"]
  few-shot-subtask-agents["few-shot-subtask-agents"]
  segment-json-composer["segment-json-composer"]
  klaviyo-segment-schema["klaviyo-segment-schema"]
  validation-suite["validation-suite"]
  llm-evaluator["llm-evaluator"]
  hand-designed-test-cases["hand-designed-test-cases"]
  natural-language-segment-request --> prompt-chain-orchestrator
  prompt-chain-orchestrator --> openai-llm-api
  prompt-chain-orchestrator --> few-shot-subtask-agents
  prompt-chain-orchestrator --> segment-json-composer
  few-shot-subtask-agents --> segment-json-composer
  segment-json-composer --> klaviyo-segment-schema
  segment-json-composer --> validation-suite
  validation-suite --> llm-evaluator
  validation-suite --> hand-designed-test-cases

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`natural-language-segment-request`	User-provided description of the desired customer segment.	`prompt-chain-orchestrator`
`prompt-chain-orchestrator`	Coordinates small, atomic LLM requests and parallel subtask execution.	`openai-llm-api`, `few-shot-subtask-agents`, `segment-json-composer`
`openai-llm-api`	Language-model provider used by Segments AI subtasks.	—
`few-shot-subtask-agents`	Specialized prompt agents seeded with annotated examples, ideal outputs, and lessons.	`segment-json-composer`
`segment-json-composer`	Composition layer that joins subtask outputs into Klaviyo's segment JSON schema.	`klaviyo-segment-schema`, `validation-suite`
`klaviyo-segment-schema`	Required structured output contract for customer segment definitions.	—
`validation-suite`	Pre-change validation package using LLM evaluators and hand-designed tests.	`llm-evaluator`, `hand-designed-test-cases`
`llm-evaluator`	LLM-as-judge strategy for creative generated outputs.	—
`hand-designed-test-cases`	Regression examples created by the team for known segment-generation cases.	—

Data flow

A user describes a desired customer segment in natural language. Segments AI breaks the task into small, separable prompt calls, uses few-shot system instructions to specialize each subtask, runs many subtasks asynchronously, composes the outputs into Klaviyo's segment JSON schema, and validates changes with a packageable suite of LLM-judged and hand-designed test cases before major releases.

3 supporting quotes

“That means when we generate content with an LLM, we try to make each request as small, atomic, and simple as possible.”article, stated

“Segment AI does many of these processes asynchronously (i.e. occurring in parallel).”article, stated

“Ultimately, we decided to use a mix of LLMs evaluating LLMs and hand-designed test cases.”article, stated

Why this architecture

Klaviyo avoided a simple ChatGPT wrapper because segment generation has to satisfy a domain-specific JSON schema and creative outputs are hard to validate deterministically. Small parallel prompt subtasks reduce chain entropy, few-shot examples specialize the model without explicit fine-tuning, and mixed validation gives directional quality signals without letting validation consume the whole project.

3 supporting quotes

“the service is far from just being a wrapper around ChatGPT.”article, stated

“Prompt chaining typically makes generations quicker and higher quality in cases where outputs can be joined together without loss of cohesion.”article, stated

“The issue of chain entropy is especially pronounced in use cases like Segments AI where generations need to adhere to a specific style”article, stated

Models used: OpenAI language models · LLM evaluators

Scale

Segments AI was released in November 2023.
“In November 2023, Klaviyo released a new segmentation feature called Segments AI”article, stated
Klaviyo released the feature to 130,000 customers.
“to its 130,000 customers. The service helps companies automate the process of building intelligent segments”article, stated
The company had developed multiple similar OpenAI-backed features over the prior 1.5 years.
“Like many other exciting features developed over the past 1.5 years, it involves an OpenAI account.”article, stated
The validation work should not consume more than one third of the team's time.
“If you find yourself spending more than a third of your time tinkering with validation, there are probably better uses of time.”article, stated

Alternatives considered (roads not taken)

One large ChatGPT wrapper prompt

Rejected because: The service decomposes work into many small subtasks because it must produce reliable segment JSON rather than freeform chat.

“That said, the service is far from just being a wrapper around ChatGPT.”article, stated

Sequential waterfall prompt chains

Rejected because: Klaviyo parallelizes separable subtasks to reduce chain entropy and avoid early errors cascading.

“we try to keep subtasks as separable and parallelized as possible such that there are fewer gates to pass through.”article, stated

Explicit fine-tuning for every subtask

Rejected because: The team seeds annotated examples into system instructions to specialize the model implicitly.

“this process can be done implicitly by seeding training examples into the system instruction.”article, stated

Only LLM-judged validation or only hand tests

Rejected because: They used both because each strategy is imperfect but useful for quick feedback and bug diagnosis.

“Both evaluation strategies are imperfect, but they were a useful, directional tool for quick feedback and bug diagnosis.”article, stated

Patterns

`prompt-chaining` (segment-definition-generation)

Klaviyo splits segment generation into small, atomic prompts that produce joinable parts of a JSON segment definition.

“Segments AI makes extensive use of prompt chaining internally. That means when we generate content with an LLM, we try to make each request as small, atomic, and simple as possible.”article, stated

`structured-output-contract` (segment-json-schema)

The generated outputs need to adhere to Klaviyo's segment JSON schema, so the system decomposes and controls generation around that contract.

“generations need to adhere to a specific style (in our case, Klaviyo’s segment JSON schema).”article, stated

`fine-tuning-adaptation` (few-shot-task-specialization)

Instead of model fine-tuning, Klaviyo seeds examples into system instructions so subtask agents behave like segment-domain specialists.

“this process can be done implicitly by seeding training examples into the system instruction.”article, stated

`llm-as-judge` (feature-validation)

The validation suite combines LLMs evaluating LLM outputs with hand-designed test cases for directional regression feedback.

“we decided to use a mix of LLMs evaluating LLMs and hand-designed test cases.”article, stated

`eval-driven-development` (customer-facing-genai)

The validation suite runs before major code changes and expands as functionality grows, even though the team acknowledges imperfect signals.

“The validation suite we created was packaged such that it could be run before major changes to the codebase.”article, stated

Tradeoffs

How to decompose segment generation.

Chose parallel/separable subtasks that can be composed into JSON over a synchronous waterfall chain of dependent prompts.

Rationale: Parallel subtasks reduce error propagation when schema-conformant JSON must be assembled from several model outputs.

“we try to keep subtasks as separable and parallelized as possible such that there are fewer gates to pass through.”article, stated

How much time to spend on validation.

Chose clear directional validation with stakeholder alignment over endless tinkering with imperfect validation suites.

Rationale: The team warns that validation matters, but users ultimately judge the feature in production.

“If you find yourself spending more than a third of your time tinkering with validation, there are probably better uses of time.”article, stated

Gotchas (what bit them)

Prompt chaining can propagate early mistakes.

Root cause: A wrong path in an early LLM subtask can contaminate downstream JSON composition.

Resolution: Klaviyo keeps subtasks separable and parallelized to reduce chain entropy and waterfall failure.

“While prompt chaining typically increases generation quality, it also creates the risk of errors propagating like an LLM game of telephone.”article, stated

Creative LLM outputs are hard to validate deterministically.

Root cause: There are many jointly valid ways to define a segment such as engaged users.

Resolution: Klaviyo uses imperfect but useful LLM-judge and hand-designed tests for quick feedback and bug diagnosis.

“when there are dozens of jointly valid ways to define a segment of “engaged users,” it becomes difficult to evaluate feature performance”article, stated

Generative AI features require more maintenance than standard data-science projects.

Root cause: Customer needs change, bugs appear, outputs are non-deterministic, and the model ecosystem changes quickly.

Resolution: Klaviyo treats the feature as ongoing operational work with close Data Science/Product collaboration.

“stochastic outputs and a rapidly changing state of the art mean that Gen AI features often require more maintenance than more standard data science projects.”article, stated

Evaluation

Klaviyo evaluates Segments AI with LLM-as-judge checks, hand-designed edge-case tests, pre-change validation-suite runs, regression analysis, and production customer feedback.

Metrics: validation-suite pass/fail · bug diagnosis feedback · regression coverage · stakeholder consistency · production customer use