Automating Supplier Ticket Management with LLM Agents

wayfair · E-commerce · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Supplier support ticket triage

Wayfair needed to automate repetitive supplier-ticket triage in SupportHub: reading unstructured tickets, looking up supplier data, and populating structured JIRA fields without losing reliability on edge cases.

Constraints:

• tickets are created in an existing JIRA-based supplier-support system
• the system must classify among 81 possible question types
• supplier IDs may require database lookup and retry logic when ticket information is imperfect
• production operators need trace visibility, latency/token monitoring, and debuggability
• pure agent autonomy caused reliability and coordination failures

Architecture

Wilma uses an event-driven LangGraph workflow triggered by SupportHub ticket events. A webhook publishes new-ticket events to Pub/Sub, a consumer invokes a graph, and the graph runs ordered LLM and tool calls for question-type classification, language detection, supplier-ID lookup, and JIRA updates. The final design is hybrid: deterministic workflow orchestration wraps a ReAct supplier-ID agent with BigQuery access for the one step that needs adaptive query reformulation.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  supporthub-ticketing-system[SupportHub ticketing system] --> pub-sub-topic[Pub/Sub topic]
  pub-sub-topic --> pub-sub-consumer[Pub/Sub consumer]
  pub-sub-consumer --> langgraph-workflow[LangGraph workflow]
  langgraph-workflow --> intent-classification-llm[Intent classification LLM]
  langgraph-workflow --> language-detection-llm[Language detection LLM]
  langgraph-workflow --> supplier-id-react-agent[Supplier ID ReAct agent]
  supplier-id-react-agent --> bigquery[BigQuery]
  langgraph-workflow --> jira-api[JIRA API]
  jira-api --> supporthub-ticketing-system
  langgraph-workflow --> arize-observability[Arize observability]

Components

Data flow

A supplier email creates a SupportHub ticket, which triggers a webhook and Pub/Sub event. The Pub/Sub consumer invokes a LangGraph graph. The graph classifies the ticket question type and language with LLM calls, invokes a ReAct agent with BigQuery access to identify the supplier ID, and then calls the JIRA API to update the ticket fields. Arize traces and alerts monitor the multi-step execution.

Why this architecture

Wayfair moved away from fully autonomous multi-agent coordination because it was unreliable and costly, then moved beyond a rigid workflow because the workflow failed on edge cases. The hybrid architecture keeps deterministic ordering for the overall triage flow while letting the supplier-ID step use agentic retry and query reformulation where it adds value.

Models used: LLM for question-type classification · LLM for language detection · ReAct LLM agent

Scale

• The system classifies tickets across 81 question-type categories.
  “By comparison, humans correctly identify the question type only 75% of the time (across 81 possible categories).”article, stated
• Wilma identifies question type with 93% accuracy.
  “accurately identifying the question type 93% of the time, the language 98% of the time, and the supplier ID 88% of the time.”article, stated
• Wilma identifies preferred language with 98% accuracy.
  “accurately identifying the question type 93% of the time, the language 98% of the time, and the supplier ID 88% of the time.”article, stated
• Wilma identifies supplier IDs with 88% accuracy.
  “accurately identifying the question type 93% of the time, the language 98% of the time, and the supplier ID 88% of the time.”article, stated
• The human baseline for question-type identification was 75%.
  “By comparison, humans correctly identify the question type only 75% of the time (across 81 possible categories).”article, stated

Alternatives considered (roads not taken)

Fully agentic supervisor with multiple specialized agents

Rejected because: Agents missed information passed between one another and the supervisor could invoke agents unnecessarily.

Pure workflow with no embedded agent

Rejected because: It was cheaper and easier to debug but failed on edge cases such as typo-tolerant supplier lookup.

Raw logs as the main debugging interface

Rejected because: Purpose-built trace navigation and search worked better once the graph became complicated.

Patterns

tool-use-orchestration (orchestration)

LangGraph coordinates LLM calls, BigQuery queries, and JIRA/SupportHub updates in one ticket-triage workflow.

async-message-orchestration (orchestration)

Ticket creation is decoupled from automation work through webhook events, Pub/Sub, and a consumer that invokes the graph.

structured-output-contract (integration)

The automation populates fixed ticket fields: supplier ID, primary question type, and preferred language.

observability-tracing (operations)

Arize provides trace navigation, debugging, and anomaly alerts for graph executions.

confidence-thresholding (quality)

The system compares automated triage accuracy against human baselines and reports separate accuracy measures for each field.

Tradeoffs

Use a hybrid workflow plus embedded agent

Chose deterministic workflow orchestration with a ReAct agent for supplier-ID lookup over a fully agentic supervisor coordinating all subtasks.

Rationale: The workflow provided reliability, debuggability, and lower cost, while the embedded agent preserved adaptive error handling for the one step that needed it.

Prefer explicit workflow calls for most steps

Chose a defined series of LLM and function calls over supervisor-mediated delegation among agents.

Rationale: The explicit workflow was more reliable, easier to debug, and cheaper because it reduced unnecessary LLM calls.

Invest in LLM-specific observability

Chose Arize trace visualization, search, and alerts over looking at raw output logs.

Rationale: Graph executions became complicated enough that searchable traces and anomaly alerts were operationally more useful than raw logs.

Gotchas (what bit them)

Pure multi-agent coordination was unreliable.

Root cause: Autonomous agents ignored information from other agents and the supervisor repeated calls unnecessarily.

Resolution: Wayfair replaced the fully agentic design with a workflow and kept agent autonomy only inside the supplier-ID lookup node.

Rigid workflows failed on messy edge cases.

Root cause: A deterministic lookup could stop at the first failed database query even when the ticket contained typo-like errors.

Resolution: Wayfair embedded a ReAct supplier-ID agent that could try different query strategies and reformulate based on observed results.

Complex LLM graphs became hard to debug with ordinary logs.

Root cause: Multi-step graph invocations require searchable execution traces rather than raw output inspection.

Resolution: The team used Arize for pre-launch development, post-launch debugging, trace navigation, and anomaly alerts.

Evaluation

Wayfair evaluated triage performance against human baselines and monitored production graph behavior with Arize traces and alerting.

Metrics: 93% question-type identification accuracy · 98% language detection accuracy · 88% supplier-ID identification accuracy · 75% human question-type identification baseline · token usage and latency anomaly alerts

Outcomes

• Wilma exceeded human question-type classification accuracy.
• The system reduced ticket processing time.
• Supplier associates could focus on complex problem solving and high-value interactions instead of manual data entry.

The article reports accuracy across 81 possible question categories but does not disclose ticket volume or latency figures.