From Mega-Prompts to Production: Lessons Learned Scaling LLMs in Enterprise Customer Support

godaddy · E-commerce · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: enterprise customer-support LLM operations

GoDaddy's Digital Care team needed to operationalize LLM-powered customer support across high-volume messaging channels while controlling prompt bloat, model reliability, latency, structured-output failures, unsafe escalation decisions, memory limits, RAG quality, and testing effort.

Constraints:

• GoDaddy receives over 60,000 daily customer contacts across SMS, WhatsApp, and web messaging
• one mega-prompt grew beyond 1500 tokens and became less accurate as topics and instructions increased
• structured outputs from LLMs remained invalid on a measurable share of requests
• customer-support chat has strict latency and reliability expectations, including a 30-second upstream timeout
• sensitive actions and human transfer decisions cannot safely be left solely to model judgment

Architecture

GoDaddy evolved its AI Assistant from a single mega-prompt into a controller-delegate, multi-prompt support architecture. A controller prompt manages the conversation and delegates to task-oriented prompts for support-topic classification, human-agent summaries, RAG/search, structured-response tasks, and self-help flows. Deterministic guardrails handle PII/offensive-content checks, stop phrases, interaction limits, human transfer, and external approvals. Supporting layers manage memory, model selection, retries/streaming/asynchronous response, RAG data preparation, and QA reporting for post-release review.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  messaging-channels["Messaging channels"] --> controller-prompt["Controller prompt"]
  controller-prompt --> task-oriented-delegate-prompts["Task-oriented delegate prompts"]
  controller-prompt --> memory-manager["Memory manager"]
  controller-prompt --> guardrail-controls["Guardrail controls"]
  task-oriented-delegate-prompts --> structured-output-validator["Structured-output validator"]
  task-oriented-delegate-prompts --> rag-and-search-tools["RAG and search tools"]
  task-oriented-delegate-prompts --> human-support-queue["Human support queue"]
  rag-and-search-tools --> rag-data-preparation["RAG data preparation"]
  guardrail-controls --> human-support-queue
  memory-manager --> model-selector["Model selector"]
  async-and-streaming-response-layer["Async and streaming response layer"] --> messaging-channels
  async-and-streaming-response-layer --> controller-prompt
  qa-reporting-and-review["QA reporting and review"] --> controller-prompt

Components

Data flow

A customer message arrives through SMS, WhatsApp, or web chat. The controller prompt continues the conversation until it can classify the support topic or delegate to a specialized task prompt. Delegates collect information, run searches/RAG when the intent is known, produce structured outputs that are validated, or summarize context for a human Guide. Guardrails and deterministic controls can override model decisions for transfer, interaction limits, sensitive actions, PII, and offensive content, while memory and model-routing layers adjust context and serving choices.

Why this architecture

GoDaddy moved toward controller-delegate prompts because a single prompt became too expensive, too long, and less accurate as support topics, questions, memory, and RAG context grew. The surrounding architecture exists to bound probabilistic model behavior, preserve customer experience under latency/reliability failures, and keep humans in the loop where actions are risky or uncertain.

Models used: ChatGPT 3.5 Turbo · ChatGPT 3.5 Turbo 0603 · ChatGPT 3.5 Turbo 1106 · ChatGPT 4.0 · OpenAI GPT models · Anthropic Claude · Titan · LLaMa · LLM agents with tools · Sparse Priming Representations

Scale

• GoDaddy receives more than 60,000 daily customer contacts in messaging channels.
  “GoDaddy receives over 60,000 customer contacts every day in our messaging channels.”article, stated
• The channels include SMS, WhatsApp, and web messaging.
  “My team, Digital Care, leverages LLMs to provide an exceptional customer experience in our messaging channels (SMS, WhatsApp, and web).”article, stated
• The first AI Assistant classified conversations into twenty support topics.
  “classify the conversation into one of twenty support topics we use to bucket our support inquiries.”article, stated
• The second experiment's mega-prompt exceeded 1500 tokens.
  “The new prompt bloated to over 1500 tokens by the time we launched our second experiment”article, stated
• ChatGPT 3.5 structured outputs were invalid on about 1% of requests and ChatGPT 4 on about 0.25%.
  “they weren't perfect (we experinece invalid output on 1% of ChatGPT 3.5 and 0.25% of ChatGPT 4 requests).”article, stated
• GoDaddy saw about 1% of chat completions fail at the model provider.
  “We've experienced an average of 1% of our chat completions failing at the model provider (e.g., OpenAI).”article, stated
• ChatGPT 4 completions under 1000 tokens averaged 3-5 seconds.
  “ChatGPT 4.0, on average, respond between 3-5 seconds for completions under 1000 tokens.”article, stated
• Some calls lasted up to 30 seconds, matching the client timeout boundary.
  “As token sizes increase, the performance degrades significantly (we've seen calls lasting up to 30 seconds — when we time out our client).”article, stated
• The upstream communication provider has a 30-second integration timeout.
  “our upstream communication provider has a 30-second timeout on calls to our integration.”article, stated
• Available context sizes referenced include 32,000 tokens for OpenAI GPT and 100,000 tokens for Anthropic Claude.
  “OpenAI GPT offers up to 32,000 tokens and Anthropic Claude up to 100,000 tokens”article, stated
• SPR experiments averaged more than 50% token reduction.
  “On average, we see an over 50% reduction in token usage”article, stated
• The testing discussion considers thousands of tests run multiple times daily from CI.
  “especially when you can run thousands of tests multiple times daily from a CI pipeline.”article, stated

Alternatives considered (roads not taken)

single mega-prompt for all customer interactions

Rejected because: The prompt became costly, hit token limits in long conversations, declined in accuracy, and handled many tasks poorly.

only task-oriented prompts

Rejected because: Task prompts improve narrow flows but are not suitable for general open-ended conversations.

deterministic code alone to route to prompts

Rejected because: The earliest blend was rigid, so the team moved toward controller prompts that decide when to delegate.

let the model decide all human-transfer decisions

Rejected because: Users were sometimes stuck with an LLM that refused to transfer, so GoDaddy moved transfer decisions into deterministic controls.

retrieve RAG documents on every prompt invocation

Rejected because: Early customer messages are often pleasantries, so premature retrieval focused the model on the wrong content and reduced accuracy.

keep raw support content unchanged for RAG

Rejected because: Flowery and redundant source documents increase token use and may hurt prediction quality.

fully automated testing for natural-language conversations

Rejected because: The range of natural-language inputs is effectively infinite, and LLM-based automated testing can be cost-prohibitive at CI scale.

Patterns

multi-agent-orchestration (architecture)

GoDaddy moved from one prompt toward a controller-delegate multi-prompt architecture where a controller decides when to hand the conversation to specialized delegates.

tool-use-orchestration (architecture)

The assistant uses agent-style prompts that can request actions with parameters, let software execute the action, and feed the result back into the prompt.

rag-retrieval (knowledge access)

GoDaddy uses RAG for support content, dynamic instructions, standard support questions, and agent-crafted search terms once enough intent information is collected.

structured-output-contract (reliability)

The team requests structured outputs when needed but validates them, uses functions when available, and considers parallel prompts for user-facing text versus machine-readable structure.

safety-policy-enforcement (guardrails)

GoDaddy adds guardrails for PII, offensive content, prompt instructions, bot interaction limits, sensitive-action approval, and model-independent human transfer controls.

human-in-the-loop-gate (control)

Human support agents receive summaries and handle uncertain or risky situations, with deterministic transfer paths and external approvals rather than model-only decisions.

fallback-degradation (resilience)

The team considers provider switching, retries, degraded operation, asynchronous responses, streaming APIs, and redundant calls to handle outages, failures, and slow completions.

context-compaction (memory)

GoDaddy manages conversation context through summarization, entity tracking, recent-context retention, tool-result removal, and possible controller/delegate memory stacks.

model-routing (routing)

The team explores changing models dynamically for outages, context limits, support-contact surges, dissatisfied customers, reliability, and cost.

eval-driven-development (quality)

The deployment process relies on continuous prompt testing, output aggregation, QA review, and multidisciplinary transcript swarms after releases.

latency-optimization (performance)

GoDaddy mitigates slow model calls through retries, asynchronous acknowledgement, outbound APIs, and streaming responses while accounting for timeout limits.

Tradeoffs

Prompt architecture

Chose controller-delegate multi-prompt architecture over one growing mega-prompt.

Rationale: The controller preserves open conversation while delegates keep individual tasks concise, cheaper, and more accurate.

Structured response handling

Chose validated structured outputs and native functions when available over trusting raw conversational model output.

Rationale: Structured outputs remained fallible, and validation was necessary to protect downstream systems.

Human transfer and risky actions

Chose deterministic controls and external approval channels over model judgment alone.

Rationale: Some support decisions are too risky for probabilistic model behavior, especially transfer refusal or sensitive customer-impacting actions.

Latency mitigation

Chose retry, asynchronous messaging, and streaming APIs over blocking synchronous chat calls only.

Rationale: LLM completions can fail or exceed upstream timeouts, and chat customers should not inherit poor model latency as a support problem.

Conversation memory

Chose selective summarization, entity tracking, recent-context retention, and tool-result pruning over always retaining every message and tool output in a large context.

Rationale: Large contexts cost more and can make the model fixate on repeated concepts, recent tokens, or tool outcomes.

RAG timing

Chose intent-aware RAG and agent-crafted search terms over running retrieval on every prompt invocation.

Rationale: The team found early retrieval based on pleasantries harmed generation accuracy, while waiting for enough intent context improved relevance.

RAG content representation

Chose SPR-compressed and clustered representations in a vector store over raw, redundant knowledge-base documents.

Rationale: Compressed and clustered representations reduce token usage and may widen the retrieved knowledge space when many similar documents exist.

Runtime model strategy

Chose adaptive model selection over one fixed provider and context-size choice.

Rationale: Different providers, context windows, outage conditions, support surges, and dissatisfied customers create different reliability, cost, and capability needs.

Gotchas (what bit them)

Mega-prompts became costly, brittle, and less accurate as scope grew.

Root cause: A single prompt absorbed more topics, questions, instructions, contexts, and RAG content than it could handle cleanly.

Resolution: Use task-oriented delegates at key transitions, coordinated by a controller prompt.

Structured responses were still invalid even with function-style support.

Root cause: LLMs designed for conversational responses can mix text and structure or produce malformed machine-readable output.

Resolution: Validate structured outputs, use native function capabilities where possible, lower temperature, and consider parallel prompts.

Prompts did not transfer reliably across models or model versions.

Root cause: Different models and versions behaved differently under the same support prompts.

Resolution: Continuously fine-tune and test prompts against the specific model/version being used.

Model-controlled transfer could strand users.

Root cause: The model sometimes refused to transfer to humans when users needed an escape hatch.

Resolution: Use deterministic stop phrases, bot-interaction limits, external approvals, and default human intervention for uncertain cases.

LLM calls were slow and unreliable for real-time chat.

Root cause: Provider failures, larger-context models, growing token sizes, and upstream timeout constraints all affected chat responsiveness.

Resolution: Use retry logic, consider redundant parallel calls, move toward asynchronous responses, and adopt streaming APIs when possible.

More context was not always better.

Root cause: Large contexts can be cost-prohibitive and can cause models to fixate on repeated or recent content.

Resolution: Retain short conversations whole, summarize earlier portions of long conversations, track entities, keep recent context, and remove tool outcomes when useful.

Premature RAG degraded generation quality.

Root cause: Initial customer messages are often pleasantries, so early retrieval selected irrelevant documents and focused the model on wrong content.

Resolution: Wait until the model has enough conversation context to craft search terms or switch to a specialized RAG prompt after intent is determined.

Raw support documents wasted tokens and narrowed retrieval diversity.

Root cause: Knowledge-base content included redundant or flowery text, and many similar documents could crowd out broader knowledge in a short context.

Resolution: Compress documents into Sparse Priming Representations, store them in a vector store, and experiment with clustering to reduce duplication.

Testing natural-language LLM systems remained labor-intensive.

Root cause: Natural-language input space is infinite, prompt changes can shift behavior, and LLM-as-tester approaches can be too expensive at CI scale.

Resolution: Aggregate LLM outputs for QA review and swarm transcripts with developers, writers, product managers, business analysts, and QA after major releases.

Evaluation

GoDaddy evaluated the assistant through experiments across prompt architectures, model versions, structured-output failure rates, latency/reliability observations, RAG quality, CI-scale test feasibility, QA reporting, and post-release transcript review.

Metrics: 60,000+ daily messaging contacts · 20 support-topic classification buckets · 1500+ token mega-prompt size · 1% invalid ChatGPT 3.5 structured output · 0.25% invalid ChatGPT 4 structured output · 1% provider chat-completion failure rate · 3-5 second ChatGPT 4 completions under 1000 tokens · up to 30 second calls before client timeout · over 50% SPR token reduction · thousands of tests multiple times daily from CI

Outcomes

• GoDaddy found that LLMs outperformed older natural language units in early customer-support experiments.
• Controller-delegate prompting simplified the code base and enhanced chatbot capability in preliminary experiments.
• Prompt tuning improved ChatGPT 3.5 behavior and reduced support-case issues.
• ChatGPT functions improved structured-output accuracy but did not eliminate invalid outputs.
• Model-crafted RAG search queries improved Knowledge Base relevancy and recommendation quality.
• SPR tests showed more than 50% average token reduction, though the team had not yet fully operationalized the technique.

The deployment operated in GoDaddy's Digital Care environment with over 60,000 daily customer contacts across SMS, WhatsApp, and web, with experiments spanning support-topic classification, self-help, RAG, model comparison, guardrails, structured output, memory, and QA review.