Building a Multi-Agent LLM Platform for Customer Service Automation

deutsche_telekom · Telecommunications · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Telecom customer-service automation platform

Deutsche Telekom needed to deploy customer-facing LLM agents across European countries, languages, chat and voice channels, and business domains without each team rebuilding routing, lifecycle, retrieval, testing, guardrails, and deployment infrastructure.

Constraints:

• the platform had to span around 10 countries, multiple languages, and chat, voice, and autonomous channels
• different countries and channels require different business processes, APIs, and prompt flows
• customer-facing LLMs were non-deterministic, fragile, jailbreakable, and not fully production-ready
• JVM investments, SDKs, client libraries, and observability systems made a Python-only stack unattractive
• multiple teams needed to build agents in parallel without prompt changes collapsing the entire chatbot

Architecture

Deutsche Telekom built LMOS, an open-source agent computing platform for Frag Magenta 1BOT. A customer-facing chatbot sends requests to an LMOS router, which uses advertised agent capabilities for intent-based routing across isolated business-domain agents. Agents are written with the Kotlin ARC DSL, package as Docker/Helm deployments, call platform services such as customer APIs and search/RAG APIs, and run under a Kubernetes/Istio control plane with channels, runtime bootstrapping, lifecycle management, filters, sensitive-data handling, and rollout controls.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  frag-magenta-chatbot["Frag Magenta chatbot"] --> channel["Channel"]
  frag-magenta-chatbot --> lmos-router["LMOS router"]
  channel --> lmos-router
  lmos-router --> billing-agent["Billing agent"]
  lmos-router --> faq-agent["FAQ agent"]
  lmos-router --> news-agent["News agent"]
  lmos-router --> weather-agent["Weather agent"]
  billing-agent --> lmos-arc["LMOS ARC"]
  billing-agent --> customer-api["Customer API"]
  billing-agent --> search-api["Search API"]
  billing-agent --> faq-agent
  faq-agent --> search-api
  agent-platform["Agent platform"] --> customer-api
  agent-platform --> search-api
  agent-platform --> lmos-kernel["LMOS kernel"]
  lmos-kernel --> lmos-arc
  lmos-control-plane["LMOS control plane"] --> lmos-router
  lmos-control-plane --> channel
  lmos-control-plane --> lmos-runtime["LMOS runtime"]
  lmos-control-plane --> lmos-arc
  lmos-control-plane --> non-arc-agents["Non-ARC agents"]

Components

Data flow

A customer request arrives through a chat, voice, web, or app channel. The LMOS runtime groups the relevant agents for that channel, and the router picks an agent based on intent and advertised capabilities. The selected isolated agent uses a dynamically generated prompt, filters, knowledge blocks, tools such as billing or contract APIs, optional fallback to FAQ/RAG, and output filters. Sensitive data is anonymized before LLM use and deanonymized before the customer response. Agents are packaged and deployed as Docker images/Helm resources, while the control plane manages lifecycle, routing, channels, and traffic.

Why this architecture

LMOS was built because rudimentary prompt abstractions and one-off functions would not scale across Deutsche Telekom's countries, channels, APIs, teams, and release needs. Isolated domain agents limit prompt-Jenga blast radius, the platform absorbs hard concerns such as routing, lifecycle, RAG, guardrails, testing, and deployment, and the Kubernetes-native control plane lets agents become first-class deployable resources.

Models used: GPT-4o · GPT-4o mini · GPT-3.5 · OpenAI tool-calling models · internal German PII model

Scale

• The deployment target covered around 10 European countries.
  “How do you deploy GenAI across our European footprint, which is around 10 countries? Also, for all of the channels through which customers reach us, which is chat channel, voice channel, and also autonomous use cases where this might come in.”article, stated
• Deutsche Telekom had around 100 million customers in Europe.
  “We have around 100 million customers only in Europe, and they reach us through all these channels. We knew that voice models are going to emerge, so we needed something fundamental, it was pretty clear.”article, stated
• The deployed LLM agents answered more than one million questions.
  “We have had so far, more than a million questions answered by the use cases for which we have deployed this, with an 89% acceptable answer rate.”article, stated
• The acceptable-answer rate was 89%.
  “more than a million questions answered by the use cases for which we have deployed this, with an 89% acceptable answer rate.”article, stated
• More than 300,000 human-agent conversations were deflected with a risk rate under 2%.
  “That is more than 300,000 human-agent conversations deflected with a risk rate under 2%. Not only that, we were able to benchmark what we built against some of the LLM powered vendor products.”article, stated
• Production A/B testing showed around 38% better agent handovers than vendor products.
  “this is around 38% agent handovers were better in comparison to the vendor products, for the same use cases that we tried.”article, stated
• Agent development time fell from two months to 10 days.
  “When we started, it was 2 months, and now it has brought down to 10 days. This involves a lot of discovery of the business processes, API integration, and everything.”article, stated
• Simple feature/use-case enhancement dropped to 2.5 days.
  “billing queries. This is the usual development lifecycle, not building agents every day. It used to take weeks, and it is now brought down to 2.5 days. Earlier we used to release only one per month. As most of you might know, the brittleness or the fragility of these kind of systems, you”article, stated
• Production release cadence improved from one per month to two per week.
  “Earlier we used to release only one per month. As most of you might know, the brittleness or the fragility of these kind of systems, you cannot release fast, especially for a company with a brand like Deutsche Telekom, it can be jailbreaked if you don't do the necessary tests.”article, stated

  “We brought it down to two per week in production. Risky answer, there are a lot of goof-ups as well, or the latest one was someone jailbreaked, or bought and turned it into a [inaudible 00:15:26] bot or something.”article, stated

Alternatives considered (roads not taken)

Rudimentary functions on top of prompts

Rejected because: The team concluded that this would not scale across countries, business processes, APIs, and specifications.

Python-first or off-the-shelf LangChain-style stack

Rejected because: Deutsche Telekom had JVM investments, SDKs, and observability libraries, and there was not enough in the JVM ecosystem for their scale needs.

One large prompt/chatbot containing all business domains

Rejected because: Prompt changes are fragile; isolated microservice agents limit failures to one domain agent rather than the whole chatbot.

Traditional weight/canary-only routing

Rejected because: Multi-agent systems need intent-based routing from agent capabilities, not just static traffic weights.

Closed ARC-only platform

Rejected because: LMOS was designed to run Python, LangChain, LlamaIndex, or other agents as long as they follow the runtime specs.

Off-the-shelf dialog-tree vendor product

Rejected because: The previous vendor bot had low solution rates, while the LMOS architecture mixed dialog predictability with LLM flexibility and beat vendor handover performance.

Patterns

multi-agent-orchestration (agents)

LMOS composes a customer-facing chatbot from multiple isolated domain agents and routes each incoming request to the right agent based on intent.

model-routing (routing)

LMOS routes work between agents by advertised capability and intent rather than only by weights or canaries.

rag-retrieval (retrieval)

RAG pipelines live behind the search API and FAQ agent rather than being reimplemented inside every business-domain agent.

guardrails-io-filtering (safety)

ARC filters detect human-agent requests, hacking attempts, fabricated links, unresolved answers, and output artifacts before continuing or falling back.

privacy-preserving-processing (safety)

The agent marks personalized conversations as sensitive, anonymizes IBANs before LLM use, and deanonymizes only before the customer response.

canary-rollout (operations)

The platform supports channel and small-percentage releases, allowing agents to be exposed to 1% or 2% of customers before broad rollout.

latency-optimization (operations)

LMOS can execute some LLM checks in parallel and uses smaller models for simpler filters to reduce response delay.

eval-driven-development (evaluation)

Flagged questions become test cases for future releases, and human annotators are used because fully automated hallucination/risk detection is not solved.

prompt-versioning (operations)

The platform treats prompts as brittle release artifacts and isolates them per agent so prompt changes do not collapse the whole chatbot.

fallback-degradation (operations)

If the primary agent cannot answer, an unresolved detector can trigger fallback to the FAQ/RAG agent.

Tradeoffs

Core platform strategy

Chose build LMOS as an open-source JVM/Kotlin-friendly platform over adopting a Python-first or existing-framework-only approach.

Rationale: Deutsche Telekom had JVM investments and needed enterprise-grade constructs unavailable in the ecosystem at the time.

Agent topology

Chose isolated microservice agents by business domain over a single monolithic multi-domain chatbot prompt.

Rationale: Multiple teams could work in parallel and prompt failures had a smaller blast radius.

Knowledge placement

Chose curated one-to-two-page knowledge blocks for most agents over RAG pipelines inside every agent.

Rationale: Modern context windows made small curated knowledge blocks enough for many agents, while central RAG stayed behind the FAQ/search layer.

Data protection inside tools

Chose anonymize sensitive data before it reaches the LLM over passing real customer data directly through tool results.

Rationale: Tool output is fed back into the LLM, so real personal data must be hidden until the final customer-visible response.

Routing mechanism

Chose capability-advertised intent routing over manual routing configuration for every new agent/channel.

Rationale: The router can bootstrap itself from deployed agent capabilities and automatically pick the right agent for a channel.

Gotchas (what bit them)

LLM prompts behaved like prompt Jenga.

Root cause: Small prompt changes could break unrelated behavior if every domain lived in one prompt.

Resolution: The platform isolates domains into separate agents so a bad prompt change breaks one agent instead of the whole chatbot.

Customer-facing models could return surplus or fabricated information.

Root cause: LLMs naturally fill space, may fabricate links or phone numbers, and mix Markdown/JSON artifacts into output.

Resolution: The team uses concise prompts, low temperature/static seed, output cleanup, link verification, and filters.

LLM testing remained difficult and still needed people in the loop.

Root cause: There was no fully reliable automated technique to identify hallucinations or risky answers.

Resolution: Human annotators, flagged-question test sets, small blast-radius releases, and isolated microservices reduce risk.

Voice and chat channels could not share naive prompt flows.

Root cause: Voice channels have different interaction constraints, such as not being able to send links.

Resolution: LMOS uses channels and tenant/channel management so agents can be scoped and adapted by country and channel.

Production LLM agents must be designed for failure and fast intervention.

Root cause: Jailbreaks and risky answers can happen despite testing.

Resolution: LMOS includes constructs that let teams intervene and deploy fixes within hours.

Evaluation

Deutsche Telekom combines human annotators, flagged-question regression tests, production A/B testing against vendor products, acceptable-answer/risk metrics, guardrail tests, and small-percentage releases to validate customer-facing agents.

Metrics: acceptable answer rate · risk rate · human-agent conversations deflected · agent handover performance against vendors · release cadence · agent development time · feature development time · flagged-question regression tests

Outcomes

• LMOS became a fully open-source multi-agent platform for Deutsche Telekom customer-facing AI agents.
• The platform answered more than one million questions with an 89% acceptable-answer rate.
• More than 300,000 human-agent conversations were deflected with under 2% risk rate.
• Production A/B testing showed around 38% better agent handovers than vendor LLM products.
• Agent development time fell from two months to 10 days, and production release cadence reached two per week.

LMOS supports a European customer-service footprint of around 10 countries, 100 million customers, multiple channels, country/channel scoping, and domain agents such as billing, contracts, weather, news, FAQ/RAG, and future non-ARC agents.