Fine-tuning and Deploying LLMs for Customer Service Contact Centers

swisscom · Telecommunications · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: telecommunications customer-service contact centers

Swisscom needed LLM-driven contact-center dialogue that could personalize responses from customer history and service data while meeting voice-channel latency, peak-volume, lifecycle-control, and production-operability requirements.

Constraints:

• responses had to be fast enough for phone interactions
• models had to handle hundreds of requests per minute during peak times and outages
• Swisscom wanted control over model lifecycle rather than forced provider deprecations
• the production path needed repeatable training, evaluation, registry, deployment, monitoring, and security controls

Architecture

Swisscom built a fine-tuned contact-center LLM path around anonymized production-like data, synthetic prompt-completion generation from a larger teacher model, SageMaker training/evaluation pipelines, SageMaker Model Registry, and a SageMaker endpoint deployed from infrastructure-as-code. The runtime sits behind the customer-service dialogue application, uses customer/account context to generate tailored answers, and is monitored through CloudWatch and internal alerting.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  customer-service-dialogue-application[Customer service dialogue application]
  customer-data-context[Customer data context]
  synthetic-data-generation[Synthetic data generation]
  sagemaker-pipelines[SageMaker Pipelines]
  sagemaker-model-registry[SageMaker Model Registry]
  fine-tuned-llama-3-1-8b-model[Fine-tuned Llama 3.1 8B model]
  monitoring-and-alerts[Monitoring and alerts]
  customer-service-dialogue-application --> customer-data-context
  customer-service-dialogue-application --> fine-tuned-llama-3-1-8b-model
  synthetic-data-generation --> sagemaker-pipelines
  sagemaker-pipelines --> sagemaker-model-registry
  sagemaker-model-registry --> fine-tuned-llama-3-1-8b-model
  monitoring-and-alerts --> fine-tuned-llama-3-1-8b-model

Components

Data flow

For training, anonymized production-like data is converted into synthetic prompt-completion pairs with a larger teacher model, stored in S3, run through SageMaker Pipelines for training and evaluation, and registered in SageMaker Model Registry. For production use, the customer-service system uses customer context, calls the fine-tuned Llama 3.1 8B model through the deployment, and returns a dialogue answer that can be converted to speech.

Why this architecture

Swisscom chose a smaller fine-tuned model because the contact-center path needed custom task accuracy, bounded voice latency, scalability under peak request volume, lifecycle control, and hourly infrastructure economics rather than per-token billing.

Models used: Llama 3.1 8B Instruct · larger teacher LLM

Scale

• The contact-center model had to handle hundreds of requests per minute at peak times and outages.
  “Our models must be capable of handling our volume of hundreds of requests per minute at peak times and during outages where our voice channel experiences significant workload.”transcript, stated
• The production deployment was serving about 50% of CIVR requests at the time of the talk.
  “Currently in production we have like 50% traffic there of all our requests in CIVR.”transcript, stated
• The deployment served with median latency below 250 milliseconds.
  “we served in with a median latency of below 250 milliseconds. That's super crucial to our use case.”transcript, stated
• The deployed base model was Llama 3.1 with 8 billion parameters and used around 90% of GPU memory.
  “the base model is llama 3.1 with 8 billion parameters um and we use around 90% of our um GPU memory with that.”transcript, stated

Alternatives considered (roads not taken)

use a larger general-purpose LLM directly

Rejected because: the production path needed a smaller model with comparable task accuracy and bounded latency.

stay in notebook-style experimentation

Rejected because: notebooks were not production-ready, so Swisscom moved to repeatable SageMaker model-building pipelines.

Patterns

fine-tuning-adaptation (model-operations)

Swisscom fine-tuned a small Llama 3.1 8B model on synthetic prompt-completion pairs generated from anonymized production-like data by a larger teacher model.

eval-driven-development (evaluation)

Training was organized as a repeatable pipeline that included an evaluation job before storing models in the registry.

observability-tracing (operations)

The deployment used CloudWatch metrics, visualization, exports to internal monitoring, and alerts as the operational visibility layer.

latency-optimization (serving)

Swisscom optimized around a smaller fine-tuned model and a deployment target that could meet a voice-channel median latency under 250 milliseconds.

Tradeoffs

Use a smaller fine-tuned model for the contact-center path.

Chose fine-tuned Llama 3.1 8B over a larger teacher/general LLM at runtime.

Rationale: The smaller model could match task accuracy while providing bounded latency and better price/performance for peak call-center loads.

Start with a fixed production deployment before adding more dynamic scaling.

Chose two always-running instances over a more complex dynamic scaling setup immediately.

Rationale: The team prioritized a simple production foundation and identified dynamic scaling as later improvement work.

Gotchas (what bit them)

Voice-channel interactions had little tolerance for slow model responses.

Root cause: Customers calling a hotline cannot wait through multi-second generation delays.

Resolution: Swisscom selected and fine-tuned a smaller model to keep latency bounded in the contact-center path.

Notebook experiments did not provide a production-ready model lifecycle.

Root cause: The team needed repeatable builds, code versioning, evaluation, registry storage, and deployable infrastructure.

Resolution: They moved the workflow into SageMaker Pipelines and SageMaker Model Registry.

Evaluation

Swisscom evaluated the fine-tuned model inside SageMaker Pipelines, comparing accuracy with the larger teacher model and checking latency/concurrency behavior for the contact-center deployment.

Metrics: accuracy comparable to the larger teacher LLM · median latency below 250 milliseconds · 50% of CIVR traffic served by the model · hundreds of requests per minute peak-load requirement

Outcomes

• fine-tuned small model delivered accuracy comparable to the larger teacher model
• production traffic reached about 50% of CIVR requests
• median latency was below 250 milliseconds
• hourly SageMaker deployment economics supported peak-load serving

The model was in production for about half of CIVR requests, served below 250ms median latency, and was designed around hundreds of requests per minute at peak load.