Optimizing Call Center Analytics with Small Language Models and Multi-Adapter Serving

convirza · Telecommunications · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Call center analytics

Convirza needed to extract many custom indicators from millions of monthly calls without the cost and complexity of one deployed model per indicator.

Constraints:

• The platform processes millions of monthly calls.
• Customers require hundreds of custom indicators.
• Separate model deployments per indicator were costly and complex.
• Insights needed to be near real-time.

Architecture

Convirza moved from OpenAI GPT and custom Longformer deployments to fine-tuned Llama 3.1 8B with LoRA adapters served through Predibase.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  call-transcript-inputs["Call Transcript Inputs"]
  llama-3-1-8b-serving["Llama 3.1 8B Serving"]
  lora-adapter-pool["LoRA Adapter Pool"]
  analytics-output["Analytics Output"]
  call-transcript-inputs --> llama-3-1-8b-serving
  llama-3-1-8b-serving --> lora-adapter-pool
  lora-adapter-pool --> analytics-output

Components

Data flow

Call transcripts are routed to a fine-tuned Llama 3.1 8B serving stack, the appropriate LoRA adapter handles each indicator extraction task, and the platform returns analytics with near real-time latency.

Why this architecture

A small language model with many adapters reduced serving cost and complexity while improving throughput and quality.

Models used: Llama 3.1 8B · LoRA adapters · Predibase

Scale

• The platform handles millions of monthly calls.
  “handle millions of monthly calls”summary, stated
• The serving stack can run 60+ adapters on a single GPU.
  “run 60+ adapters on a single GPU.”summary, stated
• Inference time is sub-0.1 second.
  “sub-0.1 second inference times”summary, stated

Alternatives considered (roads not taken)

OpenAI GPT models

Rejected because: The LoRA/Llama stack reduced cost by 10x compared with OpenAI.

Separate deployed model per indicator

Rejected because: That approach was costly and complex for hundreds of indicators.

Patterns

model-routing (serving)

Different LoRA adapters specialize the same base model for many indicators.

cost-aware-routing (operations)

Adapter serving lowered cost and improved throughput compared with GPT and separate models.

fine-tuning-adaptation (model-adaptation)

The system fine-tunes Llama 3.1 8B with LoRA adapters for indicator extraction.

Tradeoffs

Use SLM plus adapters

Chose Fine-tuned Llama 3.1 8B with LoRA over OpenAI GPT and custom Longformer deployments.

Rationale: The adapter stack improved cost, F1, throughput, and latency.

Gotchas (what bit them)

Custom indicators can explode serving complexity.

Root cause: Each indicator previously required separately deployed model behavior.

Resolution: Serve many LoRA adapters from a shared base model.

Evaluation

Convirza measured cost, F1, throughput, inference time, and adapter density.

Metrics: Cost reduction · F1 score · Throughput · Inference time · Adapters per GPU

Outcomes

• 10x lower cost than OpenAI.
• 8% higher F1 scores.
• 80% higher throughput.
• Sub-0.1 second inference.
• 60+ adapters on one GPU.

The platform handles millions of monthly calls and hundreds of custom indicators.