Multi-LoRA Serving for Agent Performance Analysis at Scale

convirza · Tech · 2024 · source Agentic: no · promotional bias: high · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Call analytics and agent-performance scoring

Convirza needed to evaluate customer-service calls across 60+ performance indicators without slow Longformer training loops or high per-indicator serving costs.

Constraints:

Legacy training iterations took 9-24+ hours.
Separate models for each indicator cost $500-$1,500 monthly.
The platform processes millions of calls per month.
Peak workloads require autoscaling GPU capacity.

Architecture

Convirza adopted Predibase multi-LoRA serving on a single Meta Llama-3-8b base model. Separate LoRA adapters specialize each of 60+ performance indicators while LoRAX dynamically loads adapters, batches requests, and autoscales A100 GPU infrastructure.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  call-analytics-platform["Call Analytics Platform"]
  indicator-adapter-router["Indicator Adapter Router"]
  lorax-multi-lora-server["LoRAX Multi-LoRA Server"]
  llama-3-8b-base-model["Llama-3-8b Base Model"]
  autoscaling-gpu-infrastructure["Autoscaling GPU Infrastructure"]
  call-analytics-platform --> indicator-adapter-router
  indicator-adapter-router --> lorax-multi-lora-server
  lorax-multi-lora-server --> llama-3-8b-base-model
  lorax-multi-lora-server --> autoscaling-gpu-infrastructure
  llama-3-8b-base-model --> lorax-multi-lora-server
  autoscaling-gpu-infrastructure --> lorax-multi-lora-server

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Call Analytics Platform`	Processes calls and produces agent-performance scorecards.	`Indicator Adapter Router`
`Indicator Adapter Router`	Selects the LoRA adapter corresponding to each performance indicator.	`LoRAX Multi-LoRA Server`
`LoRAX Multi-LoRA Server`	Loads adapters, batches requests, and applies them to the shared base model.	`Llama-3-8b Base Model`, `Autoscaling GPU Infrastructure`
`Llama-3-8b Base Model`	Shared model serving all specialized adapters.	`LoRAX Multi-LoRA Server`
`Autoscaling GPU Infrastructure`	Scales A100 GPU capacity during variable call-processing load.	`LoRAX Multi-LoRA Server`

Data flow

Call analytics requests select the relevant performance indicator adapter, LoRAX applies that LoRA adapter to the shared Llama-3-8b base model, and autoscaled GPUs serve predictions for scorecards under variable load.

“When an inference request comes in, the appropriate adapter is loaded and applied to the base model to generate predictions.”summary, stated

“The multi-LoRA serving approach (LoRAX) handles the orchestration of loading and unloading adapters efficiently.”summary, stated

Why this architecture

Multi-LoRA serving avoids deploying a full model per indicator. Small adapters share one base model, reducing memory and compute while preserving specialization and faster iteration.

“Instead of deploying 60+ separate model instances (each requiring dedicated GPU resources), Convirza can now share a single base model across all adapters.”summary, stated

“The LoRA adapters are small (typically a few megabytes) and can be loaded dynamically at inference time”summary, stated

Models used: Meta Llama-3-8b · LoRA adapters · Longformer

Scale

The legacy system used more than 60 performance indicators.
“over 60 performance indicators”summary, stated
Training iterations previously took 9-24+ hours.
“Training times were a major bottleneck, with each iteration taking anywhere from 9 to 24+ hours.”summary, stated
Convirza trained and deployed adapters for more than 20 indicators in the first month.
“In the first month after adoption, they trained and deployed adapters for over 20 indicators.”summary, stated
The platform needs double-digit A100 GPUs during peak periods.
“their workload requires scaling up to double-digit A100 GPUs during peak periods.”summary, stated

Alternatives considered (roads not taken)

Separate Longformer models per indicator

Rejected because: Training was slow and serving many separate models was expensive.

“slow training times (9-24+ hours per iteration) and high infrastructure costs ($500-$1,500 per indicator monthly) when using Longformer models”summary, stated

OpenAI API for equivalent functionality

Rejected because: The Predibase setup is reported as cheaper, higher-throughput, and more accurate for their evaluation tasks.

“10x cost reduction compared to OpenAI, 8% higher F1 scores, 80% increased throughput”summary, stated

Patterns

`fine-tuning-adaptation` (modeling)

Separate LoRA adapters specialize the base model for more than 60 call-performance indicators.

“This approach allowed Convirza to train separate LoRA adapters for each of their 60+ performance indicators”summary, stated

`model-routing` (architecture)

The server dynamically loads the correct adapter for each inference request.

“the appropriate adapter is loaded and applied to the base model to generate predictions.”summary, stated

Tradeoffs

Use managed Predibase infrastructure

Chose Managed multi-LoRA training and serving over Building custom ML infrastructure internally.

Rationale: Custom infrastructure would distract from core product development.

“would have diverted focus from their core product development.”summary, stated

Use LoRA adapters

Chose Small dynamically loaded adapters on one base model over Full fine-tuned model per indicator.

Rationale: Adapters reduce serving cost and improve iteration speed.

“Training LoRA adapters is also substantially faster than full model fine-tuning.”summary, stated

Gotchas (what bit them)

Full-model-per-indicator serving does not scale economically.

Root cause: Each indicator-specific model consumes dedicated GPU resources.

Resolution: Share one base model and serve small adapters dynamically.

“dramatically reducing the memory footprint and compute requirements compared to running multiple full models.”summary, stated

Call-processing workloads are spiky.

Root cause: Peak-hour demand requires significantly more GPU capacity than quieter periods.

Resolution: Use autoscaling GPU infrastructure with minimized cold-start times.

“call processing workloads can be highly variable with significant peak-hour demand.”summary, stated

Evaluation

Convirza compared F1 score, cost, throughput, and inference latency against OpenAI and Longformer baselines.

Metrics: F1 score · cost · throughput · mean inference time

“Cost: 10x reduction compared to OpenAI API costs for equivalent functionality”summary, stated

Outcomes

Convirza achieved a reported 10x cost reduction versus OpenAI.
The system achieved 8% higher F1 and 80% increased throughput.
It serves 60+ adapters from a single Llama-3-8b base model.

Summary-only extraction: source cache unusable. Reported scale includes millions of calls per month, 60+ indicators, double-digit A100 peak scaling, and sub-second mean inference.

“The system achieves sub-second mean inference time even while serving 60+ adapters, with an average response time under two seconds according to the quoted metrics.”summary, stated