Optimizing Production LLM Chatbot Performance Through Multi-Model Classification

idiada · Automotive · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise chatbot request routing

Applus+ IDIADA needed AIDA to route diverse workforce chatbot requests into the right processing pipelines efficiently and accurately.

Constraints:

The chatbot serves automotive workforce interactions.
Classification approaches included LLM-only, RAG-augmented LLM, and classical ML over embeddings.
The production system handles more than 1,000 daily interactions.
Latency and cost mattered alongside accuracy.

Architecture

IDIADA compared LLM classifiers against embedding-based classical ML classifiers and selected the faster, higher-performing embedding/SVM approach using Cohere multilingual embeddings.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  aida-chatbot["AIDA Chatbot"]
  embedding-classifier["Embedding Classifier"]
  processing-pipelines["Processing Pipelines"]
  aida-chatbot --> embedding-classifier
  embedding-classifier --> processing-pipelines

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`AIDA Chatbot`	Receives workforce chatbot requests.	`Embedding Classifier`
`Embedding Classifier`	Classifies request type using embeddings and classical ML.	`Processing Pipelines`
`Processing Pipelines`	Handle conversation, service, and document-translation workflows.	—

Data flow

User requests enter AIDA, embeddings are computed with Titan or Cohere models, classical classifiers route the request to Conversation, Services, or Document Translation pipelines, and the routed chatbot flow responds.

“embedding-based approaches with classical ML classifiers significantly outperformed the LLM-based methods”summary, stated

“SVM using Cohere embeddings achieving 0.99 F1 for Conversation, 0.80 for Services, and 0.93 for Document Translation”summary, stated

Why this architecture

Classical ML over embeddings beat LLM-based classifiers for this routing task in both performance and speed.

“while also being 4-60x faster than LLM inference.”summary, stated

Models used: Amazon Titan embeddings · Cohere multilingual embeddings · SVM · k-NN · ANN · Amazon Bedrock

Scale

The optimized system handles over 1,000 daily interactions.
“handles over 1,000 daily interactions with 95% accuracy”summary, stated
The system reports 95% accuracy.
“95% accuracy”summary, stated
Team productivity increased by 20%.
“20% increase in team productivity.”summary, stated

Alternatives considered (roads not taken)

Pure LLM classifiers

Rejected because: Embedding-based classical classifiers outperformed them and were faster.

“pure LLM-based classifiers, LLM with example augmentation (RAG), and traditional ML algorithms”summary, stated

Patterns

`model-routing` (routing)

A classifier routes user requests to the right chatbot processing path.

“efficiently routing diverse user requests to appropriate processing pipelines.”summary, stated

`eval-driven-development` (model-selection)

The team compared several classifier families and selected the measured winner.

“experimented with multiple classification approaches”summary, stated

`cost-aware-routing` (latency)

Classical classifiers were 4-60x faster than LLM inference.

“4-60x faster than LLM inference.”summary, stated

Tradeoffs

Use embedding-based SVM routing

Chose Cohere embeddings plus SVM over LLM inference classifiers.

Rationale: It achieved strong F1 and was much faster.

“SVM using Cohere embeddings achieving 0.99 F1 for Conversation, 0.80 for Services, and 0.93 for Document Translation”summary, stated

Gotchas (what bit them)

LLM classifiers are not automatically best for production routing.

Root cause: The routing task fit embedding/classical ML methods better.

Resolution: Benchmark LLM, RAG-augmented, and classical approaches before choosing.

“embedding-based approaches with classical ML classifiers significantly outperformed the LLM-based methods”summary, stated

Evaluation

IDIADA measured F1 by class, speed versus LLM inference, daily interactions, accuracy, and productivity lift.

Metrics: F1 by class · Inference speed · Daily interactions · Accuracy · Productivity increase

“0.99 F1 for Conversation, 0.80 for Services, and 0.93 for Document Translation”summary, stated

Outcomes

The optimized system handles 1,000+ daily interactions.
Accuracy is reported at 95%.
Team productivity increased 20%.
The chosen classifier is 4-60x faster than LLM inference.

The system routes over 1,000 daily employee chatbot interactions.

“driving a reported 20% increase in team productivity.”summary, stated