Optimizing RAG Latency Through Model Racing and Self-Hosted Infrastructure

elevenlabs · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production RAG latency

ElevenLabs needed to reduce RAG latency caused mostly by query rewriting through a single external LLM dependency.

Constraints:

Query rewriting accounted for over 80% of RAG latency.
The prior architecture depended on one externally hosted LLM.
The system needed lower latency and better provider-outage resilience.
Self-hosted models had to compete with hosted models in production.

Architecture

ElevenLabs implemented model racing, running multiple hosted and self-hosted models in parallel and accepting the first valid query-rewrite response.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  rag-query["RAG Query"]
  model-racing-layer["Model Racing Layer"]
  self-hosted-qwen-models["Self-Hosted Qwen Models"]
  retriever["Retriever"]
  hosted-model["Hosted Model"]
  rag-query --> model-racing-layer
  model-racing-layer --> hosted-model
  model-racing-layer --> self-hosted-qwen-models
  model-racing-layer --> retriever
  self-hosted-qwen-models --> model-racing-layer

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`RAG Query`	Starts the retrieval workflow.	`Model Racing Layer`
`Model Racing Layer`	Runs multiple rewrite models in parallel and selects the first valid response.	`Hosted Model`, `Self-Hosted Qwen Models`, `Retriever`
`Self-Hosted Qwen Models`	Provide low-latency fallback and racing candidates.	`Model Racing Layer`
`Retriever`	Runs retrieval after query rewriting.	—

Data flow

A RAG query is sent to several query-rewriting models in parallel, including self-hosted Qwen models; validation checks responses; the first valid rewrite wins; retrieval and answer generation proceed with lower median latency.

“multiple models (including self-hosted Qwen 3-4B and 3-30B-A3B models) process queries in parallel, with the first valid response winning.”summary, stated

“query rewriting accounted for over 80% of RAG latency due to reliance on a single externally-hosted LLM.”summary, stated

Why this architecture

Parallel racing reduced tail risk and provider dependency while letting the fastest valid model determine latency.

“providing fallbacks during provider outages and reducing dependency on external services.”summary, stated

Models used: Qwen 3-4B · Qwen 3-30B-A3B · Externally hosted LLM

Scale

Median RAG latency fell from 326ms to 155ms.
“reduced median RAG latency from 326ms to 155ms”summary, stated
Latency improved by 50%.
“a 50% improvement”summary, stated

Alternatives considered (roads not taken)

Single external LLM for query rewriting

Rejected because: It caused most latency and created provider dependency.

“reliance on a single externally-hosted LLM.”summary, stated

Patterns

`model-routing` (latency)

Model racing routes to the first valid response among parallel models.

“with the first valid response winning.”summary, stated

`cost-aware-routing` (serving)

Self-hosted models reduce dependence on external services and improve resilience.

“reducing dependency on external services.”summary, stated

Tradeoffs

Run models in parallel

Chose Model racing over Sequential fallback or one provider.

Rationale: Parallelism reduced median latency and improved outage resilience.

“process queries in parallel, with the first valid response winning.”summary, stated

Gotchas (what bit them)

A small preprocessing step can dominate RAG latency.

Root cause: Query rewriting accounted for over 80% of latency.

Resolution: Race multiple rewrite models and accept the first valid output.

“query rewriting accounted for over 80% of RAG latency”summary, stated

Evaluation

ElevenLabs measured median RAG latency before and after model racing.

Metrics: Median RAG latency · Provider resilience

“from 326ms to 155ms”summary, stated

Outcomes

Median RAG latency fell from 326ms to 155ms.
Latency improved 50%.
Provider-outage resilience improved through fallbacks.

The key production metric is median RAG latency.

“also improving system resilience by providing fallbacks during provider outages”summary, stated