Multi-Model LLM Orchestration with Rate Limit Management

bito · Tech · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI coding assistant infrastructure

Bito needed to scale an LLM-powered coding assistant despite provider TPM/RPM limits, variable latency, model-specific prompts, and local code-indexing requirements.

Constraints:

Single-provider OpenAI limits could not handle traffic growth.
Different request types need different context windows and reasoning capability.
Prompt behavior does not transfer cleanly across GPT and Claude models.
Code indexing should keep user data local.

Architecture

Bito built a custom multiplexer/load balancer routing across OpenAI accounts, Azure OpenAI, Anthropic Claude, GPT-3.5, GPT-4, Claude Instant, and Claude 2 based on capacity, context size, request type, cost, and model health, with graceful degradation and model-specific prompt repositories.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ide-extensions["IDE Extensions"]
  llm-load-balancer["LLM Load Balancer"]
  openai-accounts["OpenAI Accounts"]
  azure-openai["Azure OpenAI"]
  anthropic-claude["Anthropic Claude"]
  prompt-repository["Prompt Repository"]
  local-vector-index["Local Vector Index"]
  ide-extensions --> llm-load-balancer
  ide-extensions --> local-vector-index
  llm-load-balancer --> openai-accounts
  llm-load-balancer --> azure-openai
  llm-load-balancer --> anthropic-claude
  llm-load-balancer --> prompt-repository
  openai-accounts --> llm-load-balancer
  azure-openai --> llm-load-balancer
  anthropic-claude --> llm-load-balancer
  prompt-repository --> llm-load-balancer
  local-vector-index --> ide-extensions

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`IDE Extensions`	Expose code explanation, generation, testing, and understanding features in VS Code and JetBrains.	`LLM Load Balancer`, `Local Vector Index`
`LLM Load Balancer`	Routes requests by capacity, context, request type, cost, and model health.	`OpenAI Accounts`, `Azure OpenAI`, `Anthropic Claude`, `Prompt Repository`
`OpenAI Accounts`	Serve GPT-3.5, GPT-4, and account-specific limits.	`LLM Load Balancer`
`Azure OpenAI`	Provides additional OpenAI model capacity through Azure.	`LLM Load Balancer`
`Anthropic Claude`	Provides Claude models for fallback and larger contexts.	`LLM Load Balancer`
`Prompt Repository`	Stores model-specific prompt variants.	`LLM Load Balancer`
`Local Vector Index`	Indexes user code locally to keep code data on the user's machine.	`IDE Extensions`

Data flow

IDE extension requests enter Bito's routing layer, which checks capacity, context length, task type, cost, and provider health. It selects an LLM provider, applies model-specific prompts, falls back when limits are hit, and uses local vector indexing for code context.

“They built a custom load balancer and model orchestration system that routes requests across OpenAI (GPT-3.5 and GPT-4), Azure OpenAI, and Anthropic (Claude)”summary, stated

“The load balancer makes routing decisions based on several factors:”summary, stated

Why this architecture

Multi-provider orchestration keeps the user experience responsive when one provider hits limits or degrades, while model-specific prompts handle behavioral differences across LLMs.

“Having multiple fallback options ensures users get answers even when individual providers experience issues.”summary, stated

“prompts are not portable across models. What works on GPT-3.5 may not work on GPT-4, and neither will work identically on Anthropic's models.”summary, stated

Models used: GPT-3.5 · GPT-4 · Azure OpenAI · Claude Instant · Claude 2

Scale

Bito hit TPM and RPM limits while scaling beyond one OpenAI account.
“hitting TPM (tokens per minute) and RPM (requests per minute) limits across LLM providers.”summary, stated
Provider latency could vary from 1 second to more than 15 seconds.
“latency can vary from 1 second to 15+ seconds depending on platform load”summary, stated

Alternatives considered (roads not taken)

Stay on one model/provider

Rejected because: Rate limits and context windows could not support growing usage.

“Starting with a single OpenAI account, they quickly encountered rate limiting issues as usage grew.”summary, stated

Use one prompt across all models

Rejected because: Different models require different prompting strategies and output handling.

“GPT-4 prompts don't transfer directly.”summary, stated

Patterns

`model-routing` (architecture)

Bito routes coding requests across OpenAI, Azure OpenAI, and Anthropic by capacity, cost, context, and health.

“routes requests across OpenAI (GPT-3.5 and GPT-4), Azure OpenAI, and Anthropic (Claude) based on context size, request type, cost, and availability.”summary, stated

`rag-retrieval` (retrieval)

A local vector database indexes code context for retrieval without sending all code remotely.

“a homegrown vector database for code indexing that keeps data local to user machines.”summary, stated

`eval-driven-development` (quality)

Bito uses user feedback, human investigation, curated test cases, and regression testing.

“Running prompts against test suites, especially after model updates”summary, stated

Tradeoffs

Graceful degradation

Chose Fallback to cheaper or smaller models and reduce context when necessary over Failing requests when the preferred model is unavailable.

Rationale: The product prioritizes returning a useful answer under provider limits.

“When falling back to models with smaller context windows, reduce the provided context while still attempting to answer the question”summary, stated

Local code indexing

Chose Homegrown vector database on user machines over Centralized code index.

Rationale: Local indexing helps preserve code privacy and context availability.

“homegrown vector database for code indexing that keeps data local to user machines.”summary, stated

Gotchas (what bit them)

Prompt portability breaks across model providers.

Root cause: GPT-3.5, GPT-4, and Claude follow instructions differently.

Resolution: Maintain a prompt repository with model-specific variants.

“This leads to maintaining what Anas calls a "prompt repository" with model-specific variations.”summary, stated

Automatic LLM evaluation can reinforce mistakes.

Root cause: The evaluator may not know the correct answer for technical questions.

Resolution: Use human-verified test cases and manual investigation for hallucinations.

“Rather than relying on LLMs to evaluate LLM outputs (which Anas notes can lead to self-reinforcing errors), they maintain human-verified test cases with known correct answers”summary, stated

Evaluation

Bito combines thumbs up/down feedback, human review of hallucinations, curated test scenarios, and regression testing after model updates.

Metrics: thumbs up/down · hallucination investigations · test-suite regression

“Bito's evaluation strategy relies heavily on human feedback and manually curated test cases:”summary, stated

Outcomes

Bito scaled a coding assistant through multi-provider load balancing and fallback.
The system handles provider rate limits, context-window differences, and model-specific prompts.
The approach keeps code indexing local to user machines.

The case reports rate-limit pressure and 1s to 15+ second provider latency variation, but not user count.

“Their approach prioritizes user experience through intelligent routing while managing costs and handling the operational complexity of maintaining separate prompts for each model.”summary, stated