Privacy-Preserving University Chatbot with LiteLLM Proxy for Multi-Model Governance and Cost Control

unnamed_private_university · Education · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: University LLM gateway governance and cost control

A private university needed a privacy-preserving chatbot for students and employees that could swap models, potentially self-host models later, distribute API access to courses, enforce budgets, and handle provider capacity limits without locking applications to one LLM provider.

Constraints:

• Models needed to be easy to update because the LLM provider landscape changes quickly.
• The university wanted the option to self-host models in the future.
• Students, employees, courses, and departments needed governed API access through a consistent interface.
• Heavy power-user traffic made model costs higher than expected and required automated budget enforcement.
• Provider rate limits and concurrent usage required routing, retries, additional deployments, and fallback behavior.

Architecture

The implementation uses LiteLLM's proxy server as an OpenAI-compatible gateway between university applications and multiple model providers. The proxy is deployed from a Docker image with a mounted YAML configuration defining model names, provider credentials, and routing rules. PostgreSQL enables spend logging and budget enforcement by user/team, Redis supports load-balancing strategies across model deployments, the admin UI supports non-technical budget administration, and application code can keep using direct HTTP calls or the OpenAI client against a custom base URL.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  university-chat-application["University chat application"] --> litellm-proxy["LiteLLM proxy"]
  litellm-proxy["LiteLLM proxy"] --> university-chat-application["University chat application"]
  litellm-proxy["LiteLLM proxy"] --> model-configuration["Model configuration"]
  model-configuration["Model configuration"] --> litellm-proxy["LiteLLM proxy"]
  litellm-proxy["LiteLLM proxy"] --> postgresql-budget-database["PostgreSQL budget database"]
  postgresql-budget-database["PostgreSQL budget database"] --> litellm-proxy["LiteLLM proxy"]
  litellm-proxy["LiteLLM proxy"] --> redis-routing-cache["Redis routing cache"]
  redis-routing-cache["Redis routing cache"] --> litellm-proxy["LiteLLM proxy"]
  litellm-proxy["LiteLLM proxy"] --> model-provider-deployments["Model provider deployments"]
  model-provider-deployments["Model provider deployments"] --> litellm-proxy["LiteLLM proxy"]
  litellm-proxy["LiteLLM proxy"] --> admin-user-interface["Admin user interface"]
  admin-user-interface["Admin user interface"] --> litellm-proxy["LiteLLM proxy"]
  admin-user-interface["Admin user interface"] --> postgresql-budget-database["PostgreSQL budget database"]
  postgresql-budget-database["PostgreSQL budget database"] --> admin-user-interface["Admin user interface"]
  litellm-proxy["LiteLLM proxy"] --> observability-integrations["Observability integrations"]
  observability-integrations["Observability integrations"] --> litellm-proxy["LiteLLM proxy"]

Components

Data flow

A student or employee sends a chat request through the university application. The app calls the LiteLLM proxy, either directly over HTTP or through an OpenAI-compatible client pointed at the proxy URL. LiteLLM maps the requested model to the provider configuration, checks user/team budget state in PostgreSQL, calculates cost from model token pricing, and routes the request to an available provider deployment. If a primary model is rate-limited, router settings and Redis state can distribute traffic across multiple deployments or fall back to another model.

Why this architecture

The proxy architecture was chosen because it gave the university model flexibility, API-key distribution, OpenAI-client compatibility, governance features, and future self-hosting options without changing every downstream application when providers or models change.

Models used: GPT-4.1 · DALL-E 3 · GPT-5 · Gemma 3 · GPT OSS

Scale

• The chatbot served both students and employees at a private university.
  “there's a private university who would like to implement um their own privacy preserving chatbot um and make it accessible for both students and employees.”transcript, stated
• The initial budget hierarchy separated users into two teams: students and employees.
  “we start start easily with just having two teams. Uh one is the student team and what the other one is the employee team”transcript, stated
• Routing capacity is modeled with TPM and RPM limits for duplicated model deployments.
  “there is like a TPM and RPM with numbers TPM means tokens per minute and RPM requests per minute.”transcript, stated
• The proxy added about 200 milliseconds of latency in this setup, which was acceptable for the use case.
  “in my use cases, this this was not a problem. I don't know, I had like 200 milliseconds more of a latency um in the setup.”transcript, stated

Alternatives considered (roads not taken)

Use LiteLLM as only a Python library

Rejected because: The proxy provided more flexibility for API keys, future requirements, and OpenAI-compatible access by many applications.

Implement all provider-specific API handling in the chat application

Rejected because: The proxy abstracts different providers behind an OpenAI-compatible server and lets existing OpenAI-based tooling continue to work.

Rely entirely on LiteLLM built-in budgeting for complex university requirements

Rejected because: Specific requirements such as per-message cost education and temporary budget increases for user subsets were not available out of the box.

Use LiteLLM when there is only one provider and no governance need

Rejected because: The speaker says the tool is most useful for multiple providers, models, and governance needs like cost control.

Patterns

model-routing (gateway)

LiteLLM routes requests through a central proxy to multiple model providers and can distribute traffic across duplicate model deployments.

cost-aware-routing (governance)

The proxy calculates cost from model token prices, records it by user/team, checks budgets, and returns budget-exceeded errors when limits are reached.

fallback-degradation (reliability)

The configuration can route to duplicate deployments and then a fallback model if capacity is exceeded.

observability-tracing (operations)

Centralizing requests in LiteLLM enables one proxy configuration to send request information to observability tools.

privacy-preserving-processing (governance)

The architecture was selected for a privacy-preserving university chatbot and future self-hosting option, keeping model/provider choices under institutional control.

latency-optimization (performance)

The team explicitly considered proxy latency and accepted the observed roughly 200ms overhead for this use case.

Tradeoffs

Use a proxy server as the central gateway.

Chose LiteLLM proxy server over Only the LiteLLM Python library or direct provider clients.

Rationale: The proxy supported OpenAI compatibility, API-key distribution, centralized governance, and future flexibility.

Accept a proxy abstraction to simplify multi-provider governance.

Chose Unified LiteLLM API gateway over Application-specific integrations for each provider.

Rationale: The abstraction reduces provider complexity and centralizes governance, but may hide provider-specific capabilities.

Use built-in LiteLLM budgeting for basic cost control while accepting custom work for advanced requirements.

Chose Built-in team/user budgets plus custom processes where needed over Building all cost governance from scratch.

Rationale: LiteLLM handled automatic spend calculation and budget errors, but custom education and temporary-budget features required additional implementation.

Add Redis-backed routing and fallback models for capacity.

Chose LiteLLM routing strategies, duplicate model deployments, Redis state, and fallback models over Only retrying failed requests in the chat application.

Rationale: Retries alone could not handle too many simultaneous users when model capacity was exhausted.

Consider alternatives such as OpenRouter or TensorZero before committing to LiteLLM.

Chose Evaluate the gateway landscape for current project requirements over Assuming LiteLLM is always the best choice.

Rationale: The speaker notes that similar API gateway and budgeting capabilities are now implemented in other projects too.

Gotchas (what bit them)

LiteLLM's basic budget controls did not cover detailed educational cost attribution.

Root cause: The proxy did not allow custom metadata to be sent and stored with requests in the way the university needed.

Resolution: The suggested workaround was to capture cost information returned in response headers and implement a separate process around it.

Temporary budget increases for subsets of users were not supported in the needed form.

Root cause: The enterprise feature existed for API keys but not for individual users or teams in this implementation.

Resolution: The team would need to implement budgeting outside LiteLLM, wrap calls in custom budget checks, or contribute the feature upstream.

Pricing calculations can drift unless the proxy is updated regularly.

Root cause: Model pricing is maintained inside LiteLLM's internal model-spec data, so deployed images need updates to pick up price changes.

Resolution: The team must regularly deploy new Docker image versions to keep pricing data current.

The abstraction layer can block provider-specific features and brand-new models.

Root cause: LiteLLM has to implement provider-specific capabilities and new model support before users can access them through the proxy.

Resolution: Teams should be cautious when they need many provider-specific features or immediate access to new models.

Fast-moving features were sometimes buggy in production.

Root cause: LiteLLM developed many new features quickly, but some existing or new features did not work as expected.

Resolution: Production teams may need capacity to fix, patch, contribute, or run their own Docker image.

Evaluation

The case evaluates LiteLLM through production use in a university chatbot: whether it can support OpenAI-compatible access, budget enforcement, admin workflows, custom requirements, load balancing, routing, observability integrations, and acceptable latency.

Metrics: Budget-exceeded checks by user/team · Spend by model/provider in the admin UI · TPM and RPM capacity limits for model deployments · Approximately 200ms added proxy latency in this setup · Observed rate-limit errors under concurrent use

Outcomes

• The university got a working OpenAI-compatible proxy gateway for multiple models and providers.
• Basic cost control worked through PostgreSQL-backed spend tracking, teams, users, and budget-exceeded errors.
• The implementation exposed gaps around custom metadata, per-message cost display, temporary budget increases, provider-specific features, and feature stability.
• The architecture supported routing across duplicate model deployments, Redis-backed load balancing, fallback models, and centralized observability integrations.

The system covered students and employees, began with two teams, used TPM/RPM capacity settings for duplicate deployments, and accepted roughly 200ms of additional proxy latency.