Scaling Self-Hosted LLMs with GPU Optimization and Load Testing

fuzzy_labs · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: self-hosted LLM serving for developer support

Fuzzy Labs needed to scale a public-facing RAG assistant backed by self-hosted models so developers could get fast answers while the business controlled GPU cost and safety risk.

Constraints:

• The service needed fast response times for developer-facing interaction.
• It had to support hundreds or potentially thousands of concurrent user sessions.
• GPU infrastructure was expensive enough that simply adding hardware was not acceptable.
• Guardrails and auxiliary models had to be accounted for as part of the production serving footprint.

Architecture

The initial architecture used a FastAPI model server running Hugging Face inference pipelines with a quantized Mistral 7B model, ChromaDB for retrieval, an orchestration service, and a separate guardrails service. Load testing with Locust exposed latency and concurrency limits, so the team replaced Hugging Face inference with a vLLM-style paged-attention runtime and used Ray Serve for horizontal GPU-backed model serving and autoscaling.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Developer question"] --> B["Orchestration service"]
  B --> C["Guardrails service"]
  B --> D["ChromaDB vector database"]
  B --> E["Model server API"]
  E --> F["Mistral 7B model"]
  G["Locust load tester"] --> B
  E --> H["vLLM inference runtime"]
  H --> F
  I["Ray Serve autoscaler"] --> H
  I --> J["GPU servers"]
  H --> J
  C --> J

Components

Data flow

A developer question enters the orchestration service, which checks guardrails, retrieves context from ChromaDB, and calls the model server. The initial FastAPI/Hugging Face pipeline serves Mistral 7B on Kubernetes/GPU infrastructure; Locust load tests measure latency and concurrency; vLLM/paged attention improves single-server latency; Ray Serve then scales model serving horizontally and shares GPU capacity with auxiliary guardrail and embedding models.

Why this architecture

The team deliberately benchmarked the simple baseline first, then optimized the single-server inference bottleneck with paged attention and addressed separate concurrency scaling with Ray Serve autoscaling rather than treating all performance problems as one hardware-scaling problem.

Models used: Mistral 7B instruction-tuned quantized model · guardrail classification models · embedding model

Scale

• The target usage included hundreds or potentially thousands of concurrent sessions.
  “we want hundreds or potentially thousands of concurrent user sessions”transcript, stated
• The load-test scenarios included 10, 20, 30, and 50 concurrent users plus a failure test.
  “we've got 50 concurrent users for that this is all very hypothetical but at least it gives us a model and a framework”transcript, stated
• The baseline single-user latency was 11 seconds.
  “one concurrent user one user sending this request we have a latency of 11 seconds”transcript, stated
• Latency worsened from 11 to 40 to 65 seconds as concurrency increased.
  “it's gone up from 11 to 40 to 65 that growth rate that's a problem the system breaks down really quickly”transcript, stated
• Paged-attention/vLLM-style serving claimed a 24x improvement over Hugging Face pipelines.
  “what they claim is that they get a 24 times improvement over hugging face pipelines”transcript, stated

Alternatives considered (roads not taken)

Optimize immediately based on architectural intuition

Rejected because: The team chose to benchmark first with Locust so optimization followed measured bottlenecks.

Keep the plain FastAPI plus Hugging Face pipeline baseline

Rejected because: Even one concurrent user had 11-second latency, and small concurrency increases made the service unusable.

Add more model servers before improving single-server latency

Rejected because: The simpler first bottleneck was single-server latency; horizontal scaling came after the single-instance runtime improved.

Manually scale GPU hardware

Rejected because: Ray Serve provided demand-based autoscaling and GPU sharing so the system could avoid over-provisioning expensive GPUs.

Patterns

rag-retrieval (retrieval)

The assistant used a vector database to retrieve context for developer questions before answering with the model server.

safety-policy-enforcement (safety)

A separate guardrails service filtered inappropriate questions, toxic language, discriminatory language, off-topic requests, and PII independently of the LLM.

eval-driven-development (evaluation)

Locust load tests drove the optimization sequence by revealing latency, request-rate, and concurrency bottlenecks before the team changed serving infrastructure.

Tradeoffs

Benchmark before changing infrastructure.

Chose Locust-based traffic scenarios and failure tests over prematurely swapping FastAPI, ChromaDB, or Hugging Face components by intuition.

Rationale: The team wanted measured bottlenecks before deciding whether the model server, vector database, or scaling layer actually needed replacement.

Fix latency before scaling replicas.

Chose paged-attention/vLLM-style inference for one server over adding more servers as the first response.

Rationale: Single-server latency was the simpler bottleneck and had to improve before horizontal scaling would make economic sense.

Use Ray Serve for horizontal scaling.

Chose Ray Serve autoscaling across GPU servers over manual capacity management or a single model server.

Rationale: Ray Serve provided many-server deployment, demand-based scaling, and GPU support for LLM and auxiliary-model workloads.

Gotchas (what bit them)

The simplest self-hosted baseline was too slow before realistic traffic arrived.

Root cause: FastAPI plus Hugging Face pipelines on one GPU server had high single-user latency and degraded quickly with small concurrency increases.

Resolution: Benchmark the baseline, then replace the inference runtime with paged-attention/vLLM-style serving before horizontal scaling.

GPU cost made naive horizontal scaling risky.

Root cause: Adding enough GPUs to absorb every traffic spike could protect user experience but destroy the business case.

Resolution: Use Ray Serve autoscaling and GPU sharing so capacity grows with demand and can shrink back down.

Guardrails were not just a small wrapper around the LLM.

Root cause: The guardrails subsystem itself contained multiple independent models that could also need acceleration.

Resolution: Treat guardrails and embedding models as part of the GPU-capacity planning problem, not external afterthoughts.

Evaluation

The talk evaluates the architecture through Locust traffic scenarios, baseline latency/request-rate measurements, concurrency ramps, and comparative measurements after changing the inference runtime and scaling layer.

Metrics: latency at one user and under higher concurrency · requests per minute · throughput · concurrent user scenarios · failure-point load testing

Outcomes

• The baseline exposed an 11-second single-user latency and rapid degradation under modest concurrency.
• Paged-attention/vLLM-style inference improved latency materially versus Hugging Face pipelines.
• Combining Ray Serve with the faster inference runtime got the system to a production-ready path for public users.

The talk frames scale as maintaining latency while supporting hundreds or thousands of concurrent sessions on expensive GPU infrastructure.