Scaling and Optimizing Self-Hosted LLMs for Developer Documentation

various · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Self-hosted LLM optimization, structured outputs, and document analysis

The session covered three production LLM scaling problems: self-hosting a documentation RAG assistant with acceptable latency, producing reliable structured outputs efficiently, and reducing manual public-comment analysis with hybrid NLP and LLM workflows.

Constraints:

• The developer documentation customer required self-hosting because proprietary documentation could not be shared with third parties.
• Initial Hugging Face plus FastAPI inference had unusable latency beyond very low user counts.
• Production RAG involved multiple services, including a guardrails model and vector database, not just one LLM.
• Structured outputs need reliable parsing and lower token use across models.
• Public-comment analysis must handle tens of thousands of statements and legally salient points.

Architecture

The Fuzzy Labs RAG stack uses FastAPI orchestration, a vector database, guardrails, and self-hosted Mistral 7B Instruct on AWS GPU instances, optimized with vLLM and Ray Serve after Locust benchmarking. Boundary's BAML improves structured-output reliability and token efficiency through generated parsers and data-model-aware prompts. SAS combines traditional text analytics, entity extraction, bucketing, and LLM summarization to shrink public-comment analysis effort.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Developer query API"] --> B["Vector database"]
  A --> C["Guardrails service"]
  A --> D["Mistral model server"]
  D --> E["vLLM inference runtime"]
  D --> F["Ray Serve scaling layer"]
  C --> F
  B --> F
  G["BAML structured output layer"] --> H["Structured output consumers"]
  I["Text analytics preprocessor"] --> J["LLM summarizer"]

Components

Data flow

For the RAG assistant, developer questions pass through FastAPI, retrieval, guardrails, and a self-hosted Mistral model. Benchmarks identify latency and throughput bottlenecks, then vLLM improves per-GPU inference and Ray Serve shares/scales GPU-backed services. For structured outputs, BAML converts model text into typed data models. For public comments, text analytics filters and buckets statements before LLM summarization.

Why this architecture

The session emphasizes measuring first, then optimizing at the right layer: self-hosting satisfied data constraints, vLLM and Ray Serve addressed inference/scaling bottlenecks, BAML addressed structured-output reliability and token cost, and classic NLP reduced the amount of text the LLM had to summarize.

Models used: Mistral 7B Instruct · guardrails classifier · LLMs for structured outputs · SAS information extraction model · LLM summarization model

Scale

• The RAG benchmark scenarios included 20, 30, and 50 concurrent users, with initial failure around five users.
  “20 concurrent users is a good starting point”transcript, stated
• vLLM claimed 24x throughput over plain Hugging Face pipelines and 2.5x over Hugging Face TGI.
  “get a 24 times throughput improvement over plain hugging face pipelines”transcript, stated
• BAML reduced token use from 108 to 83 in the example, about 20-25% cheaper and faster.
  “we came down from 108 tokens to 83 tokens that's a saving of 83 and 108 just straight up almost 20 25% cheaper and 25% faster”transcript, stated
• SAS reduced one public-comment analysis workload from about 4,500 hours to closer to 600 hours.
  “going from 4500 hours the closest closer to 600 hours”transcript, stated
• Public comment workloads may include 20,000 to 50,000 unique statements.
  “I've gotten like let's say 20,000 or 50,000 unique statements out of those”transcript, stated

Alternatives considered (roads not taken)

Hosted third-party LLM APIs for proprietary documentation

Rejected because: The customer did not want proprietary documentation shared with external parties.

Plain Hugging Face pipelines plus FastAPI

Rejected because: The initial setup had very high latency and was effectively unusable beyond one user.

LLM-only public-comment summarization

Rejected because: Traditional text analytics can focus the LLM on the right buckets and statements first.

Patterns

rag-retrieval (retrieval)

The documentation assistant retrieves relevant docs and combines them with the developer's question before generation.

latency-optimization (optimization)

The team benchmarked latency and throughput, then adopted vLLM and Ray Serve for vertical and horizontal scaling.

structured-output-contract (interface)

BAML turns model output into typed structures and parsers, improving reliability and lowering token use.

safety-policy-enforcement (governance)

A separate guardrails model filters off-topic, competitor, harmful, and proprietary-information requests.

hybrid-search (retrieval)

SAS combines traditional text analytics, entity extraction, bucketing, and LLM summarization instead of using only one retrieval/generation method.

Tradeoffs

Deployment model

Chose self-hosted Mistral on GPU instances over third-party hosted model APIs.

Rationale: The customer needed to keep proprietary documentation in-house and learn internal LLM operations.

Optimization order

Chose benchmark before optimizing over premature tuning.

Rationale: Load tests exposed actual bottlenecks and informed whether vertical or horizontal scaling mattered.

Public-comment analysis design

Chose traditional NLP filtering plus LLM summarization over manual review or raw LLM summarization over everything.

Rationale: Bucketing and statement extraction focus the LLM on relevant subsets and reduce manual effort.

Gotchas (what bit them)

Self-hosted RAG was unusable without inference optimization.

Root cause: Plain Hugging Face inference plus FastAPI produced high latency and failed at low concurrency.

Resolution: Benchmark first, then adopt vLLM and Ray Serve.

Production LLM stacks may contain multiple models that all need scaling.

Root cause: The RAG assistant used both a main LLM and a guardrails model, plus vector operations.

Resolution: Use a serving layer that can share GPUs and scale several services.

Structured outputs can be unparsable even when the text looks reasonable.

Root cause: Models do not naturally emit reliable typed data without a contract/parser layer.

Resolution: Use BAML to generate parsing algorithms and data-model-aware prompts.

Evaluation

The session evaluates systems through Locust load testing, latency, throughput, request rate, failure-point testing, structured-output parsing/token metrics, and manual-hour reduction.

Metrics: latency · tokens per second · successful requests per minute · concurrent users · failure point · token count · manual hours reduced

Outcomes

• vLLM improved latency and slowed latency growth without increasing GPU count.
• Ray Serve enabled horizontal scaling and GPU sharing across model services.
• BAML reduced token usage in the example by roughly 20-25%.
• SAS reduced one public-comment analysis workload from about 4,500 to 600 hours.

The session spans infrastructure-scale RAG, structured-output token efficiency, and public-comment workloads with tens of thousands of statements.