Building Production-Grade Generative AI Applications with Comprehensive LLMOps

block_(square) · Tech · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise LLMOps for production generative AI applications across multiple business units

Block needed a flexible platform for generative AI applications across Square, Cash App, TIDAL, and TBD that could support fast-moving model choices, many serving endpoints, RAG, feedback logging, cost attribution, rate limits, sensitive-data handling, self-hosted models, fine-tuning, and monitoring without requiring every application to be rewritten whenever prompts or models changed.

Constraints:

• The generative AI space was changing fast enough that overbuilding infrastructure could become obsolete within three to six months.
• Teams had many applications calling many endpoints, reaching a couple hundred endpoints and creating visibility, cost, and quota-management problems.
• Applications needed to swap among self-hosted Llama, ChatGPT, Claude, open-source models, and fine-tuned models without changing application code each time.
• Customer-facing and sensitive-data applications could not always send data to third-party model providers.
• Most latency and cost came from generation, so teams needed options such as optimized GPU serving and fine-tuning smaller models.

Architecture

Block treats LLM chains as versioned MLflow models served behind stable endpoints. Application-specific chains handle prompts, validation, retries, fallback, retrieval, and pre/post-processing, while MLflow AI Gateway routes foundation-model calls to self-hosted Llama, OpenAI, Claude, fine-tuned models, or other vendors. Vector Search keeps RAG context decoupled from model hosting, Delta tables and Kafka feedback feed inference analysis, Lakehouse Monitoring connects data and model lineage, and optimized GPU serving/self-hosted models protect sensitive data and latency budgets. Fine-tuning smaller models is used selectively when enough endpoint volume justifies the upfront and operational costs.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  business-unit-applications["Business unit applications"] --> serving-endpoint["Serving endpoint"]
  serving-endpoint --> versioned-llm-chain["Versioned LLM chain"]
  serving-endpoint --> inference-logging["Inference logging"]
  versioned-llm-chain --> preprocessing-filters["Preprocessing filters"]
  versioned-llm-chain --> vector-search-endpoint["Vector Search endpoint"]
  vector-search-endpoint --> delta-context-tables["Delta context tables"]
  versioned-llm-chain --> mlflow-ai-gateway["MLflow AI Gateway"]
  mlflow-ai-gateway --> foundation-and-fine-tuned-models["Foundation and fine-tuned models"]
  mlflow-ai-gateway --> central-credentials-and-rate-limits["Central credentials and rate limits"]
  foundation-and-fine-tuned-models --> optimized-gpu-serving["Optimized GPU serving"]
  versioned-llm-chain --> output-validator["Output validator"]
  inference-logging --> inference-tables["Inference tables"]
  kafka-feedback-pipeline["Kafka feedback pipeline"] --> inference-tables
  inference-tables --> lakehouse-monitoring["Lakehouse Monitoring"]
  delta-context-tables --> lakehouse-monitoring
  mlflow-tracking["MLflow tracking"] --> versioned-llm-chain

Components

Data flow

A Block application sends a request to a serving endpoint that wraps an LLM chain. The chain may validate inputs, run retrieval against a Vector Search endpoint backed by Delta tables, hydrate the prompt with retrieved context, call MLflow AI Gateway for a foundation model or fine-tuned model, and validate output for toxicity or hallucination before returning to users. Inference logging captures inputs and outputs into inference tables. Application feedback such as accept rates flows through Kafka into Delta tables, where it can be joined back to inferences for A/B analysis and bad-output analysis. AI Gateway centralizes model routing, credentials, rate limits, usage, and cost attribution; Lakehouse Monitoring monitors both data and model outputs with shared lineage.

Why this architecture

Block chose a platform architecture because direct calls to individual endpoints quickly break down once teams need version tracking, A/B testing, model swaps, sensitive-data controls, endpoint proliferation management, quota enforcement, retrieval updates, and cost/latency optimization. The separation between application endpoints, gateway-routed foundation models, vector-search context, inference logs, and monitoring allows teams to iterate without rewriting applications for every prompt or model change.

Models used: GPT-4 · OpenAI ChatGPT · Anthropic Claude · Self-hosted Llama · Llama 70B · Fine-tuned 13B and 3B parameter models · BERT-based chat models under 1B parameters · Embedding models for vector search · Second-model hallucination validators

Scale

• Block spans Square, Cash App, TIDAL, and TBD, with different GenAI needs across business units.
  “We have Cash App, which is peer-to-peer payments and increasingly banking solutions. We have TIDAL, which is a music streaming service, and TBD”transcript, stated
• Block had accumulated a couple hundred serving endpoints.
  “I think we're at a couple hundred of these today.”transcript, stated
• The first structured-output example reached six or seven versions before shipping.
  “you've tried out six or seven different versions of it before you've shipped that.”transcript, stated
• Some teams faced company-wide GPT-4 rate limits.
  “where you have a company-wide rate limit on GPT-4”transcript, stated
• Fine-tuning could move a working model from 13B parameters to 3B parameters.
  “maybe it was working well enough with a 13 billion parameter model. But with fine-tuning, you might be able to take that down to a 3 billion parameter model.”transcript, stated
• Block had prior chat models below the 1B-parameter range.
  “we have successfully built like BERT-based chat models. So we know that models under the 1 billion parameter range can work”transcript, stated
• Fine-tuned model hosting on even one A10 GPU was considered relatively expensive.
  “if you have even one A10 that ends up being relatively expensive.”transcript, stated
• Databricks described proprietary model price drops as much as 10x during the year.
  “The prices have been dropping as much as 10x.”transcript, stated

Alternatives considered (roads not taken)

Directly integrate each application with one model endpoint.

Rejected because: Block found direct endpoint integrations tempting but fragile because prompt/model versions proliferate, applications need frequent updates, and AI Gateway provides stable routing and operational controls.

Host a separate GPU foundation model behind every application endpoint.

Rejected because: The number of application chains is much larger than the number of foundation models, so hosting a GPU model per chain would be egregiously expensive.

Keep RAG context inside the model-hosting process.

Rejected because: In-memory vector search works early, but it becomes operationally difficult to update context when every data change requires redeploying the model.

Send all customer-facing data to third-party hosted models.

Rejected because: For sensitive data, Block relied on self-hosted models and optimized GPU serving to avoid sending data out to a third party.

Fine-tune every use case immediately.

Rejected because: Fine-tuning only pays off when endpoint usage justifies upfront training, GPU hosting, and operational costs, so Block frames it as an iterative maturity step.

Patterns

prompt-versioning (operations)

Block treats the whole LLM stack of prompts, chains, validation, model choice, and endpoint behavior as a versioned MLflow model so teams can compare versions and avoid losing track of prompt/model performance.

model-routing (architecture)

MLflow AI Gateway provides a stable API route that can point to self-hosted Llama, ChatGPT, Claude, Bedrock, Databricks-served models, external vendors, or fine-tuned models.

rag-retrieval (architecture)

Customer-facing chat applications retrieve Block-specific context through Vector Search rather than relying on the base model's reference material.

observability-tracing (operations)

Inference logging, inference tables, customer feedback joins, cost attribution, usage visibility, and lakehouse monitoring provide operational visibility into model behavior.

guardrails-io-filtering (safety)

Customer-facing chains add pre-processing for prompt injection or bad actors and post-processing for toxicity or hallucination checks.

privacy-preserving-processing (safety)

For the most sensitive data, Block relies on self-hosted models and optimized GPU serving rather than third-party model providers.

fine-tuning-adaptation (modeling)

Fine-tuning is used selectively to reduce latency and cost by moving from larger models to smaller models when use-case scale justifies it.

cost-aware-routing (operations)

The gateway prevents duplicate GPU hosting, centralizes cost attribution, enforces company-wide rate limits, and lets fine-tuned models be reused by multiple endpoints.

Tradeoffs

Build a flexible platform but avoid overbuilding for a fast-moving space.

Chose Minimal platform capabilities that speed integration, versioning, routing, and monitoring over Heavy infrastructure that could be invalidated in three to six months.

Rationale: Block needed a general toolkit across business units while preserving speed as model and vendor choices changed.

Centralize model routing through AI Gateway.

Chose Stable gateway route with centralized credentials, usage, cost, and quota controls over Application code directly calling model-specific endpoints.

Rationale: The gateway keeps application endpoints stable while model vendors, rate limits, and foundational model choices change underneath.

Decouple vector context from model endpoints.

Chose Separate Vector Search endpoints for changing context over Keeping retrieved context in memory inside model hosting.

Rationale: Separate retrieval lets teams update knowledge without redeploying the model and lets prompt/model selection iterate independently.

Self-host sensitive workloads despite serving complexity.

Chose Self-hosted models on optimized GPU serving over Third-party hosted models for all customer data.

Rationale: Sensitive-data pipelines require security control, but usable latency required optimized GPU serving rather than naive self-hosting.

Fine-tune smaller models only when volume justifies it.

Chose Fine-tuning to reduce latency and cost for high-volume endpoints over Always using larger general models or fine-tuning every endpoint.

Rationale: Generation dominates latency and spend, but fine-tuning has upfront and hosting costs that require enough usage.

Gotchas (what bit them)

Prompt/model variants become unmanageable if teams do not version the whole LLM stack.

Root cause: A simple structured-output app can quickly accumulate prompt changes, model swaps, validation, retries, fallback, and six or seven versions before shipping.

Resolution: Package the chain as a versioned MLflow model and compare versions through A/B tests rather than rewriting application code.

Endpoint proliferation creates cost, visibility, and quota failures.

Root cause: Many application-specific chains called many model endpoints, and each endpoint otherwise had to know which model it called.

Resolution: Route foundational model calls through MLflow AI Gateway for centralized visibility, rate limits, usage, cost attribution, and model swaps.

RAG context becomes hard to update if it is embedded inside model hosting.

Root cause: In-memory vector search can work early, but adding new context requires redeploying the model when retrieval is coupled to hosting.

Resolution: Decouple retrieval into a separate Vector Search endpoint that can receive new context while prompts and model selections iterate separately.

Self-hosting large models can be too slow without optimized serving infrastructure.

Root cause: Naive containers running PyTorch for big models had token latency too high for customer-facing applications.

Resolution: Use optimized GPU serving endpoints for self-hosted or fine-tuned models.

Customer-facing model outputs need both input and output guardrails.

Root cause: Freeform customer input can include prompt injection or bad actors, and model output can be toxic or hallucinated.

Resolution: Add pre-processing filters and post-processing validators, including second-model hallucination checks where needed.

Fine-tuning has a minimum scale threshold.

Root cause: Fine-tuning has upfront data/model work and ongoing GPU hosting costs, so low-volume endpoints may not justify the investment.

Resolution: Use fine-tuning as an iterative maturity step only when it improves performance or reduces costs enough for the endpoint volume.

Evaluation

Block evaluates production GenAI systems by treating chains as versioned models, running A/B tests between versions, logging inputs/outputs through inference tables, joining feedback back to inferences, analyzing bad outputs, validating against prompt-injection/toxicity/hallucination risks, and tracking cost, usage, rate limits, latency, and monitoring lineage.

Metrics: A/B test outcomes between chain versions · Accept rates on suggested actions · Inputs that cause bad outputs · Rate-limit usage by application · Usage and cost attribution by endpoint/model · Inference latency per token · Output toxicity · Hallucination detection by a second model · Retrieval recall · Lakehouse monitoring alerts and default/custom metrics

Outcomes

• Block built a general LLMOps platform for multiple business units rather than one-off integrations per application.
• The platform reached a couple hundred serving endpoints, making gateway-level visibility and quota controls necessary.
• RAG became a dominant pattern: Block says most applications so far were retrieval-based.
• Self-hosted models with optimized GPU serving enabled sensitive-data use cases that could not send data to third-party providers.
• Fine-tuning was identified as a route to reduce latency and cost by moving from larger models to smaller task-specific models when endpoint volume justified it.

The case reports a couple hundred endpoints, six or seven versions before shipping one chain, company-wide GPT-4 rate-limit concerns, a shift from 13B to 3B fine-tuned model targets, prior sub-1B BERT-based chat models, and an A10 GPU as a meaningful hosting-cost threshold.