Production LLM Systems at Scale - Lessons from Financial Services, Legal Tech, and ML Infrastructure

nubank,_harvey_ai,_galileo_and_convirza · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production LLM and small-language-model systems across finance, legal, evaluation, and call analytics

The panelists describe how production LLM systems must move beyond prototypes into high-volume, high-stakes workflows where cost, latency, evaluation reliability, human oversight, and modular architecture determine whether the system can be trusted.

Constraints:

• High-volume systems such as conversation analytics and financial workflows make large-model inference cost-prohibitive when prompts are large.
• Legal and financial workflows require confidence indicators, citations, and human oversight because mistakes can be costly.
• Prototype behavior in controlled environments diverges from production behavior on real user data.
• Model choice must be evaluated per module against quality, throughput, latency, and cost rather than by model brand or size.

Architecture

Across the four cases, teams converge on decomposing LLM applications into narrow production modules. Nubank proves workflows with large models, then routes modules through cheaper or lower-latency models using centralized telemetry. Convirza breaks call-analysis questions into small fine-tuned SLM adapters that emit structured outputs for dashboards and reports. Harvey exposes complex legal workflows as typed subcomponents with citations, confidence, and human orchestration. Galileo replaces expensive LLM-as-judge loops with smaller evaluator models and feedback-tuned metrics.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  production-workflow-request["Production workflow request"] --> task-decomposition-layer["Task decomposition layer"]
  task-decomposition-layer["Task decomposition layer"] --> model-selection-layer["Model selection layer"]
  model-selection-layer["Model selection layer"] --> specialized-slm-adapters["Specialized SLM adapters"]
  specialized-slm-adapters["Specialized SLM adapters"] --> structured-output-layer["Structured output layer"]
  model-selection-layer["Model selection layer"] --> telemetry-proxy["Telemetry proxy"]
  telemetry-proxy["Telemetry proxy"] --> evaluation-platform["Evaluation platform"]
  structured-output-layer["Structured output layer"] --> evaluation-platform["Evaluation platform"]
  evaluation-platform["Evaluation platform"] --> human-feedback-loop["Human feedback loop"]
  task-decomposition-layer["Task decomposition layer"] --> human-orchestration-panel["Human orchestration panel"]
  structured-output-layer["Structured output layer"] --> human-orchestration-panel["Human orchestration panel"]

Components

Data flow

A production task begins as a customer workflow, legal process, evaluation request, or call-analysis question. Teams first decompose it into narrow sub-tasks, then select a model class for each step based on quality, latency, throughput, and cost. Specialized SLMs or larger models produce structured outputs, while telemetry proxies, evaluation platforms, and human feedback loops measure quality and feed corrections back into prompts, fine-tuning datasets, or evaluator metrics.

Why this architecture

The shared architectural argument is that production tasks are rarely best served by one large prompt. Decomposition gives each module a measurable contract; model routing lets teams spend model capacity only where quality requires it; and human/evaluation loops catch the real-world failures that prototype metrics miss.

Models used: small language models · fine-tuned small language models · OpenAI models · GPT-4o mini · open source models · 500 million parameter DeBERTa models · 2-8 billion parameter evaluator models · BERT-style models

Scale

• Convirza processes unstructured data from millions of phone calls.
  “Their primary challenge was processing unstructured data from human conversations at scale—specifically millions of phone calls.”summary, stated
• Convirza decomposed 5-10 high-level analysis questions into smaller yes/no, rating, and classification questions.
  “Their approach involved breaking down complex analysis questions (initially 5-10 high-level questions) into smaller, more tractable questions: yes/no responses, 1-10 scale ratings, and simple classifications.”summary, stated
• Convirza described fine-tuning SLMs as a roughly 10-minute process once datasets existed.
  “The fine-tuning process was remarkably straightforward: "it was literally just clicking the big red button and sitting there for 10 minutes."”summary, stated
• Galileo cites a customer with 10 million queries per day whose LLM-judge evaluation costs exceeded seven figures within eight months.
  “one organization with 10 million queries per day saw evaluation costs exceed seven figures within eight months simply because they had an LLM-as-judge in the loop”summary, stated
• Galileo spans evaluator models from 500M parameters to 2-8B parameter models.
  “Small end: 500 million parameter DeBERTa models that reframe RAG hallucination detection as an entailment problem”summary, stated

Alternatives considered (roads not taken)

Prompting large language models for all call-analysis questions

Rejected because: Convirza found the prompt sizes and inference economics infeasible for customer-scale phone-call analysis.

Single question-answer interactions for legal workflows

Rejected because: Harvey users needed complicated multi-step legal processes with typed subcomponents, citations, and human inspection.

Legacy NLP metrics such as ROUGE and BLEU for long-form legal or generative outputs

Rejected because: The panelists found classical metrics provided essentially no useful signal for modern generative tasks.

Always using LLM-as-judge evaluation at production scale

Rejected because: Galileo observed that LLM-judge loops could double calls and produce unsustainable evaluation cost.

Patterns

workflow-decomposition (architecture)

The cases repeatedly break broad workflows into narrow modules with explicit contracts, allowing independent model selection, evaluation, and human intervention.

model-routing (architecture)

Nubank and others evaluate model choices per module across quality, throughput, latency, and cost, rather than standardizing on one model class.

fine-tuning-adaptation (model-adaptation)

Convirza and Galileo both adapt smaller models to narrow production tasks, using existing datasets or feedback to beat broad prompting economics.

eval-driven-development (quality)

The teams build evaluation into release and improvement loops, from human review and beta feedback to small-model hallucination checks and false-positive feedback.

human-in-the-loop-gate (governance)

Harvey and Galileo keep humans in review, intervention, dogfooding, beta validation, and false-positive correction loops.

cost-aware-routing (operations)

The panel treats cost as a first-class routing and architecture constraint, especially for high-volume calls and evaluation loops.

Tradeoffs

Use specialized SLMs for narrow production tasks

Chose Fine-tuned small models and narrow adapters over Large general-purpose LLM prompts for every task.

Rationale: Narrow models can match or beat large-model prompts for constrained tasks while improving latency, throughput, and cost.

Modularize complex legal workflows

Chose Subcomponents with defined inputs, outputs, citations, and human inspection over A single opaque legal assistant response.

Rationale: Strict module contracts and user transparency make high-stakes legal workflows more verifiable and controllable.

Optimize after product-market fit

Chose Large-model prototyping followed by module-level latency and cost optimization over Prematurely optimizing every workflow before validating usefulness.

Rationale: Nubank first proves a workflow with larger models, then replaces expensive steps with cheaper or lower-latency models where the workflow is known to matter.

Use smaller evaluator models for scale

Chose Small entailment or token-probability evaluator models over Chain-of-thought LLM judging for every evaluation case.

Rationale: Many production evaluation tasks do not require deep reasoning, and smaller evaluators can deliver comparable accuracy at lower cost.

Gotchas (what bit them)

Large-model prompting can be economically infeasible even when token prices are falling.

Root cause: High-volume, long-context workflows multiply prompt size and calls across millions of customer interactions.

Resolution: Decompose questions and fine-tune specialized small models for narrow outputs before optionally using LLMs for narrative reports.

Prototype evaluation can miss production behavior shifts.

Root cause: Controlled local environments do not reproduce real-world user data and evolving application contexts.

Resolution: Use robust evaluation tooling, production telemetry, and real-world feedback rather than relying on prototype checks.

LLM-as-judge can become a hidden production cost center.

Root cause: Putting a judge model in every evaluation loop effectively doubles or multiplies LLM calls at high query volume.

Resolution: Use smaller evaluator models, token-probability methods, and feedback-tuned metrics for narrow evaluation tasks.

Classical NLP metrics may provide no useful signal for legal-grade generation.

Root cause: ROUGE/BLEU-style overlap metrics do not capture correctness, legal reasoning, or usefulness in complex generated work products.

Resolution: Use human evaluation, citations, confidence indicators, beta feedback, and task-specific external evaluators.

Evaluation

The panel describes evaluation as multi-layered: module-level quality, throughput, latency, and cost checks; human review for long-form or legal outputs; user feedback and beta validation; and smaller automated evaluator models for scalable hallucination or RAG checks.

Metrics: quality · throughput · latency · cost · thumbs up/down feedback · RAG hallucination detection · false-positive feedback · human evaluator judgments

Outcomes

• Convirza's fine-tuned SLMs outperformed their best-crafted LLM prompts for narrow call-analysis tasks.
• Nubank reused mature ML telemetry and centralized proxies for LLM workflows.
• Galileo moved evaluation toward smaller models for high-scale hallucination and RAG metrics.
• The panel presents a broader shift from experimentation toward production-grade LLM system architecture.

The strongest scale figures are Convirza's millions of phone calls, Galileo's 10 million-query/day evaluation anecdote, and Nubank's strict internal latency and transaction-cost constraints.