Evolution of ML Platform to Support GenAI Infrastructure

lyft · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: ML platform evolution for enterprise GenAI

Lyft needed to extend an existing large-scale ML platform to support LLM vendors, client libraries, centralized proxies, evaluations, guardrails, and AI assistants without losing observability, security, and model lifecycle discipline.

Constraints:

• Lyft already operated a broad ML estate across dozens of teams, repositories, and high-traffic models.
• Most LLM usage would rely on vendor APIs rather than self-hosted LLMs.
• LLM requests are more complex than tabular model calls, with natural-language prompts, temperature, structured outputs, and client-side libraries.
• The platform needed to filter PII, evaluate inputs and outputs online/offline, and log enough data for quality analysis.
• Teams wanted higher-level assistant abstractions on top of raw LLM calls.

Architecture

Lyft reused its ML serving platform by redefining an LLM API proxy as a model. It wrapped vendor client libraries so traffic flowed through Lyft's model-serving proxy, injected API keys server-side, captured granular metadata, applied online PII filtering and output guardrails, logged request/response pairs, ran offline LLM-as-judge evaluations, and began building higher-level assistant interfaces.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  vendor-client-wrapper["Vendor client wrapper"] --> llm-proxy["LLM proxy"]
  llm-proxy --> vendor-llm-apis["Vendor LLM APIs"]
  vendor-llm-apis --> evaluation-hooks["Evaluation hooks"]
  evaluation-hooks --> pii-filter["PII filter"]
  pii-filter --> llm-proxy
  evaluation-hooks --> evaluation-warehouse["Evaluation warehouse"]
  evaluation-warehouse --> assistant-interface["Assistant interface"]

Components

Data flow

A developer imports a Lyft-wrapped vendor client, builds an LLM request locally, the wrapper routes HTTP through Lyft's ML serving system, the server-side proxy injects credentials and forwards to the vendor, evaluations can filter PII or inspect outputs, request/response pairs are logged, and offline workflows score quality with another LLM.

Why this architecture

Lyft reused the ML serving platform because its lifecycle, observability, security, compute, and model management primitives still applied when the model was an external API. Wrapping vendor clients let engineers keep familiar libraries while centralizing governance and evaluation.

Models used: OpenAI API · Anthropic API · Databricks Dolly · PII filtering model · LLM-as-judge evaluators · Lyft ML model interface · LLM vendor clients

Scale

• Lyft had more than 50 models across more than 100 GitHub repositories and more than 1,000 unique models.
  “we have more than 50 models across more than 100 GitHub repos and um more than a thousand unique models”transcript, stated
• Some Lyft models received 10,000 requests per second or more.
  “some of those models get 10,000 RPS or more”transcript, stated
• About 50 engineering teams had at least one model.
  “50 engineering teams have at least one model”transcript, stated
• The LLM client/proxy playbook was applied to more than half a dozen vendors.
  “we applied this Playbook to more than half a dozen vendors”transcript, stated
• In early 2024, 100% of Lyft's LLM traffic flowed through the proxy.
  “we had 100% of our traffic going through the proxy that I was describing”transcript, stated

Alternatives considered (roads not taken)

Self-host most LLMs.

Rejected because: Lyft quickly found most users wanted vendor API access for the foreseeable future.

Let developers call vendor APIs directly.

Rejected because: Central proxying provided observability, security, standardized infrastructure, and model-management simplification.

Buy a full evaluation platform in 2024.

Rejected because: Lyft decided its immediate requirements could be met with an internal lightweight evaluation framework.

Patterns

model-routing (vendor proxy)

Lyft routes multiple vendor APIs through a centralized LLM proxy inside its ML serving system.

privacy-preserving-processing (PII filtering)

Online evaluations redact PII before sending prompts to external LLM vendors.

llm-as-judge (offline evaluation)

Lyft runs LLM-based workflows over logged request-response pairs to flag unhelpful or incomplete answers.

observability-tracing (central proxy logging)

The proxy captures granular traffic origin, user, service, environment, prompts, responses, and evaluation actions.

Tradeoffs

Vendor APIs over self-hosted LLMs for most use.

Chose API-mediated vendor access over Lyft-hosted LLM serving as the primary path.

Rationale: Users wanted vendor access, while Lyft could still preserve platform controls through the proxy.

Build a lightweight evaluation framework internally.

Chose Hooks for modular online/offline evaluations over A heavyweight external evaluation system.

Rationale: The platform needed immediate hooks for PII filtering, output guardrails, and quality scoring.

Use both client-side and server-side control.

Chose Wrapped clients plus server proxy over Server-only proxying with unmanaged clients.

Rationale: Owning both sides gave Lyft latitude to modify, log, observe, and add product features where appropriate.

Gotchas (what bit them)

Traditional ML model abstractions did not fit LLM APIs directly.

Root cause: LLM calls had no local model binary and required complex client-constructed natural-language requests.

Resolution: Treat the proxy code as the model under the ML model interface and defer actual model execution to vendors.

Centralizing traffic answers who is using LLMs but not whether usage is good.

Root cause: Proxy logs alone did not evaluate meaning, quality, or safety.

Resolution: Add modular online and offline evaluation hooks for inputs, outputs, and request-response pairs.

Vendor clients needed governance without developer friction.

Root cause: Developers expect familiar client libraries, but platforms need proxying, logging, and credential management.

Resolution: Wrap vendor packages with the same interface while overriding the transport layer.

Evaluation

Lyft evaluates LLM usage with online input PII filtering, online output guardrails, offline request/response quality scoring, logged evaluation actions, and LLM-as-judge prompts for helpfulness and information sufficiency.

Metrics: PII removed before vendor call · Online output guardrail result · Request/response quality score · Unhelpfulness flag · Information sufficiency flag · Traffic origin and service metadata · Evaluation actions warehoused

Outcomes

• Lyft standardized vendor LLM access through its existing ML platform.
• Developers could use familiar vendor packages without managing API keys directly.
• The proxy gave the platform observability, security, network, and evaluation hooks.
• The evaluation framework enabled PII filtering, output guardrails, and offline quality analysis.

The talk cites 50+ models, 100+ GitHub repos, 1,000+ unique models, 10,000+ RPS for some models, 50 teams with models, more than half a dozen vendors, and 100% of LLM traffic through the proxy in early 2024.