Building and Scaling Enterprise LLMOps Platforms: From Team Topology to Production

various · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise LLMOps platform engineering

Enterprises adopting generative AI need a way to move LLM application work out of isolated data-science pilots and into traditional product teams without forcing every team to independently solve model access, RAG infrastructure, observability, evals, feedback, governance, and developer enablement.

Constraints:

• data science teams often lack production-running habits while application teams need AI enablement
• model, framework, and vendor choices change rapidly
• LLM outputs require testing approaches beyond deterministic unit tests
• prompt chains and sub-calls need observability different from ordinary API health checks
• governance needs to cover licenses, training-data opt-outs, prompt injection, PII, and regulatory risk
• centralization should reduce duplicated work without creating premature platform overhead for small companies

Architecture

Patrick Debois proposes an enterprise AI platform model inspired by DevOps and Team Topologies. Feature teams build AI-enabled products on top of a central AI platform team that curates model access, vector/RAG infrastructure, data connectors, model registry/version control, provider abstraction/proxy access, observability, production eval monitoring, caching, and feedback services. A companion enablement function gives teams prototyping tools, frameworks, local development environments, education, and secure access to company data. Governance layers handle model access, licensing, training opt-outs, prompt injection defenses, PII, risk assessment, and guardrails, with generic central rules plus use-case-specific team rules. The pattern is recommended for organizations scaling beyond roughly ten teams, not for premature optimization in very small companies.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  feature-teams[Feature teams] --> ai-platform-team[AI platform team]
  feature-teams --> enablement-function[Enablement function]
  feature-teams --> experience-crew[Experience crew]
  feature-teams --> governance-workflow[Governance workflow]
  ai-platform-team --> model-access-proxy[Model access proxy]
  ai-platform-team --> vector-and-rag-services[Vector and RAG services]
  ai-platform-team --> data-connectors[Data connectors]
  ai-platform-team --> model-registry[Model registry]
  ai-platform-team --> observability-and-evals[Observability and evals]
  ai-platform-team --> feedback-and-caching-services[Feedback and caching services]
  ai-platform-team --> governance-workflow
  ai-platform-team --> enablement-function
  model-access-proxy --> model-registry
  model-access-proxy --> governance-workflow
  vector-and-rag-services --> data-connectors
  data-connectors --> feature-teams
  observability-and-evals --> feedback-and-caching-services

Components

Data flow

Feature teams use prototyping tools, frameworks, local development, and secure data access from the AI platform. Their applications call centrally curated model access and model-provider proxies, retrieve company data through shared connectors and vector/RAG services, and reuse centrally registered models. Production prompts and multi-step LLM calls flow into observability/tracing and eval health checks, while feedback services capture richer end-user edits. Governance teams provide generic guardrails and approval workflows, then product teams layer use-case-specific rules. Platform feedback from teams returns to enablement and the shared services backlog.

Why this architecture

A platform architecture fits the enterprise problem because GenAI adoption creates shared concerns too expensive and risky for each feature team to solve alone. Team Topologies offers interaction modes for platform teams to provide APIs, collaborate, or facilitate. Shared model proxies, data connectors, observability, evals, caching, feedback, and governance reduce duplication while letting feature teams integrate AI into their own domains. Debois cautions that the pattern is inappropriate for a ten-person company and becomes useful when organizations scale to many teams.

Models used: LLMs · helper model · LLM-as-judge · GitHub Copilot

Scale

• The talk grounds its analogy in DevOps Days starting in 2009 with 60 people and 15 years of DevOps transformation.
  “in 2009 I organized a small conference called devop days there were 60 people and this is how the world kind of got into devops as a word”transcript, stated

  “seen 15 years of a new kind of phenomenon discussion transformation taking Ro across the whole Enterprise”transcript, stated
• The presenter draws from two years of hands-on GenAI application work.
  “I've witnessed firsthand in the company after you know now two years building geni application”transcript, stated
• One production prompt may fan out into five or six prompt iterations or questions.
  “one prompt it's rarely one prompt usually one prompt leads to five six iterations questions”transcript, stated
• The organization went through about eight LLM models over two years, forcing prompt rework each time.
  “we went through I think eight different models over two years for llms to get like Improvement of quality”transcript, stated
• The proposed platform operating model is positioned for organizations scaling to 10 or more teams, not a 10-person company.
  “I would not recommend to do this for a 10 person company uh don't do premature optimization but if you're scaling out to 10 and more teams that's probably a pattern”transcript, stated

Alternatives considered (roads not taken)

Leave GenAI work inside data science teams

Rejected because: The presenter observed friction because data science teams were not used to running production systems, while GenAI app work became more integration-heavy.

Let every team choose models, connectors, observability, evals, and feedback services independently

Rejected because: Repeatedly solving shared platform services is expensive and slows teams that should focus on their domains.

Manual prompt rewrites after every model change

Rejected because: After eight model changes in two years, manual prompt rework had no guarantee of working without evals.

Central guardrails as a single universal rule set

Rejected because: The observed pattern combines central generic rules with use-case-specific rules owned by product teams.

Build a full AI platform too early

Rejected because: For a 10-person company, the platform structure is premature optimization; the recommendation applies when scaling to about 10 or more teams.

Patterns

rag-retrieval (platform)

The AI platform includes vector databases, embeddings, and shared connectors so teams can build RAG over company data without reinventing data access.

observability-tracing (operations)

The platform captures production prompts and traces multi-step LLM interactions because one user prompt can create several sub-calls.

eval-driven-development (quality)

The platform treats production eval health checks and testing as required infrastructure before model or prompt refactors.

prompt-versioning (change-management)

Repeated model changes force prompt rework, so prompt/model changes need versioning and eval coverage rather than manual edits without guarantees.

llm-as-judge (evaluation)

The testing spectrum includes using an LLM to judge another LLM's output when exact, regex, sentiment, or semantic-distance checks are insufficient.

semantic-caching (cost)

The proposed platform includes central caching services so teams do not each build expensive duplicated infrastructure.

safety-policy-enforcement (governance)

Central governance manages model licenses, training opt-outs, risk assessments, prompt injection defenses, guardrails, PII scrutiny, and model restrictions.

Tradeoffs

Organizational topology

Chose AI platform team plus feature teams over GenAI remaining inside data science silos.

Rationale: A platform lets traditional application developers scale GenAI into their own domains while a central team abstracts shared infrastructure.

Shared services

Chose central connectors, observability, evals, caching, and feedback over each team building the same services independently.

Rationale: The services are expensive, specialized, and common enough that duplicating them would slow adoption.

Evaluation strategy

Chose a mixed testing stack of exact checks, helper models, semantic distance, LLM-as-judge, and human feedback over only deterministic tests.

Rationale: LLM outputs cannot always be validated with simple regex or exact assertions, so quality checks need layered evidence.

Governance model

Chose central generic rules plus team-specific rules over purely centralized one-size-fits-all guardrails.

Rationale: Central teams can define common controls, but product teams know the use-case-specific risks.

Platform timing

Chose platform consolidation when scaling to many teams over premature platform buildout for a tiny organization.

Rationale: The cognitive and organizational overhead is justified around 10+ teams, not a 10-person company.

Gotchas (what bit them)

GenAI pilots can get trapped in data science teams that are not prepared for production operations.

Root cause: The initial work lands with data scientists, but production GenAI applications quickly become integration and operational problems.

Resolution: Move engineering capability into the work, then scale the capability out to feature teams through an AI platform.

Model changes caused costly manual prompt rewrites.

Root cause: The organization changed LLMs repeatedly and had to rework prompts without enough automated assurance.

Resolution: Add eval and testing coverage before prompt/model refactors.

Framework churn made teams retreat to lower-level DIY implementations.

Root cause: AI frameworks moved quickly enough that teams felt bitten by dependency changes.

Resolution: Use frameworks for learning and enablement, but keep platform support for observability, tracing, monitoring, and lower-level primitives.

End-user feedback can be collected but not acted on by engineers.

Root cause: Product owners understand the domain context behind feedback, while engineers are more used to error stacks.

Resolution: Treat feedback handling as a product/platform service and include domain owners in the loop.

AI code generation can increase review load and reduce situational awareness.

Root cause: Humans shift from producing code to managing and reviewing larger AI-generated PRs.

Resolution: Invest in tests, evals, monitoring, observability, failure preparation, and review practices rather than only measuring code generation speed.

Guardrail logs can become too noisy to use.

Root cause: Rules and alert logs can grow until teams ignore them, similar to intrusion detection or WAF fatigue.

Resolution: Use central guardrails, team-specific rules, and ongoing tooling improvements rather than assuming a static rule layer is sufficient.

Evaluation

The proposed platform evaluates GenAI applications through production prompt tracing, eval health checks, model-change detection, end-user behavior monitoring, data-quality monitoring, exact tests, regex checks, helper-model sentiment checks, semantic distance, LLM-as-judge, human feedback, and domain-owner feedback interpretation.

Metrics: production prompt traces · eval health checks · model-change detection · end-user behavior anomalies · data quality · exact output checks · regex checks · sentiment helper-model results · semantic distance · LLM-as-judge results · human feedback · review time and PR size

Outcomes

• the talk proposes an AI platform topology rather than reporting one product rollout
• GenAI work moved from data science toward engineers and then into feature teams through platform abstraction
• the proposed platform centralizes model access, vector/RAG services, connectors, observability, evals, caching, feedback, and governance
• the speaker reports eight LLM model changes over two years as evidence for needing evals before prompt/model refactors
• review burden and situational-awareness loss are identified as production risks from AI-generated code
• central governance plus self-service team rules is proposed for guardrails

The case is a 2023 practitioner framework talk, not a single deployed application metric report. Its concrete scale anchors are two years of GenAI app experience, eight LLM model changes, one prompt producing five or six sub-calls, and a recommendation to adopt the platform topology around 10+ teams.