Building an Enterprise GenAI Platform with Standardized LLMOps Framework

factset · Finance · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise financial GenAI platform

FactSet needed to replace fragmented, team-by-team LLM experimentation with a standardized LLMOps platform that could support governed RAG applications, model comparison, lineage, and production quality controls.

Constraints:

• LLM development was fragmented across teams without a shared platform.
• Financial workflows needed governance, auditability, lineage, and controlled access to proprietary content.
• RAG quality and latency both mattered for earnings-call summarization.
• The platform needed to support commercial, open-source, and custom models without forcing one model choice.

Architecture

FactSet standardized new LLM and AI application development on Databricks and Databricks-managed MLflow. A representative earnings-call summarization RAG pipeline chunks call transcripts by length and speaker, creates embeddings, updates AI Search indexes, retrieves relevant context, evaluates model iterations in MLflow, and uses Unity Catalog for governance, lineage, and auditability.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  application-teams[Application Teams] --> databricks-llmops-platform[Databricks LLMOps Platform]
  databricks-llmops-platform --> mlflow[MLflow]
  databricks-llmops-platform --> unity-catalog[Unity Catalog]
  databricks-llmops-platform --> rag-pipeline[RAG Pipeline]
  rag-pipeline --> ai-search-index[AI Search Index]
  rag-pipeline --> model-endpoints[Model Endpoints]
  mlflow --> model-endpoints[Model Endpoints]

Components

Data flow

Earnings-call text is chunked by length and speaker, embedded, and indexed in AI Search. RAG applications retrieve indexed context, compare model outputs and performance in MLflow, and rely on Unity Catalog for governed data and model lineage.

Why this architecture

FactSet chose a standardized Databricks/MLflow stack because the main bottleneck was not one RAG application but the lack of common governance, model comparison, lineage, and reusable LLMOps practices across many teams.

Models used: Llama 3 · DBRX · Mistral · OpenAI · Anthropic · LangChain custom models · HuggingFace custom models

Scale

• The platform targets enterprise-wide standardization of new LLM and AI application development at FactSet.
  “FactSet decided to standardize new development of LLM / AI applications on Databricks and Databricks-managed MLflow”article, stated
• A model change reduced average user request latency by more than 70%.
  “and, recently, Databricks DBRX to reduce average user request latency by over 70%.”article, stated
• A later optimization reduced end-to-end latency by about 60%.
  “Because of our investment in fine-tuning efforts, we were able to significantly reduce end-to-end latency by about 60%. Most of these fine-tuned models are from open-source LLM models, as depicted in the figure above.”article, stated

Alternatives considered (roads not taken)

Let teams keep fragmented LLMOps practices.

Rejected because: Fragmentation produced governance, lineage, and auditability problems.

Use a simple RAG workflow without deeper LLMOps iteration.

Rejected because: The team quickly hit a quality ceiling and needed evaluation and model iteration.

Commit to a single proprietary model path.

Rejected because: The platform needed to compare and support custom, open-source, commercial, and external models.

Patterns

rag-retrieval (knowledge)

FactSet uses RAG over earnings-call content to ground summarization.

eval-driven-development (llmops)

MLflow is used to compare model performance across iterations.

model-routing (platform)

The platform supports many model families and provider types rather than one fixed model.

latency-optimization (performance)

FactSet tuned model and serving choices after testers saw response times over a minute.

Tradeoffs

Platform standardization

Chose Databricks-managed MLflow and Unity Catalog over Fragmented team-local LLMOps stacks.

Rationale: Shared tools made governance, lineage, auditability, and model comparison enforceable.

Model strategy

Chose A multi-model platform over A single commercial-model pipeline.

Rationale: Different workloads needed cost, latency, and quality comparisons across model options.

Gotchas (what bit them)

A working RAG demo can still hit a quality ceiling.

Root cause: Retrieval alone did not solve model quality, evaluation, and iteration needs.

Resolution: Use MLflow and the standardized platform to compare iterations and improve the pipeline.

Enterprise LLMOps fragments without shared governance.

Root cause: Different teams used different tools, models, and governance models.

Resolution: Standardize development on Databricks, MLflow, and Unity Catalog.

Evaluation

FactSet evaluated the LLMOps stack through model-performance comparisons, latency measurements, tester feedback, and RAG quality iteration for earnings-call summarization.

Metrics: Average user request latency · End-to-end latency · Model performance across iterations · RAG quality ceiling · Governance and lineage coverage

Outcomes

• FactSet standardized new LLM and AI application development on Databricks and MLflow.
• The representative RAG pipeline improved latency through model and architecture changes.
• Unity Catalog addressed governance, lineage, and auditability gaps.

The source reports enterprise standardization, over-70% average request latency reduction, about-60% end-to-end latency reduction, and support for commercial, open-source, external, and custom models.