Scaling Custom AI Application Development Through Modular LLM Framework

blackrock · Finance · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Internal AI application development for regulated investment operations

BlackRock needed to build many custom AI applications for investment operations much faster while keeping domain experts, compliance, and human review embedded in the workflow.

Constraints:

• Investment operations span document extraction, complex workflows, Q&A, and agentic systems.
• New issue setup requires prospectus/term-sheet ingestion, domain expertise, structured outputs, transformations, and downstream integration.
• Fully agentic approaches did not work well enough because financial instruments require domain knowledge in human heads.
• Regulated environments require human-in-the-loop and four-eyes checks.
• Deployment requires access control, cluster choice, cost controls, and CI/CD-like app factory behavior.

Architecture

BlackRock built a modular framework with a Sandbox and App Factory. The Sandbox lets investment-operations domain experts configure extraction templates, prompts, data types, validations, inter-field dependencies, transformation logic, and LLM strategy. The App Factory turns sandbox definitions into deployable apps with the right data/developer platform, cluster, access control, and distribution. Human-in-the-loop review remains central for compliance.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Document ingestion pipeline"] --> B["Sandbox"]
  B --> C["Extraction template manager"]
  B --> D["Domain expert workflow"]
  C --> E["Transformation executors"]
  D --> J["Human review gate"]
  B --> F["App Factory"]
  E --> F
  F --> G["Developer platform"]
  F --> H["Deployment cluster selector"]
  F --> I["Access control layer"]
  G --> K["Downstream applications"]
  I --> K

Components

Data flow

Prospectuses or term sheets are ingested into pipelines, domain experts configure extraction templates in the Sandbox, extraction runs produce structured outputs, transformations/executors encode business logic, and the App Factory deploys the resulting application into BlackRock's app ecosystem with appropriate infrastructure and compliance controls.

Why this architecture

BlackRock moves prompt, extraction-template, and LLM-strategy iteration into the hands of domain experts, then uses an app factory to make deployment repeatable. This solves the speed problem without pretending regulated finance can remove human review.

Models used: LLMs for document extraction · LLMs for Q&A systems · LLMs for agentic systems · configurable model providers

Scale

• BlackRock reduced complex app build time from 3-8 months to a couple of days.
  “an app took us close to like 8 months somewhere between 3 to 8 months to build a single app for a complex use case”transcript, stated
• One cited workload required analyzing 500 research reports overnight, implying cluster-choice requirements.
  “hey, I need to analyze, you know, 500 research reports like overnight, can you help me do this?”transcript, stated
• The framework targets four application classes: document extraction, workflow automation, Q&A, and agentic systems.
  “it kind of falls into like four different buckets right? One is everything to do with document extraction.”transcript, stated

Alternatives considered (roads not taken)

Fully agentic systems for the workflow

Rejected because: Financial-domain complexity and embedded human knowledge made full autonomy insufficient.

Engineering-led prompt and template iteration

Rejected because: Domain experts know the extraction and validation requirements, and moving iteration to them speeds delivery.

One fixed deployment cluster for all AI apps

Rejected because: Different workloads need different compute, from GPU inference to burstable clusters.

Patterns

human-in-the-loop-gate (governance)

BlackRock explicitly designs for domain-expert and four-eyes review in regulated financial workflows.

structured-output-contract (interface)

Extraction templates define fields, data types, validations, constraints, and inter-field dependencies.

tool-use-orchestration (agent-architecture)

The framework orchestrates document ingestion, extraction templates, transformations, app deployment, and downstream integrations.

model-routing (model-ops)

Domain experts choose LLM strategies and deployment clusters appropriate to the application.

Tradeoffs

Automation design

Chose human-in-the-loop framework over fully agentic processing.

Rationale: Regulated investment operations need compliance and domain checks.

Iteration ownership

Chose domain-expert sandbox over engineering-only app development.

Rationale: Domain experts can iterate faster on extraction fields, prompts, validations, and strategies.

Deployment abstraction

Chose App Factory over bespoke deployment for each app.

Rationale: Apps need repeatable choices for clusters, access control, distribution, and cost controls.

Gotchas (what bit them)

Fully agentic workflow automation was not enough.

Root cause: Complex financial instruments require domain expertise and tacit business knowledge.

Resolution: Design a human-in-the-loop sandbox and app-factory system.

Deployment became a separate bottleneck after prompt/template work.

Root cause: AI apps need distribution, access control, cluster selection, and cost controls.

Resolution: Use an App Factory to make deployment CI/CD-like.

Simple prompt templates were insufficient for extraction.

Root cause: Financial extraction fields need QC checks, constraints, data types, and dependencies.

Resolution: Provide richer extraction-template configuration in the Sandbox.

Evaluation

BlackRock evaluates through domain-expert extraction comparison, QC checks, validation constraints, four-eyes review, deployment readiness, and time-to-app reduction.

Metrics: time to build app · field validation pass · QC check pass · inter-field dependency correctness · human review completion · deployment readiness

Outcomes

• Complex AI application development compressed from 3-8 months to a couple of days.
• Domain experts can iterate directly on extraction templates and LLM strategies.
• App Factory turns sandbox definitions into deployable internal applications.
• The framework preserves human-in-the-loop compliance for regulated financial workflows.

The framework is aimed at BlackRock investment operations, where internal app needs span extraction, workflows, Q&A, and agentic systems.