Forward Deployed Engineering: Bringing Enterprise LLM Applications to Production

openai · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: enterprise LLM application deployment

OpenAI's Forward Deployed Engineering team was created to move enterprise LLM applications from inconsistent pilots into production by embedding with customers, solving high-value workflows, adding evals and guardrails, and turning repeated customer pain into reusable platform primitives.

Constraints:

• hands-off technical support was not reliable enough to get enterprises into production
• regulated and high-stakes users need trust, evaluation, and accuracy before adoption
• enterprise workflows require data translation, integration, tracing, guardrails, and business logic rather than a standalone model call
• LLMs should not enforce hard business constraints that deterministic systems can check
• FDE capacity must stay focused on high-value, generalizable, or research-relevant problems instead of drifting into services revenue

Architecture

The FDE operating model embeds engineers with customers, identifies a core high-value workflow, builds the data/API translation layer around the customer's systems, orchestrates LLM or agent behavior with tools, traces, evals, and deterministic guardrails, pilots with domain users until trust is earned, then extracts reusable primitives into products such as Swarm, Agent SDK, and Agent Kit. The pattern is zero-to-one product discovery through production deployment, not generic consulting.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  embedded-fde-team[Embedded FDE team]
  customer-domain-workflow[Customer domain workflow]
  data-translation-layer[Data translation layer]
  llm-orchestration-layer[LLM orchestration layer]
  deterministic-guardrails[Deterministic guardrails]
  evaluation-and-telemetry-framework[Evaluation and telemetry framework]
  user-pilot-and-feedback-loop[User pilot and feedback loop]
  reusable-product-primitives[Reusable product primitives]
  embedded-fde-team --> customer-domain-workflow
  embedded-fde-team --> evaluation-and-telemetry-framework
  customer-domain-workflow --> data-translation-layer
  customer-domain-workflow --> user-pilot-and-feedback-loop
  data-translation-layer --> llm-orchestration-layer
  data-translation-layer --> deterministic-guardrails
  llm-orchestration-layer --> deterministic-guardrails
  llm-orchestration-layer --> evaluation-and-telemetry-framework
  deterministic-guardrails --> user-pilot-and-feedback-loop
  evaluation-and-telemetry-framework --> llm-orchestration-layer
  evaluation-and-telemetry-framework --> user-pilot-and-feedback-loop
  user-pilot-and-feedback-loop --> reusable-product-primitives
  reusable-product-primitives --> embedded-fde-team

Components

Data flow

A customer workflow enters through embedded discovery and domain modeling. The FDE team builds a data or metadata translation layer over existing enterprise systems, gives the LLM access to appropriate tools or simulators, wraps the workflow in evals, telemetry, and deterministic guardrails, pilots outputs with users, collects labels and feedback, and then either ships the workflow to production or distills the reusable parts into platform primitives for other customers.

Why this architecture

The FDE model exists because enterprise production failures are usually not solved by model access alone. OpenAI found that repeated production success required embedding with users, building evals and guardrails around whole workflows, and using customer deployments as design partners for reusable product/platform capabilities instead of trying to generalize too early.

Models used: GPT-4 · Codex · DALL-E 3 · Swarm · Agent SDK · Agent Kit · OpenAI Playground

Scale

• The FDE team targets problems worth tens of millions to low billions in customer value.
  “saving customers or generating customers to the tune of like you know tens of millions sometimes the low billions in terms of value.”transcript, stated
• Morgan Stanley adoption reached about 98% of wealth advisors and drove a 3x increase in research-report usage.
  “in the end about like 98% of them adopted it. It resulted in like a 3x increase in the amount of like usage of their research reports.”transcript, stated
• The Morgan Stanley technical pipeline took 6 to 8 weeks, followed by months of pilots and trust-building.
  “within like 6 to 8 weeks we had that we had the pipeline. We had some guardrails.”transcript, stated
• The semiconductor customer had engineers spending roughly 70-80% of their time fixing bugs and maintaining compatibility.
  “they have to spend like 70 80% of their time like you know just like fixing bugs, uh like maintaining compatibility with old versions of chips”transcript, stated
• The semiconductor engagement delivered 10 FDE use cases and was already seeing 20-30% savings in early divisions.
  “we delivered like 10 different FDE use cases across their across their value chain”transcript, stated

  “we're sitting is about 20 to 30% for the first couple divisions that we rolled out with.”transcript, stated
• Klarna's customer-service policy problem scaled from about 20 policies to 400 policies or more.
  “wrapping each like intent with a set of e-vals so that we could scale from like say 20 policies all the way up to 400 policies or more”transcript, stated
• The FDE team grew from 2 people to 39, with a target of 52 by year end.
  “then we're now we're now 39 and we're going to be 52 by the end of the year.”transcript, stated

Alternatives considered (roads not taken)

hands-off technical support for enterprise AI customers

Rejected because: OpenAI found that support model was not reliably getting enterprises to production.

generalize an enterprise platform before deeply solving a specific customer problem

Rejected because: The team identified premature generalization as its biggest mistake and found better product primitives by solving specific customer pain first.

let an LLM enforce hard supply-chain constraints directly

Rejected because: The FDE approach uses deterministic checks for business rules that must hold 100% of the time.

pursue services revenue as the core FDE business model

Rejected because: The team wants strategic product/platform outcomes and sees services revenue as a risk that can drag organizations away from product bets.

adopt isolated point tools for orchestration, tracing, labeling, evals, or guardrails

Rejected because: The interview argues that enterprises struggle when they must wire many partial tools together and solve integration problems themselves.

Patterns

tool-use-orchestration (agent-architecture)

FDE systems give LLMs access to customer APIs, simulators, execution environments, and workflow tools while controlling what the model can do.

eval-driven-development (evaluation)

The FDE process builds labeled expert trajectories and eval sets before considering LLM-generated work done.

safety-policy-enforcement (governance)

Hard business constraints and runtime guardrails are enforced deterministically rather than left to probabilistic model behavior.

observability-tracing (operations)

The semiconductor debugging work forked Codex and added telemetry so the team could define and measure expert action trajectories.

rag-retrieval (retrieval)

Morgan Stanley's deployment required retrieval tuning over wealth-management research reports before advisors could trust the output.

human-in-the-loop-gate (adoption)

Customer users label data, check work, and build trust before high-stakes deployments move from technical readiness to production usage.

fine-tuning-adaptation (optimization)

The interview predicts using orchestrated agent networks to generate training data and fine-tune specialist models for domains like chip design or drug discovery.

Tradeoffs

Choose embedded FDE over hands-off enterprise support.

Chose customer-embedded engineering and adoption work over remote technical support around generic model access.

Rationale: Production adoption depended on deep domain understanding and sitting with users, not only answering technical questions.

Use deterministic guardrails for constraints and LLMs for nuanced reasoning.

Chose hybrid deterministic-probabilistic architecture over letting the model handle every part of planning and validation.

Rationale: The team uses deterministic systems for rules that must always hold and reserves LLMs for orchestration, insight generation, and tradeoff exploration.

Solve customer-specific pain before platformizing.

Chose zero-to-one deployments followed by extraction of reusable primitives over building generalized enterprise products before the problem is proven.

Rationale: The team learned that deep customer work reveals generalizable primitives, while premature generalization creates high-concept products without traction.

Prioritize strategic product and research learning over pure services revenue.

Chose selective high-value FDE engagements over taking every lucrative custom-services opportunity.

Rationale: OpenAI wants FDE work to become platform capabilities or research learning, not a services treadmill.

Build a data translation layer over existing enterprise systems.

Chose metadata/API logic layers that let LLMs use data in place over moving all data into a new centralized store before building the application.

Rationale: The interview frames metadata translation as the repeated work needed for LLMs to combine raw enterprise data and business logic effectively.

Gotchas (what bit them)

Model access plus hands-off support did not reliably produce enterprise deployments.

Root cause: Production success required domain immersion, user adoption work, and workflow engineering beyond generic technical support.

Resolution: Create a forward-deployed model that embeds with customers and repeatedly translates hard enterprise workflows into production systems.

Technical readiness arrived before user trust.

Root cause: Morgan Stanley's regulated users needed evidence, labels, pilots, and feedback loops before trusting probabilistic outputs.

Resolution: Continue pilots, collect evals, and iterate with advisors after the technical pipeline works.

Enterprise LLM workflows can fail when they depend on many disconnected point tools.

Root cause: Production requires orchestration, tracing, labeling, evals, guardrails, and integrations across enterprise systems.

Resolution: Build an integrated workflow layer or platform capability rather than leaving customers to stitch the chain together.

Generalizing too early produced weak enterprise products.

Root cause: High-concept solutions were not anchored in a clear customer problem.

Resolution: Do things that do not scale first, solve a specific hard workflow, then extract reusable primitives.

Probabilistic LLMs are unsafe for hard business constraints.

Root cause: Supply-chain optimization has requirements such as supplier minimums, materials coverage, costs, and lead times that must be correct.

Resolution: Use deterministic constraints and verification widgets around the LLM's planning and simulation outputs.

Evaluation

The FDE approach relies on eval-driven development: expert-labeled trajectories, pilot feedback, telemetry, deterministic checks, and adoption or efficiency metrics for deployed customer workflows.

Metrics: 98% Morgan Stanley advisor adoption · 3x increase in Morgan Stanley research-report usage · 6-8 weeks to Morgan Stanley technical pipeline, followed by 4 months of pilots and eval iteration · 20-30% efficiency savings in early semiconductor divisions, with a 50% target · Klarna policy scaling from about 20 policies to 400 or more · FDE team growth from 2 to 39, with 52 planned by year end

Outcomes

• Morgan Stanley deployed GPT-4 for wealth-management research, reaching about 98% advisor adoption and 3x research-report usage.
• The semiconductor engagement delivered multiple FDE use cases and evolved a Codex-based debugging workflow toward fixing bugs and raising pull requests.
• The Klarna customer-service work became Swarm, then influenced Agent SDK and Agent Kit.
• The automotive supply-chain example demonstrated an LLM orchestrator with deterministic checks, simulator access, and explainable tradeoff recommendations.
• The FDE team grew rapidly and became a repeatable enterprise deployment motion at OpenAI.

The interview cites high-value target problems, tens of millions to low billions in customer value, 39 FDEs growing to 52, 400+ policy customer-service scale, 10x more complex T-Mobile follow-on work, and hundreds or thousands of simulation runs in production-style agentic planning.