Building Production-Ready AI Agents: Lessons from BeeAI Framework Development

ibm · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Open-source AI agent framework development and production-readiness lessons

IBM Research's BeeAI team wanted to help broader builders create useful AI agents, but found that production-ready agents require transparency, custom architectures, enforceable rules, security controls, clean developer experience, and evaluation processes beyond ordinary chat UI or prompt engineering.

Constraints:

• Non-experts struggled to translate generative AI capabilities into high-value problem solving.
• Successful teams needed both LLM expertise and systems-engineering expertise.
• Users did not trust agent outputs unless they could inspect how the answer was produced.
• Agent interactions with external systems amplify security risk, including irreversible actions.
• Prompting alone cannot guarantee enterprise business-rule adherence.
• No single agent architecture fits every production use case and acceptance criterion.
• Existing benchmark datasets and tooling did not fit the team's agent-evaluation needs.

Architecture

The BeeAI effort began with Peri, an agent prototype that combined tool calling, chain-of-thought prompting, web navigation, code execution, and synthesis on top of Llama 3-70B-Chat, an open-source model without built-in reasoning. The team added a trajectory explorer so users could inspect the steps used to produce a response. Those lessons fed into BeeAI Framework, a TypeScript-based open-source agent framework for full-stack developers. BeeAI evolved from an opinionated BeeAgent into ReActAgent plus workflows: a flexible multi-actor orchestration system intended to let developers compose agent architectures matched to their use case, interaction modality, and acceptance criteria. The team also built custom evaluation processes and benchmark data that test both outputs and reasoning trajectories.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  builder-needs-research[Builder needs research] --> beeai-framework[BeeAI Framework]
  builder-needs-research --> open-source-community[Open-source community]
  peri-prototype[Peri prototype] --> llama-3-70b-chat[Llama 3-70B-Chat]
  llama-3-70b-chat --> chain-of-thought-prompting[Chain-of-thought prompting]
  chain-of-thought-prompting --> tool-calling-layer[Tool calling layer]
  tool-calling-layer --> peri-prototype
  peri-prototype --> trajectory-explorer[Trajectory explorer]
  trajectory-explorer --> evaluation-process[Evaluation process]
  beeai-framework --> reactagent[ReActAgent]
  beeai-framework --> workflows[Workflows]
  workflows --> reactagent
  workflows --> agent-interaction-modalities[Agent interaction modalities]
  beeai-framework --> open-source-ui[Open-source UI]
  open-source-ui --> agent-interaction-modalities
  open-source-ui --> open-source-community
  beeai-framework --> evaluation-process
  beeai-framework --> security-and-rule-controls[Security and rule controls]
  security-and-rule-controls --> tool-calling-layer

Components

Data flow

The initial agent receives complex user queries, breaks them into sub-steps, navigates web pages, calls tools, executes code, and synthesizes an answer. A trajectory explorer records and displays the agent's reasoning and actions so users can inspect how the answer was produced. The framework layer packages these lessons into BeeAI Framework, where developers can choose ReActAgent or workflow-based multi-actor orchestration, connect tools and interaction modalities, and evaluate features against custom benchmark datasets. The open-source stack and UI provide the consumption layer for developers to interact with, consume, and demo their agents.

Why this architecture

The team built its own framework because existing tools did not meet full-stack developer needs, and because agent architectures vary too much for a single definitive design. The trajectory explorer and interaction modalities address user trust and consumption, while workflows and evaluation processes let teams adapt architecture and acceptance criteria to each production use case.

Models used: Llama 3-70B-Chat · OpenAI o1 · DeepSeek-R1 · LLMs · BeeAI Framework · ReActAgent · RAG · Tool calling · Chain-of-thought prompting

Scale

• The BeeAI lessons come from a year-long IBM Research incubation effort.
  “Over the past year, we have been laser-focused on enabling a diverse set of builders to harness the power of AI agents”article, stated
• The team made its initial premises early in 2024.
  “Early in 2024, we made two key observations from our prior work incubating Generative AI models”article, stated
• The early prototype used the open-source Llama 3-70B-Chat model.
  “it relied on an open-source model, Llama 3–70B-Chat”article, stated
• BeeAI Framework is a TypeScript-based library.
  “the BeeAI Framework, a TypeScript-based library”article, stated
• The open-source stack launched last year and found an audience with TypeScript developers.
  “BeeAI Open-Source Stack that launched last year.”article, stated

  “We quickly found an audience with TypeScript developers, who leveraged BeeAI’s capabilities to build innovative applications.”article, stated
• Python feature parity is described as a top priority because Python dominates AI development.
  “Python is key to unlocking broader adoption, as it remains the dominant language for AI development”article, stated

  “Bringing the Python library to feature parity with Typescript is our top priority.”article, stated
• BeeAI's community examples include Bee Canvas and UI Builder.
  “Bee Canvas and [UI Builder](https://www.youtube.com/watch?v=hH7vY04LRZ4”article, stated

Alternatives considered (roads not taken)

Traditional chat GUI only

Rejected because: Users needed to inspect the agent's reasoning trajectory, so the team added a visual trajectory explorer.

Using only existing agent frameworks

Rejected because: The team found a gap for full-stack developers and built BeeAI Framework to address it.

Closed-source validation of the stack

Rejected because: The team open-sourced the entire stack to see what resonated and measure traction.

One definitive agent design

Rejected because: The team renamed BeeAgent to ReActAgent and added workflows after concluding that production architectures must vary by use case.

Serving everyday builders and developers equally from the start

Rejected because: Initial traction came from developers, so the team chose to focus on one persona at a time.

Patterns

tool-use-orchestration (agent-runtime)

The prototype and framework combine LLMs with designated tool environments so agents can retrieve documents, navigate pages, execute code, and act beyond chat.

prompt-chaining (reasoning)

The early agent used chain-of-thought prompting plus tool calling to obtain agentic behavior from Llama 3-70B-Chat despite the model lacking built-in reasoning.

observability-tracing (trust-and-debugging)

The trajectory explorer exposes the agent's intermediate steps, making reasoning inspectable and increasing user confidence.

eval-driven-development (quality)

BeeAI created custom benchmark data, defined every supported feature, regression-tested new capabilities, and evaluated both outputs and reasoning trajectories.

multi-agent-orchestration (agent-architecture)

BeeAI added workflows as a flexible multi-actor orchestration system so developers can compose custom agent architectures.

human-in-the-loop-gate (interaction-design)

The framework roadmap treats agent-to-human interaction as a core design space, and the trajectory explorer makes human inspection part of trusting agent work.

safety-policy-enforcement (enterprise-readiness)

The team explicitly identifies security amplification and business-rule enforcement as production requirements that cannot be solved by prompting alone.

Tradeoffs

Use an open-source non-reasoning model for the early agent prototype

Chose Llama 3-70B-Chat plus agent architecture, tool calling, and chain-of-thought prompting over A model with built-in reasoning such as OpenAI o1 or DeepSeek-R1.

Rationale: The team showed that agentic behavior could be harnessed through architecture even when the model itself lacked built-in reasoning.

Add trajectory exploration to the user experience

Chose A visual trajectory explorer over A traditional final-answer chat interface.

Rationale: The team found that inspecting the reasoning path was necessary for transparency and user trust.

Build BeeAI Framework

Chose A custom TypeScript-based agent framework over Existing agent tooling.

Rationale: Existing tools left a gap for full-stack developers, and BeeAI could encode the team's requirements throughout the stack.

Open-source the entire stack

Chose Open community validation and traction measurement over Keeping the framework closed during early development.

Rationale: Open-sourcing let the team see what resonated with different users and gather real-world adoption signals.

Move from BeeAgent to ReActAgent plus workflows

Chose A named single-agent implementation and flexible multi-actor workflow system over A single opinionated framework-default architecture.

Rationale: The team concluded that production users need architecture flexibility tied to specific use cases and acceptance criteria.

Gotchas (what bit them)

Correct answers are insufficient if users cannot inspect the reasoning path.

Root cause: Agent behavior is opaque in a traditional chat GUI, so users lack confidence even when the final response is plausible.

Resolution: Expose trajectories through a visual explorer that lets users investigate the steps taken to generate a response.

Users distrust reasoning methods that violate their mental model.

Root cause: Accuracy alone does not capture whether the path to the answer feels appropriate for the task.

Resolution: Design agent behavior and tools to align with expected reasoning, such as using calculators for math rather than scraped search results.

Agentic systems expand the security blast radius.

Root cause: Agents dynamically interact with external systems and can modify state or take irreversible actions.

Resolution: Treat rule enforcement and security controls as framework-level enterprise requirements, not as prompt-only concerns.

Prompting cannot guarantee enterprise rule compliance.

Root cause: LLM instructions can guide behavior but do not strictly enforce business rules under all conditions.

Resolution: Add comprehensive rule-enforcement mechanisms around agent behavior.

Trying to serve multiple personas too early diluted product focus.

Root cause: The team wanted everyday builders but observed stronger traction from developers.

Resolution: Double down on developers first, including Python feature parity and a cleaner developer experience.

Fragmented repositories created developer-experience friction.

Root cause: A spread of repositories made it harder for newcomers to get value from the stack.

Resolution: Consolidate repositories and create a more seamless developer experience.

Evaluation

BeeAI's evaluation process was custom-built because existing benchmarks and tooling did not match the framework's needs. The team defined supported features, built a custom benchmark dataset, checked simple controls such as tone and verbosity as well as complex reasoning accuracy, regression-tested new capabilities, and inspected reasoning trajectories in addition to final outputs.

Metrics: Supported feature coverage · Custom benchmark dataset performance · Tone control · Verbosity control · Complex reasoning task accuracy · Regression impact on existing functionality · Reasoning trajectory analysis

Outcomes

• The team built Peri, an early agent prototype that outperformed commercially available search solutions at the time.
• The trajectory explorer improved transparency and user trust by letting users inspect agent reasoning steps.
• BeeAI Framework emerged as an open-source TypeScript framework for full-stack developers.
• The open-source stack found traction with TypeScript developers and community implementations such as Bee Canvas and UI Builder.
• The team narrowed focus to developers, prioritized Python feature parity, and began consolidating repositories for a cleaner developer experience.
• Production-agent lessons were codified around architecture flexibility, interaction modalities, evaluation, security, and business-rule enforcement.

The source describes a year of IBM Research incubation rather than a mature enterprise deployment. Concrete scale markers include the early-2024 starting premises, Llama 3-70B-Chat as the prototype model, a TypeScript BeeAI Framework, community projects such as Bee Canvas and UI Builder, and Python feature parity as a top priority for broader adoption.