Building a Natural Language Agent Builder with Comprehensive LLMOps Practices

vellum · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Natural-language agent builder and LLMOps platform

Vellum wanted to let users build production-grade agents through conversation, while still preserving the tool design, testing, tracing, deployment, and UX discipline needed for reliable agent systems.

Constraints:

• users should not have to learn a new framework, SDK, or drag-and-drop editor before building an agent
• tool misuse is a major source of agent failures
• large documentation corpora are too slow and expensive to put entirely in the system prompt
• agent changes can regress earlier capabilities without a rigorous test suite
• users need progress visibility and interactive UI, not only raw text

Architecture

Vellum built a beta natural-language agent builder: users chat with an agent that builds workflows or agents, consults a summarized knowledge base through a read tool, uses carefully designed tool abstractions, records execution traces, supports tests for desired behaviors, parses text outputs into richer UI elements, and can deploy generated agents as SDK code, hosted API endpoints, or one-click AI apps.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-chat["User chat"] --> natural-language-agent-builder["Natural language agent builder"]
  natural-language-agent-builder --> tool-abstraction-layer["Tool abstraction layer"]
  natural-language-agent-builder --> knowledge-base["Knowledge base"]
  natural-language-agent-builder --> execution-tracer["Execution tracer"]
  natural-language-agent-builder --> ui-renderer["UI renderer"]
  natural-language-agent-builder --> deployment-options["Deployment options"]
  test-suite["Test suite"] --> natural-language-agent-builder
  test-suite --> execution-tracer
  ui-renderer --> user-chat

Components

Data flow

A user describes the agent they want. The builder agent interprets the request, consults concise documentation summaries in its prompt, calls a read-knowledge-base tool when it needs full documentation, uses higher-level tools rather than raw API calls to change the workflow, produces text outputs that Vellum parses into workflow updates and interactive UI elements, stores the full execution trace for review, and eventually ships the result as SDK code, a hosted API endpoint, or an app.

Why this architecture

Vellum chose a conversational builder on top of its existing SDK and editor because it wanted agent creation to be accessible without learning framework mechanics. The internal design emphasizes high-level tools, selective knowledge retrieval, stored executions, rigorous tests when needed, and richer UI parsing because those are the operational controls that keep a meta-agent from becoming opaque or brittle.

Models used: LLMs · agent builder model

Scale

• Vellum had spent three years building production-grade agent tooling.
  “you guys. So, you know, we've spent three years making it easy to build and deploy production grade agents here at Vellum. But unfortunately, I'm retiring from being an agent builder myself. You know, it's very sad. Uh but the reason for that is, you know, I had a good run,”transcript, stated
• The prior drag-and-drop editor and SDK took two and a half of those three years.
  “know, we spent the first two and a half of those three years building this super flexible drag and drop editor and, you know, a really robust, powerful SDK for people to build agents. And, you know, that that's all cool, but however, drag and drop is dead. Like, what is”transcript, stated
• The documentation corpus was about 100,000 tokens.
  “for example, in our agent builder, we have like a 100,000 tokens of like Vellum docs that, you know, talk about how to build agents in Vellum, use our SDK, all that. You know, we could load that all into the system prompt, but like that's going to be really slow and expensive.”transcript, stated
• The knowledge base used about 30 documents.
  “what we found to work well is uh we just created a knowledge base of about 30 documents and put summaries of those really small summaries kind of like you know like the abstract of a paper that's like makes you want to read more into the system prompt. So the model can call”transcript, stated
• Agent builder generations currently take roughly 30 seconds to 2 minutes.
  “know like anywhere from like 30 seconds to like 2 minutes later it hopefully spits out a workflow or like an agent that does what you want. We want to be able to show like more real-time updates in terms of what agent builder is doing so you can see the workflow as it's”transcript, stated
• One bad execution review yielded about 20 takeaways.
  “this still. Well, like we saw this, we looked at this execution, there were like 20 takeaways from it and then we fixed them, right? So that's that's the iteration process that's worked really well for us.”transcript, stated

Alternatives considered (roads not taken)

Require users to learn a framework, SDK, or drag-and-drop builder

Rejected because: The natural-language builder aims to make agent creation conversational and lower-friction.

Expose underlying APIs directly as agent tools

Rejected because: Low-level tools force the agent to rediscover API sequences every run and increase mistakes.

Load all documentation into the system prompt

Rejected because: The 100,000-token documentation corpus would be slow and expensive.

Use only vibes-based testing or only rigorous tests

Rejected because: Vibes help early and for UX feel, but rigorous suites are needed to distinguish high pass rates and prevent regressions.

Patterns

tool-use-orchestration (tools)

Vellum designs tools as high-level actions, groups sequential API calls, simplifies arguments, and gives descriptive errors so the agent can recover.

rag-retrieval (retrieval)

The builder uses a lightweight knowledge base: summaries in the system prompt plus a read tool for full documentation entries.

eval-driven-development (evaluation)

Vellum adds tests for desired behaviors that currently fail and iterates until the agent passes them, while also using vibes-based testing for early UX feel.

observability-tracing (operations)

The agent builder stores and reviews execution traces, thinking steps, tool calls, import/type errors, and user-visible behavior to improve the agent.

guardrails-io-filtering (safety)

The tool-design guidance treats descriptive errors as guardrails that let the model distinguish retryable timeouts from bad arguments.

Tradeoffs

Tool granularity

Chose coherent high-level action tools over blindly exposing low-level APIs.

Rationale: Designing the abstraction once avoids forcing the agent to reason through the same API sequence every runtime execution.

Knowledge access

Chose summaries plus read-knowledge-base tool over putting 100,000 tokens of documentation in the system prompt.

Rationale: The summary/read-tool approach avoids slow and expensive prompts while letting the model retrieve relevant full docs.

Testing strategy

Chose mix vibes-based testing and concrete test suites over treating one testing mode as universally correct.

Rationale: Vibes work early and for UX feel, while concrete tests catch regressions and fine-grained pass-rate changes.

Deployment model

Chose multiple export/deployment options over one fixed hosting path.

Rationale: Users may want SDK code in their own codebase, hosted API endpoints, or one-click apps depending on control and convenience needs.

Gotchas (what bit them)

Tool abstractions are the most common source of agent failures.

Root cause: Poorly designed tools cause missed calls, wrong calls, too many calls, or wrong arguments.

Resolution: Design tools from first principles as high-level actions, simplify arguments, group API sequences, and make similar tools distinct or combined with differentiating arguments.

Undescriptive tool errors block agent recovery.

Root cause: A generic error string gives the model no clue whether to retry, change arguments, or stop.

Resolution: Return error messages that identify timeout versus bad argument and explain what was wrong.

Agent improvements regressed earlier capabilities before a stronger test suite existed.

Root cause: Manual/vibes testing was not enough to detect non-local regressions as the system evolved.

Resolution: Vellum built more rigorous tests and adds tests for behaviors the agent should pass.

Long-running agent-builder executions can feel like a black box.

Root cause: Users may wait 30 seconds to 2 minutes without seeing the workflow change in real time.

Resolution: Vellum plans real-time workflow updates and richer UI to show what the builder is doing as it works.

Evaluation

Vellum uses a blend of vibes-based testing, concrete test suites, regression tests for desired behaviors, and execution-trace review. The team stores executions, reviews failures, derives takeaways from traces, and uses those insights to improve the agent builder.

Metrics: test-suite pass rate · regression coverage · manual UX/vibes assessment · execution traces reviewed · tool-call errors · import/type errors · user-visible execution quality

Outcomes

• Vellum launched a beta natural-language agent builder.
• The builder can produce workflows or agents from chat in roughly 30 seconds to 2 minutes.
• The system supports SDK code export, hosted API endpoints, and one-click AI apps.
• Reviewing one bad execution produced about 20 takeaways that the team fixed.
• The same tracing and visibility tooling is used internally and offered to users.

Vellum's case includes three years of production-agent tooling work, a 100,000-token documentation corpus summarized into about 30 knowledge-base documents, and generation times currently around 30 seconds to 2 minutes.