Building a Tool Calling Platform for LLM Agents

arcade_ai · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM agent infrastructure

Arcade AI argues that agents cannot become broadly useful until tool execution, tool management, and user-delegated authentication are separated from ad hoc orchestration code.

Constraints:

• Current tool calling often predicts parameters but leaves execution to developers or orchestration frameworks.
• Tools may need different runtimes, compute resources, dependencies, scaling behavior, and authorization scopes.
• API-key-only tools limit agents to public or bot-token style actions instead of acting as a user.
• Tool behavior needs CI/CD-style testing, versioning, packaging, and monitoring.

Architecture

Arcade AI provides a tool-calling platform with an opinionated Tool SDK, an engine exposing Auth, Tools, and LLM APIs, and actor workers that serve Python tools. The LLM API is OpenAI-compatible, injects and executes tools through the Tools API, and relies on a user-scoped OAuth/Auth API so agents can perform authenticated actions such as Notion, Twilio, Gmail, and Outlook operations.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  client-application[Client Application] --> llm-api[LLM API]
  llm-api --> tools-api[Tools API]
  tools-api --> auth-api[Auth API]
  auth-api --> token-storage[Token Storage]
  tools-api --> actor-workers[Actor Workers]
  tool-sdk[Tool SDK] --> actor-workers
  actor-workers --> external-services[External Services]
  tool-sdk --> tool-evaluation-framework[Tool Evaluation Framework]

Components

Data flow

A client sends a prompt and available tool definitions to an LLM/Arcade endpoint, the model predicts tool parameters, Arcade routes execution through its LLM API to the Tools API, the Tools API obtains user-scoped authorization from the Auth API, and an actor worker runs the tool in the appropriate runtime before the result is fed back into the LLM response.

Why this architecture

Arcade separates tool execution from orchestration so tools can be packaged, scaled, authenticated, tested, and run in the environments they require instead of being trapped inside a Python orchestration process or API-key-only helper.

Models used: large language models

Scale

• The platform reflects about a year and a half of Arcade-building experience.
  “let us down bring us home talk to us about what you've learned in the last year and a half building our Kade wouldn't dare I think I've never had a harder time picking which slides I want to put in a presentation so we're going to try and get through it but as usual I”transcript, stated
• The speaker frames tool selection as hard when an agent may have hundreds or thousands of tools.
  “if we can facilitate the usage of an agent with hundreds if not thousands of tools the capability of that agent grows significantly”transcript, stated
• Smaller models may degrade around about 20 tools in context.
  “models especially smaller models start to fall off at about 20 or so tools so managing those is a non-trivial task”transcript, stated
• Arcade's tools can be hosted with about four lines of Modal code in the cited Slackbot example.
  “we actually have a slackbot hosted on modal right now um and it is incredibly easy to host it's about four lines of code”transcript, stated
• Arcade runs CI/CD-style tests for its tool packages every night.
  “every single one cicd runs the tools every night um we have to do that for all of the packages every single one um we have a difference between Community packages and arcade packages arcade packages will be tested that way community packages will not be great question I have a lot”transcript, stated

Alternatives considered (roads not taken)

Collocate tool execution inside the agent orchestration framework

Rejected because: Collocation mixes compute resources and prevents scaling tool execution independently from the agent.

Treat tools as notebook helpers or repo subpackages

Rejected because: The speaker argues tools need standard software practices: packaging, pinned dependencies, versions, tests, and monitoring.

API-key bot-token tools

Rejected because: API-key tools limit agents to low-impact actions and do not let agents act with the user's delegated permissions.

Developer-managed tool execution and prompt reinjection

Rejected because: Arcade's LLM API moves execution, input/output handling, and result reinjection behind a single HTTPS request.

Patterns

tool-use-orchestration (agent infrastructure)

Arcade centralizes tool registration, grouping, selection, execution, scaling, and result handling outside the agent orchestration framework.

structured-output-contract (tooling)

Tool definitions require typed inputs, typed outputs, descriptions, and execution methods so LLMs can predict parameters.

eval-driven-development (quality)

Arcade includes a tool-calling evaluation framework intended to run in CI/CD against new models, prompts, and tool capability changes.

safety-policy-enforcement (security)

OAuth scopes, user IDs, token storage, and deployment-boundary choices constrain what agents can do on behalf of users.

cost-aware-routing (operations)

The architecture allows tools to run in the runtime they need, such as GPU Modal for image generation, while the agent runs elsewhere.

Tradeoffs

Tool runtime placement

Chose Dedicated distributed tool runtimes over Tool execution inside orchestration frameworks.

Rationale: Separate runtimes let tools scale, use specialized compute, and be maintained independently from agent logic.

Tool SDK design

Chose Opinionated typed Python tool package scaffold over Loose docstring parsing and arbitrary helper functions.

Rationale: Parameter annotations and packaging reduce ambiguity and make tools easier to serve to LLMs.

Authorization model

Chose User-scoped OAuth-style auth over Bot-token/API-key access.

Rationale: Agents become more useful when they can act as the user within authorized scopes.

Deployment responsibility

Chose Support Arcade cloud or customer-deployed Arcade environments over Only hosted execution.

Rationale: Security responsibility and token residency can align with the organization running the runtime.

Gotchas (what bit them)

Tool calling is often confused with tool execution.

Root cause: Model APIs typically return predicted parameters, but execution remains the developer's responsibility.

Resolution: Design an execution platform where predicted parameters flow into managed tools and runtimes.

Python orchestration frameworks can constrain tool runtime and concurrency.

Root cause: Collocated tool execution is limited by the orchestration runtime and can hit Python/GIL-style constraints for compute-heavy tools.

Resolution: Separate tool execution into actor runtimes that can be scaled independently or deployed on appropriate infrastructure.

Too many available tools can overwhelm model context and tool selection.

Root cause: Hundreds of tools cannot all fit in a single useful prompt, and smaller models degrade around 20 tools.

Resolution: Invest in tool grouping, actor-specific toolkits, and future tool selection systems.

Tool integrations can silently change behavior under the agent.

Root cause: External services and integration packages evolve independently.

Resolution: Run CI/CD against fake OAuth accounts and every tool package nightly.

Evaluation

Arcade evaluates tool capability through a tool-calling-specific evaluation framework, critics/rubrics, CI/CD checks on model and prompt changes, nightly integration tests with fake OAuth accounts, and package-level testing for Arcade-owned tools.

Metrics: Tool-calling capability under new models · Agent capability under prompt/model changes · Tool package nightly CI pass/fail · Integration behavior with OAuth-backed services · Production capability drift

Outcomes

• Arcade demonstrated authenticated tool actions such as creating a Notion document, sending a Twilio message, and reading Gmail.
• The platform provides SDK, Auth API, Tools API, LLM API, and actor worker pieces for tool-calling agents.
• The speaker reported prebuilt toolkits and an open-source/downloadable platform with a closed-source engine binary.

The talk is an early product/platform presentation rather than a mature production case study; the strongest operating detail is nightly CI for Arcade-owned tool packages.