Building Production AI Agents at Scale with Temporal and KGoose

block · Finance · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production financial-services agent platform

Block needed to turn Goose-style local agents into KGoose, a shared service that could power Cash App, Square, and internal products while handling long-running sessions, tool calls, non-deterministic LLM outputs, financial-data compliance, and frequent AI workflow iteration.

Constraints:

• A single user input could produce dozens of tool calls and frequent timeouts under a synchronous chat API.
• Agent sessions need to run for a long time while preserving state and allowing interruption, continuation, and version rollout.
• LLM and tool payloads can be variably sized, large, and non-deterministic.
• Financial-services products require guardrails, PII protection, data isolation, encryption, and product-specific compliance boundaries.
• Temporal history, LLM context windows, and DynamoDB access patterns impose different limits that have to be managed separately.
• The same platform must serve Money Bot, Manager Bot, G2, subagents, memory extraction, automations, and ad-hoc chats.

Architecture

KGoose evolved from a stateless query API to a DynamoDB-backed synchronous chat API and then to an asynchronous Temporal-backed agent workflow. Clients push messages and poll for results. The API persists session/task/message state in DynamoDB, starts or updates a Temporal chat workflow for the session, and uses Temporal state variables to coordinate one task at a time. The workflow runs an await-driven agent loop, fetches messages from DynamoDB, calls the LLM through activities, stores responses, executes tool calls through MCP, and clears the current task when complete. Activities are split between chat/state/tool work and inference/model-resolution work. The platform adds lifecycle termination, interruption cleanup, human-in-the-loop patterns, context compaction, continue-as-new, model adapters, guardrails, error categorization, Get Version/replay testing, and monitoring workflows.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  product-agents["Product agents"] --> kgoose-api["KGoose API"]
  kgoose-api --> dynamodb-session-store["DynamoDB session store"]
  kgoose-api --> temporal-chat-workflow["Temporal chat workflow"]
  temporal-chat-workflow --> dynamodb-session-store
  temporal-chat-workflow --> chat-activities["Chat activities"]
  temporal-chat-workflow --> llm-activities["LLM activities"]
  temporal-chat-workflow --> human-approval-surface["Human approval surface"]
  temporal-chat-workflow --> continue-as-new-lifecycle["Continue-as-new lifecycle"]
  temporal-chat-workflow --> monitoring-workflow["Monitoring workflow"]
  chat-activities --> dynamodb-session-store
  chat-activities --> mcp-tools["MCP tools"]
  chat-activities --> guardrail-services["Guardrail services"]
  llm-activities --> model-adapter-layer["Model adapter layer"]
  llm-activities --> guardrail-services
  message-compaction["Message compaction"] --> temporal-chat-workflow
  message-compaction --> dynamodb-session-store
  replay-and-versioning-tests["Replay and versioning tests"] --> temporal-chat-workflow
  monitoring-workflow --> temporal-chat-workflow
  monitoring-workflow --> dynamodb-session-store
  human-approval-surface --> product-agents

Components

Data flow

A client sends a V3 push message with a message, model choice, and enabled tools. The API validates it, stores the session and task in DynamoDB, starts or finds the Temporal chat workflow, checks whether the session is busy, and calls the start-task update method. The workflow waits until a task is available, tool requests are resolved, and the session is not interrupted. It fetches current messages from DynamoDB, invokes the LLM through an LLM activity, saves the response, processes MCP tool calls through chat activities, and exposes updates to clients through DynamoDB-backed polling. Guardrails, context compaction, continue-as-new, interruption cleanup, model fallback, and monitoring workflows handle production failure cases.

Why this architecture

Block moved to an asynchronous Temporal architecture because synchronous agent loops timed out under real production tool use. Temporal was already trusted inside Block for durable tier-zero workflows, while DynamoDB was better suited for low-latency client-facing data. The final design intentionally separates orchestration from data and keeps workflows lean while activities hold business logic, retries, and mutable AI behavior, allowing a small team to share one workflow across customer-facing and internal products.

Models used: LLMs accessed through KGoose · Model X for fast quality responses · Separate vision-capable model adapter · External guardrail models · Anthropic models · OpenAI models

Scale

• The platform powers Square, Cash App, Afterpay, Money Bot, Manager Bot, G2, automations, and internal productivity use cases.
  3 supporting quotes

  “I work for Block the parent company of Square, Cash App and Afterpay.”transcript, stated

  “Money Bot, a financial assistant inside the Cash App that allows users to send peer-to-peer payments.”transcript, stated

  “Manager Bot is a business partner for Square merchants.”transcript, stated
• Temporal was already used at Block for more than 100 services, including tier-zero payments and ledger workflows.
  “We've been using it for over a hundred services including some tier zero critical features like payments and the Square ledger.”transcript, stated
• KGoose keeps hot workflows alive but terminates them after one hour of inactivity, checking every 10 minutes.
  “Workflows can remain active while the session is hot and are terminated after one hour of an activity.”transcript, stated

  “it basically wakes up every 10 minutes and checks if it's been an hour since the last user interaction occurred.”transcript, stated
• Temporal event history limits are 50 MB or 50,000 events, while one agent turn sometimes added 3.4 MB.
  “Temporal has a hard limit of 50 megabytes or 50,000 events before hard terminating the workflow execution.”transcript, stated

  “each turn was adding about 3.4 megabytes of data into event history”transcript, stated
• Manager Bot must handle merchants with catalogs containing thousands of items.
  “some merchants just have a catalog with thousands of catalog items.”transcript, stated
• Some tool doom loops continued for over 100 consecutive tool calls before the task had to be terminated.
  “the doom loop will still happen over a hundred consecutive tool calls and at some point we just need to kind of give up and terminate the task”transcript, stated
• The chat workflow had been in production for over 10 months by the time of the talk.
  “since launching a chat workflow for our first use case over 10 months ago now”transcript, stated
• Internal and Cash App use cases process over 100 million weekly activities at 300 to 400 actions per second.
  “Our internal and Cash App use cases are at about over a hundred million weekly activities or the 300 to 400 actions per second range.”transcript, stated
• Block reports over 10,000 concurrent workflows running in each namespace.
  “we're about over 10,000 workflows that's running concurrently in each namespace at any given time.”transcript, stated

Alternatives considered (roads not taken)

Keep the V1 one-message query API.

Rejected because: It had no agent capability and forced clients to track message history themselves.

Keep the V2 synchronous stateful chat API.

Rejected because: Long tool-call chains frequently timed out and degraded production use.

Store primary application data inside Temporal workflow state.

Rejected because: Temporal event histories and queries were poor fits for low-latency APIs, streaming, schema evolution, and data access outside the agent lifecycle.

Terminate and rehydrate every workflow on every user message.

Rejected because: Rehydrating on every request would add latency, so Block chose an inactivity-based lifecycle compromise.

Let agents directly execute every sensitive action after approval.

Rejected because: For high-trust operations, Block sometimes returns structured recommendations to the client instead, simplifying auth by avoiding direct agent capabilities.

Use Temporal's default continue-as-new suggestion alone.

Rejected because: The SDK suggestion was too conservative for KGoose's large agent payloads, so custom thresholds and future reference-passing were needed.

Patterns

durable-execution (reliability)

KGoose runs each long-lived chat session as a Temporal workflow, using async polling, update/signal/query methods, continue-as-new, lifecycle termination, and recovery workflows to keep agent loops durable.

tool-use-orchestration (agent_tools)

The workflow treats a task as a user message plus all subsequent tool calls, uses MCP for tools, and coordinates tool request resolution before the loop advances.

human-in-the-loop-gate (safety)

KGoose supports both approval-gated executions and recommendation-only actions when clients should decide or execute sensitive operations.

guardrails-io-filtering (safety)

The platform checks user inputs for complaints/escalations, LLM outputs for PII and out-of-band statements, and agent-specific nonsense through regex and external guardrail services.

model-routing (model_operations)

KGoose keeps the agent loop model-agnostic, uses adapters for model capability gaps, and falls back to another model during provider outages or rate limits.

observability-tracing (operations)

Block uses production conversation monitoring, error traces, per-session inspection, and stuck-workflow monitoring to discover and recover from agent failures.

eval-driven-development (evaluation)

Harness changes to prompts, tools, MCP definitions, and guardrails are paired with product-specific eval suites and human testing.

agent-memory (context_management)

The chat workflow is used as a unit of LLM inference for memory extraction, while session state and message history persist outside the workflow in DynamoDB.

Tradeoffs

Move KGoose V3 to asynchronous Temporal orchestration.

Chose Client submit-and-poll with a Temporal chat workflow over Synchronous stateful chat requests.

Rationale: Async orchestration avoids request timeouts from long tool chains and gives the workflow durable control over long-running agent loops.

Separate orchestration state from application data.

Chose Temporal for lightweight orchestration, DynamoDB for messages and client-facing data over Putting full application data in Temporal event history.

Rationale: This gives Temporal the durability/retry role while DynamoDB handles low-latency APIs, streaming workarounds, schema evolution, and data access outside agent lifecycles.

Keep workflows lean and activities complex.

Chose Business logic, state management, retries, and model work in activities over Heavy workflow logic.

Rationale: Activity-level changes are easier to test, retry, time out, and evolve without constantly versioning deterministic workflow code.

Use hot-session inactivity termination.

Chose Keep active sessions warm and terminate after one hour idle over Never-ending workflows or rehydrating on every request.

Rationale: The policy balances responsiveness for active sessions with operational flexibility for versioning and deprecating old logic.

Offer both direct approval and recommendation-only human-in-the-loop modes.

Chose Approval-gated execution for some tools and structured recommendations for others over One universal agent-executes-after-approval model.

Rationale: Recommendation-only mode supports many suggested actions and high-trust operations where granting direct agent execution would complicate authentication.

Recover context-limit failures by changing the tool response rather than appending user-visible messages.

Chose Replace oversized tool output with a narrower-query error and retry over Expose immutable-context failure directly to users.

Rationale: The agent can adapt its next request while the original oversized message remains hidden for debugging.

Accept that some non-deterministic workflow executions must be discarded.

Chose Recover user sessions by replacing stuck workflow executions over Assuming Get Version and replay tests prevent all determinism bugs.

Rationale: Block uses versioning and tests first, but production mistakes can still affect many workflows at once; session state in DynamoDB enables recovery with a new workflow.

Gotchas (what bit them)

Synchronous chat APIs time out once agents call many tools.

Root cause: One user input could produce dozens of server-side tool calls that took too long to complete before the response returned.

Resolution: Move to an asynchronous Temporal workflow where clients submit inputs and poll for outputs.

Long-lived workflows create lifecycle and versioning pressure.

Root cause: Keeping every chat workflow alive forever makes it harder to deprecate old logic, while rehydrating on every request adds latency.

Resolution: Use an inactivity-based lifecycle policy that keeps hot sessions active but terminates after one hour of inactivity.

Interrupting a tool-using agent can corrupt message history for the next LLM call.

Root cause: Different providers impose message-history rules, and open tool requests must be followed by corresponding tool responses.

Resolution: Check interruption flags at multiple loop points and stub or add cleanup messages so history remains valid.

Temporal history grows too quickly if activities pass full messages.

Root cause: Agent turns can add megabytes of payload to event history, pushing close to continue-as-new thresholds.

Resolution: Use custom continue-as-new thresholds and plan to pass message references by ID or use large-payload storage.

Incorrect continue-as-new state initialization can cause duplicate work or loops.

Root cause: The original message was re-added to history during continuation, making the agent treat it as new work.

Resolution: Fix continuation state handling so only current work resumes and monitor continue-as-new counts.

Large tool responses blow past LLM context limits after successful tool calls.

Root cause: Merchant catalogs can contain thousands of items, so the next LLM call receives too much tool-response content.

Resolution: Replace the oversized tool response with an error asking for a narrower query, hide the original for debugging, and retry the LLM call.

LLMs can emit tool doom loops or nonsensical outputs even when calls are valid.

Root cause: The model may repeatedly request the same valid tool or produce malformed/wrong-language responses despite advanced capabilities.

Resolution: Track cumulative tool calls, substitute dummy responses to steer the model, add regex checks, and eventually terminate tasks when loops persist.

Workflow non-determinism can make sessions inoperable.

Root cause: AI teams frequently change prompts, tools, and control flow while old workflows are still active and must replay with compatible code paths.

Resolution: Use Get Version, replay tests, and a monitoring workflow that recovers stuck sessions by starting a replacement chat workflow.

Evaluation

Block evaluates agent behavior through human testing, product-specific eval suites, active monitoring of production conversations, error traces, aggregate session inspection, per-session inspection, guardrail/generation-error bucketing, replay tests for Temporal determinism, and future deeper observability for product teams.

Metrics: Human testing results · Product-specific eval-suite outcomes · Production conversation monitoring · Error traces and error categories · Aggregate and per-session inspection · Recorded Temporal event-history replay tests · Stuck workflow detection · Agent performance observability still planned for deeper product-team use

Outcomes

• KGoose became a shared Temporal-backed agent harness for Money Bot, Manager Bot, G2, automations, subagents, and memory extraction.
• The platform evolved from a synchronous timeout-prone API into an asynchronous workflow model with DynamoDB-backed client data.
• Block reports more than 100 million weekly activities and 300 to 400 actions per second for internal and Cash App use cases.
• The system supports over 10,000 concurrent workflows in each namespace.
• The architecture includes explicit mitigations for history bloat, context limits, interruption cleanup, guardrails, tool doom loops, fallback model calls, and workflow non-determinism.

The talk reports over 100 Temporal-backed services at Block, over 100 million weekly internal/Cash App activities, 300-400 actions per second, over 10,000 concurrent workflows in each namespace, one-hour idle workflow termination, 10-minute lifecycle checks, 50 MB/50,000-event Temporal limits, and some turns adding 3.4 MB to history. It does not disclose user counts, accuracy, cost, or revenue impact.