Managing Memory and Scaling Issues in Production AI Agent Systems

gradient_labs · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production AI-agent infrastructure reliability

Gradient Labs needed its Cloud Run and Temporal-based customer-conversation agent to keep responding reliably while memory growth, workflow cache sizing, provider latency, and autoscaling behavior interacted in production.

Constraints:

• The agent participates in active customer conversations, so incident response had to ensure customers were not left waiting.
• The system runs long-running Temporal workflows that manage conversation state, timers, and child response-generation workflows.
• Agent components process variable-sized documents and run parallel work, making memory use naturally variable.
• Cloud Run autoscaling did not naturally scale on Temporal polling workload signals, exposing a mismatch between workflow execution demand and platform scaling metrics.

Architecture

Gradient Labs runs a Go-based AI agent on Google Cloud Run, with each conversation represented as a long-running Temporal workflow that manages state, timers, and child workflows. Platform and agent metrics page on-call engineers. When memory alerts fired, the team used redeployment, profiler flame graphs, hypothesis tests, and Temporal cache tuning. When latency later rose, they adjusted LLM failover first, then diagnosed Cloud Run instance under-scaling and restored throughput with minimum instance tuning.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Conversations["Customer conversation workload"] --> Agent["Cloud Run agent service"]
  Agent --> Workflows["Temporal workflows"]
  Workflows --> Cache["Temporal workflow cache"]
  Workflows --> Child["Child response workflows"]
  Child --> Failover["LLM provider failover"]
  Agent --> Metrics["Platform and agent metrics"]
  Metrics --> Profiler["Profiler investigation"]
  Profiler --> Cache
  Agent --> Autoscaler["Cloud Run autoscaler"]
  Workflows --> TemporalCloud["Temporal Cloud"]
  Cache --> TemporalCloud

Components

Data flow

Customer conversations enter the AI agent service on Cloud Run. Each conversation is handled as a Temporal workflow that stores state, manages timers, and invokes child workflows to generate and safeguard replies. Temporal workers use a workflow cache to avoid repeatedly fetching histories from Temporal Cloud. Metrics and profiler traces feed incident investigation; cache-size and memory changes affect container stability, while Cloud Run instance counts determine how quickly scheduled Temporal activities can start executing.

Why this architecture

Temporal gives each conversation durable state, timers, and child workflow structure, which fits long-running customer interactions. Cloud Run provides managed container deployment and autoscaling, but the incident shows that workflow polling does not automatically produce the HTTP, event, or CPU signals Cloud Run expects. The architecture therefore needs explicit cache and instance-count tuning, provider failover, and deep observability across the model and infrastructure stack.

Models used: LLMs from multiple providers · AI agent

Scale

• The incident involved active production customer conversations.
  “Our top priority is always to ensure that no customers are left hanging: i.e., that our agent is replying in all its active conversations.”article, stated
• The first diagnostic change increased agent memory to 5x the prior allocation.
  “The first change was to boost the agent’s memory to 5x what it previously had. Memory usage continued to grow but eventually plateaued.”article, stated
• The second diagnostic change reduced the worker cache size by 10x.
  “We then deployed a change to decrease the worker’s cache size down by 10x. The container’s memory continued to grow, but plateaued at a much lower value.”article, stated
• Temporal activity scheduling-to-start latency sometimes grew more than 10x during the autoscaling incident.
  “the time between scheduling and starting to execute our Temporal activities had grown substantially (sometimes more than 10x), while our rate of executions had dropped.”article, stated
• The autoscaling bottleneck was identified and fixed in less than an hour.
  “By manually editing the minimum instance count, we saw an immediate uptick in activity executions and everything returned to normal. It took us less than an hour to identify and fix the problem.”article, stated

Alternatives considered (roads not taken)

Treat the memory alert as a conventional application memory leak

Rejected because: Memory-intensive agent components and parallel processing were inspected, but no retained resources explained the growth.

Only increase container memory

Rejected because: More memory bought time and showed a plateau, but the durable fix was lowering the Temporal worker cache size.

Assume increased latency was only an LLM provider incident

Rejected because: Provider failover initially helped, but the recurring cross-provider latency was actually Temporal activities waiting for too few workers.

Rely entirely on default Cloud Run autoscaling

Rejected because: The Temporal polling workload did not drive the HTTP, event, or CPU signals that Cloud Run used for scaling.

Patterns

durable-execution (runtime)

Each customer conversation runs as a long-running Temporal workflow that preserves state, timers, and child workflow orchestration across response generation.

observability-tracing (operations)

The team used alerts, platform/agent metrics, Google Cloud Profiler flame graphs, and activity-execution metrics to isolate memory and scaling failures.

fallback-strategy (resilience)

When latency looked like an LLM provider incident, the team adjusted its LLM failover system to route around provider degradation.

latency-optimization (operations)

Temporal cache and Cloud Run instance counts were tuned against latency, network calls, infrastructure cost, and activity-start delays.

Tradeoffs

Tune Temporal workflow cache size

Chose Smaller worker cache sized to container memory over Larger cache that reduces Temporal Cloud calls but risks crashes.

Rationale: The smaller cache made memory sustainable while accepting possible latency and network-call costs.

Buy time during the memory incident

Chose Redeploy with more memory to clear the immediate alert over Keep the service under memory pressure while investigating.

Rationale: The immediate priority was maintaining customer replies while preserving time for deeper root-cause analysis.

Set minimum Cloud Run instances explicitly

Chose Manual minimum-instance tuning over Let Cloud Run infer Temporal workload demand from default autoscaling signals.

Rationale: Temporal polling and low CPU did not create the signals Cloud Run needed, so default autoscaling throttled activity execution.

Gotchas (what bit them)

A workflow cache can look like a memory leak when its plateau exceeds container limits.

Root cause: Temporal's workflow cache stores execution histories, and the prior container memory limit was below the cache's plateau value.

Resolution: Gradient Labs tested memory and cache-size changes, then tuned the worker cache size down.

Fixing crashes removed an accidental autoscaling signal.

Root cause: Cloud Run had been keeping enough instances alive partly because it gracefully handled crashing containers; after cache tuning stopped crashes, the service scaled down too far.

Resolution: The team manually increased the minimum instance count, restoring activity execution throughput.

Temporal polling did not fit Cloud Run's default autoscaling signals.

Root cause: The agent polled Temporal Cloud instead of receiving HTTP requests or consuming events, and CPU utilization was low.

Resolution: The team recognized Cloud Run's signal mismatch and adjusted minimum instances rather than trusting default autoscaling.

A first plausible latency explanation can hide an internal bottleneck.

Root cause: LLM provider incidents are a common latency cause, but the recurrence cut across providers and prompts.

Resolution: The team checked activity scheduling metrics and found scheduled activities waiting for available workers.

Evaluation

Gradient Labs used production alerts, profiler flame graphs, deliberate memory and cache-size experiments, latency monitoring, provider-outage checks, Temporal activity scheduling metrics, and manual instance-count intervention to validate root causes and fixes.

Metrics: memory boosted to 5x previous allocation during hypothesis testing · worker cache size decreased by 10x · Temporal scheduling-to-start delay sometimes more than 10x · autoscaling bottleneck identified and fixed in less than an hour · activity execution rate recovered after increasing minimum instances

Outcomes

• The memory incident was traced to Temporal workflow cache plateau behavior rather than an ordinary leak.
• Reducing worker cache size stabilized memory at a lower plateau.
• The later latency regression was traced to Cloud Run scaling down too far for Temporal activity execution.
• Increasing minimum instances restored activity execution and normal behavior.
• The incident exposed the need to tune prompts, LLM providers, databases, and containers together.

The case describes a production customer-conversation AI agent, 5x memory and 10x cache-size experiments, more than 10x Temporal activity scheduling delay, and a sub-hour autoscaling fix.