Building Stateful AI Agents with In-Context Learning and Memory Management

letta · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: stateful agent platform

Letta addresses the limitation that LLM agents are typically stateless, forget prior experience, and can degrade as context accumulates, making them weaker as long-running coworkers or production agents.

Constraints:

Architecture

Letta implements stateful agents by treating the context window as mutable memory. Developers configure memory blocks, models, tools, MCP connections, and context budgets through UI/API/SDK; agents update memory through tool calls, older context is summarized to stay within token budgets, and cloud agents can use local tools through human-in-the-loop client execution.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  developer-platform["developer platform"] --> agent-runtime["agent runtime"]
  developer-platform --> memory-blocks["memory blocks"]
  developer-platform --> context-window-viewer["context window viewer"]
  agent-runtime --> memory-tools["memory tools"]
  memory-tools --> memory-blocks
  memory-blocks --> context-window-viewer
  tool-schemas["tool schemas"] --> context-window-viewer
  message-buffer["message buffer"] --> context-window-viewer
  message-buffer --> summarization-system["summarization system"]
  summarization-system --> context-window-viewer
  agent-runtime --> cloud-api-service["cloud API service"]
  cloud-api-service --> cli-client["CLI client"]
  cloud-api-service --> local-client-tool-executor["local client tool executor"]
  local-client-tool-executor --> human-approval-gate["human approval gate"]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
developer platform UI/API/SDK surface for creating, configuring, and messaging stateful agents. agent runtime, memory blocks, context window viewer
agent runtime Runs the selected model with instructions, tools, runtime state, and configured context. memory tools, tool schemas, summarization system, cloud API service
memory blocks Human-readable in-context memory sections with labels, descriptions, and character limits. memory tools, context window viewer
memory tools Allow agents to rewrite their own memory blocks through explicit tool calls. memory blocks, agent runtime
context window viewer Shows developers the exact prompt/context sent to the model for debugging. memory blocks, tool schemas, message buffer, summarization system
summarization system Compacts older context when token budgets are exceeded while preserving state. context window viewer, message buffer
cloud API service Hosts persistent agents and exposes them to UI, SDK, API, and CLI clients. CLI client, local client tool executor
local client tool executor Executes local filesystem or command tools only after client-side human approval. cloud API service, human approval gate
human approval gate Lets the user approve or reject local tool operations requested by the cloud agent. local client tool executor
CLI client Connects local terminals to cloud-hosted agents by agent ID. cloud API service
message buffer Stores recent raw conversation history inside the context window. context window viewer, summarization system
tool schemas Expose available tool definitions to the model within the context window. context window viewer, agent runtime

Data flow

A developer creates an agent in the Letta platform, configures memory blocks and tools, sends messages through UI/API/CLI, and the agent reasons about new information. When it learns something, it calls memory tools to rewrite context-window blocks; the context viewer exposes system instructions, tool schemas, metadata, core memory, compacted summaries, and message buffers. For local coding, a cloud agent requests client-side tool execution and the user approves file operations.

4 supporting quotes
“this is the Letter developer platform. Um, feel free to kind of open this up. You can just sign in and create an account at app.letter.com.”transcript, stated
“this agent right now just has some simple memory related tools and then also conversation search. And on this right hand side, we also have the memory blocks. So this is essentially the in context memory of the agent. So um if I open this up, you can see”transcript, stated
“it's going to call this memory replace tool. And you can see that there's an update that just happened um to the human section of the memory blocks.”transcript, stated
“when the agent wants to execute one of those tools, for example, like GP or ls on my machine, it actually goes back and asks for permission from the client”transcript, stated

Why this architecture

Letta chooses in-context learning because human-readable memory is inspectable, model-agnostic, and usable with frontier APIs where parameter updates are limited. The platform wraps that idea in memory blocks, context visibility, summarization, and explicit tool execution modes so developers can operate long-running agents without treating the model as a black box.

3 supporting quotes
“engineering etc. But I think a lot of these phrases are kind of circling around this similar idea of what if you can learn information into the context window itself. So in the same way that if you get get some information you update your model parameters instead you update the context”transcript, stated
“first of all it's extremely interpretable. It's just English or whatever language. So you can just read it to tell if it's right.”transcript, stated
“it's also model agnostic. If you learn some context, some information, you can pass that to OpenAI, you can pass it to anthropic, to Gemini, open models, whatever.”transcript, stated

Models used: Claude Sonnet · GPT-4o · GPT-5 · OpenAI models · Anthropic models · Gemini · open models

Scale

Alternatives considered (roads not taken)

Fine-tuning, RL, or other parameter updates for learning.

Rejected because: The speaker says parameter learning is difficult to productionize, hard to evaluate online, and limited for frontier-model APIs.

“one of them is I think it's actually very hard to do online or regular updates of parameters just because it's very difficult to evaluate like what has the model actually learned”transcript, stated
“most people today are using frontier models who use GBT cloud for it etc and in those cases you don't you don't really get to control the model parameters”transcript, stated

Naive truncation or compaction that drops agent state.

Rejected because: Letta uses summarization and budget controls because simple compaction can destroy coding-agent state.

“you don't have to ever compact. I don't know if if you've ever used cloud code like the compaction kind of destroys your agent a lot of the time.”transcript, stated
“used cloud code like the compaction kind of destroys your agent a lot of the time. Um but this is using like let us summarization in the back end. So um you know you can customize the context window budget. So I can like lower this to be like just like a 100,000. Um”transcript, stated

Pure cloud execution for local development tools.

Rejected because: The coding demo keeps the agent in the cloud but routes local filesystem and command operations through client approval.

“even though it's running in the cloud, it's still able to do things like GP to search through all the files locally.”transcript, stated
“it actually goes back and asks for permission from the client and asks for the client to execute that. And so that's how I'm able to have this like cloud agent basically access my local computer um to do things like, you know, make code suggestions. So it”transcript, stated

Treat agents as stateless workflow automations.

Rejected because: The central product thesis is that stateless agents cannot learn and therefore remain limited as AI coworkers.

“right now I think when we talk about agents we talk about them purely in a state as as essentially stateless functions”transcript, stated
“we also are missing this incredibly important piece that is what about the ability to learn you obviously cannot learn if you have no memory and I would actually even go as far as to argue that the biggest missing piece in terms of you know this dream of AGI or”transcript, stated

Patterns

agent-memory (agent architecture)

Letta makes memory blocks first-class context sections that the agent can update over time.

“on this right hand side, we also have the memory blocks. So this is essentially the in context memory of the agent.”transcript, stated
“these blocks also have a description and a label associated with them as well as a character limit.”transcript, stated

tool-use-orchestration (agent architecture)

Agents learn and act through explicit tools, including memory replacement, conversation search, MCP, file search, and local operations.

“Um, and then because it's learned some new information, it's going to call this memory replace tool. And you can see that there's an update that just happened um to the human section of the memory blocks.”transcript, stated
“there's also custom tools and you can also attach MCP into this.”transcript, stated

context-compaction (context management)

The platform summarizes older context when token budgets are exceeded rather than dropping state blindly.

“Um, and then we haven't done any summarization yet, but if we did do some compaction, the summary would go in here. And then there's of course like just the raw message buffer.”transcript, stated
“as the agent kind of runs it'll like actually like compact things and make sure that the uh or actually I think it just happened.”transcript, stated

human-in-the-loop-gate (safety)

Local filesystem and command tools are executed through client-side approval rather than directly by the cloud agent.

“web page or conversation search these both both execute on the server as opposed to the all these other tools we actually consider them like variants of human in the loop. So when the agent wants to execute one of those tools, for example, like GP or ls on my machine, it actually goes back”transcript, stated
“So it just requested to edit a file and I approved it. Um so yeah, the other nice thing about this too is like you don't have to ever compact.”transcript, stated

observability-tracing (debugging)

The context window viewer exposes raw context sections and tool schemas so developers can debug agent state.

“we really kind of prioritize in Leta is helping you understand like what is exactly in the context window.”transcript, stated
“So you can actually see like the raw schema of the tools in here. Um there's summary about external metadata that's available and then there's also the core memory.”transcript, stated

Tradeoffs

Learn into the context window instead of model parameters.

Chose Mutable in-context memory over Fine-tuning, RL, or parametric memory.

Rationale: Context memory is interpretable, flexible, model-agnostic, and usable with frontier API models where parameter control is limited.

3 supporting quotes
“context engineering etc. But I think a lot of these phrases are kind of circling around this similar idea of what if you can learn information into the context window itself. So in the same way that if you get get some information you update your model parameters instead you update the context”transcript, stated
“this actually has huge practical advantages because it means that first of all it's extremely interpretable.”transcript, stated
“Um, and then it's also model agnostic. If you learn some context, some information, you can pass that to OpenAI, you can pass it to anthropic, to Gemini, open models, whatever.”transcript, stated

Expose the full context window to developers.

Chose A context window viewer over Opaque agent state.

Rationale: The presenter argues most agent failures can be traced to context, so developers need raw visibility into instructions, tools, memory, summaries, and messages.

“you can pop over this context window viewer and these are kind of all the different actually is everyone here familiar with like a lm context window.”transcript, stated
“90% of the time when your agents something goes wrong, you can actually debug it to something that's wrong in the context.”transcript, stated

Let cloud agents access local resources through approval.

Chose Cloud-hosted state plus client-side human-in-the-loop tools over Purely cloud execution or unrestricted local access.

Rationale: This keeps persistent cloud state while making sensitive local file operations explicit and user-approved.

“even though it's running in the cloud, it's still able to do things like GP to search through all the files locally.”transcript, stated
“it actually goes back and asks for permission from the client and asks for the client to execute that.”transcript, stated

Keep the platform model-agnostic.

Chose Memory and tools independent of provider over Provider-specific learned state.

Rationale: The same learned context can be passed to multiple model providers, allowing provider swaps for cost, quality, or availability.

“If you learn some context, some information, you can pass that to OpenAI, you can pass it to anthropic, to Gemini, open models, whatever.”transcript, stated

Gotchas (what bit them)

Agent performance can degrade as context accumulates.

Root cause: Existing state may contain mistakes or pollution, and models can recover worse from polluted history than from a fresh state.

Resolution: Make context state visible, manageable, and summarizable instead of letting it accumulate opaquely.

3 supporting quotes
“They they degrade over time rather than getting better. So, um you might have noticed if you're running like coding agents or like if you're chatting with a chatbot for a really long time that the agent seems to kind of get like crazier and crazier or like worse and worse.”transcript, stated
“the performance really severely drops once you have an existing history that maybe has some mistakes”transcript, stated
“So we try to make that really transparent. And of course, let model agnostic. And and I think one of the biggest benefits that you get from Leta is being able to control your own state and context.”transcript, stated

Memory updates can teach the agent wrong facts.

Root cause: The agent itself decides what to store, so learning quality depends on model reasoning and memory-management prompts.

Resolution: Keep memory human-readable and inspectable so developers can audit learned state.

“It's just English or whatever language. So you can just read it to tell if it's right.”transcript, stated
“you can apply LLM as judge. Um, and then it's also model agnostic. If you learn some context, some information, you can pass that to OpenAI, you can pass it to anthropic, to Gemini, open models, whatever.”transcript, stated

Wrapping a strong model can accidentally regress its base capabilities.

Root cause: Harness logic, context management, or tool design can interfere with model behavior.

Resolution: Evaluate the harness as an additive layer and explicitly check that it does not degrade tasks like coding.

“if cloud for sonnet is really good at coding but let's not and it's running on that model that means that we've kind of degraded it.”transcript, stated
“making sure that you know we are only adding capabilities on top of the models and not actually regressing things with the way that we've designed our harness.”transcript, stated

The local-coding feature still had rough edges.

Root cause: Letta Code was described as a new experimental project rather than a mature product.

Resolution: Treat the cloud-local coding architecture as experimental until its reliability and UX stabilize.

“we have a new experimental project which is essentially allowing leta agents to connect to your local environment.”transcript, stated
“this is like a really new tool. Um, you know, would recommend trying it out. It still has a couple rough edges”transcript, stated

Evaluation

The talk cites Chroma context-rot research, Letta's RecoveryBench, real deployments such as Ezra, and qualitative demos of UI, memory, and local coding behavior rather than a single formal production scorecard.

Metrics: RecoveryBench fresh-state versus polluted-state comparison · Ezra learning for approximately one month · 20,000-character memory block example · 100,000-token context-budget demo · Claim that 90% of agent failures can be debugged to context

3 supporting quotes
“We also had this benchmark called recovery bench which was basically measuring how well does an agent do from a fresh state versus like a sort of polluted or existing state”transcript, stated
“Ezra is like our support agent that has been in training for approximately a month.”transcript, stated
“This creates an explicit, trackable learning process where every memory update is logged and auditable.”summary, stated

Outcomes

The transcript gives limited production metrics, but includes a one-month support-agent example, a 20,000-character memory block, a 100,000-token budget demo, and customer examples.

4 supporting quotes
“we work with Built Rewards, which is um, a rent uh, payment credit card company to build recommendation agents that learn from their users.”transcript, stated
“we also work with another company, 11X, for example, um, that's building deep research agents on Leta.”transcript, stated
“Ezra actually really feels like something that was fine-tuned even though everything is actually being done in context.”transcript, stated
“I can actually download this agent as an agent file and send it to my friend and they can upload it back.”transcript, stated