Context Management and Memory Strategies for Production AI Agents

arize · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Observability agent context management

Arize's Alex agent helps users build AI applications by analyzing traces, spans, prompts, metadata, and long conversations inside the Arize observability platform. The agent's own diagnostic workload created a context-window loop: the more Alex inspected traces, the more context it produced, and the more likely it was to fail late in long sessions.

Constraints:

Trace and span data can be large before user conversation history is added.
Follow-up questions need earlier context, so naive truncation breaks reasoning.
Users prefer to stay in one chat while moving across pages and workflows.
Heavy data operations must not overload the main conversational context.
Long-term memory was still incomplete, so the production system needed pragmatic memory-store retrieval.

Architecture

Alex separates visible context from retrievable memory. The main chat keeps lightweight context, smart truncation preserves the head and tail of long messages, truncated middle content is stored for tool-based retrieval, and heavy Arize data searches are delegated to sub-agents that keep bulky trace context outside the main conversation.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  arize-observability-data["Arize observability data"] --> alex-main-agent["Alex main agent"]
  alex-main-agent --> smart-truncation-layer["Smart truncation layer"]
  smart-truncation-layer --> memory-store["Memory store"]
  memory-store --> alex-main-agent
  alex-main-agent --> trace-search-sub-agents["Trace search sub-agents"]
  trace-search-sub-agents --> arize-observability-data
  alex-main-agent --> long-session-evaluation-harness["Long-session evaluation harness"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Alex main agent`	Conversational AI harness that helps users build AI applications and coordinates trace-analysis workflows.	`Arize observability data`, `Smart truncation layer`, `Memory store`, `Trace search sub-agents`, `Long-session evaluation harness`
`Arize observability data`	Traces, spans, prompts, metadata, and application context that Alex analyzes for users.	`Alex main agent`, `Trace search sub-agents`
`Smart truncation layer`	Keeps the first and last parts of long messages, removes duplicate/tool-call noise, and stores omitted context for retrieval.	`Alex main agent`, `Memory store`
`Memory store`	Stores truncated middle content and exposes IDs/tools so Alex can retrieve context when it becomes relevant later.	`Alex main agent`, `Smart truncation layer`
`Trace search sub-agents`	Handle data-heavy trace search and analysis outside the main conversation context.	`Alex main agent`, `Arize observability data`
`Long-session evaluation harness`	Tests Alex on long conversations so late-session forgetting is detected before users report it.	`Alex main agent`

Data flow

A user asks Alex to inspect observability data. The main agent sees lightweight conversation context, trace summaries, and tool IDs; long tool outputs are truncated while the omitted middle is stored in memory. When the task requires deeper trace analysis, the main agent delegates the heavy data work to sub-agents and receives back compact results for the ongoing conversation.

“Alex would run on our our traces span data. The spans would grow, there would be too much data, we'd hit a context limit, Alex would fail.”transcript, stated

“we truncate the middle, uh, keep the head and the tail, and then at any point of Alex feels like there's a tool call that was important or a message from the the previous conversation that's important, it can always go back and grab that context.”transcript, stated

Why this architecture

Arize found that neither naive truncation nor plain summarization preserved enough task state. Splitting visible context, retrievable memory, and delegated sub-agent work let Alex keep the main chat usable while preserving access to important details.

“the main takeaway from this was that over truncation had broke the reasoning. It couldn't remember.”transcript, stated

“I think I was surprised the most by the fact that summarization didn't work. I think that was again like the obvious choice for us, but uh the combination of truncation along with being able to store it in memory”transcript, stated

Models used: Alex LLM agent · Trace-search sub-agents

Scale

Alex had been under development for close to a year.
“my team and I have been building for honestly close to a year now, which is our AI agent Alex.”transcript, stated
The agent had more than 40 built-in skills.
“We have advanced planning, 40-plus skills built into it, core workflows across prompt engineering like prompt optimization, data gen, data augmentation, annotations, etc.”transcript, stated
A single trace stack can contain hundreds of spans.
“there can be hundreds of spans within it and Alex needs to figure out what data it should look at.”transcript, stated
Observed user sessions grew from fewer than 10 turns to more than 20 turns.
“I think when we started, I was seeing like less than 10 turns per conversation with Alex, and now I'm seeing folks really go, you know, push the limits up to like 20 plus.”transcript, stated

Alternatives considered (roads not taken)

Naive beginning-only truncation

Rejected because: It made follow-ups look like new conversations and removed context needed for reasoning.

“Can we just take the beginning of it? Is just the beginning important? Is that enough information to give Alex for her to actually perform the analysis that's needed?”transcript, stated

Over-truncate the conversation

Rejected because: Arize explicitly observed that over-truncation broke reasoning and memory.

“Follow-ups looked like new conversations. If I asked one question, Alex would respond, and then I said, you know, ask a follow-up like, you know, what are the most common inputs?”transcript, stated

Rely on summarization for context control

Rejected because: The speaker says summarization was the obvious choice but did not work as well as truncation plus memory.

“summarization didn't work. I think that was again like the obvious choice for us, but uh the combination of truncation along with being able to store it in memory uh is something that we found to be really successful.”transcript, stated

Keep all trace search work inside one agent

Rejected because: Heavy trace search created too much context for the main conversation, so Arize moved that work to sub-agents.

“This was all handled in one agent. And then after basically what we have is these this main agent plus a sub-agent.”transcript, stated

Patterns

`context-compaction` (context)

Alex keeps the head and tail of long context while storing the middle for later retrieval.

“we take the beginning still 100 characters. We also take 100 off of the tail.”transcript, stated

`agent-memory` (memory)

Truncated context is stored behind IDs/tools so the agent can recover important prior messages or tool calls.

“Basically, we have it saved off in a database with IDs. And so, what Alex can do is it has a tool where it has all the IDs and like where in the conversation”transcript, stated

`multi-agent-orchestration` (orchestration)

The main agent delegates bulky trace-search work to sub-agents that keep heavy data context separate.

“we have the main conversation with the chat and light context only. We keep it pretty light. What it can do is it can delegate to the sub-agents.”transcript, stated

`eval-driven-development` (evaluation)

Arize added long-session tests because late failures only appeared after conversations grew.

“it seemed like it was working, um but then as we saw these longer and longer conversations, there were failures happening and we didn't know about them too late”transcript, stated

Tradeoffs

Keep only selected context in the main conversation

Chose smart truncation plus memory retrieval over sending full trace and conversation context to the main agent.

Rationale: The main agent stays within context limits while retaining a path back to important omitted material.

“context aside with the model sees memory decides what survives and so this is kind of a system we built with the um smart truncation there.”transcript, stated

Delegate heavy trace search work

Chose main agent plus sub-agents over a single agent carrying all trace search context.

Rationale: Sub-agents keep heavy data context outside the main chat and return compact results.

“that's where the heavy data stays. So, we can keep all of the heavy data context in our sub-agent. And then once it gets a result, we can kind of pass that over”transcript, stated

Use pragmatic memory-store retrieval before full long-term memory

Chose database IDs and retrieval tools over waiting for complete persistent long-term memory.

Rationale: The current approach works enough for context recovery while the team focuses on the deeper long-term memory problem.

“right now it's working. So, we're kind of focused more on the long term memory cuz I feel like that's where I'm getting the most complaints.”transcript, stated

Gotchas (what bit them)

The agent analyzing traces generated the same context pressure it was supposed to manage.

Root cause: Alex ran over traces and span data that grew until it exceeded the context limit.

Resolution: Separate visible context, retrievable memory, and delegated sub-agent work.

“So the system analyzing the data was constrained by the data, and that was a major problem for us.”transcript, stated

Naive truncation made follow-up questions lose their conversational history.

Root cause: Only the beginning of context was kept, so the agent forgot prior details.

Resolution: Use smart truncation with memory retrieval rather than beginning-only truncation.

“Follow-ups looked like new conversations. If I asked one question, Alex would respond, and then I said, you know, ask a follow-up”transcript, stated

Failures showed up late in long conversations.

Root cause: Early tests did not exercise the long multi-page sessions users actually preferred.

Resolution: Add long-session evaluations and focus on detecting forgetting before users report it.

“our failures appear late. So, uh when we first did this smart truncation, it seemed like it was working, um but then as we saw these longer and longer conversations”transcript, stated

Long-term memory remained an unsolved product gap.

Root cause: The deployed memory mechanism was primarily context-store retrieval, not persistent cross-chat memory.

Resolution: Arize was actively adding real long-term memory so users could reference prior issues across chats.

“we're in the process of adding long-term memory, and I think that's something that'll be a real game-changer for us.”transcript, stated

Evaluation

Arize evaluates context-management behavior through long-session tests and observed production failures in Alex conversations.

Metrics: Late-session failures · Conversation turn count · Ability to retrieve relevant prior tool calls or messages · Sub-agent handling of heavy trace-search work

“the solution that we came up here is long session evals and I want to include this because, you know, it's maybe not”transcript, stated

Outcomes

Smart truncation plus memory was successful enough to be used in Alex.
Sub-agents became the recurring pattern for handling heavy context.
Alex still needed improved long-term memory for cross-chat issue recall.

Alex had 40-plus skills, trace stacks could contain hundreds of spans, and user sessions had grown to 20-plus turns.

“we've we found that this combination of truncation and memory has been really really uh successful.”transcript, stated

“the pattern we keep returning to is sub agents. Uh we just keep breaking things up and having context handled by different parts.”transcript, stated